packages feed

crypton 2.0.1 → 2.2.5

raw patch · 555 files changed

This diff is very large; some files are shown as “too large to diff”. Download the raw patch for the complete diff.

Files

CHANGELOG.md view
@@ -1,1398 +1,1026 @@ # CHANGELOG for crypton -## 2.0.1--* fix(cpu): stop reading Intel's SDBG bit as AMD's XOP, which crashed SHA-512-  and ChaCha20 on Broadwell and later.  The vendored assembly dispatches on a-  capability word this library fills, and reads bit 11 of its second dword as-  XOP -- which is CPUID leaf 0x80000001 ECX bit 11, an AMD extended leaf.-  crypton put the raw leaf 1 ECX there, whose bit 11 is SDBG, the silicon-  debug interface, which Intel has reported since Broadwell.  On such a-  processor `sha512-x86_64.S` took `.Lxop_shortcut` and-  `chacha-x86_64.S` took `.Lcrypton_chacha20_asm_4xop`, and the first-  `vprotq` is an invalid opcode: SIGILL, for SHA-512 and SHA-384, and for-  ChaCha20 and so ChaCha20-Poly1305.  OpenSSL clears leaf 1's bit 11 before-  merging the real flag in; crypton now clears it and leaves it clear, so the-  XOP paths are never taken.  Nothing is lost -- XOP ran from AMD's Bulldozer-  to Excavator and Zen dropped it -- and it is the reason the AVX-512 bits-  beside it are cleared as well.  Reported by @lucasdicioccio, who-  disassembled the trap-  [#202](https://github.com/kazu-yamamoto/crypton/issues/202)--## 2.0.0--* fix(docs): export the names the documentation already referred to.-  `Crypto.Number.ModArithmetic` throws `CoprimesAssertionError` and-  `ModulusAssertionError` from `inverseCoprimes` and `squareRoot`, and said so-  in the haddock, without exporting either, so a caller could not name the-  exception it was told to expect; `Crypto.PubKey.Rabin.Types.generatePrimes`-  takes a `PrimeCondition` in its exported signature and that synonym was not-  exported either.  All three are now exported-  [#195](https://github.com/kazu-yamamoto/crypton/pull/195)--* Breaking change: fix(bcrypt): refuse a cost bcrypt does not have rather than-  substituting one.  A cost below 4 came back as a cost-10 hash and a cost-  above 31 as a cost-31 one, with nothing said either way, so a caller asking-  for something bcrypt does not do was answered with something else and had no-  way to tell -- `hashPassword 3` and `hashPassword 10` returned the same-  thing.  Both ends are now reported as `CryptoError_ParameterInvalid`, which-  is what every other KDF here already did for a refused parameter.-  `hashPassword` can therefore fail where it could not before, so-  `tryHashPassword` is added beside it; the salt it generates is always the-  right length, so the cost is the only thing it can report-  [#59](https://github.com/kazu-yamamoto/crypton/issues/59)--* Breaking change: fix(poly1305): take a checked key, so that initializing-  cannot fail.  A Poly1305 key is thirty-two bytes and nothing else about it-  can be wrong, so `initialize` returning a `CryptoFailable` put an error case-  in front of every caller for a length most of them know is right -- and they-  answered it with `throwCryptoError`, this library included: the one in-  `Crypto.Cipher.ChaChaPoly1305` guarded a `B.take 32`, and `auth` did not-  even do that, it called `error`.  There is now a `Key` with `key` to build-  one, the length is checked there, and `initialize :: Key -> State` and-  `auth :: Key -> ba -> Auth` are total.  A caller checks once and then-  initializes as often as it likes with nothing to handle.  `initialize k`-  becomes `initialize <$> key k` where the length is unknown, and where it is-  known the check moves to where the key is made.  `Key` has no `Show`, as-  key material should not-  [#28](https://github.com/kazu-yamamoto/crypton/issues/28)--* fix(pubkey): stop printing private keys.  `Show` is what `print`, a message-  built with `error`, an exception and a test framework's failure output all-  reach for, so it is the instance a key travels on when nobody meant to send-  it anywhere; the library already kept that promise for the secret keys held-  in a `ScrubbedBytes`, and the documentation for `ScrubbedBytes` advertises-  it, while ten other types printed theirs in full.  Those ten ---  `RSA.PrivateKey` and `RSA.KeyPair`, `DSA.PrivateKey` and `DSA.KeyPair`,-  `ECDSA.PrivateKey` and `ECDSA.KeyPair`, `DH.PrivateNumber`, and the-  `PrivateKey` of `Rabin.Basic`, `Rabin.Modified` and `Rabin.RW` -- now render-  the public part and `<secret>` for the rest.  Nothing else about them-  changes: `Read`, `Eq`, `Data`, `Generic` and `NFData` are all still derived.-  The new `Crypto.Debug` exports a class `DebugShow` whose `debugShow` returns-  exactly what the derived `Show` used to return, for those ten and for the-  five `ScrubbedBytes` secret keys as well, which never had a `Show` that-  spoke.  **Code that serialized a key through `show` has to say `debugShow`-  instead**; `read (debugShow k) == k` still holds, but `read` given the-  output of `show` will now fail at run time, which the compiler cannot point-  at-  [#72](https://github.com/kazu-yamamoto/crypton/issues/72)--* deprecate(ecc): the curves over a binary field.  They are obsolete, they are-  the curves a cofactor makes delicate, and pyca/cryptography deprecated them-  for removal in the release that fixed CVE-2026-26007.  A `DEPRECATED` pragma-  now covers the eighteen `SEC_t*` constructors of `CurveName` and the-  eighteen types of the same names in `Crypto.ECC.Simple.Types`; nothing is-  removed, so the only effect is a warning where one of them is named, and-  they will go in a later major version.  The prime curves with a cofactor,-  `SEC_p112r2` and `SEC_p128r2`, are not deprecated: the check above covers-  them-  [#66](https://github.com/kazu-yamamoto/crypton/issues/66)--* fix(ecc): refuse a public point outside the prime-order subgroup.  A point-  that satisfies the curve equation is not necessarily in the subgroup the-  base point generates; the two coincide only where the cofactor is 1.  Of the-  curves in `CurveName` twenty have a cofactor -- the eighteen binary ones,-  and `SEC_p112r2` and `SEC_p128r2`, whose cofactor is 4 -- and on those the-  other party could offer a point of small order, at which point the value-  that came back depended on our private number only through its residue-  modulo that order, and offering it and watching the answer handed them those-  bits.  This is the flaw pyca/cryptography fixed as CVE-2026-26007; what is-  fixed here is the same one, found by following that report.-  `Crypto.PubKey.ECC.DH.getShared` and `tryGetShared`, and-  `Crypto.ECC.Simple.Prim.pointFromIntegers`, now require the point to be in-  the subgroup and report `CryptoError_PointSubgroupInvalid` when it is not.-  The check is `isPointInSubgroup`, newly exported from both prim modules:-  where the cofactor is 1 it answers without work, and otherwise it multiplies-  by the group order and requires the point at infinity, which is what-  OpenSSL's `EC_KEY_check_key` does and costs one further scalar-  multiplication -- an exchange on an affected curve is about twice the price,-  and one on every other curve is unchanged.  The typed `Crypto.ECC` interface-  offers only cofactor-1 curves and dedicated implementations, so nothing-  reaching elliptic curves through it, `tls` among them, was affected--* perf(p256): inline the field arithmetic on AArch64.  `felem_mul` and-  `felem_square` end in `felem_reduce_degree`, a carry chain the whole width-  of the number, and that chain is what their latency is: one product feeding-  the next costs 18.1 ns on an Apple M4, while four independent ones cost 11.4-  ns each.  The curve arithmetic has independent products to offer -- the two-  squarings that open a point doubling, the multiplication and the squaring-  that close it -- but only if the compiler inlines the reduction instead of-  calling it, since a call is a fence.  Plain `inline` does not change its-  mind; `always_inline` does, and it is worth asking where there are registers-  to hold two carry chains at once and not where there are not: 1.23x on an-  M4, 1.12x and 1.05x on a Neoverse under clang and gcc, and 0.95x and 0.82x-  on x86-64, whose fifteen general-purpose registers are not enough.  So it is-  gated on the architecture and x86-64 is left byte-identical.  ECDH P-256 on-  an M4: 69.16 to 56.24 us, against openssl's 24.68, so 0.36 becomes 0.44;-  ECDSA P-256 signing and verification move with it.  The cost is code, 30 to-  116 kilobytes of it-  [#188](https://github.com/kazu-yamamoto/crypton/pull/188)-* perf(number): count bytes from the bit count, not from base 256.  `numBytes`-  asked GMP how many base-256 digits a number has, and GHC's bignum answers-  that by dividing the number down to nothing, one digit at a time, where the-  same question in base two is a look at the highest limb.  On a 2048-bit-  `Integer` that is 1.65 us against 0.01.  Every serialization here asks for-  the size before it allocates and `i2ospOf` asks twice, so the cost landed on-  every RSA, DSA and DH operation leaving the `Integer` world: `i2ospOf_` at-  256 bytes goes from 2.87 to 0.09 us and RSA-2048 verification from 18.4 to-  15.5 us on an M4.  Signing moves by a percent; it is two exponentiations and-  hardly touches this-  [#187](https://github.com/kazu-yamamoto/crypton/pull/187)-* perf(ecc): stop sharing the doublings in the double multiplication.-  `pointAddTwoMuls` was Shamir's trick, one pass over the bits of both scalars-  at once in `Integer` arithmetic, which is the right trade when the two-  multiplications would cost the same.  They have not for a while: `pointMul`-  goes to C, and over a prime field it multiplies the base point through a-  table of its multiples at about a third of the price -- and the base point-  is one of the two, since ECDSA verification is the only caller.  Doing them-  separately and adding: ECDSA P-384 verification 1397 to 698 us on an M4 in-  the typed API, 9839 to 738 in the older one, and a curve over a binary field-  641 ms to 2.6.  P-256 keeps the double multiplication it has in C-  [#186](https://github.com/kazu-yamamoto/crypton/pull/186)-* perf(sha3): take the CRYPTOGAMS Keccak for x86-64 as well.  The same module-  as [#181](https://github.com/kazu-yamamoto/crypton/pull/181) on the other-  architecture, and the reason it was not taken at the time was a measurement-  taken on the wrong machine: crypton on Apple silicon against openssl on-  x86-64, which said there was nothing to gain.  Measured on one machine there-  was: SHA3-256 on an EPYC 7763 goes from 109 to 421 MB/s, against openssl's-  426, so 0.26 becomes 0.99-  [#184](https://github.com/kazu-yamamoto/crypton/pull/184)-* perf(sha1): take the CRYPTOGAMS SHA-1 for AArch64.  The instructions are the-  ones [#170](https://github.com/kazu-yamamoto/crypton/pull/170) put in, and-  the arrangement is what the module has over them: the message schedule of-  the next four rounds runs against the rounds of this one, which is not-  something a C function is going to be made to do ---  [#179](https://github.com/kazu-yamamoto/crypton/pull/179) tried the one-  thing C can do here, handing over a run of blocks, and on this processor it-  measured nothing at all.  The entry point for processors that have the-  instructions is not exported, so the module's own dispatch picks it and the-  answer to the runtime question goes into the word that dispatch reads.  On-  Apple silicon: 3155 to 3379 MB/s at 16 KiB, against openssl's 3350 on the-  same machine, so where this was at 0.94 it is now a shade ahead.  The-  intrinsics stay for the block a message ends with, and for any processor-  that has the instructions but is built without the assembly-  [#182](https://github.com/kazu-yamamoto/crypton/pull/182)-* perf(sha3): take the CRYPTOGAMS Keccak for AArch64.  The instructions are-  the ones [#171](https://github.com/kazu-yamamoto/crypton/pull/171) put in ---  EOR3, RAX1, XAR and BCAX -- and what this module does with them is take a-  run of blocks rather than one at a time, and schedule the round it is in-  against the next one.  The absorb loop hands over the whole run, which also-  drops the alignment trampoline on that path: the assembly reads the message-  as bytes.  SHA3-256 on Apple silicon: 1002 to 1104 MB/s at 16 KiB, against-  openssl's 1058 on the same machine.  Only the absorb side is handed over;-  the squeeze, which SHAKE uses to produce output, is entangled with this-  side's buffer bookkeeping and is not where the time goes-  [#181](https://github.com/kazu-yamamoto/crypton/pull/181)-* perf(xts): double the XTS tweak in the integer registers.  The tweak-  advances by doubling in GF(2^128) once per block, and it was doing that in a-  vector register: six operations on the same units that are running the-  rounds and the exclusive ors, in a chain where each waits for the one-  before.  On a processor whose AES is fast that is not a detail -- taking the-  doubling out of a diagnostic build, which gives the wrong answer but says-  where the time goes, left XTS running at the speed of ECB.  It costs three-  integer operations instead, and the integer units have nothing else to do-  here; what crosses over is one move per block.  On x86-64 that also gets-  eight values out of a register file with sixteen entries, so the round keys-  stay where they were.  AES-128-XTS at 16 KiB: 9644 to 18646 MB/s on Apple-  silicon and 4504 to 7660 on a Haswell-generation x86-64, against openssl's-  17382 and 6997 on the same machines, so both are now a little ahead where-  they were at 0.55 and 0.64.  No assembly: the AArch64 module in CRYPTOGAMS-  has no XTS, and the x86-64 one's is inside a module this does not otherwise-  want-  [#180](https://github.com/kazu-yamamoto/crypton/pull/180)-* perf(sha1): hand the SHA-1 block loop a run of blocks rather than one at a-  time.  A block at a time means the state goes out to memory and comes back-  either side of every block, with the two shuffles that put it in the order-  the instructions want; against the hundred-odd cycles a block costs with the-  SHA extensions that is most of what stood between this and openssl.  On an-  EPYC 7763: 1364 to 1677 MB/s, against openssl's 1670 on the same machine,-  and on a Xeon 8370C 1506 to 1619.  On Apple silicon it measures nothing at-  all -- that processor hides the cost -- and is kept there only so the two-  paths have one shape.  The intended file for this was CRYPTOGAMS'-  `sha1-x86_64.pl`, which turns out to be the 2006 scalar implementation: no-  SSSE3, no AVX, no SHA extensions.  Processors without the extensions are-  therefore where they were, 664 MB/s against openssl's 791 on a Haswell-  [#179](https://github.com/kazu-yamamoto/crypton/pull/179)-* perf(sha2): take the CRYPTOGAMS SHA-256 and SHA-512 for x86-64.  One-  generator gives both, as on AArch64, and each dispatches on what the-  processor has: the SHA extensions, AVX2, AVX, SSSE3 or plain integer code.-  That replaces everything written here for x86-64 -- `sha256_x86.c` and-  `sha512_x86.c` go -- since it is ahead of all of it either way.  At 16 KiB:-  on an EPYC 7763, SHA-256 1430 to 1584 MB/s and SHA-512 423 to 769; on a-  Haswell-generation part, SHA-256 318 to 379 and SHA-512 488 to 593, where-  openssl reports 378 and 589.  The block loops hand over the whole run of-  blocks rather than one at a time, and the alignment trampoline goes with it.-  This also fixes a bug in the capability word-  [#176](https://github.com/kazu-yamamoto/crypton/pull/176) added: bit 29 of-  leaf 7 EBX is the SHA extensions, not an AVX-512 bit, and was being cleared-  along with them -- which cost the SHA-256 assembly two thirds of its speed-  on a processor that has them, and which no machine here could have shown,-  since none has them-  [#178](https://github.com/kazu-yamamoto/crypton/pull/178)-* perf(chacha): take the CRYPTOGAMS ChaCha20 for x86-64 as well.  The C here-  vectorises from eight blocks up and takes anything shorter one block at a-  time, so a message of a few hundred bytes -- a QUIC packet, a small TLS-  record -- ran at a fifth of the bulk rate.  The module has vector code for-  those lengths and is a few per cent ahead in bulk besides: 494 to 1091 MB/s-  at 256 bytes, 477 to 701 at 128, and 2201 to 2374 at 16 KiB, which is-  openssl's 2389 on the same machine.  It is handed everything from one block-  up, where the AArch64 module is handed nothing below three, that one's-  scalar path measuring level with the C.  Keystream generation is still the-  C on both, having no input to exclusive-or-  [#177](https://github.com/kazu-yamamoto/crypton/pull/177)-* perf(poly1305): take the CRYPTOGAMS Poly1305 for x86-64 as well.  The same-  module for the other architecture, through the same three functions, so what-  this adds is the capability word: where the AArch64 one reads-  `crypton_armcap_P`, this one reads `crypton_ia32cap_P`, which is cpuid's own-  words in the order OpenSSL keeps them, filled with the bits for anything the-  operating system will not preserve cleared.  What it brings over the AVX2-  written here is a hand-scheduled scalar path, which is what a message of a-  few hundred bytes actually uses, and an AVX path for machines with no AVX2:-  on a Haswell-generation x86-64, 4298 to 5345 MB/s at 16 KiB and 556 to 1573-  at 64 bytes, against the roughly 5270 openssl reaches there.  `poly1305_avx2.c`-  goes the way the NEON did.  The module's AVX-512 paths are not taken: the-  generator chooses what to emit from the version of the assembler it is told-  about, and it is now told one that predates them, no machine here being able-  to run them and an assembler still in use being unable to assemble them.-  Pinning that version also makes the checked-in assembly independent of the-  host that produced it-  [#176](https://github.com/kazu-yamamoto/crypton/pull/176)-* perf(sha256): take the CRYPTOGAMS SHA-256 for AArch64.  The instructions are-  the ones the intrinsics here already use; what the module does with them is-  schedule them across a whole run of blocks rather than one at a time, and-  keep the message schedule of the next block moving while the rounds of this-  one are still going, which a function that is handed one block and returns-  cannot do whatever it is written in.  So the block loop hands over the whole-  run, which also drops the alignment trampoline on this path -- the assembly-  reads the message as bytes and wants neither the alignment nor the copy.  On-  Apple silicon: 2637 to 3279 MB/s at 16 KiB, against openssl's 3323 on the-  same machine, and 1576 to 1966 at 64 bytes.  SHA-512, which the same-  generator emits, is not taken: 1876 here against openssl's 1880, the-  ARMv8.2 instructions for it having gone in with #110-  [#175](https://github.com/kazu-yamamoto/crypton/pull/175)-* perf(poly1305): take the CRYPTOGAMS Poly1305 for AArch64.  One-  multiplication modulo 2^130 - 5 depends on the one before it, so what there-  is to win is in how the multiplies and the carries are laid against each-  other, and in keeping the accumulator in whichever base costs less: the-  module works in base 2^64 while the message is short and switches to base-  2^26 for the four-way vector loop, deciding that for itself.  Unlike the-  other two it is the whole of the arithmetic rather than a bulk loop bolted-  to the side, so the context now holds either the 26-bit limbs the C works in-  or the 192 bytes the assembly keeps, as a union, and grows from 84 bytes to-  232.  On Apple silicon: 4269 to 8060 MB/s at 16 KiB and 1542 to 4355 at 64-  bytes, and ChaCha20-Poly1305 together, which is what this is for, 1416 to-  2284 against openssl's 2180 on the same machine.  `poly1305_neon.c`, which-  [#169](https://github.com/kazu-yamamoto/crypton/pull/169) added, goes: the-  assembly is faster at every length on every target that gets it, and the-  scalar C remains for the targets that do not.  The tests came first and-  found that the chunking property here had been testing nothing -- it used-  the all-zero key, whose r is zero, so both sides were the nonce whatever-  they did, which is how it came to feed the chunks to `update` in reverse-  order and pass-  [#174](https://github.com/kazu-yamamoto/crypton/pull/174)-* perf(chacha): take the CRYPTOGAMS ChaCha20 for AArch64.  The vector-  registers hold four ChaCha states and there is no room for a fifth, so once-  four blocks are in flight the only place further parallelism can come from-  is the integer side: that module runs a fifth block through the general-  registers alongside four in the vector ones, and above 512 bytes two-  alongside six.  Which register holds which word is the whole of the trick-  and C has no way to say it, which is why the intrinsics here sat at about-  0.63 of what openssl gets out of this very file.  On Apple silicon, a-  message per call: 1911 to 3069 MB/s at 512 bytes, 1913 to 3093 at 4 KiB and-  2056 to 3112 at 64 KiB, against openssl 3.6's 3164 on the same machine,-  which is this code.  It is handed only the states it fits -- twenty rounds,-  a 256-bit key, and as many blocks as the 32-bit counter has room for, since-  crypton's counter is 64 bits wide and carries where the assembly wraps ---  and nothing below 192 bytes, where its own vector path starts.  The tests-  came first: the properties here generated one shape of state, so the-  256-bit constants were never exercised by them-  [#173](https://github.com/kazu-yamamoto/crypton/pull/173)-* perf(gcm): take the CRYPTOGAMS stitched AES-GCM for x86-64, which is the-  first assembly in the package.  Counter-mode AES and GHASH do not compete-  for the same execution ports, so a loop that interleaves them at-  instruction granularity runs both in about the time the rounds alone take;-  written in C that interleaving does not survive the compiler, which sinks-  every multiply to the end of the group, and the disassembly of what-  [#160](https://github.com/kazu-yamamoto/crypton/pull/160) produced says so.-  On a Haswell-generation x86-64, a message per call, AES-128-GCM: 2672 to-  3172 MB/s at 1152 bytes, 3394 to 4271 at 4 KiB and 3657 to 5110 at 16 KiB,-  where openssl speed on the same machine reports 4896; decryption within a-  couple of points of that, and AES-256-GCM 3147 to 4297 at 16 KiB against-  openssl's 4206.  `cbits/asm` holds the module, the translator it needs and the-  generated assembly, one file per object format, so that building needs no-  perl; `cbits/asm/README.md` records where it came from and what was done to-  it, which is to rename the entry points, a program linking both crypton-  and openssl being entitled to object to two definitions of-  `aesni_gcm_encrypt`.  What the assembly reads is laid out OpenSSL's way and-  is built per message in `cbits/aes/gcm_x86_asm.c`, the powers of H being-  the ones crypton already has, shifted up a bit.  Short messages are not-  handed over at all.  `cabal-version` is now 3.0, for `asm-sources`-  [#172](https://github.com/kazu-yamamoto/crypton/pull/172)-* perf(sha3): use the ARMv8.2 SHA-3 instructions.  Keccak was the plain C-  everywhere, a round at a time over tables of rotation amounts and lane-  positions, at half of what openssl manages on the same machine.  EOR3,-  RAX1, XAR and BCAX exist for exactly this permutation and take a round from-  around a hundred and fifty operations to sixty-six; they come with the-  SHA-512 extension the tree already asks for.  Rho and pi move one lane of-  every row into every other row, so the round cannot be done in place, and-  four rounds go in an iteration, which is worth a fifth over one.  SHA3-256-  551 to 991 MB/s (openssl 1064), SHAKE128 700 to 1166, Keccak-256 559 to-  944.  The body is generated from the definitions in FIPS 202 rather than-  copied in, and the script that worked out the rotations and the lane-  permutation checked itself against the published digests of the empty-  string and of "abc" before emitting any C, which is how a first attempt-  with chi reading lanes another row had already overwritten was caught.  x86-  is untouched: nothing there has instructions for this-  [#171](https://github.com/kazu-yamamoto/crypton/pull/171)-* perf(sha1): use the ARMv8 SHA-1 instructions.  The AArch64 paths for-  SHA-256 and SHA-512 went in with #104 and #110 and x86 got its SHA-1-  instructions in [#165](https://github.com/kazu-yamamoto/crypton/pull/165),-  but the AArch64 SHA-1 ones were never used -- and they are part of the same-  optional feature as the SHA-256 ones, so every processor that has those has-  these.  SHA1C, SHA1P and SHA1M each do four rounds with one of the three-  round functions, SHA1H carries E from one group to the next, and SHA1SU0-  and SHA1SU1 do the message schedule between them.  On Apple silicon: 1272-  to 3180 MB/s, against openssl 3.6's 3350 on the same machine.  Checked-  against the hardware rather than through an emulation of the instructions,-  the machine here having them: the digests agree with the generic-  implementation over every message length from 0 to 2000, with each input-  split in two updates-  [#170](https://github.com/kazu-yamamoto/crypton/pull/170)-* perf(poly1305): four blocks at a time with NEON.  AArch64 had only the-  scalar loop, whose five 26-bit limbs and 32-bit multiplies are the shape a-  32-bit machine wants.  This is the arithmetic of the AVX2 path in NEON,-  written as a transliteration of that file rather than a fresh formulation,-  since the maths there is already pinned by the known-answer tests; what-  differs is the width, AVX2 holding four 64-bit products in a register where-  NEON holds two, so each product becomes a pair and the limbs are packed-  back into four 32-bit lanes before the next multiply.  On Apple silicon:-  Poly1305 2783 to 4840 MB/s, and ChaCha20-Poly1305 together 1164 to 1368.-  Checked against the scalar implementation over forty keys and every message-  length from 0 to 400, with each input split in two updates.  Also measured-  and left alone: BLAKE2b at 1612 MB/s against openssl's 1378, the reference-  C being the faster of the two-  [#169](https://github.com/kazu-yamamoto/crypton/pull/169)-* perf(modes): stop the generic cipher modes allocating per byte.  Counter-  mode with a cipher whose modes are not in C ran at a third of what the same-  cipher managed in ECB, and at an eighth for Blowfish, for two reasons-  outside the cipher.  The counters were built one at a time by `ivAdd`,-  which allocates a block and walks the whole width of the counter from the-  original for each of them; they are now one buffer filled in place.  And-  the exclusive or was `Data.ByteArray`'s, which walks a byte at a time-  through an IO applicative -- 420 MB of heap for 8 MiB of counter mode,-  against 17 MB for the same data through ECB, which is fifty bytes allocated-  per byte produced and cost more than the cipher did.  There is a-  `crypton_memxor` to call instead, a pass of words, which the modes and CMAC-  use.  The serial modes also took each block as a copy and take shared-  slices now.  On Apple silicon, counter mode: Camellia-128 79.7 to 285.9-  MB/s, Blowfish 60.4 to 282.6, DES 46.2 to 114.9, CAST5 40.2 to 86.6,-  Twofish-128 37.4 to 57.6, 3DES 25.2 to 36.3, and CBC and CMAC by a third to-  a half as much again.  AES is unchanged: its modes are in C and never came-  this way-  [#168](https://github.com/kazu-yamamoto/crypton/pull/168)-* build: compile the C at -O3, which is what came of looking at P-256 against-  openssl.  The comparison in the problem list was wrong -- a base point-  multiplication here against openssl's ECDH, which is a variable point one ---  and measured properly P-256 is 2.8 to 3.3 times slower rather than the 1.27-  claimed.  The time is in the field arithmetic, five 51-bit limbs in-  Montgomery form at 44.4 ns a multiplication, against hand-written assembly-  using `mulx`, `adcx` and `adox`; a four-limb saturated Montgomery-  multiplication written in C to see what a compiler would give measured 41.3-  ns, so that is not the way in.  What did move is the optimisation level GHC-  passes: a P-256 base point multiplication goes from 71.0 to 59.8 us on x86-64-  and 26.0 to 24.3 on Apple silicon, AES-128-GCM from 3455 to 3708 MB/s and-  AES-128-OCB from 2187 to 2484, with ChaCha20, Poly1305, SHA-1 and MD5 within-  a couple of per cent either way.  The masked selections in the curve and-  field code compile to no conditional jumps at either level-  [#167](https://github.com/kazu-yamamoto/crypton/pull/167)-* refactor(aes): drop the keystream generator nobody can call.  `genCTR` and-  `genCounter` are exported from a module in `other-modules`, so nothing-  outside the library could reach them and nothing inside used them; the only-  mention left was a test commented out since the cryptonite days.  They were-  also the slowest thing in the file, a block at a time through the-  single-block entry point at 715 MB/s where counter mode does 5788, and the-  three ways of fixing that are each worse than removing them: counter mode-  over zeros costs Apple silicon a fifth, counters through ECB costs both, and-  a keystream loop written out per key size is eighty lines for an API no-  caller can see.  Also declares `crypton_aes_encrypt_ctr` and-  `crypton_aes_encrypt_c32` in the header, which had them defined and imported-  but never declared-  [#166](https://github.com/kazu-yamamoto/crypton/pull/166)-* perf(sha1): use the Intel SHA extensions on x86-64.  The extension that-  carries the SHA-256 instructions carries four for SHA-1 as well, and the same-  cpuid bit answers for both, so this is one file and one branch.  On an AMD-  EPYC 7763: 725.1 to 1363.8 MB/s, against openssl's 1668.2 on the same-  machine.  1.9x, where the SHA-256 instructions were worth 4.8x -- SHA-1's-  rounds are cheaper to begin with, so there is less for an instruction to-  replace.  The sequence was checked by replacing the four instructions with C-  that follows the SDM and comparing against the generic implementation over-  every length from 0 to 1024, which found the same missing schedule step-  [#155](https://github.com/kazu-yamamoto/crypton/pull/155) had-  [#165](https://github.com/kazu-yamamoto/crypton/pull/165)-* docs(sidechannel): say what the modules that still work in `Integer` keep-  from the clock, and fix the two places where something could be done about-  it.  ElGamal inverted the shared secret with the extended Euclidean-  algorithm, whose steps follow the bits it is given -- the modulus is prime,-  so Fermat reaches it.  Its signing inverts the ephemeral value modulo an even-  number, where Fermat does not reach, so `sign` blinds instead: the algorithm-  is handed that value times a fresh random unit and the blinder divided out-  afterwards.  What is left is written down rather than fixed -- the Jacobi-  symbols Rabin takes modulo its private primes, and the cost of `Integer`-  arithmetic following the size of the numbers -- and `Crypto.Cipher.AES` now-  says which implementation a machine gets and that the fallback, being-  table-driven, is not constant time-  [#164](https://github.com/kazu-yamamoto/crypton/pull/164)-* perf(poly1305): shorten the carry chain and stop the AVX2 loop spilling.  The-  carries go in pairs, since the two halves of that chain do not depend on each-  other; the powers of r are read from memory, there being sixteen registers-  and ten of them wanted for the accumulator and the products; and the message-  is added limb by limb as the block comes apart rather than five limbs being-  formed first.  4203 to 4452 MB/s, and ChaCha20-Poly1305 together from 1412 to-  1488.  What is left is the instruction count: 107 per 64 bytes, of which 25-  are the multiply-  [#163](https://github.com/kazu-yamamoto/crypton/pull/163)-* perf(chacha): combine as the keystream comes out of the registers.  All three-  vector implementations wrote it to a buffer on the stack and read it back to-  exclusive-or it with the input, which is a pass over every byte for something-  the registers were already holding.  2128 to 2230 MB/s on x86-64, and nothing-  on Apple silicon, where the round trip was free.  Measured while doing it:-  the AVX2 path already did eight blocks at a time, and what is left of the gap-  to openssl in this AEAD is Poly1305 rather than the cipher-  [#162](https://github.com/kazu-yamamoto/crypton/pull/162)-* perf(sha): compute the message schedule in vector registers on x86.  SHA-512-  has no instruction there and SHA-256 has none on a processor older than-  Goldmont or Zen, which includes the Ice Lake and Cascade Lake server parts.-  The rounds are a chain and stay where they are; the schedule is a quarter of-  the work, comes out four words at a time and depends on nothing but the-  message, so it goes into the vector registers and runs alongside rounds that-  need the general ones.  SHA-256 228 to 314 MB/s, SHA-512 354 to 480-  [#161](https://github.com/kazu-yamamoto/crypton/pull/161)-* perf(gcm): take the GHASH of the group before, alongside this group's rounds.-  Held a group apart the multiply and the rounds run through each other, where-  in step neither could start until the other finished.  With it, the multiply-  called directly rather than through a branch pointer the compiler cannot see-  through, and the round keys read from memory rather than spilled: AES-128-GCM-  2797 to 3458 MB/s and AES-256-GCM 2458 to 3053.  openssl does 4895 and 4205-  on the same machine; the rest of that is instruction-level interleaving,-  which does not survive being written in intrinsics-  [#160](https://github.com/kazu-yamamoto/crypton/pull/160)-* perf(ocb): drive OCB through the ECB paths a group at a time.  It ran one-  block at a time through the single-block entry point and so cost four times-  what GCM costs, for a mode that does less work than GCM.  The offsets have to-  be worked out in order but the block cipher calls under them do not depend on-  each other, so eight go through ECB together.  OCB-128 1130 to 3500 MB/s on-  Apple silicon and 694 to 2173 on x86-64, the authenticated data 1141 to 5900-  and 692 to 2526.  CCM is unchanged and stays that way: what is left there is-  CBC-MAC, where each block waits for the one before it-  [#159](https://github.com/kazu-yamamoto/crypton/pull/159)-* test(aes): run the XTS vectors, and add OCB and CCM at 192 and 256 bits.  The-  XTS known-answer tests never ran: the call was commented out and the test it-  would have called did not compile, so vectors at both key sizes sat in the-  tree unused.  XTS is defined only for a 128-bit block, which the general KAT-  runner cannot promise, so it gains a counterpart for a cipher that can.  OCB-  and CCM had vectors at 128 bits only.  2613 examples to 2679-  [#158](https://github.com/kazu-yamamoto/crypton/pull/158)-* perf(aes): build the AArch64 key schedule with the instructions rather than-  the S-box table.  The AArch64 path expanded a key by calling the generic-  implementation and then inverting the round keys, so every schedule went-  through sixteen lookups at addresses derived from the key -- a small thing-  next to the per-block indexing the extensions exist to remove, but a key-  schedule is what an attacker most wants out of a cache, and x86 has never-  needed the table.  AArch64 has no counterpart to AESKEYGENASSIST, but AESE-  against a zero key is SubBytes and ShiftRows, and a word given to it in all-  four columns comes back as SubWord in each of them.  The words stay in-  vector registers throughout, which is what makes it free: moving each one to-  a general register for the instruction and back cost more than the-  instruction did, 87 to 144 ns for an AES-128 schedule, where keeping them in-  registers gives 81.4-  [#157](https://github.com/kazu-yamamoto/crypton/pull/157)-* perf(aes): AES-192 through the processor's AES instructions.  Every 192-bit-  slot in the branch table was left at the generic code, on x86 and on AArch64-  alike, so a 192-bit key got the table-driven software AES while 128 and 256-  got the instructions.  It was 164 times slower for counter mode on the x86-  machine measured and 62 on Apple silicon, and it was also the only key size-  whose data path indexes a table with bytes derived from the key -- a caller-  who picks AES-192 over AES-128 for a wider margin was quietly given a weaker-  one.  Counter mode then GCM, before and after: Apple silicon 152.7 to 9452.0-  MB/s and 112.3 to 7049.3, x86-64 40.4 to 6635.1 and 39.9 to 2633.5.  Both-  implementations were already written once per key size, so this instantiates-  them again at twelve rounds; x86 also needed the 192-bit schedule, which-  does not fall into 128-bit pieces the way the other two do-  [#156](https://github.com/kazu-yamamoto/crypton/pull/156)-* perf(sha256): use the Intel SHA extensions on x86-64, which is what issue-  [#31](https://github.com/kazu-yamamoto/crypton/issues/31) reports -- SHA-256-  four to eight times slower than sha256sum and openssl, both of which use the-  processor's instructions.  AArch64 got its instructions in #104 and is at-  parity with them; x86 had nothing.  SHA256RNDS2 does two rounds at a time and-  SHA256MSG1 and SHA256MSG2 help with the message schedule, so a block costs-  four groups of sixteen instructions instead of sixty-four rounds of scalar-  work.  On an AMD EPYC 9V74: 338.3 to 1612.7 MB/s, against openssl's 1783.8 on-  the same machine.  The extensions arrived with Goldmont and Ice Lake at Intel-  and with Zen at AMD, far later than AES-NI, so a processor without them is-  ordinary rather than ancient: the code sits behind a target attribute and a-  cpuid question, and the plain C stays for everything else-  [#155](https://github.com/kazu-yamamoto/crypton/pull/155)-* perf(bcrypt): Blowfish, and the key setup bcrypt wraps it in, in C.  bcrypt-  is a cost parameter and a promise that the cost is paid, and what pays it is-  the Blowfish key schedule; in Haskell that cost about twice what the usual-  implementations charge, so a hash of a given length of time had to be asked-  for with a lower cost than elsewhere.  Cost 8 goes from 25.97 to 9.98 ms,-  cost 10 from 102.01 to 39.79, cost 12 from 418.78 to 159.41, `bcrypt_pbkdf`-  from 109.76 to 40.38, and Blowfish over 4 KiB from 0.05 to 0.01 -- at cost-  10 that is 39.8 ms against the 52 `htpasswd` takes on the same machine.  The-  Haskell cipher goes with it, so there is one implementation rather than two,-  and nothing exposed changes-  [#154](https://github.com/kazu-yamamoto/crypton/pull/154)-* perf(prime): fewer Miller-Rabin rounds for a candidate nobody chose.  A-  number handed over may have been built to pass, and against that the only-  thing to go on is that a round catches three quarters of the composites-  there are, so `isProbablyPrime`, `findPrimeFrom` and `findPrimeFromWith`,-  which all take their number from the caller, keep their thirty rounds.  A-  candidate drawn here is the case Damgard, Landrock and Pomerance worked out-  and Table 4.4 of the Handbook of Applied Cryptography tabulates:-  `generatePrime` and `generateSafePrime` now use twice what it asks for one-  chance in 2^80, capped at the thirty they had, which leaves the chance far-  under one in 2^100 at every size.  With the candidates held fixed,-  `generatePrime 1024` goes from 28.5 to 16.9 ms and an RSA-2048 key from 52.9-  to 39.2-  [#153](https://github.com/kazu-yamamoto/crypton/pull/153)-* fix(rsa): work the private exponent out without the extended Euclidean-  algorithm.  The modulus is the secret there, so multiplying the value by a-  random number hides nothing; what does is that `e` is public.  Whatever `d`-  is, `e * d = 1 + k * phi` for some `k` under `e`, and reading that modulo-  `e` gives `k` as an inverse modulo a number of a handful of bits, which for-  a prime `e` is Fermat; `d` is then an exact division.  What phi touches is a-  remainder and a division, and nothing in either follows it-  [#152](https://github.com/kazu-yamamoto/crypton/pull/152)-* perf(f2m): ask aarch64 for its carry-less multiply as well.  #148 used PMULL-  only where the compiler had been told the machine has the crypto-  extensions, which is so on Apple and not on a Linux built for the bare-  ARMv8 baseline, though every processor that runs such a build has it.  It is-  now compiled behind an attribute and the machine asked at run time, through-  the auxiliary vector on Linux and Android, elf_aux_info on FreeBSD and a-  sysctl on Apple: sect283k1 421.2 to 168.9 us there, sect571r1 2289.2 to-  579.9-  [#151](https://github.com/kazu-yamamoto/crypton/pull/151)-* refactor(ecc): one multiplication for both of the curve APIs, and one place-  for each buffer's size.  Which path a point multiplication takes was written-  out twice, and the copy in `Crypto.ECC.Simple.Prim` cannot be reached from-  outside the library on a curve over a binary field, so the suite never ran-  it; it moves to the internal module both already share, which makes the copy-  nobody can call the same code everybody runs.  The two buffers for a C call-  that still had their size written out separately from the offsets into them-  now take both from one list, as the one that was wrong in #141 does -- the-  note there records that neither valgrind nor the debug RTS catches that-  mistake, both having been tried-  [#150](https://github.com/kazu-yamamoto/crypton/pull/150)-* perf(f2m): use the x86 carry-less multiply where the processor has it.-  PCLMULQDQ is not part of the x86-64 baseline, so the cpuid the package-  already runs for AES-NI reports one more bit and the multiplication that-  uses the instruction sits behind an attribute.  Measured through Rosetta,-  which translates rather than runs it, so the ratio is what to read:-  sect283k1 560.4 to 177.5 us, sect571r1 3049.6 to 623.9-  [#149](https://github.com/kazu-yamamoto/crypton/pull/149)-* perf(f2m): do the binary field arithmetic in C.  The ladder of #142 spent-  nearly all its time on one thing -- a carry-less multiplication, which-  ordinary arithmetic does not give and which in Haskell was `Integer` shifts-  and exclusive ors, about 4 us for a 283-bit multiplication.  The field and-  the ladder over it are now C, with the processor's instruction where there-  is one and four interleaved groups of bits where there is not, folding for-  the reduction and Fermat for the inverse.  sect163k1 3431 to 67.4 us with-  the instruction and 117.3 without, sect283k1 10181 to 157.6 and 386.2,-  sect571r1 40469 to 521.6 and 2101.0.  It is also constant time, which the-  Haskell ladder was not-  [#148](https://github.com/kazu-yamamoto/crypton/pull/148)-* perf(bignum): start the doubling for `R^2 mod m` at the highest power of two-  under the modulus rather than at one, which for a modulus that fills its-  limbs is half the steps.  Two to three percent of a curve operation, and-  every curve operation and every `expSafe` pays for it once.  Folding instead-  of Montgomery for the primes shaped `2^k - c` was written and measured-  alongside it and is not here: it is slower in this representation, 87.8 ns-  against 76.8 for a 521-bit multiplication, because the shift down by `k`-  costs more than the reduction pass it replaces when `k` does not land on a-  limb boundary-  [#147](https://github.com/kazu-yamamoto/crypton/pull/147)-* perf(ecc): keep a table of the multiples of each curve's base point, which-  is the point signing and making a key multiply and the only one worth a-  table.  A multiplication with it is one addition per four bits and no-  doublings: secp256k1 211.1 to 59.8 us, secp384r1 519.3 to 144.2, secp521r1-  1047.7 to 283.0, and ECDSA P-384 signing 556.4 to 179.9 on both elliptic-  curve APIs, which share the table.  A table is built when a curve is first-  asked for one -- 2.8 ms for secp256k1, 5.5 for secp384r1, 10.6 for secp521r1-  -- and is 221 KB and 456 KB for the last two, so it pays for itself after-  about fifteen multiplications-  [#146](https://github.com/kazu-yamamoto/crypton/pull/146)-* perf(bignum): take the limbs four and two at a time as well as eight in the-  loop every modular multiplication is built out of.  Four and six limbs, which-  is what most of the curves want, fell entirely to the one-at-a-time tail-  before: a field multiplication at six limbs goes from about 58 to 49 ns,-  secp384r1 scalar multiplication from 596.7 to 519.3 us and ECDSA P-384-  signing from 645.0 to 556.4.  Specialising the sizes further, which is what a-  generated implementation would do, measures about 4% more and is not here-  [#145](https://github.com/kazu-yamamoto/crypton/pull/145)-* fix(rsa): keep the blinding factor out of the extended Euclidean algorithm.-  The blinder is a random number and its inverse, and the inverse went through-  an algorithm whose steps follow the number handed to it -- the number the-  blinding rests on, and unlike the other inverses this one is worked out once-  per operation rather than once per key.  `n` being composite leaves no-  Fermat to fall back on, so the algorithm is handed the factor multiplied by-  sixteen fresh random bytes and its answer multiplied by them again, which-  leaves the inverse wanted and shows the algorithm nothing to do with it.  In-  IO, where every draw of randomness goes to the system, `generateBlinder`-  goes from 74 to about 120 us and a PKCS#1 v1.5 `signSafer` from 719 to about-  765; under a DRG the caller carries, 24.7 to 24.9-  [#144](https://github.com/kazu-yamamoto/crypton/pull/144)-* fix(rsa): work `qinv` out without the extended Euclidean algorithm.  Making-  a key inverts one prime modulo the other and both of them are the key-  itself, so that inverse is now Fermat's little theorem through `expSafe`,-  which the other prime being prime allows: 308.6 us against 10.1, on a key-  that takes tens of milliseconds to make.  Making a key cannot be constant-  time -- the search for the primes takes as long as it takes -- but what that-  leaks is about the search rather than about the primes it settles on, and-  the haddock now says which is which-  [#143](https://github.com/kazu-yamamoto/crypton/pull/143)-* perf(ecc): a ladder for the curves over a binary field.  These were the last-  multiplication whose cost followed the scalar: an affine double-and-add, one-  addition for every bit that was set and none for the others, which on-  sect283k1 ran from 9665 us for a scalar with two bits set to 17958 for one-  with 270.  It is now Montgomery's ladder, which carries the multiples of two-  consecutive numbers -- their difference being the point is what lets it-  carry only their x coordinates -- and spends one addition and one doubling-  on every bit whichever way it goes, working the y out at the end from the-  two x it is left with, so one division does for the whole multiplication-  where the affine code had one per step.  The multiplication is now flat, and-  quicker: sect163k1 4428 to 3431 us, sect233r1 9304 to 6806, sect283k1 13797-  to 10181, sect409k1 30826 to 20584, sect571r1 61931 to 40469.  Uniform is-  not constant time -- these are `Integer` operations, whose cost follows the-  values -- and the point with no x, which is its own negation, keeps the code-  that was there-  [#142](https://github.com/kazu-yamamoto/crypton/pull/142)-* perf(ecc): multiply points in C on curves over a prime field.  P-256 has had-  a C implementation all along; every other prime curve -- P-384, P-521,-  secp256k1 and the rest -- multiplied points with `Integer` arithmetic, which-  cannot be constant time, since what an `Integer` operation costs follows the-  value it is given.  The C walks four bits of scalar at a time, taking the-  multiple to add from a table of sixteen that it reads by touching every-  entry and keeping one with a mask, and its addition and doubling are the-  complete formulas of Renes, Costello and Batina, which answer for every pair-  of points with no case to choose between.  A P-384 multiplication goes from-  1557 to 585 us and no longer follows the scalar, ECDSA P-384 signing from-  1700 to 636 us, P-521 from 1942 to 1123.  Binary curves are unchanged, and a-  point that is not on the curve keeps the answer the Haskell gives it-  [#141](https://github.com/kazu-yamamoto/crypton/pull/141)-* fix(ecc): add at every bit in the prime-curve multiplication, which laziness-  was skipping.  The multiplication adds at every bit, set or not, so that its-  cost follows the width of the curve's order rather than the scalar -- but-  the addition was a binding only one branch of the following `if` used, so at-  a bit that was not set it stayed a thunk and was never worked out.  The cost-  followed the number of bits set in the scalar, which is the nonce when-  signing and the private key in ECDH: on P-384, 765.8 us for a scalar with-  two bits set against 1671.5 for one with 383, in a straight line between.-  Both copies of the multiplication had it, so both elliptic curve APIs were-  affected on every prime curve but P-256-  [#140](https://github.com/kazu-yamamoto/crypton/pull/140)-* fix(ecdsa): keep the P-256 signature out of `Integer` arithmetic.  The-  scalar handed to the C implementation was reduced with `mod`, a division,-  whose steps follow the number being divided -- the nonce when signing, the-  private key in ECDH.  Twice the order is more than 256 bits hold, so a-  scalar that fits is brought under the order by one masked subtraction-  instead.  The second half of a signature, `kInv * (z + r * d)`, was-  `Integer` arithmetic as well, and now goes through `scalarAdd` and-  `scalarMul`, which on P-256 are the C implementation's fixed-width-  arithmetic.  What is left on that curve is the conversion between `Integer`-  and fixed-width scalars, which is also what it costs: signing goes from 34.1-  to 39.3 us, and ECDH and the other curves are unchanged-  [#139](https://github.com/kazu-yamamoto/crypton/pull/139)-* fix(dsa,ecdsa): invert the signing nonce without a side channel.  Both-  inverted it with the extended Euclidean algorithm, whose step count and-  branches follow the bits of what it is given -- and a handful of signatures-  whose nonces are partly known give the private key away, so the nonce is-  worth as much as the key.  `Crypto.Number.ModArithmetic.inverseSafe` works-  the inverse out with Fermat's little theorem through `expSafe` instead,-  falling back on `inverse` when the modulus turns out not to be prime, so-  every answer is the one it was.  On P-256 the C implementation does it.-  Signing costs a little more: ECDSA P-256 30.6 to 34.1 us, ECDSA P-384 678.7-  to 703.4, DSA-2048 422.5 to 437.1.  Verification inverts a value that-  arrives in the signature and is left alone-  [#138](https://github.com/kazu-yamamoto/crypton/pull/138)-* perf(number): square, and multiply, faster in `expSafe`.  The product and-  the Montgomery reduction are now a full product followed by a reduction-  rather than interleaved, built out of one loop that takes its limbs eight at-  a time, and squaring works out only the products on one side of the diagonal-  and doubles their sum.  At 2048 bits the constant-time exponentiation goes-  from 3.59 to 2.15 ms, which is 1.4x GMP's own rather than 2.4x; RSA-2048-  signing goes from 0.80 to 0.63 ms and DH-2048 `getShared` from 2.43 to 1.67-  [#137](https://github.com/kazu-yamamoto/crypton/pull/137)-* fix(number): make `expSafe` hide the exponent again.  It asked integer-gmp-  for `powModSecInteger` and fell back on the ordinary `powModInteger` when-  that was missing; since integer-gmp 1.1 it is always missing, so on every-  GHC this package supports `expSafe` was the same windowed exponentiation as-  `expFast`, table indexed by the exponent's bits, for RSA, DSA, DH, ElGamal-  and Rabin alike.  It now goes to C: four bits of exponent at a time, the-  table of sixteen read by touching every entry and keeping one with a mask,-  and a Montgomery multiplication whose final subtraction is masked too.  The-  exponent's length is still visible, rounded up to a whole 64-bit word, which-  is what GMP's own `mpz_powm_sec` lets slip.  Hiding the exponent costs 1.6x-  at 512 bits and 2.4x at 2048: RSA-2048 signing goes from 0.46 to 0.80 ms and-  DH-2048 `getShared` from 1.04 to 2.43-  [#136](https://github.com/kazu-yamamoto/crypton/pull/136)-* perf(prime): stop running a Fermat test that Miller-Rabin subsumes.  Every-  candidate was tested to base 2 before the Miller-Rabin rounds, which begin-  with the same base and prove more; the primes it passed paid for it twice-  and the composites it caught were nearly all caught by trial division first.-  RSA-2048 key generation goes from 55.0 to 32.8 ms-  [#135](https://github.com/kazu-yamamoto/crypton/pull/135)-* perf(f2m): reduce the binary field by folding the top back in rather than-  taking a step per bit of excess, square a byte at a time through a table of-  the patterns a byte spreads into, and take four bits of a multiplier at a-  time rather than one.  On the 283-bit field, squaring goes from 5440 to 2068-  ns and multiplication from 7526 to 4086.  A scalar multiplication there is-  still affine, so it inverts once per addition, which is where its time now-  goes-  [#134](https://github.com/kazu-yamamoto/crypton/pull/134)-* perf(ecc): fold instead of dividing in the generic prime-curve arithmetic,-  and add the point being multiplied as the affine point it is.  These primes-  are `2^k - c` with `c` far smaller, so the top half of a product folds back-  in with a shift, a multiplication and an addition, where dividing costs four-  times as much -- above 256 bits, below which the folding costs more than it-  saves.  P-521 scalar multiplication goes from 892 to 492 us and P-384 from-  684 to 572-  [#133](https://github.com/kazu-yamamoto/crypton/pull/133)-* perf(ecc): route P-256 through the C implementation the library already had.-  `Crypto.PubKey.ECDSA` reached `cbits/p256`; `Crypto.PubKey.ECC.*`, the older-  and more widely used API, never did.  ECDSA signing goes from 590 to 28.7 us,-  verification from 726 to 91.4, and `getShared` from 1177 to 96.  On P-256-  that multiplication is now constant time, where the generic code branches on-  the scalar at every bit-  [#132](https://github.com/kazu-yamamoto/crypton/pull/132)-* Breaking change: perf(camellia): put Camellia in C, 40 to 321 MiB/s.  The-  round function ran a byte at a time in Haskell; generating the tables that-  take a byte straight to its contribution gained 14%, and the rest was the-  language.  Input that is not a whole number of blocks now raises, where the-  tail of the answer used to be uninitialised memory-  [#131](https://github.com/kazu-yamamoto/crypton/pull/131)-* Breaking change: perf(twofish): walk the blocks once and carry them in words-  rather than appending each result to what came before and going through lists-  per block.  2 MiB goes from 0.14 to 56 MiB/s, and the rate no longer falls as-  the message grows.  Input that is not a whole number of blocks now raises,-  where it used to come back longer than it went in-  [#130](https://github.com/kazu-yamamoto/crypton/pull/130)-* perf(modes): cut the message without copying the rest of it in the generic-  block cipher modes, which every cipher but AES uses, and hand whole slices to-  the cipher in the modes whose blocks do not depend on one another.  Camellia-  in CBC goes from 1.8 to 22.9 MiB/s at 1 MiB, DES CBC decryption from 0.5 to-  83, and every figure is now flat in the message length where it used to fall-  [#129](https://github.com/kazu-yamamoto/crypton/pull/129)-* Breaking change: perf(des): put DES in C.  It was carried over lists of-  `Bool`, one cons cell per bit, with the key schedule recomputed for every-  block: 0.04 MiB/s, and 3DES 0.013, against 105 and 41 for OpenSSL.  They are-  now 112 and 37.  Input that is not a whole number of blocks now raises, where-  the tail of the answer used to be uninitialised memory-  [#128](https://github.com/kazu-yamamoto/crypton/pull/128)-* perf(cmac): slice the message rather than copying what is left of it once per-  block, and chain through CBC, which is what CMAC's chaining is.  A MAC over-  4 MiB goes from 0.36 to 1628 MiB/s, which is the speed of AES-CBC itself-  [#127](https://github.com/kazu-yamamoto/crypton/pull/127)-* fix(rabin): decode OAEP without early exits, as-  `Crypto.PubKey.RSA.OAEP.unpad` has since #91.  The difference is not-  measurable against the cost of mask generation, and is structural: the scan-  across the padding no longer depends on the data-  [#126](https://github.com/kazu-yamamoto/crypton/pull/126)-* Breaking change: fix(rabin): refuse a ciphertext or a signature that is not-  below the modulus, and a ciphertext carrying a leading zero octet.  Squaring-  and the square roots that undo it work modulo n, so Basic and Rabin-Williams-  decrypted `c + n` to whatever `c` decrypted to, and all three schemes verified-  `s + n`, and `-s`, wherever they verified `s`.  `Basic.signWith` also refuses a-  padding whose first octet is zero, which the signature cannot carry: about one-  signature in 256 was one its own `verify` rejected-  [#125](https://github.com/kazu-yamamoto/crypton/pull/125)-* fix(prime): derive the Miller-Rabin witnesses from the number being tested and-  from a secret drawn once per process.  They came from one generator made once-  and shared by every call, so the witnesses for one number were the witnesses-  for every number, and testing a number again told the caller nothing it had-  not already been told.  This is the path every GHC since 9.0 takes, integer-gmp-  1.1 having no Miller-Rabin of its own-  [#124](https://github.com/kazu-yamamoto/crypton/pull/124)-* docs(elgamal): say what `signWith` requires of its ephemeral value: the range-  is 1 to p-2, not the "between 0 and p-1" the haddock claimed, and the value is-  a private key that a signature discloses if it is reused or revealed-  [#123](https://github.com/kazu-yamamoto/crypton/pull/123)-* Breaking change: fix(afis): give `split` and `merge` one answer for a parameter-  they cannot use.  They had four between them, including a division by zero for-  an expand count of zero and, for a count of one, handing the diffused data back-  as though it were the secret-  [#122](https://github.com/kazu-yamamoto/crypton/pull/122)-* Breaking change: fix(rsa): refuse a ciphertext or a signature whose integer-  representative is not below the modulus, which RFC 8017 requires in sections-  5.1.2 and 5.2.2.  `PKCS15.decrypt` and `OAEP.decrypt` decrypted `c + n` to the-  same message as `c`, and `PSS.verifyDigest` accepted `s + n` wherever it-  accepted `s`-  [#121](https://github.com/kazu-yamamoto/crypton/pull/121)-* fix(otp): search the HOTP resynchronization window without early exits.  The-  time taken read out both where in the window the client's counter was found-  and how many of the submitted values were right -- the second of which the-  answer itself does not give, being `Nothing` either way.  A call now costs one-  HMAC per counter in the window plus one per extra value, every time-  [#120](https://github.com/kazu-yamamoto/crypton/pull/120)-* Breaking change: fix(kdf): report a refused parameter as a `CryptoError` rather-  than as an `ErrorCall` carrying a string, with a `'`-suffixed variant of each-  entry point returning `CryptoFailable`.  PBKDF2 had no validation at all: a-  negative output length reached `memSet` and killed the process with SIGBUS, and-  an iteration count of zero returned 32 bytes of zeroes-  [#119](https://github.com/kazu-yamamoto/crypton/pull/119)-* perf(xts): take eight blocks at a time on AArch64 and x86-64, and dispatch XTS-  decryption through the branch table, which it had never used.  AArch64 goes-  from 1200 to 7742 MiB/s encrypting and 1166 to 7763 decrypting, x86-64 from-  1220 to 3464 and from 594 to 3461-  [#118](https://github.com/kazu-yamamoto/crypton/pull/118)-* perf(poly1305): take four blocks at a time with AVX2 on x86-64, folding the-  lanes back together weighted by the powers of r.  1347 to 4137 MiB/s-  [#117](https://github.com/kazu-yamamoto/crypton/pull/117)-* perf(ecc): work in Jacobian coordinates in both generic prime-field scalar-  multiplications, and say in `Crypto.ECC` which curves branch on a secret-  scalar.  P-384 and P-521 ECDSA are 2.3x: signing goes from 3.36 to 1.46 ms and-  from 5.96 to 2.61 ms.  P-256, which has its own C implementation, is unaffected-  [#116](https://github.com/kazu-yamamoto/crypton/pull/116)-* Breaking change: fix(padding): bound PKCS#7 padding by the block rather than by-  the whole input, which had let a block of sixteen accept a claim of twenty, and-  refuse a `ZERO` size of zero rather than dividing by it.  What `ZERO` can and-  cannot undo is now written down-  [#115](https://github.com/kazu-yamamoto/crypton/pull/115)-* perf(gcm): give x86 its own GCM decryption loop.  It fell to the generic one,-  which calls the block function once per block, and ran at a quarter the speed-  of encryption; both directions now take eight blocks at a time and fold their-  GHASH into one reduction.  AES-256-GCM decryption goes from 561 to 2733 MiB/s-  and AES-128 from 667 to 3150-  [#114](https://github.com/kazu-yamamoto/crypton/pull/114)-* perf(chacha): take eight blocks at a time with AVX2 where the machine has it,-  with the cpuid and XGETBV checks that decide.  ChaCha20 on x86-64 goes from-  900 to 2074 MiB/s-  [#113](https://github.com/kazu-yamamoto/crypton/pull/113)-* perf(chacha): do four blocks at a time with SSE2 on x86-64, where the cipher-  had no vector code at all.  ChaCha20 goes from 493 to 900 MiB/s-  [#112](https://github.com/kazu-yamamoto/crypton/pull/112)-* perf(chacha): do four blocks at a time with NEON on AArch64.  ChaCha20 goes-  from 1025 to 1955 MiB/s-  [#111](https://github.com/kazu-yamamoto/crypton/pull/111)-* feat(sha512): use the ARMv8.2 SHA-512 instructions on AArch64, which SHA-384-  and the truncated SHA-512/t variants share.  Hashing 1 MiB goes from 1.53 ms-  to 597 us.  The extension is optional, so it is asked for at runtime on both-  Apple and Linux rather than assumed-  [#110](https://github.com/kazu-yamamoto/crypton/pull/110)-* perf(gcm): drive GCM from AArch64 rather than the generic loop, with a group-  of eight blocks folding into a single GHASH reduction.  AES-128-GCM goes from-  4030 to 8266 MiB/s and AES-256 from 4043 to 7172-  [#109](https://github.com/kazu-yamamoto/crypton/pull/109)-* perf(aes): specialise the AArch64 code by key size and interleave eight-  blocks, and give CTR its own loop.  AES-256 ECB goes from 3886 to 15991-  MiB/s, CTR from 2935 to 13567 and CBC decryption from 4366 to 15807-  [#108](https://github.com/kazu-yamamoto/crypton/pull/108)--* perf(aes): build the AES-NI paths on Windows, which was missing from the list of-  systems that compile them.  Windows builds have been doing AES, and GHASH with it,-  in the generic C-  [#107](https://github.com/kazu-yamamoto/crypton/pull/107)-* fix(armv8): compile the AArch64 sources on a toolchain whose baseline lacks the-  crypto extensions.  They had not built with GCC on AArch64 Linux since #100; CI now-  builds and tests there-  [#106](https://github.com/kazu-yamamoto/crypton/pull/106)-* perf(gcm): fold four GHASH blocks into one reduction.  AES-256-GCM is 1.6x at 1 KiB-  and 2.6x at 64 KiB on Apple silicon, and the x86 paths gain the same structure-  [#105](https://github.com/kazu-yamamoto/crypton/pull/105)-* perf(sha256): use the ARMv8 SHA-2 instructions on AArch64.  SHA-256 and SHA-224 are-  5.5x-  [#104](https://github.com/kazu-yamamoto/crypton/pull/104)-* ci: keep the macOS jobs from queueing behind each other, and supersede a branch's-  earlier run-  [#103](https://github.com/kazu-yamamoto/crypton/pull/103)-* perf(aes): use PMULL for GHASH on AArch64-  [#102](https://github.com/kazu-yamamoto/crypton/pull/102)-* ci: ask cabal where its caches live rather than assuming, and keep the build-  products in the cache-  [#101](https://github.com/kazu-yamamoto/crypton/pull/101)-* perf(aes): use the ARMv8 cryptographic extensions on AArch64.  With the GHASH work-  in #102 and #105, AES-256-ECB goes from 121 to 2992 MiB/s and AES-256-GCM from 92 to-  2318 MiB/s on Apple silicon-  [#100](https://github.com/kazu-yamamoto/crypton/pull/100)-* build(bench): move the benchmarks from gauge, which is no longer maintained, to-  tasty-bench, and let them resolve on a current GHC-  [#99](https://github.com/kazu-yamamoto/crypton/pull/99)-* Breaking change: fix(padding): reject a `PKCS7` block size outside 1..255.  `pad`-  raises and `unpad` returns `Nothing`, where both previously narrowed the size to a-  `Word8` and silently agreed on the wrong value-  [#98](https://github.com/kazu-yamamoto/crypton/pull/98)-* feat(elgamal): fix `Crypto.PubKey.ElGamal` and expose it-  [#97](https://github.com/kazu-yamamoto/crypton/pull/97)-* docs(bcrypt): say that only the first 72 bytes of a password count-  [#96](https://github.com/kazu-yamamoto/crypton/pull/96)-* test: move the test suite from tasty to hspec, with hspec-discover.  `cabal-version`-  is now 2.0-  [#95](https://github.com/kazu-yamamoto/crypton/pull/95)--* feat(aead): add `tryAeadSimpleDecrypt`, which takes the tag length as its own argument instead of reading it off the supplied tag-  [#94](https://github.com/kazu-yamamoto/crypton/pull/94)-* Breaking change: feat(dh): add `tryGetShared` to `Crypto.PubKey.DH` and `Crypto.PubKey.ECC.DH`, reporting a rejected peer value as `CryptoFailable`; `getShared` is now defined in terms of it and so raises a `CryptoError` rather than an `ErrorCall`-  [#93](https://github.com/kazu-yamamoto/crypton/pull/93)-* fix(otp): compare TOTP candidates without an early exit-  [#92](https://github.com/kazu-yamamoto/crypton/pull/92)-* fix(rsa): drop the early exits from PKCS#1 v1.5 and OAEP unpadding-  [#91](https://github.com/kazu-yamamoto/crypton/pull/91)-* Breaking change: fix(argon2): report invalid options as `CryptoFailed` rather than raising, adding `CryptoError_ParameterInvalid` to `CryptoError`-  [#90](https://github.com/kazu-yamamoto/crypton/pull/90)-* Breaking change: fix(dh): validate the peer public number, and size the shared secret from `p` rather than `params_bits`-  [#89](https://github.com/kazu-yamamoto/crypton/pull/89)-* fix(dsa): do not crash on values that are not invertible modulo `q`-  [#88](https://github.com/kazu-yamamoto/crypton/pull/88)-* Breaking change: fix(ecdh): validate the peer point before the exchange-  [#87](https://github.com/kazu-yamamoto/crypton/pull/87)-* Breaking change: fix(pkcs15): reject PKCS#1 v1.5 signatures of the wrong length or out of range-  [#86](https://github.com/kazu-yamamoto/crypton/pull/86)-* Breaking change: fix(otp): require a digest long enough for RFC 4226 dynamic truncation, which was reading past the end of the MAC-  [#85](https://github.com/kazu-yamamoto/crypton/pull/85)-* fix(ecc): accept zero-x P-256 shared secret-  [#84](https://github.com/kazu-yamamoto/crypton/pull/84)-* fix(p256): accept valid edge-case points-  [#83](https://github.com/kazu-yamamoto/crypton/pull/83)-* Breaking change: fix(hkdf): enforce output length limit-  [#82](https://github.com/kazu-yamamoto/crypton/pull/82)-* Breaking change: fix(ed25519): reject non-canonical signatures-  [#81](https://github.com/kazu-yamamoto/crypton/pull/81)-* Support GHC 9.14; `tested-with` now covers 9.10.2, 9.12.4 and 9.14.1-  [#74](https://github.com/kazu-yamamoto/crypton/pull/74)--### API changes--* New exports: `Crypto.OTP.minimumDigestSize`, `Crypto.PubKey.DH.tryGetShared`,-  `Crypto.PubKey.ECC.DH.tryGetShared`, `Crypto.Cipher.Types.AEAD.tryAeadSimpleDecrypt`,-  `Crypto.Number.ModArithmetic.inverseSafe`, `Crypto.PubKey.ECC.Prim.scalarInverse`,-  `scalarAdd` and `scalarMul`, `Crypto.PubKey.ECC.P256.scalarReduce`,-  and the whole of `Crypto.PubKey.ElGamal`, which was present but not exposed.-  The variant of an entry point that reports a refusal rather than raising is-  named `try` followed by the name it varies, `tryExpand` beside `expand`.  A-  trailing apostrophe was the obvious spelling and is what these were called-  until shortly before release; it collides too easily, since a caller that-  imports one of these modules unqualified and has its own `expand'` or-  `split'` no longer compiles, and `tls` did.  `Safe` was considered and set-  aside: this library already uses that suffix for something else, in-  `Crypto.Number.ModArithmetic.expSafe` and `inverseSafe` and in-  `Crypto.PubKey.ECC.P256.scalarInvSafe`, where it means the value being-  worked on stays out of the timing.-  The KDFs gained a variant of each entry point that can refuse its parameters,-  returning `CryptoFailable` instead of raising: `Crypto.KDF.Scrypt.tryGenerate`,-  `Crypto.KDF.BCrypt.tryBcrypt`, `Crypto.KDF.BCryptPBKDF.tryGenerate` and-  `tryHashInternal`, `Crypto.KDF.HKDF.tryExpand`, `Crypto.KDF.PBKDF2.tryGenerate` and-  `tryFastPBKDF2_SHA1`, `tryFastPBKDF2_SHA256` and `tryFastPBKDF2_SHA512`, and-  `Crypto.Data.AFIS.trySplit` and `tryMerge`.  These are additions and break nothing.-* Breaking change: `CryptoError_ParameterInvalid` is added to `CryptoError`.  It is-  appended, so the `Enum` values of the existing constructors are unchanged, but an-  exhaustive `case` without a wildcard will warn.  Adding a constructor to an exported-  datatype is what requires a major version bump under the PVP, which would have been-  1.2.0; this release goes to 2.0.0.  Everything else below changes behaviour rather-  than types.-* Breaking change: `getShared` in both DH modules raises a `CryptoError` where it-  previously raised an `ErrorCall`, since it is now defined in terms of `tryGetShared`.-  The same is now true of `Crypto.KDF.Scrypt.generate`, `Crypto.KDF.BCrypt.bcrypt`,-  `Crypto.KDF.BCryptPBKDF.generate` and `hashInternal`, and `Crypto.Data.AFIS.split`-  and `merge`, each of which is defined in terms of the variant above.-* Breaking change: input that used to be accepted is now rejected -- a digest shorter-  than 20 bytes in `Crypto.OTP.hotp`, a signature of the wrong length or out of range-  in `Crypto.PubKey.RSA.PKCS15.verify`, an off-curve peer point or a peer public number-  outside `1 < y < p-1` in `getShared`, an output beyond 255 blocks in-  `Crypto.KDF.HKDF.expand`, a non-canonical Ed25519 signature, and `Options` the-  implementation refuses in `Crypto.KDF.Argon2.hash`.-* Breaking change: a value at or above the modulus is now rejected where it used to be-  reduced and accepted -- a ciphertext in `Crypto.PubKey.RSA.PKCS15.decrypt` and-  `Crypto.PubKey.RSA.OAEP.decrypt`, a signature in `Crypto.PubKey.RSA.PSS.verify`, and-  both, along with a negated signature and a ciphertext with a leading zero octet, in-  the three `Crypto.PubKey.Rabin.*` schemes.-* Breaking change: parameters that used to be accepted are now refused -- an iteration-  count below one or a negative output length in `Crypto.KDF.PBKDF2`, an expand count-  below two or a secret of no bytes in `Crypto.Data.AFIS`, a `PKCS7` claim longer than-  the block and a `ZERO` size of zero in `Crypto.Data.Padding`, and a signature padding-  whose first octet is zero in `Crypto.PubKey.Rabin.Basic.signWith`.-* Breaking change: DES, 3DES, Twofish and Camellia now raise on input that is not a-  whole number of blocks, as AES already did.  Before, DES and Camellia returned an-  answer whose tail was never written -- uninitialised memory -- and Twofish returned-  more than it was given, the missing bytes read as zero.-* Breaking change: `Crypto.Data.Padding.pad` raises on a `PKCS7` block size outside-  1..255, and `unpad` returns `Nothing` for one, where both used to narrow the size to-  a `Word8` and hand back something other than what was padded.-* No exported function changed its signature.--## 1.1.5--* fix(aead): reject undersized tags-  [#80](https://github.com/kazu-yamamoto/crypton/pull/80)-* fix(aes): refuse a zero-length AES-GCM IV-  [#79](https://github.com/kazu-yamamoto/crypton/pull/79)-* fix(p256): prevent crashes when validating valid points-  [#78](https://github.com/kazu-yamamoto/crypton/pull/78)-* feat(asn1): add SHA-3 HashAlgorithmASN1 instances for PKCS#1 v1.5-  [#77](https://github.com/kazu-yamamoto/crypton/pull/77)-* OCB3 conformance-  [#76](https://github.com/kazu-yamamoto/crypton/pull/76)--## 1.1.4--* Generic instance for RSA PublicKey and PrivateKey--## 1.1.3--* Ensure that `pointAdd` in `PubKey.ECC.P256` treats the point at infinity as the additive identity.-  [#73](https://github.com/kazu-yamamoto/crypton/pull/73)--## 1.1.2--* Preparing `ram` v0.22.-* Generalizing RSA encrypt/decrypt to manipulate ScrubbedBytes directly.--## 1.1.1--* On iOS, ScrubbedBytes based hashing is used for seedNew. On other-  plateforms, entropy is used directly as used to be.-  [#71](https://github.com/kazu-yamamoto/crypton/pull/71)--## 1.1.0--* Removing "basement" and "memory".-  [#67](https://github.com/kazu-yamamoto/crypton/pull/67)---## 1.0.7--* Stop depending on basement, use upstream dependencies instead-* Stop transitively depending on basement by depending on ram.--## 1.0.6--* Fix test failures on less common 64-bit arches.-  [#65](https://github.com/kazu-yamamoto/crypton/pull/65)--## 1.0.5--* Setter/Getter for ChaCha counter.-  [#63](https://github.com/kazu-yamamoto/crypton/pull/63)-* Add simple interface to generate full blocks-  [#60](https://github.com/kazu-yamamoto/crypton/pull/60)-* Avoid `ghc-prim` dependency.-  [#61](https://github.com/kazu-yamamoto/crypton/pull/61)--## 1.0.4--* Ed448.sign: avoid extra re-derive of public key.-  [#48](https://github.com/kazu-yamamoto/crypton/pull/48)--## 1.0.3--* Make sign of Ed25519/Ed448 safer. The public key parameter is-  ignored and its public key is generated from the secret key-  parameter to prevent Double Public Key Signing Function Oracle-  Attack.-  [#47](https://github.com/kazu-yamamoto/crypton/pull/47)--## 1.0.2--* Deterministic Nonce Generation for ECDSA-  [#46](https://github.com/kazu-yamamoto/crypton/pull/46)-* ECDSA Signature Normalization.-  [#45](https://github.com/kazu-yamamoto/crypton/pull/45)-* Add Full Test Suite from RFC 6979.-  [#44](https://github.com/kazu-yamamoto/crypton/pull/44)-* ECDSA with Public Key Recovery.-  [#43](https://github.com/kazu-yamamoto/crypton/pull/43)-* Providing necessary features for HPKE.-  [#42](https://github.com/kazu-yamamoto/crypton/pull/42)--## 1.0.1--* Update decaf library.-  [#38](https://github.com/kazu-yamamoto/crypton/pull/38)-* Add TypeOperators language extension to EdDSA.hs.-  [#36](https://github.com/kazu-yamamoto/crypton/pull/36)--## 1.0.0--* Versions follow the standard version policy.-* Removing pthread stuff.-  [#32](https://github.com/kazu-yamamoto/crypton/pull/32)--## 0.34--* Hashing getRandomBytes before using as Seed for ChaChaDRG-  [#24](https://github.com/kazu-yamamoto/crypton/pull/24)-* Add support for XChaCha and XChaChaPoly1305-  [#18](https://github.com/kazu-yamamoto/crypton/pull/18)-* Strict byteArray of IV c-  [#16](https://github.com/kazu-yamamoto/crypton/pull/16)--## 0.33--* Add "crypton_" prefix to the final C symbols.-  [#9](https://github.com/kazu-yamamoto/crypton/pull/9)--## 0.32--* All C symbols now have the "crypton_" prefix.-  [#7](https://github.com/kazu-yamamoto/crypton/pull/7)-  [#8](https://github.com/kazu-yamamoto/crypton/pull/8)--## 0.31--* Crypton is forked from cryptonite with the original authors permission.-* Ignoring exceptons from hClose to read the next entropy-  [#1](https://github.com/kazu-yamamoto/crypton/pull/1)-* Enabling the support_pclmuldq flag by default.--## 0.30--* Fix some C symbol blake2b prefix to be cryptonite_ prefix (fix mixing with other C library)-* add hmac-lazy-* Fix compilation with GHC 9.2-* Drop support for GHC8.0, GHC8.2, GHC8.4, GHC8.6--## 0.29--* advance compilation with gmp breakage due to change upstream-* Add native EdDSA support--## 0.28--* Add hash constant time capability-* Prevent possible overflow during hashing by hashing in 4GB chunks--## 0.27--* Optimise AES GCM and CCM-* Optimise P256R1 implementation-* Various AES-NI building improvements-* Add better ECDSA support-* Add XSalsa derive-* Implement square roots for ECC binary curve-* Various tests and benchmarks--## 0.26--* Add Rabin cryptosystem (and variants)-* Add bcrypt_pbkdf key derivation function-* Optimize Blowfish implementation-* Add KMAC (Keccak Message Authentication Code)-* Add ECDSA sign/verify digest APIs-* Hash algorithms with runtime output length-* Update blake2 to latest upstream version-* RSA-PSS with arbitrary key size-* SHAKE with output length not divisible by 8-* Add Read and Data instances for Digest type-* Improve P256 scalar primitives-* Fix hash truncation bug in DSA-* Fix cost parsing for bcrypt-* Fix ECC failures on arm64-* Correction to PKCS#1 v1.5 padding-* Use powModSecInteger when available-* Drop GHC 7.8 and GHC 7.10 support, refer to pkg-guidelines-* Optimise GCM mode-* Add little endian serialization of integer--## 0.25--* Improve digest binary conversion efficiency-* AES CCM support-* Add MonadFailure instance for CryptoFailable-* Various misc improvements on documentation-* Edwards25519 lowlevel arithmetic support-* P256 add point negation-* Improvement in ECC (benchmark, better normalization)-* Blake2 improvements to context size-* Use gauge instead of criterion-* Use haskell-ci for CI scripts-* Improve Digest memory representation to be 2 less Ints and one less boxing-  moving from `UArray` to `Block`--## 0.24--* Ed25519: generateSecret & Documentation updates-* Repair tutorial-* RSA: Allow signing digest directly-* IV add: fix overflow behavior-* P256: validate point when decoding-* Compilation fix with deepseq disabled-* Improve Curve448 and use decaf for Ed448-* Compilation flag blake2 sse merged in sse support-* Process unaligned data better in hashes and AES, on architecture needing alignment-* Drop support for ghc 7.6-* Add ability to create random generator Seed from binary data and-  loosen constraint on ChaChaDRG seed from ByteArray to ByteArrayAccess.-* Add 3 associated types with the HashAlgorithm class, to get-  access to the constant for BlockSize, DigestSize and ContextSize at the type level.-  the related function that this replaced will be deprecated in later release, and-  eventually removed.--API CHANGES:--* Improve ECDH safety to return failure for bad inputs (e.g. public point in small order subgroup).-  To go back to previous behavior you can replace `ecdh` by `ecdhRaw`. It's recommended to-  use `ecdh` and handle the error appropriately.-* Users defining their own HashAlgorithm needs to define the-  HashBlockSize, HashDigest, HashInternalContextSize associated types--## 0.23--* Digest memory usage improvement by using unpinned memory-* Fix generateBetween to generate within the right bounds-* Add pure Twofish implementation-* Fix memory allocation in P256 when using a temp point-* Consolidate hash benchmark code-* Add Nat-length Blake2 support (GHC > 8.0)-* Update tutorial--## 0.22--* Add Argon2 (Password Hashing Competition winner) hash function-* Update blake2 to latest upstream version-* Add extra blake2 hashing size-* Add faster PBKDF2 functions for SHA1/SHA256/SHA512-* Add SHAKE128 and SHAKE256-* Cleanup prime generation, and add tests-* Add Time-based One Time Password (TOTP) and HMAC-based One Time Password (HOTP)-* Rename Ed448 module name to Curve448, old module name still valid for now--## 0.21--* Drop automated tests with GHC 7.0, GHC 7.4, GHC 7.6. support dropped, but probably still working.-* Improve non-aligned support in C sources, ChaCha and SHA3 now probably work on arch without support for unaligned access. not complete or tested.-* Add another ECC framework that is more flexible, allowing different implementations to work instead of-  the existing Pure haskell NIST implementation.-* Add ECIES basic primitives-* Add XSalsa20 stream cipher-* Process partial buffer correctly with Poly1305--## 0.20--* Fixed hash truncation used in ECDSA signature & verification (Olivier Chéron)-* Fix ECDH when scalar and coordinate bit sizes differ (Olivier Chéron)-* Speed up ECDSA verification using Shamir's trick (Olivier Chéron)-* Fix rdrand on windows--## 0.19--* Add tutorial (Yann Esposito)-* Derive Show instance for better interaction with Show pretty printer (Eric Mertens)--## 0.18--* Re-used standard rdrand instructions instead of bytedump of rdrand instruction-* Improvement to F2m, including lots of tests (Andrew Lelechenko)-* Add error check on salt length in bcrypt--## 0.17--* Add Miyaguchi-Preneel construction (Kei Hibino)-* Fix buffer length in scrypt (Luke Taylor)-* build fixes for i686 and arm related to rdrand--## 0.16--* Fix basepoint for Ed448--* Enable 64-bit Curve25519 implementation--## 0.15--* Fix serialization of DH and ECDH--## 0.14--* Reduce size of SHA3 context instead of allocating all-size fit memory. save-  up to 72 bytes of memory per context for SHA3-512.-* Add a Seed capability to the main DRG, to be able to debug/reproduce randomized program-  where you would want to disable the randomness.-* Add support for Cipher-based Message Authentication Code (CMAC) (Kei Hibino)-* *CHANGE* Change the `SharedKey` for `Crypto.PubKey.DH` and `Crypto.PubKey.ECC.DH`,-  from an Integer newtype to a ScrubbedBytes newtype. Prevent mistake where the-  bytes representation is generated without the right padding (when needed).-* *CHANGE* Keep The field size in bits, in the `Params` in `Crypto.PubKey.DH`,-  moving from 2 elements to 3 elements in the structure.--## 0.13--* *SECURITY* Fix buffer overflow issue in SHA384, copying 16 extra bytes from-  the SHA512 context to the destination memory pointer leading to memory-  corruption, segfault. (Mikael Bung)--## 0.12--* Fix compilation issue with Ed448 on 32 bits machine.--## 0.11--* Truncate hashing correctly for DSA-* Add support for HKDF (RFC 5869)-* Add support for Ed448-* Extends support for Blake2s to 224 bits version.-* Compilation workaround for old distribution (RHEL 4.1)-* Compilation fix for AIX-* Compilation fix with AESNI and ghci compiling C source in a weird order.-* Fix example compilation, typo, and warning--## 0.10--* Add reference implementation of blake2 for non-SSE2 platform-* Add support\_blake2\_sse flag--## 0.9--* Quiet down unused module imports-* Move Curve25519 over to Crypto.Error instead of using Either String.-* Add documentation for ChaChaPoly1305-* Add missing documentation for various modules-* Add a way to create Poly1305 Auth tag.-* Added support for the BLAKE2 family of hash algorithms-* Fix endianness of incrementNonce function for ChaChaPoly1305--## 0.8--* Add support for ChaChaPoly1305 Nonce Increment (John Galt)-* Move repository to the haskell-crypto organisation--## 0.7--* Add PKCS5 / PKCS7 padding and unpadding methods-* Fix ChaChaPoly1305 Decryption-* Add support for BCrypt (Luke Taylor)--## 0.6--* Add ChaChaPoly1305 AE cipher-* Add instructions in README for building on old OSX-* Fix blocking /dev/random Andrey Sverdlichenko--## 0.5--* Fix all strays exports to all be under the cryptonite prefix.--## 0.4--* Add a System DRG that represent a referentially transparent of evaluated bytes-  while using lazy evaluation for future entropy values.--## 0.3--* Allow drgNew to run in any MonadRandom, providing cascading initialization-* Remove Crypto.PubKey.HashDescr in favor of just having the algorithm-  specified in PKCS15 RSA function.-* Fix documentation in cipher sub section (Luke Taylor)-* Cleanup AES dead functions (Luke Taylor)-* Fix Show instance of Digest to display without quotes similar to cryptohash-* Use scrubbed bytes instead of bytes for P256 scalar--## 0.2--* Fix P256 compilation and exactness, + add tests-* Add a raw memory number serialization capability (i2osp, os2ip)-* Improve tests for number serialization-* Improve tests for ECC arithmetics-* Add Ord instance for Digest (Nicolas Di Prima)-* Fix entropy compilation on windows 64 bits.--## 0.1--* Initial release+## 2.2.5++**RSA private-key operations check their CRT result.**  A fault in one+half of the computation could hand out a signature that reveals a prime of+the key; the result is now raised back to the public exponent first, as+OpenSSL does, at a cost of two or three per cent on signing.++* perf(rsa): verification in C at every RSA size on Apple AArch64+  [#340](https://github.com/kazu-yamamoto/crypton/pull/340)+* perf(rsa): a generated squaring at 48 limbs on AArch64+  [#341](https://github.com/kazu-yamamoto/crypton/pull/341)+* perf(rsa): verification without lists or Integer round trips+  [#342](https://github.com/kazu-yamamoto/crypton/pull/342)+* ci: time out instead of waiting on a stalled package mirror+  [#343](https://github.com/kazu-yamamoto/crypton/pull/343)+* fix(rsa): check the CRT result before letting it out+  [#344](https://github.com/kazu-yamamoto/crypton/pull/344)+* perf(hash): one context for a one-shot hash and for MGF1+  [#345](https://github.com/kazu-yamamoto/crypton/pull/345)+* perf(rsa): RSA-3072's 24-limb square from 8-limb pieces on x86-64+  [#346](https://github.com/kazu-yamamoto/crypton/pull/346)++## 2.2.4++**New module `Crypto.PubKey.RSA.Blinded`.**  A `BlindedKey` carries RSA+blinding from one signature or decryption to the next.  The existing+`*Safer` functions use it through a small process-wide table, and+verification keeps R^2 per modulus in another.++* perf(rsa): keep the blinding between operations+  [#335](https://github.com/kazu-yamamoto/crypton/pull/335)+* perf(rsa): a generated Montgomery multiplication for AArch64+  [#336](https://github.com/kazu-yamamoto/crypton/pull/336)+* fix: the C warnings CI reports+  [#337](https://github.com/kazu-yamamoto/crypton/pull/337)+* perf(rsa): both CRT halves at once with AVX-512 IFMA+  [#338](https://github.com/kazu-yamamoto/crypton/pull/338)+* perf(rsa): keep R^2 per modulus for verification+  [#339](https://github.com/kazu-yamamoto/crypton/pull/339)++## 2.2.3++**crypton builds on Windows again.**  2.2.2's tarball lacked a header that+only Windows includes, so it built from git but not from Hackage.  With+2.2.2, `-f-support_s2n_bignum` works around it.++* bench: RSA-3072 in the tables+  [#331](https://github.com/kazu-yamamoto/crypton/pull/331)+* perf(rsa): a generated Montgomery squaring for AArch64+  [#333](https://github.com/kazu-yamamoto/crypton/pull/333)+* fix: ship the Windows wrapper for s2n-bignum+  [#334](https://github.com/kazu-yamamoto/crypton/pull/334)++## 2.2.2++**crypton builds on an Intel Mac again.**  From 2.1.2 onwards, Apple's+assembler rejected one of the vendored RSA assembly files, so the default+configuration did not build on x86-64 macOS.  With an older release,+`-f-support_s2n_bignum` works around it.++* doc: the Ed25519 verify step was the AVX2 flag, and the tables move to 2.2.1+  [#318](https://github.com/kazu-yamamoto/crypton/pull/318)+* test: the window the exponentiation walks in, and the macro that hid it+  [#319](https://github.com/kazu-yamamoto/crypton/pull/319)+* perf: the Montgomery constant in assembly, on both architectures+  [#320](https://github.com/kazu-yamamoto/crypton/pull/320)+* perf: verification need not hide an exponent that is public+  [#321](https://github.com/kazu-yamamoto/crypton/pull/321)+* doc: the generic AArch64 routines, measured at last+  [#322](https://github.com/kazu-yamamoto/crypton/pull/322)+* doc: a VM's CPUID is not the hardware: two descriptions that said otherwise+  [#323](https://github.com/kazu-yamamoto/crypton/pull/323)+* test: switches to assume what CPUID hides, for testing only+  [#324](https://github.com/kazu-yamamoto/crypton/pull/324)+* perf: RSA-3072's halves through s2n-bignum on x86-64+  [#325](https://github.com/kazu-yamamoto/crypton/pull/325)+* perf: 3072- and 4096-bit secure exponentiation through s2n-bignum+  [#326](https://github.com/kazu-yamamoto/crypton/pull/326)+* test: the two AES differential tests, in CI+  [#327](https://github.com/kazu-yamamoto/crypton/pull/327)+* perf: s2n-bignum on x86-64 Windows+  [#328](https://github.com/kazu-yamamoto/crypton/pull/328)+* fix: crypton builds on an Intel Mac again+  [#329](https://github.com/kazu-yamamoto/crypton/pull/329)+* ci: link with the Command Line Tools on macOS+  [#330](https://github.com/kazu-yamamoto/crypton/pull/330)++## 2.2.1++**A binary built on one x86-64 machine runs on another again.**  Since+2.1.8 the `-mavx2 -mbmi2` written for the ML-KEM and ML-DSA backends have+been compiling every C file in the library -- `cc-options` in a conditional+block belong to the component, not to the block -- so the compiler put AVX2+and BMI2 into ordinary code that has no dispatch and asks nothing at+runtime: Ed25519, Argon2, Curve25519, P-256, Ed448, and the older hashes+among them.  A binary built where AVX2 exists then died with `SIGILL` where+it does not, in Ed25519 rather than in anything post-quantum.++The two backends also chose their implementation when the library was+compiled and called it unchecked.  They ask the processor now, as everything+else in crypton that uses an instruction beyond the baseline already did,+and fall back to the portable C where the answer is no.  AArch64 asks about+the ARMv8.4 Keccak instructions for the same reason.++Nothing in the API changes.++* fix: stop raising the x86-64 baseline for the whole library+  [#317](https://github.com/kazu-yamamoto/crypton/pull/317)+* fix: ask the processor before using the ML-KEM and ML-DSA backends+  [#317](https://github.com/kazu-yamamoto/crypton/pull/317)+* test(ct): link the processor check into the ML-KEM backend build+  [#317](https://github.com/kazu-yamamoto/crypton/pull/317)++## 2.2.0++`ProcessorOption` is no longer a sum of constructors.  It is a number with+pattern synonyms for the names it knows, so it has `Ord` now and has lost+`Enum` and `Data`, and a `case` over the names needs a catch-all.  `RDRAND`+is gone from it, and with it the `support_rdrand` flag.  An AArch64 machine+reports `ARMAES` and `ARMPMULL` where it used to report `AESNI` and `PCLMUL`.++Two more things change under a caller without the type checker saying so:+`MonadRandom IO` draws from a per-thread generator rather than reading the+system on every call, and a missing or failing entropy source raises+`EntropyError` rather than an `ErrorCall` or an `IOException`.++* feat: take entropy from the kernel without a file descriptor+  [#300](https://github.com/kazu-yamamoto/crypton/pull/300)+* feat: a ChaCha20 generator per operating system thread+  [#300](https://github.com/kazu-yamamoto/crypton/pull/300)+* feat: MonadRandom IO goes through the per-thread generator+  [#300](https://github.com/kazu-yamamoto/crypton/pull/300)+* fix: say that the system gave no entropy, rather than error+  [#300](https://github.com/kazu-yamamoto/crypton/pull/300)+* fix: stop using RDRAND at all, since it cannot add anything+  [#300](https://github.com/kazu-yamamoto/crypton/pull/300)+* feat: name every processor feature crypton dispatches on, not three of them+  [#302](https://github.com/kazu-yamamoto/crypton/pull/302)+* fix: an AArch64 machine no longer reports AESNI and PCLMUL+  [#302](https://github.com/kazu-yamamoto/crypton/pull/302)+* chore(rsa): deprecate `sign`, `signSafer` and `verify` in favour of them+  [#305](https://github.com/kazu-yamamoto/crypton/pull/305)+* fix: compute the portable AES and GHASH without a table+  [#308](https://github.com/kazu-yamamoto/crypton/pull/308)+* feat: use the AES instructions on 32-bit ARM+  [#309](https://github.com/kazu-yamamoto/crypton/pull/309)+* test: skip the accelerated constant-time driver where the processor has no instructions+  [#310](https://github.com/kazu-yamamoto/crypton/pull/310)+* feat: use the POWER8 AES and GHASH instructions on ppc64le+  [#311](https://github.com/kazu-yamamoto/crypton/pull/311)+* test: the big-endian harness now asks about AES+  [#312](https://github.com/kazu-yamamoto/crypton/pull/312)+* doc(tutorial): compile the examples, and teach the things people come for+  [#313](https://github.com/kazu-yamamoto/crypton/pull/313)++## 2.1.10++* build: ask the processor for AES-NI on every x86 system, not four of them+  [#307](https://github.com/kazu-yamamoto/crypton/pull/307)+* fix: ask one place which ARMv8 instruction sets the machine has+  [#307](https://github.com/kazu-yamamoto/crypton/pull/307)++## 2.1.9++* feat(rsa): PKCS#1 v1.5 operations that take a digest, and ones that take a DigestInfo+  [#305](https://github.com/kazu-yamamoto/crypton/pull/305)+* feat(dsa): sign and verify over a digest+  [#305](https://github.com/kazu-yamamoto/crypton/pull/305)+* feat(elgamal): sign and verify over a digest+  [#305](https://github.com/kazu-yamamoto/crypton/pull/305)+* feat(rabin): Rabin-Williams and Modified Rabin sign and verify over a digest+  [#305](https://github.com/kazu-yamamoto/crypton/pull/305)++## 2.1.8++* chore: stop hiding foldl' from Prelude+  [#294](https://github.com/kazu-yamamoto/crypton/pull/294)+* chore: ask for hidden visibility only where the format has it+  [#295](https://github.com/kazu-yamamoto/crypton/pull/295)+* feat: ML-KEM and ML-DSA, through mlkem-native and mldsa-native+  [#297](https://github.com/kazu-yamamoto/crypton/pull/297)++## 2.1.7++RSA-PSS verification was accepting an encoding RFC 8017 says to refuse.  It+is a conformance fault rather than a forgery: producing such a signature takes+the private key, so a third party holding a valid signature cannot turn it+into one of these.++* chore(p256): drop two declarations nothing defines+  [#292](https://github.com/kazu-yamamoto/crypton/pull/292)+* fix(pss): refuse an encoding with a bit set outside emBits+  [#293](https://github.com/kazu-yamamoto/crypton/pull/293)++## 2.1.6++The `license:` field now says what the tree holds -- `BSD-3-Clause AND MIT+AND ISC` -- and `license-files:` lists all five texts.  **Nothing is required+of a user that was not required before**; the field was simply incomplete.++* Carry the MIT notice for the parts that follow fusion, and say what the tree holds+  [#267](https://github.com/kazu-yamamoto/crypton/pull/267)+* fix: two preconditions the Haskell layer did not enforce+  [#288](https://github.com/kazu-yamamoto/crypton/pull/288)+* fix(x86): stop casting a packed block to __m128i *+  [#289](https://github.com/kazu-yamamoto/crypton/pull/289)+* fix(c): say that the digest pointers are never null+  [#290](https://github.com/kazu-yamamoto/crypton/pull/290)+* fix(pbkdf2): refuse a digest larger than its block at compile time+  [#291](https://github.com/kazu-yamamoto/crypton/pull/291)++## 2.1.5++2.1.3 and 2.1.4 cannot be built with GCC 14 or newer.  This release is+that fix.++* Watch for a primitive fallen off its fast path+  [#277](https://github.com/kazu-yamamoto/crypton/pull/277)+* The performance tables for 2.1.4+  [#278](https://github.com/kazu-yamamoto/crypton/pull/278)+* The x86-64 table, on the machine the README names+  [#279](https://github.com/kazu-yamamoto/crypton/pull/279)+* perf(armv8): GHASH against a twisted H, a quarter faster+  [#281](https://github.com/kazu-yamamoto/crypton/pull/281)+* ct: run the constant-time harness on AArch64, and put its AES to it+  [#283](https://github.com/kazu-yamamoto/crypton/pull/283)+* Fix crypton_sha1_x86_do_chunk signature+  [#284](https://github.com/kazu-yamamoto/crypton/pull/284)+* ci: build the C with a compiler stricter than this matrix has+  [#285](https://github.com/kazu-yamamoto/crypton/pull/285)++## 2.1.4++2.1.3 could not be built from Hackage in the default configuration, and is+deprecated there.  This release is that fix.++* Add p256 header files to cabal extra-source-files+  [#271](https://github.com/kazu-yamamoto/crypton/pull/271)+* Build what is published, not only what is checked out+  [#272](https://github.com/kazu-yamamoto/crypton/pull/272)+* SHA-256 on AArch64 is 5.4x slower in 2.1.3 than in 2.1.2+  [#274](https://github.com/kazu-yamamoto/crypton/pull/274)+* The target attribute spelling gcc before 13 understands+  [#276](https://github.com/kazu-yamamoto/crypton/pull/276)++## 2.1.3++* fix(aead): reject missing and oversized authentication tags+  [#233](https://github.com/kazu-yamamoto/crypton/pull/233)+* fix(chacha): remove redundant Word8 import+  [#234](https://github.com/kazu-yamamoto/crypton/pull/234)+* test(number): cover the two modulus sizes the assembly runs at+  [#235](https://github.com/kazu-yamamoto/crypton/pull/235)+* perf(rsa): swap the buffers instead of copying them back+  [#236](https://github.com/kazu-yamamoto/crypton/pull/236)+* perf(rsa): scan the exponentiation's table four limbs at a time+  [#237](https://github.com/kazu-yamamoto/crypton/pull/237)+* fix(gcm): write the field doubling from its definition+  [#238](https://github.com/kazu-yamamoto/crypton/pull/238)+* doc(gcm): carry the MIT notice for the parts that follow fusion+  [#239](https://github.com/kazu-yamamoto/crypton/pull/239)+* perf(ed25519): the base point multiplication through s2n-bignum+  [#240](https://github.com/kazu-yamamoto/crypton/pull/240)+* perf(gcm): AES-GCM through the 512-bit VAES and VPCLMULQDQ+  [#241](https://github.com/kazu-yamamoto/crypton/pull/241)+* doc: the README said AVX-512 was not used, and it is+  [#242](https://github.com/kazu-yamamoto/crypton/pull/242)+* perf(ecdsa): P-256 verification multiplies both scalars at once+  [#243](https://github.com/kazu-yamamoto/crypton/pull/243)+* perf(rsa): four limbs and two carry chains on AArch64+  [#244](https://github.com/kazu-yamamoto/crypton/pull/244)+* perf(rsa): write out the ragged end of the AArch64 row+  [#245](https://github.com/kazu-yamamoto/crypton/pull/245)+* perf(rsa): build R^2 by squaring, not by doubling+  [#246](https://github.com/kazu-yamamoto/crypton/pull/246)+* perf(rsa): stop clearing the scratch a Montgomery multiply writes over+  [#247](https://github.com/kazu-yamamoto/crypton/pull/247)+* doc(gcm): write down what the AArch64 GHASH is short of+  [#248](https://github.com/kazu-yamamoto/crypton/pull/248)+* security(aes): refuse a nonce of no bytes in Crypto.Cipher.AES.GCM+  [#250](https://github.com/kazu-yamamoto/crypton/pull/250)+* ci: build and test the C the other architectures use+  [#251](https://github.com/kazu-yamamoto/crypton/pull/251)+* perf(p256): five teeth to a comb block, over the signed representation+  [#252](https://github.com/kazu-yamamoto/crypton/pull/252)+* security(cipher): stop truncating message lengths on the way to the C+  [#253](https://github.com/kazu-yamamoto/crypton/pull/253)+* fix(c): two left shifts the standard leaves undefined+  [#254](https://github.com/kazu-yamamoto/crypton/pull/254)+* ci: run the C under the sanitizers+  [#255](https://github.com/kazu-yamamoto/crypton/pull/255)+* fix(c): read words out of a block rather than pointing at it+  [#256](https://github.com/kazu-yamamoto/crypton/pull/256)+* fix(internal): drop an import nothing uses any more+  [#257](https://github.com/kazu-yamamoto/crypton/pull/257)+* fix(c): decide the dispatch table once, not on every key+  [#258](https://github.com/kazu-yamamoto/crypton/pull/258)+* Fix the three cabal flag settings that were broken+  [#259](https://github.com/kazu-yamamoto/crypton/pull/259)+* Run the C that only 32-bit architectures get, and fix what that found+  [#260](https://github.com/kazu-yamamoto/crypton/pull/260)+* Let the last addition of each scalar multiplication be a complete one+  [#261](https://github.com/kazu-yamamoto/crypton/pull/261)+* Ask whether the secrets decide anything+  [#262](https://github.com/kazu-yamamoto/crypton/pull/262)+* Ask a big-endian machine the same questions+  [#263](https://github.com/kazu-yamamoto/crypton/pull/263)+* Ask what the secrets leave behind+  [#264](https://github.com/kazu-yamamoto/crypton/pull/264)+* Stop taking eighteen runner slots to test three things+  [#265](https://github.com/kazu-yamamoto/crypton/pull/265)+* Make the scrubs ones the compiler cannot drop, and finish round ten+  [#266](https://github.com/kazu-yamamoto/crypton/pull/266)+* Feed the parsers bytes nobody chose+  [#268](https://github.com/kazu-yamamoto/crypton/pull/268)++## 2.1.2++* perf(p256): 255 squarings for the field inversion, not 287+  [#223](https://github.com/kazu-yamamoto/crypton/pull/223)+* perf(p256): ECDH through s2n-bignum, 2.7x+  [#224](https://github.com/kazu-yamamoto/crypton/pull/224)+* perf(ecc): P-384 and P-521 through s2n-bignum, 7x and 9x+  [#225](https://github.com/kazu-yamamoto/crypton/pull/225)+* perf(p256): ECDSA signing 2.4x and verification 2.2x+  [#226](https://github.com/kazu-yamamoto/crypton/pull/226)+* perf(rsa): the Montgomery multiplication through s2n-bignum on x86-64+  [#227](https://github.com/kazu-yamamoto/crypton/pull/227)+* perf(ecdsa): invert modulo the order in division steps, not an exponentiation+  [#228](https://github.com/kazu-yamamoto/crypton/pull/228)+* perf(x25519): X25519 through s2n-bignum, and a table for key generation+  [#229](https://github.com/kazu-yamamoto/crypton/pull/229)+* perf(gcm): AES-GCM through VAES and VPCLMULQDQ+  [#230](https://github.com/kazu-yamamoto/crypton/pull/230)+* perf(gcm): compile the wide loop once per key length+  [#231](https://github.com/kazu-yamamoto/crypton/pull/231)++## 2.1.1++* feat(ecdsa): RFC 6979 deterministic nonces for Crypto.PubKey.ECDSA+  [#219](https://github.com/kazu-yamamoto/crypton/pull/219)+* feat(gcm): a decrypt that hands back the tag instead of comparing it+  [#220](https://github.com/kazu-yamamoto/crypton/pull/220)+* feat(chachapoly): ChaCha20-Poly1305 a message at a time+  [#221](https://github.com/kazu-yamamoto/crypton/pull/221)+* docs(rsa): say in the haddock what the optional blinder covers+  [#222](https://github.com/kazu-yamamoto/crypton/pull/222)++## 2.1.0++* fix(cpu): stop reading Intel's SDBG bit as AMD's XOP+  [#204](https://github.com/kazu-yamamoto/crypton/pull/204)+* fix(bench): build the benchmark against the checked ChaCha20-Poly1305 key+  [#205](https://github.com/kazu-yamamoto/crypton/pull/205)+* ci: key the cache on the package version+  [#206](https://github.com/kazu-yamamoto/crypton/pull/206)+* ci: build the benchmarks+  [#207](https://github.com/kazu-yamamoto/crypton/pull/207)+* perf(gcm): a fused AES-GCM for x86-64+  [#208](https://github.com/kazu-yamamoto/crypton/pull/208)+* perf(gcm): a fused AES-GCM for AArch64+  [#209](https://github.com/kazu-yamamoto/crypton/pull/209)+* perf(gcm): build the counter in vector registers+  [#210](https://github.com/kazu-yamamoto/crypton/pull/210)+* perf(gcm): a spare lane for E(K,Y0), and a cheaper short block+  [#211](https://github.com/kazu-yamamoto/crypton/pull/211)+* perf(gcm): unroll the tail pass, and take its blocks from registers+  [#212](https://github.com/kazu-yamamoto/crypton/pull/212)+* perf(gcm): read a short block where it lies+  [#213](https://github.com/kazu-yamamoto/crypton/pull/213)+* perf(gcm): the length block and the counter, in registers+  [#214](https://github.com/kazu-yamamoto/crypton/pull/214)+* perf(gcm): let the one-call interface specialise+  [#215](https://github.com/kazu-yamamoto/crypton/pull/215)+* perf(gcm): decryption takes the fused path too+  [#216](https://github.com/kazu-yamamoto/crypton/pull/216)+* perf(gcm): GHASH takes the ciphertext from the output buffer+  [#217](https://github.com/kazu-yamamoto/crypton/pull/217)+* perf(p256): a signed five-bit window for the variable-point multiply+  [#218](https://github.com/kazu-yamamoto/crypton/pull/218)++## 2.0.1++* feat(hash): Skein with the digest size as a type parameter+  [#197](https://github.com/kazu-yamamoto/crypton/pull/197)+* fix(chachapoly1305): take a checked key, so that initializing cannot fail+  [#198](https://github.com/kazu-yamamoto/crypton/pull/198)+* feat(aes): Crypto.Cipher.AES.GCM, for many short messages under one key+  [#199](https://github.com/kazu-yamamoto/crypton/pull/199)+* build: say which platforms the fallback AES sources are for+  [#200](https://github.com/kazu-yamamoto/crypton/pull/200)+* feat(aes): encryptWithMask, for the QUIC header protection mask+  [#201](https://github.com/kazu-yamamoto/crypton/pull/201)+* fix(cpu): stop reading Intel's SDBG bit as AMD's XOP+  [#203](https://github.com/kazu-yamamoto/crypton/pull/203)++## 2.0.0++**Breaking changes.**  Input that used to be accepted is now refused: a value+at or above an RSA or Rabin modulus, a signature of the wrong length or out of+range, a digest too short for HOTP's dynamic truncation, a non-canonical+Ed25519 signature, a PKCS#7 block size outside 1..255, and block cipher input+that is not a whole number of blocks.  A refused KDF, Argon2 or bcrypt+parameter is reported as a `CryptoError` rather than raised as an `ErrorCall`,+and `CryptoError_ParameterInvalid` is appended to `CryptoError`;+`tryGetShared` is added beside `getShared`.  No exported function changed its+signature.++**Deprecated.**  The eighteen curves over a binary field in+`Crypto.ECC.Simple.Types`.  They are obsolete, they are the curves whose+cofactor is not 1, and they will go in a later major version.  Prefer a prime+curve, or X25519.++* Add GHC 9.14 to CI+  [#74](https://github.com/kazu-yamamoto/crypton/pull/74)+* fix(ed25519): reject non-canonical signatures+  [#81](https://github.com/kazu-yamamoto/crypton/pull/81)+* fix(hkdf): enforce RFC 5869 output limit+  [#82](https://github.com/kazu-yamamoto/crypton/pull/82)+* fix(p256): accept valid edge-case points+  [#83](https://github.com/kazu-yamamoto/crypton/pull/83)+* fix(ecc): accept zero-x P-256 shared secrets+  [#84](https://github.com/kazu-yamamoto/crypton/pull/84)+* fix(otp): require a digest long enough for dynamic truncation+  [#85](https://github.com/kazu-yamamoto/crypton/pull/85)+* fix(pkcs15): reject malformed PKCS#1 v1.5 signatures+  [#86](https://github.com/kazu-yamamoto/crypton/pull/86)+* fix(ecdh): validate the peer point before the exchange+  [#87](https://github.com/kazu-yamamoto/crypton/pull/87)+* fix(dsa): do not crash on non-invertible values+  [#88](https://github.com/kazu-yamamoto/crypton/pull/88)+* fix(dh): validate the peer public number+  [#89](https://github.com/kazu-yamamoto/crypton/pull/89)+* fix(argon2): report invalid options as CryptoFailed+  [#90](https://github.com/kazu-yamamoto/crypton/pull/90)+* fix(rsa): drop the early exits from PKCS#1 v1.5 and OAEP unpadding+  [#91](https://github.com/kazu-yamamoto/crypton/pull/91)+* fix(otp): compare TOTP candidates without an early exit+  [#92](https://github.com/kazu-yamamoto/crypton/pull/92)+* feat(dh): add getShared' reporting rejections as CryptoFailable+  [#93](https://github.com/kazu-yamamoto/crypton/pull/93)+* feat(aead): add aeadSimpleDecrypt' taking the tag length+  [#94](https://github.com/kazu-yamamoto/crypton/pull/94)+* test: move the suite to hspec, with hspec-discover+  [#95](https://github.com/kazu-yamamoto/crypton/pull/95)+* docs(bcrypt): say that only the first 72 bytes of a password count+  [#96](https://github.com/kazu-yamamoto/crypton/pull/96)+* feat(elgamal): fix and expose Crypto.PubKey.ElGamal+  [#97](https://github.com/kazu-yamamoto/crypton/pull/97)+* fix(padding): reject a PKCS7 block size outside 1..255+  [#98](https://github.com/kazu-yamamoto/crypton/pull/98)+* build(bench): move the benchmarks from gauge to tasty-bench+  [#99](https://github.com/kazu-yamamoto/crypton/pull/99)+* feat(aes): use the ARMv8 cryptographic extensions on AArch64+  [#100](https://github.com/kazu-yamamoto/crypton/pull/100)+* ci: stop throwing the cache away, and keep the build products in it+  [#101](https://github.com/kazu-yamamoto/crypton/pull/101)+* feat(aes): use PMULL for GHASH on AArch64+  [#102](https://github.com/kazu-yamamoto/crypton/pull/102)+* ci: cut the macOS queueing and supersede stale branch runs+  [#103](https://github.com/kazu-yamamoto/crypton/pull/103)+* feat(sha256): use the ARMv8 SHA-2 instructions on AArch64+  [#104](https://github.com/kazu-yamamoto/crypton/pull/104)+* perf(gcm): fold four GHASH blocks into one reduction+  [#105](https://github.com/kazu-yamamoto/crypton/pull/105)+* ci: build and test on aarch64 Linux+  [#106](https://github.com/kazu-yamamoto/crypton/pull/106)+* fix(cabal): build the AES-NI paths on Windows too+  [#107](https://github.com/kazu-yamamoto/crypton/pull/107)+* perf(aes): specialise by key size and interleave eight blocks on AArch64+  [#108](https://github.com/kazu-yamamoto/crypton/pull/108)+* perf(gcm): drive GCM from AArch64 rather than the generic loop+  [#109](https://github.com/kazu-yamamoto/crypton/pull/109)+* feat(sha512): use the ARMv8.2 SHA-512 instructions on AArch64+  [#110](https://github.com/kazu-yamamoto/crypton/pull/110)+* perf(chacha): do four blocks at a time with NEON on AArch64+  [#111](https://github.com/kazu-yamamoto/crypton/pull/111)+* perf(chacha): do four blocks at a time with SSE2 on x86-64+  [#112](https://github.com/kazu-yamamoto/crypton/pull/112)+* perf(chacha): take eight blocks with AVX2 where the machine has it+  [#113](https://github.com/kazu-yamamoto/crypton/pull/113)+* perf(gcm): give x86 its own decryption loop, and eight blocks either way+  [#114](https://github.com/kazu-yamamoto/crypton/pull/114)+* fix(padding): bound PKCS7 padding by the block, and check ZERO's size+  [#115](https://github.com/kazu-yamamoto/crypton/pull/115)+* ecc: say which curves branch on a secret scalar, and work in Jacobian coordinates+  [#116](https://github.com/kazu-yamamoto/crypton/pull/116)+* perf(poly1305): take four blocks at a time with AVX2 on x86-64+  [#117](https://github.com/kazu-yamamoto/crypton/pull/117)+* perf(xts): drive XTS eight blocks at a time, and dispatch its decryption+  [#118](https://github.com/kazu-yamamoto/crypton/pull/118)+* Report refused KDF parameters as CryptoError, and fix a PBKDF2 SIGBUS+  [#119](https://github.com/kazu-yamamoto/crypton/pull/119)+* Search the HOTP resynchronization window without early exits+  [#120](https://github.com/kazu-yamamoto/crypton/pull/120)+* Refuse an RSA representative that is not below the modulus+  [#121](https://github.com/kazu-yamamoto/crypton/pull/121)+* Give AFIS one answer for a parameter it cannot use+  [#122](https://github.com/kazu-yamamoto/crypton/pull/122)+* Say what ElGamal's signWith requires of k+  [#123](https://github.com/kazu-yamamoto/crypton/pull/123)+* Draw Miller-Rabin witnesses per number, not once per process+  [#124](https://github.com/kazu-yamamoto/crypton/pull/124)+* Refuse Rabin values that are not below the modulus, and keep the padding that was signed+  [#125](https://github.com/kazu-yamamoto/crypton/pull/125)+* Decode Rabin's OAEP without early exits+  [#126](https://github.com/kazu-yamamoto/crypton/pull/126)+* Make CMAC linear, and chain it through CBC+  [#127](https://github.com/kazu-yamamoto/crypton/pull/127)+* Put DES in C+  [#128](https://github.com/kazu-yamamoto/crypton/pull/128)+* Make the generic block cipher modes linear, and bulk where the blocks allow+  [#129](https://github.com/kazu-yamamoto/crypton/pull/129)+* Walk Twofish's blocks once, and carry them in words+  [#130](https://github.com/kazu-yamamoto/crypton/pull/130)+* Put Camellia in C+  [#131](https://github.com/kazu-yamamoto/crypton/pull/131)+* Route P-256 through the C implementation it already had+  [#132](https://github.com/kazu-yamamoto/crypton/pull/132)+* Fold instead of dividing in the generic curve arithmetic+  [#133](https://github.com/kazu-yamamoto/crypton/pull/133)+* Reduce the binary field by folding, and work a byte and a nibble at a time+  [#134](https://github.com/kazu-yamamoto/crypton/pull/134)+* Stop running a Fermat test Miller-Rabin subsumes+  [#135](https://github.com/kazu-yamamoto/crypton/pull/135)+* Make expSafe hide the exponent again+  [#136](https://github.com/kazu-yamamoto/crypton/pull/136)+* Square, and multiply, faster in expSafe+  [#137](https://github.com/kazu-yamamoto/crypton/pull/137)+* Invert the signing nonce without a side channel+  [#138](https://github.com/kazu-yamamoto/crypton/pull/138)+* Keep the P-256 signature out of Integer arithmetic+  [#139](https://github.com/kazu-yamamoto/crypton/pull/139)+* Add at every bit in the prime-curve multiplication, which laziness was skipping+  [#140](https://github.com/kazu-yamamoto/crypton/pull/140)+* Multiply points in C on curves over a prime field+  [#141](https://github.com/kazu-yamamoto/crypton/pull/141)+* A ladder for the curves over a binary field+  [#142](https://github.com/kazu-yamamoto/crypton/pull/142)+* Work RSA's qinv out without the extended Euclidean algorithm+  [#143](https://github.com/kazu-yamamoto/crypton/pull/143)+* Keep the RSA blinding factor out of the extended algorithm+  [#144](https://github.com/kazu-yamamoto/crypton/pull/144)+* Unroll the inner loop at four and two as well+  [#145](https://github.com/kazu-yamamoto/crypton/pull/145)+* Keep a table for each curve's base point+  [#146](https://github.com/kazu-yamamoto/crypton/pull/146)+* Start R squared at the top of the modulus, and why folding did not pay+  [#147](https://github.com/kazu-yamamoto/crypton/pull/147)+* Do the binary field arithmetic in C+  [#148](https://github.com/kazu-yamamoto/crypton/pull/148)+* Use the x86 carry-less multiply where the processor has it+  [#149](https://github.com/kazu-yamamoto/crypton/pull/149)+* Close the two testing gaps: one multiplication for both APIs, one place for each buffer's size+  [#150](https://github.com/kazu-yamamoto/crypton/pull/150)+* Ask aarch64 for its carry-less multiply as well+  [#151](https://github.com/kazu-yamamoto/crypton/pull/151)+* Work the RSA private exponent out without the extended algorithm+  [#152](https://github.com/kazu-yamamoto/crypton/pull/152)+* Fewer Miller-Rabin rounds for a candidate nobody chose+  [#153](https://github.com/kazu-yamamoto/crypton/pull/153)+* Blowfish, and the key setup bcrypt wraps it in, in C+  [#154](https://github.com/kazu-yamamoto/crypton/pull/154)+* perf(sha256): use the Intel SHA extensions on x86-64+  [#155](https://github.com/kazu-yamamoto/crypton/pull/155)+* perf(aes): AES-192 through the processor's AES instructions+  [#156](https://github.com/kazu-yamamoto/crypton/pull/156)+* perf(aes): build the AArch64 key schedule with AESE, not the S-box table+  [#157](https://github.com/kazu-yamamoto/crypton/pull/157)+* test(aes): run the XTS vectors, and OCB and CCM at 192 and 256 bits+  [#158](https://github.com/kazu-yamamoto/crypton/pull/158)+* perf(ocb): drive OCB through the ECB paths a group at a time+  [#159](https://github.com/kazu-yamamoto/crypton/pull/159)+* perf(gcm): take the GHASH of the group before, alongside this group's rounds+  [#160](https://github.com/kazu-yamamoto/crypton/pull/160)+* perf(sha): compute the message schedule in vector registers on x86+  [#161](https://github.com/kazu-yamamoto/crypton/pull/161)+* perf(chacha): combine as the keystream comes out of the registers+  [#162](https://github.com/kazu-yamamoto/crypton/pull/162)+* perf(poly1305): shorten the carry chain and stop spilling the loop+  [#163](https://github.com/kazu-yamamoto/crypton/pull/163)+* docs(sidechannel): say what the prime-field modules keep from the clock+  [#164](https://github.com/kazu-yamamoto/crypton/pull/164)+* perf(sha1): use the Intel SHA extensions on x86-64+  [#165](https://github.com/kazu-yamamoto/crypton/pull/165)+* refactor(aes): drop the keystream generator nobody can call+  [#166](https://github.com/kazu-yamamoto/crypton/pull/166)+* build: compile the C at -O3+  [#167](https://github.com/kazu-yamamoto/crypton/pull/167)+* perf(modes): stop the generic cipher modes allocating per byte+  [#168](https://github.com/kazu-yamamoto/crypton/pull/168)+* perf(poly1305): four blocks at a time with NEON+  [#169](https://github.com/kazu-yamamoto/crypton/pull/169)+* perf(sha1): use the ARMv8 SHA-1 instructions+  [#170](https://github.com/kazu-yamamoto/crypton/pull/170)+* perf(sha3): use the ARMv8.2 SHA-3 instructions+  [#171](https://github.com/kazu-yamamoto/crypton/pull/171)+* perf(gcm): the CRYPTOGAMS stitched AES-GCM on x86-64+  [#172](https://github.com/kazu-yamamoto/crypton/pull/172)+* perf(chacha): the CRYPTOGAMS ChaCha20 on AArch64+  [#173](https://github.com/kazu-yamamoto/crypton/pull/173)+* perf(poly1305): the CRYPTOGAMS Poly1305 on AArch64+  [#174](https://github.com/kazu-yamamoto/crypton/pull/174)+* perf(sha256): the CRYPTOGAMS SHA-256 on AArch64+  [#175](https://github.com/kazu-yamamoto/crypton/pull/175)+* perf(poly1305): the CRYPTOGAMS Poly1305 on x86-64 too+  [#176](https://github.com/kazu-yamamoto/crypton/pull/176)+* perf(chacha): the CRYPTOGAMS ChaCha20 on x86-64 too+  [#177](https://github.com/kazu-yamamoto/crypton/pull/177)+* perf(sha2): the CRYPTOGAMS SHA-256 and SHA-512 on x86-64+  [#178](https://github.com/kazu-yamamoto/crypton/pull/178)+* perf(sha1): hand the SHA-1 block loop a run of blocks, not one at a time+  [#179](https://github.com/kazu-yamamoto/crypton/pull/179)+* perf(xts): double the tweak in the integer registers+  [#180](https://github.com/kazu-yamamoto/crypton/pull/180)+* perf(sha3): take the CRYPTOGAMS Keccak for AArch64+  [#181](https://github.com/kazu-yamamoto/crypton/pull/181)+* perf(sha1): take the CRYPTOGAMS SHA-1 for AArch64+  [#182](https://github.com/kazu-yamamoto/crypton/pull/182)+* docs: put the performance tables in the README+  [#183](https://github.com/kazu-yamamoto/crypton/pull/183)+* perf(sha3): take the CRYPTOGAMS Keccak for x86-64 as well+  [#184](https://github.com/kazu-yamamoto/crypton/pull/184)+* docs: rebuild the performance tables+  [#185](https://github.com/kazu-yamamoto/crypton/pull/185)+* perf(ecc): stop sharing the doublings in the double multiplication+  [#186](https://github.com/kazu-yamamoto/crypton/pull/186)+* perf(number): count bytes from the bit count, not from base 256+  [#187](https://github.com/kazu-yamamoto/crypton/pull/187)+* perf(p256): inline the field arithmetic on AArch64+  [#188](https://github.com/kazu-yamamoto/crypton/pull/188)+* fix(api): name the reporting variants try..., not with an apostrophe+  [#189](https://github.com/kazu-yamamoto/crypton/pull/189)+* fix(ecc): require a public point to be in the prime-order subgroup+  [#190](https://github.com/kazu-yamamoto/crypton/pull/190)+* fix(pubkey): stop printing private keys, and add Crypto.Debug+  [#191](https://github.com/kazu-yamamoto/crypton/pull/191)+* fix(poly1305): take a checked key, so that initializing cannot fail+  [#192](https://github.com/kazu-yamamoto/crypton/pull/192)+* fix(bcrypt): refuse a cost bcrypt does not have rather than substituting one+  [#194](https://github.com/kazu-yamamoto/crypton/pull/194)+* chore: build without a warning+  [#195](https://github.com/kazu-yamamoto/crypton/pull/195)+* docs: build the documentation without a warning+  [#196](https://github.com/kazu-yamamoto/crypton/pull/196)++## 1.1.5++* fix(aead): reject undersized tags+  [#80](https://github.com/kazu-yamamoto/crypton/pull/80)+* fix(aes): refuse a zero-length AES-GCM IV+  [#79](https://github.com/kazu-yamamoto/crypton/pull/79)+* fix(p256): prevent crashes when validating valid points+  [#78](https://github.com/kazu-yamamoto/crypton/pull/78)+* feat(asn1): add SHA-3 HashAlgorithmASN1 instances for PKCS#1 v1.5+  [#77](https://github.com/kazu-yamamoto/crypton/pull/77)+* OCB3 conformance+  [#76](https://github.com/kazu-yamamoto/crypton/pull/76)++## 1.1.4++* Generic instance for RSA PublicKey and PrivateKey++## 1.1.3++* Ensure that `pointAdd` in `PubKey.ECC.P256` treats the point at infinity as the additive identity.+  [#73](https://github.com/kazu-yamamoto/crypton/pull/73)++## 1.1.2++* Preparing `ram` v0.22.+* Generalizing RSA encrypt/decrypt to manipulate ScrubbedBytes directly.++## 1.1.1++* On iOS, ScrubbedBytes based hashing is used for seedNew. On other+  plateforms, entropy is used directly as used to be.+  [#71](https://github.com/kazu-yamamoto/crypton/pull/71)++## 1.1.0++* Removing "basement" and "memory".+  [#67](https://github.com/kazu-yamamoto/crypton/pull/67)+++## 1.0.7++* Stop depending on basement, use upstream dependencies instead+* Stop transitively depending on basement by depending on ram.++## 1.0.6++* Fix test failures on less common 64-bit arches.+  [#65](https://github.com/kazu-yamamoto/crypton/pull/65)++## 1.0.5++* Setter/Getter for ChaCha counter.+  [#63](https://github.com/kazu-yamamoto/crypton/pull/63)+* Add simple interface to generate full blocks+  [#60](https://github.com/kazu-yamamoto/crypton/pull/60)+* Avoid `ghc-prim` dependency.+  [#61](https://github.com/kazu-yamamoto/crypton/pull/61)++## 1.0.4++* Ed448.sign: avoid extra re-derive of public key.+  [#48](https://github.com/kazu-yamamoto/crypton/pull/48)++## 1.0.3++* Make sign of Ed25519/Ed448 safer. The public key parameter is+  ignored and its public key is generated from the secret key+  parameter to prevent Double Public Key Signing Function Oracle+  Attack.+  [#47](https://github.com/kazu-yamamoto/crypton/pull/47)++## 1.0.2++* Deterministic Nonce Generation for ECDSA+  [#46](https://github.com/kazu-yamamoto/crypton/pull/46)+* ECDSA Signature Normalization.+  [#45](https://github.com/kazu-yamamoto/crypton/pull/45)+* Add Full Test Suite from RFC 6979.+  [#44](https://github.com/kazu-yamamoto/crypton/pull/44)+* ECDSA with Public Key Recovery.+  [#43](https://github.com/kazu-yamamoto/crypton/pull/43)+* Providing necessary features for HPKE.+  [#42](https://github.com/kazu-yamamoto/crypton/pull/42)++## 1.0.1++* Update decaf library.+  [#38](https://github.com/kazu-yamamoto/crypton/pull/38)+* Add TypeOperators language extension to EdDSA.hs.+  [#36](https://github.com/kazu-yamamoto/crypton/pull/36)++## 1.0.0++* Versions follow the standard version policy.+* Removing pthread stuff.+  [#32](https://github.com/kazu-yamamoto/crypton/pull/32)++## 0.34++* Hashing getRandomBytes before using as Seed for ChaChaDRG+  [#24](https://github.com/kazu-yamamoto/crypton/pull/24)+* Add support for XChaCha and XChaChaPoly1305+  [#18](https://github.com/kazu-yamamoto/crypton/pull/18)+* Strict byteArray of IV c+  [#16](https://github.com/kazu-yamamoto/crypton/pull/16)++## 0.33++* Add "crypton_" prefix to the final C symbols.+  [#9](https://github.com/kazu-yamamoto/crypton/pull/9)++## 0.32++* All C symbols now have the "crypton_" prefix.+  [#7](https://github.com/kazu-yamamoto/crypton/pull/7)+  [#8](https://github.com/kazu-yamamoto/crypton/pull/8)++## 0.31++* Crypton is forked from cryptonite with the original authors permission.+* Ignoring exceptons from hClose to read the next entropy+  [#1](https://github.com/kazu-yamamoto/crypton/pull/1)+* Enabling the support_pclmuldq flag by default.++## 0.30++* Fix some C symbol blake2b prefix to be cryptonite_ prefix (fix mixing with other C library)+* add hmac-lazy+* Fix compilation with GHC 9.2+* Drop support for GHC8.0, GHC8.2, GHC8.4, GHC8.6++## 0.29++* advance compilation with gmp breakage due to change upstream+* Add native EdDSA support++## 0.28++* Add hash constant time capability+* Prevent possible overflow during hashing by hashing in 4GB chunks++## 0.27++* Optimise AES GCM and CCM+* Optimise P256R1 implementation+* Various AES-NI building improvements+* Add better ECDSA support+* Add XSalsa derive+* Implement square roots for ECC binary curve+* Various tests and benchmarks++## 0.26++* Add Rabin cryptosystem (and variants)+* Add bcrypt_pbkdf key derivation function+* Optimize Blowfish implementation+* Add KMAC (Keccak Message Authentication Code)+* Add ECDSA sign/verify digest APIs+* Hash algorithms with runtime output length+* Update blake2 to latest upstream version+* RSA-PSS with arbitrary key size+* SHAKE with output length not divisible by 8+* Add Read and Data instances for Digest type+* Improve P256 scalar primitives+* Fix hash truncation bug in DSA+* Fix cost parsing for bcrypt+* Fix ECC failures on arm64+* Correction to PKCS#1 v1.5 padding+* Use powModSecInteger when available+* Drop GHC 7.8 and GHC 7.10 support, refer to pkg-guidelines+* Optimise GCM mode+* Add little endian serialization of integer++## 0.25++* Improve digest binary conversion efficiency+* AES CCM support+* Add MonadFailure instance for CryptoFailable+* Various misc improvements on documentation+* Edwards25519 lowlevel arithmetic support+* P256 add point negation+* Improvement in ECC (benchmark, better normalization)+* Blake2 improvements to context size+* Use gauge instead of criterion+* Use haskell-ci for CI scripts+* Improve Digest memory representation to be 2 less Ints and one less boxing+  moving from `UArray` to `Block`++## 0.24++* Ed25519: generateSecret & Documentation updates+* Repair tutorial+* RSA: Allow signing digest directly+* IV add: fix overflow behavior+* P256: validate point when decoding+* Compilation fix with deepseq disabled+* Improve Curve448 and use decaf for Ed448+* Compilation flag blake2 sse merged in sse support+* Process unaligned data better in hashes and AES, on architecture needing alignment+* Drop support for ghc 7.6+* Add ability to create random generator Seed from binary data and+  loosen constraint on ChaChaDRG seed from ByteArray to ByteArrayAccess.+* Add 3 associated types with the HashAlgorithm class, to get+  access to the constant for BlockSize, DigestSize and ContextSize at the type level.+  the related function that this replaced will be deprecated in later release, and+  eventually removed.++API CHANGES:++* Improve ECDH safety to return failure for bad inputs (e.g. public point in small order subgroup).+  To go back to previous behavior you can replace `ecdh` by `ecdhRaw`. It's recommended to+  use `ecdh` and handle the error appropriately.+* Users defining their own HashAlgorithm needs to define the+  HashBlockSize, HashDigest, HashInternalContextSize associated types++## 0.23++* Digest memory usage improvement by using unpinned memory+* Fix generateBetween to generate within the right bounds+* Add pure Twofish implementation+* Fix memory allocation in P256 when using a temp point+* Consolidate hash benchmark code+* Add Nat-length Blake2 support (GHC > 8.0)+* Update tutorial++## 0.22++* Add Argon2 (Password Hashing Competition winner) hash function+* Update blake2 to latest upstream version+* Add extra blake2 hashing size+* Add faster PBKDF2 functions for SHA1/SHA256/SHA512+* Add SHAKE128 and SHAKE256+* Cleanup prime generation, and add tests+* Add Time-based One Time Password (TOTP) and HMAC-based One Time Password (HOTP)+* Rename Ed448 module name to Curve448, old module name still valid for now++## 0.21++* Drop automated tests with GHC 7.0, GHC 7.4, GHC 7.6. support dropped, but probably still working.+* Improve non-aligned support in C sources, ChaCha and SHA3 now probably work on arch without support for unaligned access. not complete or tested.+* Add another ECC framework that is more flexible, allowing different implementations to work instead of+  the existing Pure haskell NIST implementation.+* Add ECIES basic primitives+* Add XSalsa20 stream cipher+* Process partial buffer correctly with Poly1305++## 0.20++* Fixed hash truncation used in ECDSA signature & verification (Olivier Chéron)+* Fix ECDH when scalar and coordinate bit sizes differ (Olivier Chéron)+* Speed up ECDSA verification using Shamir's trick (Olivier Chéron)+* Fix rdrand on windows++## 0.19++* Add tutorial (Yann Esposito)+* Derive Show instance for better interaction with Show pretty printer (Eric Mertens)++## 0.18++* Re-used standard rdrand instructions instead of bytedump of rdrand instruction+* Improvement to F2m, including lots of tests (Andrew Lelechenko)+* Add error check on salt length in bcrypt++## 0.17++* Add Miyaguchi-Preneel construction (Kei Hibino)+* Fix buffer length in scrypt (Luke Taylor)+* build fixes for i686 and arm related to rdrand++## 0.16++* Fix basepoint for Ed448++* Enable 64-bit Curve25519 implementation++## 0.15++* Fix serialization of DH and ECDH++## 0.14++* Reduce size of SHA3 context instead of allocating all-size fit memory. save+  up to 72 bytes of memory per context for SHA3-512.+* Add a Seed capability to the main DRG, to be able to debug/reproduce randomized program+  where you would want to disable the randomness.+* Add support for Cipher-based Message Authentication Code (CMAC) (Kei Hibino)+* *CHANGE* Change the `SharedKey` for `Crypto.PubKey.DH` and `Crypto.PubKey.ECC.DH`,+  from an Integer newtype to a ScrubbedBytes newtype. Prevent mistake where the+  bytes representation is generated without the right padding (when needed).+* *CHANGE* Keep The field size in bits, in the `Params` in `Crypto.PubKey.DH`,+  moving from 2 elements to 3 elements in the structure.++## 0.13++* *SECURITY* Fix buffer overflow issue in SHA384, copying 16 extra bytes from+  the SHA512 context to the destination memory pointer leading to memory+  corruption, segfault. (Mikael Bung)++## 0.12++* Fix compilation issue with Ed448 on 32 bits machine.++## 0.11++* Truncate hashing correctly for DSA+* Add support for HKDF (RFC 5869)+* Add support for Ed448+* Extends support for Blake2s to 224 bits version.+* Compilation workaround for old distribution (RHEL 4.1)+* Compilation fix for AIX+* Compilation fix with AESNI and ghci compiling C source in a weird order.+* Fix example compilation, typo, and warning++## 0.10++* Add reference implementation of blake2 for non-SSE2 platform+* Add support\_blake2\_sse flag++## 0.9++* Quiet down unused module imports+* Move Curve25519 over to Crypto.Error instead of using Either String.+* Add documentation for ChaChaPoly1305+* Add missing documentation for various modules+* Add a way to create Poly1305 Auth tag.+* Added support for the BLAKE2 family of hash algorithms+* Fix endianness of incrementNonce function for ChaChaPoly1305++## 0.8++* Add support for ChaChaPoly1305 Nonce Increment (John Galt)+* Move repository to the haskell-crypto organisation++## 0.7++* Add PKCS5 / PKCS7 padding and unpadding methods+* Fix ChaChaPoly1305 Decryption+* Add support for BCrypt (Luke Taylor)++## 0.6++* Add ChaChaPoly1305 AE cipher+* Add instructions in README for building on old OSX+* Fix blocking /dev/random Andrey Sverdlichenko++## 0.5++* Fix all strays exports to all be under the cryptonite prefix.++## 0.4++* Add a System DRG that represent a referentially transparent of evaluated bytes+  while using lazy evaluation for future entropy values.++## 0.3++* Allow drgNew to run in any MonadRandom, providing cascading initialization+* Remove Crypto.PubKey.HashDescr in favor of just having the algorithm+  specified in PKCS15 RSA function.+* Fix documentation in cipher sub section (Luke Taylor)+* Cleanup AES dead functions (Luke Taylor)+* Fix Show instance of Digest to display without quotes similar to cryptohash+* Use scrubbed bytes instead of bytes for P256 scalar++## 0.2++* Fix P256 compilation and exactness, + add tests+* Add a raw memory number serialization capability (i2osp, os2ip)+* Improve tests for number serialization+* Improve tests for ECC arithmetics+* Add Ord instance for Digest (Nicolas Di Prima)+* Fix entropy compilation on windows 64 bits.++## 0.1++* Initial release+
+ Crypto/Cipher/AES/GCM.hs view
@@ -0,0 +1,256 @@+-- |+-- Module      : Crypto.Cipher.AES.GCM+-- License     : BSD-style+-- Maintainer  : Kazu Yamamoto <kazu@iij.ad.jp>+-- Stability   : experimental+-- Portability : unknown+--+-- AES-GCM for callers that send many short messages under one key, which is+-- what a datagram transport does.+--+-- The interface in "Crypto.Cipher.Types" builds a state from the key /and/+-- the nonce and then walks it through appending the additional data,+-- encrypting and finalizing, copying the state at each step.  For a stream+-- that is nothing next to the encryption.  For a QUIC packet it is most of+-- the work: the key schedule and the table of multiples of @H@ depend on the+-- key alone, and rebuilding them for every nonce costs more than encrypting+-- 1440 bytes.+--+-- So here a t'Context' is built from the key once and holds both, and+-- 'encrypt' takes a nonce and a whole message and answers in one call.+--+-- > ctx <- throwCryptoError <$> pure (newContext key)+-- > let packet = encrypt ctx nonce header plaintext 16+--+-- This runs on AES-NI and carry-less multiply, or on the ARMv8 cryptographic+-- extension, and makes no branch and no memory access that depends on the key+-- or on the data.  Where the processor has neither, AES falls back to a table+-- driven implementation that is /not/ constant time; see the side channels+-- section of the README, and 'Crypto.System.CPU.processorOptions' for which is+-- in use.+--+-- The result is the ciphertext with the tag after it, which is the shape a+-- packet wants.  'decrypt' takes that shape back, compares the tag itself and+-- answers 'Nothing' when it does not match.+--+-- This computes the same thing as the general interface; the tests hold it to+-- that on the same vectors.+module Crypto.Cipher.AES.GCM (+    Context,+    newContext,+    encrypt,+    decrypt,+    decryptWithTag,++    -- * Header protection+    HeaderKey,+    newHeaderKey,+    encryptWithMask,+) where++import Crypto.Cipher.AES.Primitive (+    AES,+    AESGCMKey,+    gcmFullDecrypt,+    gcmFullDecryptTag,+    gcmFullEncrypt,+    gcmFullEncryptMask,+    gcmKeyInit,+    initAES,+ )+import Crypto.Cipher.Types (AuthTag)+import Crypto.Cipher.Types.AEAD (minimumTagLength)+import Crypto.Error+import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess)+import qualified Crypto.Internal.ByteArray as B+import Data.Word (Word8)+import Foreign.Ptr (Ptr)++-- | Everything a key determines: the AES key schedule and the table of+-- multiples of @H@.  Build it once and encrypt as many messages under it as+-- the key is good for.+data Context = Context !AES !AESGCMKey++-- | Take a key of 16, 24 or 32 bytes.  Any other length is reported as+-- 'CryptoError_KeySizeInvalid'.+newContext :: ByteArrayAccess key => key -> CryptoFailable Context+newContext k = do+    aes <- initAES k+    return $ Context aes (gcmKeyInit aes)++-- | Encrypt one message: the nonce, the additional data that is+-- authenticated but not encrypted, the plaintext, and how many bytes of tag+-- to produce, which GCM allows between 4 and 16.  Any other length throws+-- 'CryptoError_AuthenticationTagSizeInvalid', and so does 'decryptWithTag'.+--+-- The answer is the ciphertext followed by the tag.+--+-- A nonce must not be used twice with the same t'Context'.  Twelve bytes is+-- the size GCM is defined for and the only one that does not cost a further+-- pass.  A nonce of no bytes is refused: it would hand out the key GCM+-- authenticates with, so 'encrypt' and 'decryptWithTag' throw+-- 'CryptoError_IvSizeInvalid' for it, 'decrypt' gives 'Nothing' and+-- 'encryptWithMask' gives 'False'.+{-# INLINABLE encrypt #-}+encrypt+    :: ( ByteArrayAccess nonce+       , ByteArrayAccess aad+       , ByteArrayAccess ba+       , ByteArray output+       )+    => Context+    -> nonce+    -> aad+    -> ba+    -> Int+    -> output+encrypt (Context aes gk) nonce aad input taglen+    | tooLongForC aad input =+        throwCryptoError (CryptoFailed CryptoError_ParameterInvalid)+    | badNonce nonce =+        throwCryptoError (CryptoFailed CryptoError_IvSizeInvalid)+    | badTagLength taglen =+        throwCryptoError (CryptoFailed CryptoError_AuthenticationTagSizeInvalid)+    | otherwise = gcmFullEncrypt aes gk nonce aad input taglen++-- | Decrypt one message, in the shape 'encrypt' produced: the ciphertext with+-- its tag after it.  The tag is compared here, every byte of it whatever the+-- answer, and a message whose tag does not match gives 'Nothing' rather than+-- the plaintext.+--+-- 'Nothing' also comes back when the nonce has no bytes, the input is+-- shorter than the tag, or the tag length is outside 4 to 16.+{-# INLINABLE decrypt #-}+decrypt+    :: (ByteArrayAccess nonce, ByteArrayAccess aad, ByteArray ba)+    => Context+    -> nonce+    -> aad+    -> ba+    -> Int+    -> Maybe ba+decrypt (Context aes gk) nonce aad input taglen+    | tooLongForC aad input = Nothing+    | badNonce nonce || badTagLength taglen || B.length input < taglen = Nothing+    | otherwise = gcmFullDecrypt aes gk nonce aad body tag+  where+    (body, tag) = B.splitAt (B.length input - taglen) input++-- | Decrypt one message, the tag kept apart, and hand back the tag this end+-- computed.+--+-- For a caller whose protocol hands it the tag separately from the+-- ciphertext, so that 'decrypt' -- which wants the two together and compares+-- them itself -- does not fit.  Compare the two tags with '=='; the 'Eq'+-- instance of t'AuthTag' is a constant-time comparison, and taking them apart+-- to compare the bytes is how this goes wrong.+--+-- Nothing here says whether the message is authentic.  Until the comparison+-- is made and has come out equal, what this returns is not plaintext, it is+-- what the ciphertext turns into, and a caller must not act on it.+{-# INLINABLE decryptWithTag #-}+decryptWithTag+    :: (ByteArrayAccess nonce, ByteArrayAccess aad, ByteArray ba)+    => Context+    -> nonce+    -> aad+    -> ba+    -> Int+    -> (ba, AuthTag)+decryptWithTag (Context aes gk) nonce aad input taglen+    | tooLongForC aad input =+        throwCryptoError (CryptoFailed CryptoError_ParameterInvalid)+    | badNonce nonce =+        throwCryptoError (CryptoFailed CryptoError_IvSizeInvalid)+    | badTagLength taglen =+        throwCryptoError (CryptoFailed CryptoError_AuthenticationTagSizeInvalid)+    | otherwise = gcmFullDecryptTag aes gk nonce aad input taglen++----------------------------------------------------------------++-- | The key schedule for header protection, which QUIC keeps separately from+-- the one it encrypts with.  Built once, like a t'Context'.+newtype HeaderKey = HeaderKey AES++-- | Take a header protection key of 16, 24 or 32 bytes.+newHeaderKey :: ByteArrayAccess key => key -> CryptoFailable HeaderKey+newHeaderKey k = HeaderKey <$> initAES k++-- | Encrypt one message and, from a sample of the ciphertext it just+-- produced, make the header protection mask -- in one call, into two buffers+-- the caller already has.+--+-- QUIC takes its sample from the ciphertext, so the mask cannot be had before+-- the encryption.  It can be had before coming back, and with the buffers+-- already there nothing is allocated for either.  On an Apple M4 the mask+-- then costs about 0.02 us, where asking for it separately costs 0.11.+--+-- The sealed message wants @length input + taglen@ bytes and the mask+-- sixteen.  @sampleOffset@ says where the sixteen bytes of sample begin in+-- the sealed message, counting the tag as part of it.+--+-- 'False' comes back, and nothing is written, when the nonce has no bytes,+-- the sample would not fit, or the tag length is outside 4 to 16.+{-# INLINABLE encryptWithMask #-}+encryptWithMask+    :: (ByteArrayAccess nonce, ByteArrayAccess aad, ByteArrayAccess ba)+    => Context+    -> HeaderKey+    -> nonce+    -> aad+    -> ba+    -> Int+    -- ^ tag length+    -> Int+    -- ^ sample offset+    -> Ptr Word8+    -- ^ where the sealed message goes+    -> Ptr Word8+    -- ^ where the sixteen bytes of mask go+    -> IO Bool+encryptWithMask (Context aes gk) (HeaderKey hp) nonce aad input taglen off outp maskp+    | tooLongForC aad input = return False+    | badNonce nonce = return False+    | off < 0 || badTagLength taglen || off + 16 > B.length input + taglen =+        return False+    | otherwise = do+        gcmFullEncryptMask aes gk hp nonce aad input taglen off outp maskp+        return True++-- | The C behind all four takes its lengths as @uint32_t@, so a message or+-- its additional data from 2^32 bytes up cannot be handed to it: the length+-- would be truncated and most of the buffer left untouched, with nothing to+-- say so.  There is no splitting the work here -- the C does the whole+-- message in one call, tag and all -- so such a message is refused.+tooLongForC+    :: (ByteArrayAccess aad, ByteArrayAccess ba) => aad -> ba -> Bool+tooLongForC aad input =+    B.overCLength (B.length aad) || B.overCLength (B.length input)++-- | SP 800-38D 5.2.1.1 asks for at least one byte of IV, and this is why.+--+-- GCM builds its pre-counter block from a nonce that is not twelve bytes as+-- @J0 = GHASH_H(IV || 0^s || [0]_64 || [len(IV)]_64)@.  For an empty IV that+-- input is one block of zeros, so @J0@ is zero, and the tag of a message+-- becomes @GHASH_H(A, C) XOR E(K, 0^128)@ -- where @E(K, 0^128)@ is the+-- definition of @H@.  The tag of an empty message under an empty nonce is+-- therefore @H@ itself, and any other full tag gives @H@ as the root of a+-- known polynomial.+--+-- @H@ belongs to the key, not to the nonce.  An attacker holding it, plus one+-- genuine message under any nonce, has @E(K, J0)@ for that nonce and can make+-- a tag that verifies for data of their own -- under a correct twelve-byte+-- nonce, and for every nonce they have seen.  One encryption with an empty+-- nonce and a full tag ends the authenticity of everything under the key.+--+-- 'Crypto.Cipher.AES.Primitive.gcmAeadInit' refuses the empty IV for this+-- reason, and these four have to as well.  Only the empty one is refused:+-- SP 800-38D allows every length from one byte up.+badNonce :: ByteArrayAccess nonce => nonce -> Bool+badNonce nonce = B.length nonce == 0++-- | GCM makes a sixteen-byte tag and a shorter one is a prefix of it.  Below+-- 'minimumTagLength' it authenticates next to nothing, and past sixteen the+-- C code would read beyond the tag it computed.+badTagLength :: Int -> Bool+badTagLength t = t < minimumTagLength || t > 16
Crypto/Cipher/AES/Primitive.hs view
@@ -40,6 +40,12 @@     -- * Incremental GCM     gcmMode,     gcmInit,+    AESGCMKey,+    gcmKeyInit,+    gcmFullEncrypt,+    gcmFullEncryptMask,+    gcmFullDecrypt,+    gcmFullDecryptTag,     gcmAeadInit,      -- * Incremental OCB@@ -158,6 +164,14 @@ sizeGCM :: Int sizeGCM = 320 +-- | The size of what a key determines, which is the 320 bytes above and the+-- powers of H the fused path reads: sixteen of them, and sixteen more for+-- the term the Karatsuba multiplication would otherwise work out every time.+-- The same on every platform, so that this is one number rather than one per+-- architecture; the powers are filled only where that path is compiled in.+sizeGCMKey :: Int+sizeGCMKey = 832+ sizeOCB :: Int sizeOCB = 160 @@ -257,6 +271,7 @@     -- ^ ciphertext output encryptCTR ctx iv input     | len <= 0 = B.empty+    | B.overCLength len = error tooLongMessage     | B.length iv /= 16 =         error $             "AES error: IV length must be block size (16). Its length is: "@@ -351,6 +366,20 @@         c_aes_encrypt_c32 (castPtr o) k v i (fromIntegral len)     len = B.length input +-- | What the AES modes say when a message cannot be given to the C, whose+-- lengths are @uint32_t@.  Above that the length is truncated on the way+-- down and most of the buffer is left as it was found, with nothing to say+-- so, which is worse than refusing.+--+-- Unlike the stream ciphers, these cannot be done in pieces: the C is handed+-- the IV and does not hand it back, so a second call would start from the+-- wrong place.  ECB, CBC and XTS count blocks rather than bytes, so their+-- limit is sixteen times further out than CTR's.+tooLongMessage :: String+tooLongMessage =+    "AES error: message too long for this implementation, whose C takes its "+        ++ "lengths as uint32_t"+ {-# INLINE doECB #-} doECB     :: ByteArray ba@@ -359,6 +388,7 @@     -> ba     -> ba doECB f ctx input+    | B.overCLength nbBlocks = error tooLongMessage     | len == 0 = B.empty     | r /= 0 =         error $@@ -382,6 +412,7 @@     -> ba     -> ba doCBC f ctx (IV iv) input+    | B.overCLength nbBlocks = error tooLongMessage     | len == 0 = B.empty     | r /= 0 =         error $@@ -405,6 +436,7 @@     -> ba     -> ba doXTS f (key1, key2) iv spoint input+    | B.overCLength nbBlocks = error tooLongMessage     | len == 0 = B.empty     | r /= 0 =         error $@@ -429,6 +461,190 @@             c_aes_gcm_init (castPtr gcmStPtr) k v (fromIntegral $ B.length iv)     return $ AESGCM sm +-- | How long a message may be and still be handed to an unsafe foreign call.+-- Four kibibytes is about half a microsecond of work, and it takes in a+-- datagram of any size a network will carry.+shortMessage :: Int+shortMessage = 4096++-- | The part of a GCM state the key alone determines: H, which is the key+-- applied to a block of zeroes, and the table of its multiples.  That is 256+-- of the 320 bytes of a GCM state, and it is the same for every message sent+-- under one key, so a caller that keeps a key can build this once rather than+-- once for every message.+newtype AESGCMKey = AESGCMKey ScrubbedBytes++-- | Build the key part of a GCM state.+{-# NOINLINE gcmKeyInit #-}+gcmKeyInit :: AES -> AESGCMKey+gcmKeyInit ctx = AESGCMKey $ B.allocAndFreeze sizeGCMKey $ \p ->+    keyToPtr ctx $ \k -> c_aes_gcm_key_init (castPtr p) k++-- | Authenticate and encrypt one message in a single call: the nonce, the+-- additional data, the plaintext and the tag, with no state crossing back+-- into Haskell in between.  The result is the ciphertext followed by the tag.+{-# INLINABLE gcmFullEncrypt #-}+gcmFullEncrypt+    :: (ByteArrayAccess iv, ByteArrayAccess aad, ByteArrayAccess ba, ByteArray output)+    => AES -> AESGCMKey -> iv -> aad -> ba -> Int -> output+gcmFullEncrypt ctx (AESGCMKey gk) iv aad input taglen =+    B.allocAndFreeze (B.length input + taglen) $ \out ->+        B.withByteArray gk $ \gkp ->+            keyToPtr ctx $ \k ->+                B.withByteArray iv $ \ivp ->+                    B.withByteArray aad $ \aadp ->+                        B.withByteArray input $ \inp ->+                            call+                                out+                                (castPtr gkp)+                                k+                                ivp+                                (fromIntegral $ B.length iv)+                                aadp+                                (fromIntegral $ B.length aad)+                                inp+                                (fromIntegral $ B.length input)+                                (fromIntegral taglen)+  where+    -- An unsafe call keeps a capability for as long as it runs, so it is only+    -- right for work that is over quickly.  A message this side of+    -- 'shortMessage' is, and it is the short ones the saving matters for: a+    -- safe call costs about 0.075 us whatever the length, which is a fifth of+    -- a 1440-byte packet and a percent of a 16 KiB record.+    call+        | B.length input <= shortMessage = c_aes_gcm_full_encrypt_unsafe+        | otherwise = c_aes_gcm_full_encrypt++-- | Encrypt, and from a sample of the ciphertext just produced make the+-- header protection mask, into buffers the caller owns.  QUIC takes its+-- sample from the ciphertext, so the mask cannot be had before the+-- encryption; it can be had before coming back, and with the buffers already+-- there nothing is allocated for either.+--+-- @sampleoff@ is where the sixteen bytes of sample begin in the output.+{-# INLINABLE gcmFullEncryptMask #-}+gcmFullEncryptMask+    :: (ByteArrayAccess iv, ByteArrayAccess aad, ByteArrayAccess ba)+    => AES+    -> AESGCMKey+    -> AES+    -> iv+    -> aad+    -> ba+    -> Int+    -> Int+    -> Ptr Word8+    -> Ptr Word8+    -> IO ()+gcmFullEncryptMask ctx (AESGCMKey gk) hpctx iv aad input taglen sampleoff outp maskp =+    B.withByteArray gk $ \gkp ->+        keyToPtr ctx $ \k ->+            keyToPtr hpctx $ \hk ->+                B.withByteArray iv $ \ivp ->+                    B.withByteArray aad $ \aadp ->+                        B.withByteArray input $ \inp ->+                            call+                                outp+                                (castPtr gkp)+                                k+                                ivp+                                (fromIntegral $ B.length iv)+                                aadp+                                (fromIntegral $ B.length aad)+                                inp+                                (fromIntegral $ B.length input)+                                (fromIntegral taglen)+                                hk+                                (fromIntegral sampleoff)+                                maskp+  where+    call+        | B.length input <= shortMessage = c_aes_gcm_full_encrypt_mask_unsafe+        | otherwise = c_aes_gcm_full_encrypt_mask++-- | The same the other way, with the tag compared here rather than by the+-- caller: 'Nothing' when it does not match, and every byte of it is looked at+-- either way.  The ciphertext comes in without its tag, which is given+-- separately.+{-# INLINABLE gcmFullDecrypt #-}+gcmFullDecrypt+    :: ( ByteArrayAccess iv+       , ByteArrayAccess aad+       , ByteArrayAccess ba+       , ByteArrayAccess tag+       , ByteArray output+       )+    => AES -> AESGCMKey -> iv -> aad -> ba -> tag -> Maybe output+gcmFullDecrypt ctx (AESGCMKey gk) iv aad input tag = unsafeDoIO $ do+    (r, out) <- B.allocRet (B.length input) $ \outp ->+        B.withByteArray gk $ \gkp ->+            keyToPtr ctx $ \k ->+                B.withByteArray iv $ \ivp ->+                    B.withByteArray aad $ \aadp ->+                        B.withByteArray input $ \inp ->+                            B.withByteArray tag $ \tagp ->+                                call+                                    outp+                                    (castPtr gkp)+                                    k+                                    ivp+                                    (fromIntegral $ B.length iv)+                                    aadp+                                    (fromIntegral $ B.length aad)+                                    inp+                                    (fromIntegral $ B.length input)+                                    tagp+                                    (fromIntegral $ B.length tag)+    return $ if r /= 0 then Just out else Nothing+  where+    call+        | B.length input <= shortMessage = c_aes_gcm_full_decrypt_unsafe+        | otherwise = c_aes_gcm_full_decrypt++-- | Decrypt one message and hand back the tag that was computed over it,+-- rather than comparing it here.+--+-- For a caller that holds the expected tag in a form of its own and will+-- compare it itself.  Compare the two t'AuthTag's with '==', whose instance+-- for that type is a constant-time comparison; taking them apart and+-- comparing the bytes is how this goes wrong.+--+-- Where the tag simply arrives after the ciphertext, 'gcmFullDecrypt' is the+-- one to use: it compares in C and never puts a tag in the caller's hands.+{-# INLINABLE gcmFullDecryptTag #-}+gcmFullDecryptTag+    :: ( ByteArrayAccess iv+       , ByteArrayAccess aad+       , ByteArrayAccess ba+       , ByteArray output+       )+    => AES -> AESGCMKey -> iv -> aad -> ba -> Int -> (output, AuthTag)+gcmFullDecryptTag ctx (AESGCMKey gk) iv aad input taglen = unsafeDoIO $ do+    (tagbs, out) <- B.allocRet (B.length input) $ \outp ->+        B.alloc taglen $ \tagp ->+            B.withByteArray gk $ \gkp ->+                keyToPtr ctx $ \k ->+                    B.withByteArray iv $ \ivp ->+                        B.withByteArray aad $ \aadp ->+                            B.withByteArray input $ \inp ->+                                call+                                    outp+                                    tagp+                                    (castPtr gkp)+                                    k+                                    ivp+                                    (fromIntegral $ B.length iv)+                                    aadp+                                    (fromIntegral $ B.length aad)+                                    inp+                                    (fromIntegral $ B.length input)+                                    (fromIntegral taglen)+    return (out, AuthTag $ B.convert (tagbs :: B.Bytes))+  where+    call+        | B.length input <= shortMessage = c_aes_gcm_full_decrypt_tag_unsafe+        | otherwise = c_aes_gcm_full_decrypt_tag+ -- | append data which is only going to be authenticated to the GCM context. -- -- needs to happen after initialization and before appending encryption/decryption data.@@ -439,7 +655,8 @@     doAppend =         withNewGCMSt gcmSt $ \gcmStPtr ->             withByteArray input $ \i ->-                c_aes_gcm_aad gcmStPtr i (fromIntegral $ B.length input)+                B.inCLengths (B.length input) $ \off n ->+                    c_aes_gcm_aad gcmStPtr (i `plusPtr` off) (fromIntegral n)  -- | append data to encrypt and append to the GCM context --@@ -453,7 +670,13 @@     doEnc gcmStPtr aesPtr =         B.alloc len $ \o ->             withByteArray input $ \i ->-                c_aes_gcm_encrypt (castPtr o) gcmStPtr aesPtr i (fromIntegral len)+                B.inCLengths len $ \off n ->+                    c_aes_gcm_encrypt+                        (castPtr o `plusPtr` off)+                        gcmStPtr+                        aesPtr+                        (i `plusPtr` off)+                        (fromIntegral n)  -- | append data to decrypt and append to the GCM context --@@ -467,7 +690,13 @@     doDec gcmStPtr aesPtr =         B.alloc len $ \o ->             withByteArray input $ \i ->-                c_aes_gcm_decrypt (castPtr o) gcmStPtr aesPtr i (fromIntegral len)+                B.inCLengths len $ \off n ->+                    c_aes_gcm_decrypt+                        (castPtr o `plusPtr` off)+                        gcmStPtr+                        aesPtr+                        (i `plusPtr` off)+                        (fromIntegral n)  -- | Generate the Tag from GCM context {-# NOINLINE gcmFinish #-}@@ -500,9 +729,11 @@ -- The tag length is expressed in bytes and must be in [0..16]. -- The IV length must be in [1..15] bytes per RFC 7253. {-# NOINLINE ocbInitWithTagLength #-}-ocbInitWithTagLength :: ByteArrayAccess iv => AES -> iv -> Int -> CryptoFailable AESOCB+ocbInitWithTagLength+    :: ByteArrayAccess iv => AES -> iv -> Int -> CryptoFailable AESOCB ocbInitWithTagLength ctx iv taglen-    | taglen < 0 || taglen > 16 = CryptoFailed CryptoError_AuthenticationTagSizeInvalid+    | taglen < 0 || taglen > 16 =+        CryptoFailed CryptoError_AuthenticationTagSizeInvalid     | ivlen < 1 || ivlen > 15 = CryptoFailed CryptoError_IvSizeInvalid     | otherwise = CryptoPassed $ unsafeDoIO $ do         sm <- B.alloc sizeOCB $ \ocbStPtr ->@@ -522,7 +753,9 @@ -- need to happen after initialization and before appending encryption/decryption data. {-# NOINLINE ocbAppendAAD #-} ocbAppendAAD :: ByteArrayAccess aad => AES -> AESOCB -> aad -> AESOCB-ocbAppendAAD ctx ocb input = unsafeDoIO (snd `fmap` withOCBKeyAndCopySt ctx ocb doAppend)+ocbAppendAAD ctx ocb input+    | B.overCLength (B.length input) = error tooLongMessage+    | otherwise = unsafeDoIO (snd `fmap` withOCBKeyAndCopySt ctx ocb doAppend)   where     doAppend ocbStPtr aesPtr =         withByteArray input $ \i ->@@ -534,7 +767,9 @@ -- need to happen after AAD appending, or after initialization if no AAD data. {-# NOINLINE ocbAppendEncrypt #-} ocbAppendEncrypt :: ByteArray ba => AES -> AESOCB -> ba -> (ba, AESOCB)-ocbAppendEncrypt ctx ocb input = unsafeDoIO $ withOCBKeyAndCopySt ctx ocb doEnc+ocbAppendEncrypt ctx ocb input+    | B.overCLength (B.length input) = error tooLongMessage+    | otherwise = unsafeDoIO $ withOCBKeyAndCopySt ctx ocb doEnc   where     len = B.length input     doEnc ocbStPtr aesPtr =@@ -548,7 +783,9 @@ -- need to happen after AAD appending, or after initialization if no AAD data. {-# NOINLINE ocbAppendDecrypt #-} ocbAppendDecrypt :: ByteArray ba => AES -> AESOCB -> ba -> (ba, AESOCB)-ocbAppendDecrypt ctx ocb input = unsafeDoIO $ withOCBKeyAndCopySt ctx ocb doDec+ocbAppendDecrypt ctx ocb input+    | B.overCLength (B.length input) = error tooLongMessage+    | otherwise = unsafeDoIO $ withOCBKeyAndCopySt ctx ocb doDec   where     len = B.length input     doDec ocbStPtr aesPtr =@@ -608,7 +845,9 @@ -- needs to happen after initialization and before appending encryption/decryption data. {-# NOINLINE ccmAppendAAD #-} ccmAppendAAD :: ByteArrayAccess aad => AES -> AESCCM -> aad -> AESCCM-ccmAppendAAD ctx ccm input = unsafeDoIO $ snd <$> withCCMKeyAndCopySt ctx ccm doAppend+ccmAppendAAD ctx ccm input+    | B.overCLength (B.length input) = error tooLongMessage+    | otherwise = unsafeDoIO $ snd <$> withCCMKeyAndCopySt ctx ccm doAppend   where     doAppend ccmStPtr aesPtr =         withByteArray input $ \i -> c_aes_ccm_aad ccmStPtr aesPtr i (fromIntegral $ B.length input)@@ -619,7 +858,9 @@ -- needs to happen after AAD appending, or after initialization if no AAD data. {-# NOINLINE ccmEncrypt #-} ccmEncrypt :: ByteArray ba => AES -> AESCCM -> ba -> (ba, AESCCM)-ccmEncrypt ctx ccm input = unsafeDoIO $ withCCMKeyAndCopySt ctx ccm cbcmacAndIv+ccmEncrypt ctx ccm input+    | B.overCLength (B.length input) = error tooLongMessage+    | otherwise = unsafeDoIO $ withCCMKeyAndCopySt ctx ccm cbcmacAndIv   where     len = B.length input     cbcmacAndIv ccmStPtr aesPtr =@@ -633,7 +874,9 @@ -- needs to happen after AAD appending, or after initialization if no AAD data. {-# NOINLINE ccmDecrypt #-} ccmDecrypt :: ByteArray ba => AES -> AESCCM -> ba -> (ba, AESCCM)-ccmDecrypt ctx ccm input = unsafeDoIO $ withCCMKeyAndCopySt ctx ccm cbcmacAndIv+ccmDecrypt ctx ccm input+    | B.overCLength (B.length input) = error tooLongMessage+    | otherwise = unsafeDoIO $ withCCMKeyAndCopySt ctx ccm cbcmacAndIv   where     len = B.length input     cbcmacAndIv ccmStPtr aesPtr =@@ -682,6 +925,131 @@ foreign import ccall "crypton_aes.h crypton_aes_encrypt_c32"     c_aes_encrypt_c32         :: CString -> Ptr AES -> Ptr Word8 -> CString -> CUInt -> IO ()++foreign import ccall unsafe "crypton_aes.h crypton_aes_gcm_key_init"+    c_aes_gcm_key_init :: Ptr AESGCM -> Ptr AES -> IO ()++foreign import ccall "crypton_aes.h crypton_aes_gcm_full_encrypt"+    c_aes_gcm_full_encrypt+        :: Ptr Word8+        -> Ptr AESGCM+        -> Ptr AES+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> CUInt+        -> IO ()++foreign import ccall "crypton_aes.h crypton_aes_gcm_full_decrypt"+    c_aes_gcm_full_decrypt+        :: Ptr Word8+        -> Ptr AESGCM+        -> Ptr AES+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> IO CInt++foreign import ccall "crypton_aes.h crypton_aes_gcm_full_decrypt_tag"+    c_aes_gcm_full_decrypt_tag+        :: Ptr Word8+        -> Ptr Word8+        -> Ptr AESGCM+        -> Ptr AES+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> CUInt+        -> IO ()++foreign import ccall unsafe "crypton_aes.h crypton_aes_gcm_full_decrypt_tag"+    c_aes_gcm_full_decrypt_tag_unsafe+        :: Ptr Word8+        -> Ptr Word8+        -> Ptr AESGCM+        -> Ptr AES+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> CUInt+        -> IO ()++foreign import ccall unsafe "crypton_aes.h crypton_aes_gcm_full_encrypt"+    c_aes_gcm_full_encrypt_unsafe+        :: Ptr Word8+        -> Ptr AESGCM+        -> Ptr AES+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> CUInt+        -> IO ()++foreign import ccall unsafe "crypton_aes.h crypton_aes_gcm_full_decrypt"+    c_aes_gcm_full_decrypt_unsafe+        :: Ptr Word8+        -> Ptr AESGCM+        -> Ptr AES+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> IO CInt++foreign import ccall "crypton_aes.h crypton_aes_gcm_full_encrypt_mask"+    c_aes_gcm_full_encrypt_mask+        :: Ptr Word8+        -> Ptr AESGCM+        -> Ptr AES+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> CUInt+        -> Ptr AES+        -> CUInt+        -> Ptr Word8+        -> IO ()++foreign import ccall unsafe "crypton_aes.h crypton_aes_gcm_full_encrypt_mask"+    c_aes_gcm_full_encrypt_mask_unsafe+        :: Ptr Word8+        -> Ptr AESGCM+        -> Ptr AES+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> CUInt+        -> Ptr AES+        -> CUInt+        -> Ptr Word8+        -> IO ()  foreign import ccall "crypton_aes.h crypton_aes_gcm_init"     c_aes_gcm_init :: Ptr AESGCM -> Ptr AES -> Ptr Word8 -> CUInt -> IO ()
Crypto/Cipher/ChaCha.hs view
@@ -183,10 +183,19 @@     | B.null src = (B.empty, prevSt)     | otherwise = unsafeDoIO $ do         (out, st) <- B.copyRet prevStMem $ \ctx ->-            B.alloc (B.length src) $ \dstPtr ->+            B.alloc n $ \dstPtr ->                 B.withByteArray src $ \srcPtr ->-                    ccrypton_chacha_combine dstPtr ctx srcPtr (fromIntegral $ B.length src)+                    -- in pieces the C's uint32_t length can hold; it carries+                    -- the state in ctx, so it can simply be called again+                    B.inCLengths n $ \off len ->+                        ccrypton_chacha_combine+                            (dstPtr `plusPtr` off)+                            ctx+                            (srcPtr `plusPtr` off)+                            (fromIntegral len)         return (out, State st)+  where+    n = B.length src  -- | Generate a number of bytes from the ChaCha output directly generate@@ -201,7 +210,11 @@     | otherwise = unsafeDoIO $ do         (out, st) <- B.copyRet prevStMem $ \ctx ->             B.alloc len $ \dstPtr ->-                ccrypton_chacha_generate dstPtr ctx (fromIntegral len)+                B.inCLengths len $ \off n ->+                    ccrypton_chacha_generate+                        (dstPtr `plusPtr` off)+                        ctx+                        (fromIntegral n)         return (out, State st)  -- | similar to 'generate' but assume certains values
+ Crypto/Cipher/ChaCha/Poly1305.hs view
@@ -0,0 +1,335 @@+-- |+-- Module      : Crypto.Cipher.ChaCha.Poly1305+-- License     : BSD-style+-- Maintainer  : Kazu Yamamoto <kazu@iij.ad.jp>+-- Stability   : experimental+-- Portability : Good+--+-- ChaCha20-Poly1305 (RFC 8439) a message at a time.+--+-- "Crypto.Cipher.ChaChaPoly1305" takes a message in pieces: a state is+-- started, the additional data appended, the body encrypted and the tag+-- taken, each a step of its own.  That is what a protocol wants when the+-- message arrives in pieces, and it is eight foreign calls and the+-- allocations between them when the message was already whole.+--+-- Here the whole message goes in one call.+--+-- The functions are the same shape as "Crypto.Cipher.AES.GCM", so a protocol+-- that offers both ciphers can hold them the same way.+module Crypto.Cipher.ChaCha.Poly1305 (+    Context,+    newContext,+    encrypt,+    decrypt,+    decryptWithTag,+) where++import Crypto.Cipher.Types (AuthTag (..))+import Crypto.Error+import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess)+import qualified Crypto.Internal.ByteArray as B+import Crypto.Internal.Compat (unsafeDoIO)+import Crypto.Internal.Imports+import Foreign.C.Types (CInt (..), CUInt (..))+import Foreign.Ptr (Ptr, plusPtr)++-- | A key, checked once.+--+-- ChaCha20-Poly1305 has nothing to precompute from a key: the one-time+-- Poly1305 key comes from the nonce, so it differs for every message.  This+-- holds the thirty-two bytes and the knowledge that they are thirty-two, and+-- exists so that the interface is the one "Crypto.Cipher.AES.GCM" has.+newtype Context = Context B.ScrubbedBytes++instance NFData Context where+    rnf (Context k) = k `seq` ()++-- | Take a key of 32 bytes.  Any other length is reported as+-- 'CryptoError_KeySizeInvalid'.+newContext :: ByteArrayAccess key => key -> CryptoFailable Context+newContext k+    | B.length k /= 32 = CryptoFailed CryptoError_KeySizeInvalid+    | otherwise = CryptoPassed $ Context (B.convert k)+{-# INLINABLE newContext #-}++-- | Encrypt one message.  The result is the ciphertext with the tag after it,+-- which is the shape 'decrypt' expects.+--+-- The nonce is the twelve bytes RFC 8439 defines; any other length gives+-- 'CryptoError_IvSizeInvalid'.  RFC 8439 requires a 16-byte tag.+{-# INLINABLE encrypt #-}+encrypt+    :: ( ByteArrayAccess nonce+       , ByteArrayAccess aad+       , ByteArrayAccess ba+       , ByteArray output+       )+    => Context+    -> nonce+    -> aad+    -> ba+    -> Int+    -> CryptoFailable output+encrypt (Context k) nonce aad input taglen+    | tooLongForC aad input = CryptoFailed CryptoError_ParameterInvalid+    | not (validNonce nonce) = CryptoFailed CryptoError_IvSizeInvalid+    | badTag taglen = CryptoFailed CryptoError_AuthenticationTagSizeInvalid+    | otherwise =+        CryptoPassed $+            unsafeDoIO $+                B.alloc (B.length input + taglen) $ \out ->+                    B.withByteArray k $ \kp ->+                        B.withByteArray nonce $ \np ->+                            B.withByteArray aad $ \ap ->+                                B.withByteArray input $ \ip ->+                                    (callE (B.length input))+                                        out+                                        (out `plusPtr` B.length input)+                                        (fromIntegral taglen)+                                        kp+                                        np+                                        (fromIntegral $ B.length nonce)+                                        ap+                                        (fromIntegral $ B.length aad)+                                        ip+                                        (fromIntegral $ B.length input)++-- | Decrypt one message, in the shape 'encrypt' produced: the ciphertext with+-- its tag after it.  The tag is compared here, every byte of it whatever the+-- answer, and a message whose tag does not match gives 'Nothing' rather than+-- the plaintext.+--+-- 'Nothing' also comes back when the input is shorter than the tag, or the+-- nonce is not twelve bytes.+{-# INLINABLE decrypt #-}+decrypt+    :: (ByteArrayAccess nonce, ByteArrayAccess aad, ByteArray ba)+    => Context+    -> nonce+    -> aad+    -> ba+    -> Int+    -> Maybe ba+decrypt (Context k) nonce aad input taglen+    | tooLongForC aad input = Nothing+    | not (validNonce nonce) = Nothing+    | badTag taglen || B.length input < taglen = Nothing+    | otherwise = unsafeDoIO $ do+        (r, out) <- B.allocRet bodylen $ \outp ->+            B.withByteArray k $ \kp ->+                B.withByteArray nonce $ \np ->+                    B.withByteArray aad $ \ap ->+                        B.withByteArray body $ \ip ->+                            B.withByteArray tag $ \tp ->+                                (callD bodylen)+                                    outp+                                    tp+                                    (fromIntegral taglen)+                                    kp+                                    np+                                    (fromIntegral $ B.length nonce)+                                    ap+                                    (fromIntegral $ B.length aad)+                                    ip+                                    (fromIntegral bodylen)+        return $ if r /= 0 then Just out else Nothing+  where+    bodylen = B.length input - taglen+    (body, tag) = B.splitAt bodylen input++-- | Decrypt one message, the tag kept apart, and hand back the tag this end+-- computed.+--+-- For a caller whose protocol carries the tag separately from the ciphertext,+-- so that 'decrypt' -- which wants the two together and compares them itself+-- -- does not fit.  Compare the two tags with '=='; the 'Eq' instance of+-- t'AuthTag' is a constant-time comparison, and taking them apart to compare+-- the bytes is how this goes wrong.+--+-- Nothing here says whether the message is authentic.  Until the comparison+-- is made and has come out equal, what this returns is not plaintext, it is+-- what the ciphertext turns into, and a caller must not act on it.+{-# INLINABLE decryptWithTag #-}+decryptWithTag+    :: (ByteArrayAccess nonce, ByteArrayAccess aad, ByteArray ba)+    => Context+    -> nonce+    -> aad+    -> ba+    -> Int+    -> CryptoFailable (ba, AuthTag)+decryptWithTag (Context k) nonce aad input taglen+    | tooLongForC aad input = CryptoFailed CryptoError_ParameterInvalid+    | not (validNonce nonce) = CryptoFailed CryptoError_IvSizeInvalid+    | badTag taglen = CryptoFailed CryptoError_AuthenticationTagSizeInvalid+    | otherwise = CryptoPassed $ unsafeDoIO $ do+        (tagbs, out) <- B.allocRet (B.length input) $ \outp ->+            B.alloc taglen $ \tagp ->+                B.withByteArray k $ \kp ->+                    B.withByteArray nonce $ \np ->+                        B.withByteArray aad $ \ap ->+                            B.withByteArray input $ \ip ->+                                (callT (B.length input))+                                    outp+                                    tagp+                                    (fromIntegral taglen)+                                    kp+                                    np+                                    (fromIntegral $ B.length nonce)+                                    ap+                                    (fromIntegral $ B.length aad)+                                    ip+                                    (fromIntegral $ B.length input)+        return (out, AuthTag $ B.convert (tagbs :: B.Bytes))++-- | The C takes its lengths as @uint32_t@, so a message or its additional+-- data from 2^32 bytes up cannot be handed to it: the length would be+-- truncated and most of the buffer left untouched, with nothing to say so.+-- The C does the whole message in one call, so there is no splitting it.+tooLongForC+    :: (ByteArrayAccess aad, ByteArrayAccess ba) => aad -> ba -> Bool+tooLongForC aad input =+    B.overCLength (B.length aad) || B.overCLength (B.length input)++-- RFC 8439 is the twelve-byte nonce.  ChaCha20 will take eight, but that is+-- the other construction, with a 64-bit block counter, and it is not what+-- this AEAD is defined over -- so it is refused here rather than quietly+-- encrypting under a scheme nobody asked for.+validNonce :: ByteArrayAccess nonce => nonce -> Bool+validNonce n = B.length n == 12++badTag :: Int -> Bool+badTag t = t /= 16++-- | An unsafe call keeps a capability for as long as it runs, so it is only+-- for a message short enough that the run is short.  Four kibibytes is what+-- the AES side uses, and it takes in a datagram of any size a network will+-- carry.+shortMessage :: Int+shortMessage = 4096++callE :: Int -> CEncrypt+callE n+    | n <= shortMessage = c_chachapoly_encrypt_unsafe+    | otherwise = c_chachapoly_encrypt++callD :: Int -> CDecrypt+callD n+    | n <= shortMessage = c_chachapoly_decrypt_unsafe+    | otherwise = c_chachapoly_decrypt++callT :: Int -> CEncrypt+callT n+    | n <= shortMessage = c_chachapoly_decrypt_tag_unsafe+    | otherwise = c_chachapoly_decrypt_tag++type CEncrypt =+    Ptr Word8+    -> Ptr Word8+    -> CUInt+    -> Ptr Word8+    -> Ptr Word8+    -> CUInt+    -> Ptr Word8+    -> CUInt+    -> Ptr Word8+    -> CUInt+    -> IO ()++type CDecrypt =+    Ptr Word8+    -> Ptr Word8+    -> CUInt+    -> Ptr Word8+    -> Ptr Word8+    -> CUInt+    -> Ptr Word8+    -> CUInt+    -> Ptr Word8+    -> CUInt+    -> IO CInt++foreign import ccall "crypton_chachapoly.h crypton_chachapoly_encrypt"+    c_chachapoly_encrypt+        :: Ptr Word8+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> IO ()++foreign import ccall unsafe "crypton_chachapoly.h crypton_chachapoly_encrypt"+    c_chachapoly_encrypt_unsafe+        :: Ptr Word8+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> IO ()++foreign import ccall "crypton_chachapoly.h crypton_chachapoly_decrypt"+    c_chachapoly_decrypt+        :: Ptr Word8+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> IO CInt++foreign import ccall unsafe "crypton_chachapoly.h crypton_chachapoly_decrypt"+    c_chachapoly_decrypt_unsafe+        :: Ptr Word8+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> IO CInt++foreign import ccall "crypton_chachapoly.h crypton_chachapoly_decrypt_tag"+    c_chachapoly_decrypt_tag+        :: Ptr Word8+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> IO ()++foreign import ccall unsafe "crypton_chachapoly.h crypton_chachapoly_decrypt_tag"+    c_chachapoly_decrypt_tag_unsafe+        :: Ptr Word8+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> Ptr Word8+        -> CUInt+        -> IO ()
Crypto/Cipher/ChaChaPoly1305.hs view
@@ -1,3 +1,5 @@+{-# LANGUAGE GeneralizedNewtypeDeriving #-}+ -- | -- Module      : Crypto.Cipher.ChaChaPoly1305 -- License     : BSD-style@@ -28,7 +30,7 @@ -- >    -> ByteString -- input plaintext to be encrypted -- >    -> CryptoFailable ByteString -- ciphertext with a 128-bit tag attached -- >encrypt nonce key header plaintext = do--- >    st1 <- C.nonce12 nonce >>= C.initialize key+-- >    st1 <- C.initialize <$> C.key key <*> C.nonce12 nonce -- >    let -- >        st2 = C.finalizeAAD $ C.appendAAD header st1 -- >        (out, st3) = C.encrypt plaintext st2@@ -41,6 +43,8 @@      -- * Low level     State,+    Key,+    key,     Nonce,     XNonce,     nonce12,@@ -68,6 +72,7 @@  ) import qualified Crypto.Internal.ByteArray as B import Crypto.Internal.Imports+import qualified Crypto.Internal.Poly1305 as PolyKey import qualified Crypto.MAC.Poly1305 as Poly1305 import qualified Data.ByteArray.Pack as P import Data.Memory.Endian@@ -174,44 +179,39 @@ -- -- The key length need to be 256 bits, and the nonce -- procured using either `nonce8` or `nonce12`-initialize-    :: ByteArrayAccess key-    => key -> Nonce -> CryptoFailable State-initialize key (Nonce8 nonce) = initialize' key nonce-initialize key (Nonce12 nonce) = initialize' key nonce+-- | A ChaCha20Poly1305 key: thirty-two bytes, checked once here rather than+-- at every use, so that 'initialize' and 'initializeX' cannot fail.+newtype Key = Key ScrubbedBytes+    deriving (ByteArrayAccess, Eq, NFData) -initialize'-    :: ByteArrayAccess key-    => key -> Bytes -> CryptoFailable State-initialize' key nonce-    | B.length key /= 32 = CryptoFailed CryptoError_KeySizeInvalid-    | otherwise = CryptoPassed $ initFromRootState rootState-  where-    rootState = ChaCha.initialize 20 key nonce+-- | Take thirty-two bytes for a key.  A different length is reported as+-- 'CryptoError_KeySizeInvalid'; nothing else about a key can be wrong.+key :: ByteArrayAccess ba => ba -> CryptoFailable Key+key k+    | B.length k /= 32 = CryptoFailed CryptoError_KeySizeInvalid+    | otherwise = CryptoPassed $ Key $ B.convert k +initialize :: Key -> Nonce -> State+initialize k (Nonce8 nonce) = initialize' k nonce+initialize k (Nonce12 nonce) = initialize' k nonce++initialize' :: Key -> Bytes -> State+initialize' k nonce = initFromRootState (ChaCha.initialize 20 k nonce)+ initFromRootState :: ChaCha.State -> State initFromRootState rootState = State encState polyState 0 0   where     (polyKey, encState) = ChaCha.generate rootState 64-    -- 64 bytes are generated so the ChaCha state advances a whole block; the-    -- first 32 of them are the key, so the length is right by construction-    polyState =-        Poly1305.initialize $-            throwCryptoError $-                Poly1305.key (B.take 32 polyKey :: ScrubbedBytes)+    -- 64 bytes are generated so the ChaCha state advances a whole block, and+    -- the first 32 of them are the key, so there is no length left to check+    polyState = Poly1305.initialize (PolyKey.Key (B.take 32 polyKey))  -- | Initialize a new XChaChaPoly1305 State -- -- The key length needs to be 256 bits, and the nonce -- procured using `nonce24`.-initializeX-    :: ByteArrayAccess key-    => key -> XNonce -> CryptoFailable State-initializeX key (Nonce24 nonce)-    | B.length key /= 32 = CryptoFailed CryptoError_KeySizeInvalid-    | otherwise = CryptoPassed $ initFromRootState rootState-  where-    rootState = ChaCha.initializeX 20 key nonce+initializeX :: Key -> XNonce -> State+initializeX k (Nonce24 nonce) = initFromRootState (ChaCha.initializeX 20 k nonce)  -- | Append Authenticated Data to the State and return -- the new modified State.@@ -267,8 +267,8 @@ aeadChacha20poly1305Init     :: (ByteArrayAccess k, ByteArrayAccess n)     => k -> n -> CryptoFailable (AEAD ChaCha20Poly1305)-aeadChacha20poly1305Init key nonce = do-    st0 <- nonce12 nonce >>= initialize key+aeadChacha20poly1305Init k nonce = do+    st0 <- initialize <$> key k <*> nonce12 nonce     return $ AEAD model st0   where     model =
Crypto/Cipher/RC4.hs view
@@ -98,7 +98,14 @@     B.allocRet len $ \outptr ->         B.withByteArray clearText $ \clearPtr -> do             st <- B.copy prevSt $ \stPtr ->-                c_rc4_combine (castPtr stPtr) clearPtr (fromIntegral len) outptr+                -- in pieces the C's uint32_t length can hold; the state it+                -- keeps means it can simply be called again+                B.inCLengths len $ \off n ->+                    c_rc4_combine+                        (castPtr stPtr)+                        (clearPtr `plusPtr` off)+                        (fromIntegral n)+                        (outptr `plusPtr` off)             return $! State st   where     -- return $! (State st, B.PS outfptr 0 len)
Crypto/Cipher/Salsa.hs view
@@ -70,10 +70,19 @@     | B.null src = (B.empty, prevSt)     | otherwise = unsafeDoIO $ do         (out, st) <- B.copyRet prevStMem $ \ctx ->-            B.alloc (B.length src) $ \dstPtr ->-                B.withByteArray src $ \srcPtr -> do-                    ccrypton_salsa_combine dstPtr ctx srcPtr (fromIntegral $ B.length src)+            B.alloc n $ \dstPtr ->+                B.withByteArray src $ \srcPtr ->+                    -- in pieces the C's uint32_t length can hold; it carries+                    -- the state in ctx, so it can simply be called again+                    B.inCLengths n $ \off len ->+                        ccrypton_salsa_combine+                            (dstPtr `plusPtr` off)+                            ctx+                            (srcPtr `plusPtr` off)+                            (fromIntegral len)         return (out, State st)+  where+    n = B.length src  -- | Generate a number of bytes from the Salsa output directly generate@@ -88,7 +97,11 @@     | otherwise = unsafeDoIO $ do         (out, st) <- B.copyRet prevStMem $ \ctx ->             B.alloc len $ \dstPtr ->-                ccrypton_salsa_generate dstPtr ctx (fromIntegral len)+                B.inCLengths len $ \off n ->+                    ccrypton_salsa_generate+                        (dstPtr `plusPtr` off)+                        ctx+                        (fromIntegral n)         return (out, State st)  foreign import ccall "crypton_salsa_init"
Crypto/Cipher/Twofish/Primitive.hs view
@@ -15,9 +15,8 @@ import Crypto.Internal.WordArray import Crypto.Internal.Words (Word128 (..)) import Data.Bits-import Data.List (foldl')+import qualified Data.List as L import Data.Word-import Prelude hiding (foldl')  -- Based on the Golang referance implementation -- https://github.com/golang/crypto/blob/master/twofish/twofish.go@@ -113,7 +112,7 @@     b' = b `xor` arrayRead32 ks 1     c' = c `xor` arrayRead32 ks 2     d' = d `xor` arrayRead32 ks 3-    (!a'', !b'', !c'', !d'') = foldl' shuffle (a', b', c', d') [0 .. 7]+    (!a'', !b'', !c'', !d'') = L.foldl' shuffle (a', b', c', d') [0 .. 7]     ts =         ( c'' `xor` arrayRead32 ks 4         , d'' `xor` arrayRead32 ks 5@@ -180,7 +179,7 @@     b' = d `xor` arrayRead32 ks 7     c' = a `xor` arrayRead32 ks 4     d' = b `xor` arrayRead32 ks 5-    (!a'', !b'', !c'', !d'') = foldl' unshuffle (a', b', c', d') [8, 7 .. 1]+    (!a'', !b'', !c'', !d'') = L.foldl' unshuffle (a', b', c', d') [8, 7 .. 1]     ixs =         ( a'' `xor` arrayRead32 ks 0         , b'' `xor` arrayRead32 ks 1@@ -284,7 +283,7 @@             ( \wordIndex ->                 map                     ( \rsRow ->-                        foldl'+                        L.foldl'                             ( \acc (!rsVal, !colIndex) ->                                 acc `xor` gfMult rsPolynomial (B.index key $ 8 * wordIndex + colIndex) rsVal                             )@@ -443,7 +442,7 @@         b' = rotateL b 8  h :: ByteArray ba => [Word8] -> KeyPackage ba -> Int -> Word32-h input keyPackage offset = foldl' xorMdsColMult 0 $ zip [y0f, y1f, y2f, y3f] $ enumFrom Zero+h input keyPackage offset = L.foldl' xorMdsColMult 0 $ zip [y0f, y1f, y2f, y3f] $ enumFrom Zero   where     key = rawKeyBytes keyPackage     [y0, y1, y2, y3] = take 4 input
Crypto/ConstructHash/MiyaguchiPreneel.hs view
@@ -15,8 +15,7 @@     MiyaguchiPreneel, ) where -import Data.List (foldl')-import Prelude hiding (foldl')+import qualified Data.List as L  import Crypto.Cipher.Types import Crypto.Cipher.Types.Utils (chunk)@@ -41,7 +40,7 @@     -> MiyaguchiPreneel cipher     -- ^ output tag compute' g =-    MP . foldl' (step $ g) (B.replicate bsz 0) . chunks . pad (ZERO bsz) . B.convert+    MP . L.foldl' (step $ g) (B.replicate bsz 0) . chunks . pad (ZERO bsz) . B.convert   where     bsz = blockSize (g B.empty {- dummy to get block size -})     -- 'chunk' slices rather than splitting the message, which copied whatever
Crypto/ECC.hs view
@@ -49,13 +49,12 @@ import qualified Crypto.ECC.Simple.Prim as Simple import qualified Crypto.ECC.Simple.Types as Simple import Crypto.Error+import Crypto.KEM (SharedSecret (..)) import Crypto.Internal.ByteArray (     ByteArray,     ByteArrayAccess,-    ScrubbedBytes,  ) import qualified Crypto.Internal.ByteArray as B-import Crypto.Internal.Imports import Crypto.Number.Basic (numBits) import Crypto.Number.Serialize (i2ospOf_, os2ip) import qualified Crypto.Number.Serialize.LE as LE@@ -74,16 +73,6 @@     { keypairGetPublic :: !(Point curve)     , keypairGetPrivate :: !(Scalar curve)     }---- | Secret shared via key exchange-newtype SharedSecret = SharedSecret ScrubbedBytes-    deriving (Eq, ByteArrayAccess, NFData)--instance Semigroup SharedSecret where-    SharedSecret x <> SharedSecret y = SharedSecret (x <> y)--instance Monoid SharedSecret where-    mempty = SharedSecret mempty  class EllipticCurve curve where     -- | Point on an Elliptic Curve
Crypto/Error/Types.hs view
@@ -57,6 +57,10 @@       -- the base point generates, so multiplying it would answer modulo a       -- small order.  Appended for the same reason as the constructor above.       CryptoError_PointSubgroupInvalid+    | -- | A public key is the right length but is not a well-formed encoding+      -- of one, so no honest party produced it.  Appended for the same+      -- reason as the two constructors above.+      CryptoError_PublicKeyStructureInvalid     deriving (Show, Eq, Enum, Data)  instance E.Exception CryptoError
Crypto/Hash.hs view
@@ -53,12 +53,20 @@ import qualified Crypto.Internal.ByteArray as B import qualified Data.ByteString.Lazy as L import Data.Int (Int32)+import Data.Word (Word8)+import Foreign.Marshal.Alloc (allocaBytes) import qualified Foreign.Marshal.Utils as FMU import Foreign.Ptr (Ptr, castPtr, plusPtr)  -- | Hash a strict bytestring into a digest.-hash :: (ByteArrayAccess ba, HashAlgorithm a) => ba -> Digest a-hash bs = hashFinalize $ hashUpdate hashInit bs+hash :: forall ba a. (ByteArrayAccess ba, HashAlgorithm a) => ba -> Digest a+hash bs = Digest $+    allocAndFreezePrim (hashDigestSize (undefined :: a)) $ \(dig :: Ptr (Digest a)) ->+        -- one context, used in place: the pure steps would copy it three times+        allocaBytes (hashInternalContextSize (undefined :: a)) $ \(ctx :: Ptr (Context a)) -> do+            hashInternalInit ctx+            B.withByteArray bs (updateBlocks ctx (B.length bs))+            hashInternalFinalize ctx dig  -- | Hash the first N bytes of a bytestring, with code path independent from N. hashPrefix@@ -92,20 +100,19 @@ hashUpdates c l     | null ls = c     | otherwise = Context $ B.copyAndFreeze c $ \(ctx :: Ptr (Context a)) ->-        mapM_ (\b -> B.withByteArray b (processBlocks ctx (B.length b))) ls+        mapM_ (\b -> B.withByteArray b (updateBlocks ctx (B.length b))) ls   where     ls = filter (not . B.null) l-    -- process the data in 2GB chunks to fit in uint32_t and Int on 32 bit systems-    processBlocks ctx bytesLeft dataPtr-        | bytesLeft == 0 = return ()-        | otherwise = do-            hashInternalUpdate ctx dataPtr (fromIntegral actuallyProcessed)-            processBlocks-                ctx-                (bytesLeft - actuallyProcessed)-                (dataPtr `plusPtr` actuallyProcessed)-      where-        actuallyProcessed = min bytesLeft (fromIntegral (maxBound :: Int32))++-- | Process the data in 2GB chunks to fit in uint32_t and Int on 32 bit systems+updateBlocks :: HashAlgorithm a => Ptr (Context a) -> Int -> Ptr Word8 -> IO ()+updateBlocks ctx bytesLeft dataPtr+    | bytesLeft == 0 = return ()+    | otherwise = do+        hashInternalUpdate ctx dataPtr (fromIntegral actuallyProcessed)+        updateBlocks ctx (bytesLeft - actuallyProcessed) (dataPtr `plusPtr` actuallyProcessed)+  where+    actuallyProcessed = min bytesLeft (fromIntegral (maxBound :: Int32))  -- | Finalize a context and return a digest. hashFinalize
Crypto/Hash/Algorithms.hs view
@@ -48,8 +48,10 @@     Blake2bp (..),     Blake2s (..),     Blake2sp (..),+    Skein256 (..),     Skein256_224 (..),     Skein256_256 (..),+    Skein512 (..),     Skein512_224 (..),     Skein512_256 (..),     Skein512_384 (..),
Crypto/Hash/Skein256.hs view
@@ -1,7 +1,11 @@ {-# LANGUAGE DataKinds #-} {-# LANGUAGE DeriveDataTypeable #-} {-# LANGUAGE ForeignFunctionInterface #-}+{-# LANGUAGE KindSignatures #-}+{-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TypeFamilies #-}+{-# LANGUAGE TypeOperators #-}+{-# LANGUAGE UndecidableInstances #-}  -- | -- Module      : Crypto.Hash.Skein256@@ -13,14 +17,17 @@ -- Module containing the binding functions to work with the -- Skein256 cryptographic hash. module Crypto.Hash.Skein256 (+    Skein256 (..),     Skein256_224 (..),     Skein256_256 (..), ) where  import Crypto.Hash.Types+import Crypto.Internal.Nat import Data.Data import Data.Word (Word32, Word8) import Foreign.Ptr (Ptr)+import GHC.TypeLits (KnownNat, Nat, type (+))  -- | Skein256 (224 bits) cryptographic hash algorithm data Skein256_224 = Skein256_224@@ -51,6 +58,38 @@     hashInternalInit p = c_skein256_init p 256     hashInternalUpdate = c_skein256_update     hashInternalFinalize p = c_skein256_finalize p 256++-- | Skein256 with the digest size given as a type parameter of kind 'Nat',+-- in bits.  @t'Skein256' 256@ is @t'Skein256_256'@; the sizes with a type of+-- their own+-- above are there for their names, and this one also takes the sizes that+-- have none.+--+-- A size that is not a whole number of bytes is rounded up to the next one,+-- as the implementation underneath does.+--+-- The output is produced in counter mode, a block of it per Threefish call,+-- so one large digest is a good deal cheaper than the same number of bytes+-- taken from repeated small ones: on an Apple M4, 512 KiB arrives at 947 MB/s+-- in one digest against 172 MB/s as 8192 separate @t'Skein256_256'@ ones.+--+-- Note the digest size goes into the configuration block, so it changes the+-- value the message is hashed from: a longer digest is /not/ an extension of+-- a shorter one.  That is the opposite of how t'Crypto.Hash.SHAKE.SHAKE128'+-- behaves.+data Skein256 (bitlen :: Nat) = Skein256+    deriving (Show, Data)++instance KnownNat bitlen => HashAlgorithm (Skein256 bitlen) where+    type HashBlockSize (Skein256 bitlen) = 32+    type HashDigestSize (Skein256 bitlen) = Div8 (bitlen + 7)+    type HashInternalContextSize (Skein256 bitlen) = 96+    hashBlockSize _ = 32+    hashDigestSize _ = byteLen (Proxy :: Proxy bitlen)+    hashInternalContextSize _ = 96+    hashInternalInit p = c_skein256_init p (integralNatVal (Proxy :: Proxy bitlen))+    hashInternalUpdate = c_skein256_update+    hashInternalFinalize p = c_skein256_finalize p (integralNatVal (Proxy :: Proxy bitlen))  foreign import ccall unsafe "crypton_skein256_init"     c_skein256_init :: Ptr (Context a) -> Word32 -> IO ()
Crypto/Hash/Skein512.hs view
@@ -1,7 +1,11 @@ {-# LANGUAGE DataKinds #-} {-# LANGUAGE DeriveDataTypeable #-} {-# LANGUAGE ForeignFunctionInterface #-}+{-# LANGUAGE KindSignatures #-}+{-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TypeFamilies #-}+{-# LANGUAGE TypeOperators #-}+{-# LANGUAGE UndecidableInstances #-}  -- | -- Module      : Crypto.Hash.Skein512@@ -13,6 +17,7 @@ -- Module containing the binding functions to work with the -- Skein512 cryptographic hash. module Crypto.Hash.Skein512 (+    Skein512 (..),     Skein512_224 (..),     Skein512_256 (..),     Skein512_384 (..),@@ -20,9 +25,11 @@ ) where  import Crypto.Hash.Types+import Crypto.Internal.Nat import Data.Data import Data.Word (Word32, Word8) import Foreign.Ptr (Ptr)+import GHC.TypeLits (KnownNat, Nat, type (+))  -- | Skein512 (224 bits) cryptographic hash algorithm data Skein512_224 = Skein512_224@@ -83,6 +90,38 @@     hashInternalInit p = c_skein512_init p 512     hashInternalUpdate = c_skein512_update     hashInternalFinalize p = c_skein512_finalize p 512++-- | Skein512 with the digest size given as a type parameter of kind 'Nat',+-- in bits.  @t'Skein512' 512@ is @t'Skein512_512'@; the sizes with a type of+-- their own+-- above are there for their names, and this one also takes the sizes that+-- have none.+--+-- A size that is not a whole number of bytes is rounded up to the next one,+-- as the implementation underneath does.+--+-- The output is produced in counter mode, a block of it per Threefish call,+-- so one large digest is a good deal cheaper than the same number of bytes+-- taken from repeated small ones: on an Apple M4, 512 KiB arrives at 947 MB/s+-- in one digest against 172 MB/s as 8192 separate @t'Skein512_512'@ ones.+--+-- Note the digest size goes into the configuration block, so it changes the+-- value the message is hashed from: a longer digest is /not/ an extension of+-- a shorter one.  That is the opposite of how t'Crypto.Hash.SHAKE.SHAKE128'+-- behaves.+data Skein512 (bitlen :: Nat) = Skein512+    deriving (Show, Data)++instance KnownNat bitlen => HashAlgorithm (Skein512 bitlen) where+    type HashBlockSize (Skein512 bitlen) = 64+    type HashDigestSize (Skein512 bitlen) = Div8 (bitlen + 7)+    type HashInternalContextSize (Skein512 bitlen) = 160+    hashBlockSize _ = 64+    hashDigestSize _ = byteLen (Proxy :: Proxy bitlen)+    hashInternalContextSize _ = 160+    hashInternalInit p = c_skein512_init p (integralNatVal (Proxy :: Proxy bitlen))+    hashInternalUpdate = c_skein512_update+    hashInternalFinalize p = c_skein512_finalize p (integralNatVal (Proxy :: Proxy bitlen))  foreign import ccall unsafe "crypton_skein512_init"     c_skein512_init :: Ptr (Context a) -> Word32 -> IO ()
Crypto/Hash/Types.hs view
@@ -103,6 +103,26 @@ -- layout is architecture dependent, may contain uninitialized data fragments, -- and change in future versions.  The bytearray should not be used as input to -- cryptographic algorithms.+--+-- __A context is not erased when it is finished with.__  A hash algorithm+-- buffers its input a block at a time, and finalizing does not clear what is+-- left there.  How much survives depends on where the message ended relative+-- to the block: with SHA-256, a 32-byte message is still in the context in+-- full afterwards, and a 100-byte one leaves its last 36 bytes.+-- @hashFinalize@ works on a copy, so the caller's own context keeps what it+-- had as well.  Nothing clears either of them: this is 'Bytes' rather than+-- @ScrubbedBytes@, and the C clears nothing.  They go to the garbage+-- collector as they are, and a core file, a crash dump or a swapped page can+-- carry them away afterwards.+--+-- That is a deliberate trade rather than an oversight, and there is no way+-- to ask for the other side of it: no operation here clears a context.+-- Scrubbing them all was measured at about 70% of a 32-byte hash and a third+-- of an incremental one, because the allocation is most of the work when the+-- message is short -- and short hashes are the common case, in HMAC, in+-- HKDF, and anywhere a key or an identifier is hashed.  A 64 KB hash does+-- not notice it.  Anything that must not be left in memory this way is+-- better not hashed through this interface at all. newtype Context a = Context Bytes     deriving (ByteArrayAccess, NFData)
Crypto/Internal/ByteArray.hs view
@@ -14,6 +14,8 @@     module Data.ByteArray.Mapping,     module Data.ByteArray.Encoding,     constAllZero,+    overCLength,+    inCLengths,     allocAndFreezePrimIO,     allocAndFreezePrim,     bxor,@@ -30,6 +32,51 @@ import Foreign.Storable (peekByteOff)  import Crypto.Internal.Compat (unsafeDoIO)++-- | Whether a length is too large to reach the C, which takes its lengths as+-- @uint32_t@.+--+-- From 2^32 up the value is truncated on the way down, and the C then works+-- on the low bits of it and leaves the rest of the buffer as it found it --+-- which for a fresh allocation is zeros.  What comes back is as long as the+-- caller asked for, with nothing to say that most of it was never written:+-- a 4 GiB message through Crypto.Cipher.ChaCha.combine came back with 2^32+-- bytes of zeros where the ciphertext should have been.+--+-- Every place that hands a caller's length to the C either turns it away+-- with this or cuts the work into pieces small enough to pass.+--+-- The round trip through 'Word32' rather than a comparison against 2^32,+-- which a 32-bit 'Int' cannot hold.  There every non-negative 'Int' passes,+-- which is the right answer: there is no such buffer to be had.+overCLength :: Int -> Bool+overCLength n = fromIntegral (fromIntegral n :: Word32) /= n++-- | Walk a length in pieces small enough to reach the C, calling the action+-- with the offset and the size of each.+--+-- The same 2 GiB step "Crypto.Hash" has always taken, and for the same+-- reason: the C takes its lengths as @uint32_t@, and a 32-bit 'Int' cannot+-- hold a whole one either.  This is for the C that keeps its state in a+-- context and can simply be called again -- the stream ciphers, the MACs --+-- where a long message can be enciphered in pieces rather than refused.+inCLengths :: Int -> (Int -> Int -> IO ()) -> IO ()+inCLengths total f = go 0+  where+    go !off+        | off >= total = return ()+        | otherwise = f off n >> go (off + n)+      where+        !n = min (total - off) cChunk++-- | The step 'inCLengths' takes: the largest multiple of 64 that a signed+-- 32-bit integer holds.+--+-- Under 2^31 because a 32-bit 'Int' cannot hold more, and a+-- multiple of 64 because some of the C this feeds -- the AEAD modes -- will+-- take a piece that is not a whole number of blocks only as the last one.+cChunk :: Int+cChunk = 0x7fffffc0  -- | Allocate a pinned 'Prim.ByteArray' of the given size, populate it via a -- 'Ptr', then freeze and return it.  The pointer must not be retained after
+ Crypto/Internal/Poly1305.hs view
@@ -0,0 +1,37 @@+{-# LANGUAGE GeneralizedNewtypeDeriving #-}++-- |+-- Module      : Crypto.Internal.Poly1305+-- License     : BSD-style+-- Maintainer  : Kazu Yamamoto <kazu@iij.ad.jp>+-- Stability   : experimental+-- Portability : unknown+--+-- The Poly1305 key with its constructor, for the modules here that build one+-- from bytes whose length they already know.  "Crypto.MAC.Poly1305" exports+-- the type without the constructor, so that outside this library a key can+-- only be made by 'key', which checks.+module Crypto.Internal.Poly1305 (+    Key (..),+    key,+) where++import Crypto.Error+import Crypto.Internal.ByteArray (ByteArrayAccess, ScrubbedBytes)+import qualified Crypto.Internal.ByteArray as B+import Crypto.Internal.DeepSeq++-- | A Poly1305 key: thirty-two bytes, and the length is checked here rather+-- than at every use.  'Crypto.MAC.Poly1305.initialize' and+-- 'Crypto.MAC.Poly1305.auth' take one of these and cannot fail, so a caller+-- that holds a key does not carry an error case for a length it already knows+-- is right.+newtype Key = Key ScrubbedBytes+    deriving (ByteArrayAccess, Eq, NFData)++-- | Take thirty-two bytes for a key.  A different length is reported as+-- 'CryptoError_MacKeyInvalid'; nothing else about a key can be wrong.+key :: ByteArrayAccess ba => ba -> CryptoFailable Key+key k+    | B.length k /= 32 = CryptoFailed CryptoError_MacKeyInvalid+    | otherwise = CryptoPassed $ Key $ B.convert k
+ Crypto/KEM.hs view
@@ -0,0 +1,121 @@+-- |+-- Module      : Crypto.KEM+-- License     : BSD-style+-- Maintainer  : Kazu Yamamoto <kazu@iij.ad.jp>+-- Stability   : experimental+-- Portability : unknown+--+-- Key encapsulation: one side publishes a key, the other draws a secret and+-- returns something only the first side can turn back into it.+--+-- The shape is not ML-KEM's alone, which is why this is a class in a module+-- of its own rather than part of any one algorithm: 'Crypto.PubKey.MLKEM'+-- is one instance of it, and a caller that does not care which it has can+-- be written against this.+--+-- A bare Diffie-Hellman exchange is deliberately __not__ an instance, even+-- though the shapes line up.  The KEM a Diffie-Hellman group gives is+-- DHKEM, of RFC 9180 section 4.1, and that is not the raw exchange: its+-- shared secret is the exchange's output run through HKDF with the+-- ephemeral and the recipient public keys as context, under labels that+-- name the ciphersuite.  The binding to those two keys, and the separation+-- between suites, are what the KEM security argument rests on; the raw+-- value has neither.  A protocol can supply the binding at its own level --+-- TLS 1.3 does, in the key schedule -- but then the binding belongs to the+-- protocol, not to this class, and offering the raw exchange here would+-- invite its use somewhere that supplies nothing.+--+-- DHKEM proper is an instance, and it is not here either: the labels it+-- derives under carry the HPKE ciphersuite identifier, which is an IANA+-- registry value rather than anything a primitive knows, so the instances+-- live beside the registry in the @hpke@ package.+{-# LANGUAGE GeneralizedNewtypeDeriving #-}+{-# LANGUAGE TypeFamilies #-}++module Crypto.KEM (+    KEM (..),+    SharedSecret (..),+) where++import Data.Kind (Type)++import Crypto.Error (CryptoFailable)+import Crypto.Internal.ByteArray (ByteArrayAccess, ScrubbedBytes)+import Crypto.Internal.Imports+import Crypto.Random (MonadRandom)++-- | Secret shared via key exchange.+newtype SharedSecret = SharedSecret ScrubbedBytes+    deriving (Eq, ByteArrayAccess, NFData)++instance Show SharedSecret where+    show _ = "SharedSecret <redacted>"++instance Semigroup SharedSecret where+    SharedSecret x <> SharedSecret y = SharedSecret (x <> y)++instance Monoid SharedSecret where+    mempty = SharedSecret mempty++-- | A key encapsulation mechanism.+--+-- The three values are named for what they do rather than for what they are+-- in any one algorithm.  In ML-KEM the encapsulation key and the ciphertext+-- are what the names say; in a Diffie-Hellman exchange both are public+-- values of the group, and the \"ciphertext\" is the ephemeral one the+-- encapsulating side generates.  Keeping them apart in the types is what+-- stops one being passed where the other belongs, which they are not+-- interchangeable for even where they have the same representation.+class KEM kem where+    -- | What the encapsulating side is given.+    type EncapsulationKey kem :: Type++    -- | What the decapsulating side keeps.+    type DecapsulationKey kem :: Type++    -- | What travels back, and is decapsulated.+    type Ciphertext kem :: Type++    -- | The randomness 'encapsulate' draws, for the instances that let a+    -- caller supply it.  In ML-KEM it is @m@ of FIPS 203, a string of+    -- bytes; in DHKEM it is the ephemeral secret key, a scalar of the+    -- group.  Both are secret, and both determine the shared secret+    -- completely.+    type Coins kem :: Type++    -- | Generate a key pair for the decapsulating side.+    generateKeyPair+        :: MonadRandom m+        => proxy kem -> m (EncapsulationKey kem, DecapsulationKey kem)++    -- | Draw a secret and encapsulate it against the key.+    --+    -- This can fail, and does for some instances: a Diffie-Hellman exchange+    -- refuses a peer value that would make the secret degenerate, where+    -- ML-KEM has nothing to refuse.+    encapsulate+        :: MonadRandom m+        => proxy kem+        -> EncapsulationKey kem+        -> m (CryptoFailable (Ciphertext kem, SharedSecret))++    -- | Encapsulate with the randomness supplied rather than drawn.+    --+    -- The secret this produces is a deterministic function of the key and+    -- these coins, so they must come from a source no other party can+    -- predict or repeat, and must not be used twice.  'encapsulate' is the+    -- entry point for ordinary use; this one is for test vectors, and for a+    -- protocol that has to name the ephemeral value it used -- HPKE lets a+    -- sender supply its own, and the RFC 9180 vectors are written that way.+    encapsulateWith+        :: proxy kem+        -> EncapsulationKey kem+        -> Coins kem+        -> CryptoFailable (Ciphertext kem, SharedSecret)++    -- | Recover the secret.+    decapsulate+        :: proxy kem+        -> DecapsulationKey kem+        -> Ciphertext kem+        -> CryptoFailable SharedSecret
Crypto/MAC/Poly1305.hs view
@@ -35,6 +35,7 @@  ) import qualified Crypto.Internal.ByteArray as B import Crypto.Internal.DeepSeq+import Crypto.Internal.Poly1305 (Key (..), key) import Data.Word import Foreign.C.Types import Foreign.Ptr@@ -48,20 +49,6 @@ newtype State = State ScrubbedBytes     deriving (ByteArrayAccess) --- | A Poly1305 key: thirty-two bytes, and the length is checked here rather--- than at every use.  'initialize' and 'auth' take one of these and cannot--- fail, so a caller that holds a key does not carry an error case for a--- length it already knows is right.-newtype Key = Key ScrubbedBytes-    deriving (ByteArrayAccess, Eq, NFData)---- | Take thirty-two bytes for a key.  A different length is reported as--- 'CryptoError_MacKeyInvalid'; nothing else about a key can be wrong.-key :: ByteArrayAccess ba => ba -> CryptoFailable Key-key k-    | B.length k /= 32 = CryptoFailed CryptoError_MacKeyInvalid-    | otherwise = CryptoPassed $ Key $ B.convert k- -- | Poly1305 State. use State instead of Ctx type Ctx = State @@ -105,7 +92,10 @@ update :: ByteArrayAccess ba => State -> ba -> State update (State prevCtx) d = State $ B.copyAndFreeze prevCtx $ \ctxPtr ->     B.withByteArray d $ \dataPtr ->-        c_poly1305_update (castPtr ctxPtr) dataPtr (fromIntegral $ B.length d)+        -- in pieces the C's uint32_t length can hold; the context carries+        -- across, so it can simply be called again+        B.inCLengths (B.length d) $ \off n ->+            c_poly1305_update (castPtr ctxPtr) (dataPtr `plusPtr` off) (fromIntegral n) {-# NOINLINE update #-}  -- | updates a context with multiples bytestring@@ -114,7 +104,9 @@   where     loop [] _ = return ()     loop (x : xs) ctxPtr = do-        B.withByteArray x $ \dataPtr -> c_poly1305_update ctxPtr dataPtr (fromIntegral $ B.length x)+        B.withByteArray x $ \dataPtr ->+            B.inCLengths (B.length x) $ \off n ->+                c_poly1305_update ctxPtr (dataPtr `plusPtr` off) (fromIntegral n)         loop xs ctxPtr {-# NOINLINE updates #-} @@ -137,5 +129,9 @@         B.withByteArray k $ \keyPtr -> do             c_poly1305_init (castPtr ctxPtr) keyPtr             B.withByteArray d $ \dataPtr ->-                c_poly1305_update (castPtr ctxPtr) dataPtr (fromIntegral $ B.length d)+                B.inCLengths (B.length d) $ \off n ->+                    c_poly1305_update+                        (castPtr ctxPtr)+                        (dataPtr `plusPtr` off)+                        (fromIntegral n)             c_poly1305_finalize dst (castPtr ctxPtr)
Crypto/Number/Basic.hs view
@@ -85,7 +85,12 @@  -- | Compute the number of bits for an integer numBits :: Integer -> Int-numBits n = gmpSizeInBits n `onGmpUnsupported` (if n == 0 then 1 else computeBits 0 n)+-- GMP sizes the magnitude and calls zero zero bits, and every caller here --+-- 'numBytes' above all -- is written against that.  The fallback used to+-- answer 1 for zero, and to divide a negative number by 256 forever, because+-- the quotient never reaches zero.+numBits n =+    gmpSizeInBits n `onGmpUnsupported` (if n == 0 then 0 else computeBits 0 (abs n))   where     computeBits !acc i         | q == 0 =
Crypto/Number/F2m.hs view
@@ -36,9 +36,8 @@     (.&.),     (.|.),  )-import Data.List (foldl')+import qualified Data.List as L import Data.Word (Word32)-import Prelude hiding (foldl')  -- | Binary Polynomial represented by an integer type BinaryPolynomial = Integer@@ -89,7 +88,7 @@         | otherwise =             fold                 es-                (foldl' (\acc e -> acc `xor` (hi `shiftL` e)) (n .&. mask) es)+                (L.foldl' (\acc e -> acc `xor` (hi `shiftL` e)) (n .&. mask) es)       where         hi = n `shiftR` lfx {-# INLINE modF2m #-}@@ -211,7 +210,7 @@   where     spread :: Int -> Word32     spread b =-        foldl' (\acc i -> if testBit b i then setBit acc (2 * i) else acc) 0 [0 .. 7]+        L.foldl' (\acc i -> if testBit b i then setBit acc (2 * i) else acc) 0 [0 .. 7] {-# NOINLINE spreadTable #-}  {-# INLINE squareF2m' #-}
Crypto/Number/ModArithmetic.hs view
@@ -43,6 +43,7 @@ import Crypto.Internal.Compat (unsafeDoIO) import Crypto.Number.Basic import Crypto.Number.Compat+import Crypto.Number.ModArithmetic.Pub (expPub) import qualified Crypto.Number.Serialize.Internal as Internal import Data.Memory.PtrMethods (memSet) import Data.Word (Word32, Word8)@@ -141,6 +142,7 @@     -- what GMP's own secure exponentiation lets slip     !eLen = 8 * ((numBytes e + 7) `div` 8) + foreign import ccall safe "crypton_powm_sec"     c_powm_sec         :: Ptr Word8@@ -158,6 +160,14 @@ -- -- Use this function when all the parameters are public, -- otherwise 'expSafe' should be preferred.+--+-- A short exponent on an odd modulus -- RSA verification, where the exponent+-- is 65537 and comes out of the public key -- is walked a bit at a time in+-- C, squaring at each step and multiplying where a bit is set, which reads+-- the exponent and branches on it and is why this is not 'expSafe'.  That+-- path is taken only where it is the faster of the two, which the C decides;+-- everywhere else, and for every longer exponent, this is GMP's windowed+-- exponentiation as it has always been. expFast     :: Integer     -- ^ base@@ -167,13 +177,29 @@     -- ^ modulo     -> Integer     -- ^ result-expFast b e m = gmpPowModInteger b e m `onGmpUnsupported` exponentiation b e m+expFast b e m+    | odd m && m > 1 && e > 0 = case expPub (b `mod` m) e m of+        Just r -> r+        Nothing -> slow+    | otherwise = slow+  where+    slow = gmpPowModInteger b e m `onGmpUnsupported` exponentiation b e m  -- | @exponentiation@ computes modular exponentiation as /b^e mod m/ -- using repetitive squaring.+--+-- The corner cases are held to what GMP answers, since that is what this+-- computes on every build that has it: a modulus of one is zero whatever+-- else is asked, and a negative exponent is a request for the inverse of+-- the base raised to its magnitude, which is zero when no inverse exists.+-- Read literally, the recursion below walked a negative exponent from -1 to+-- -2 and back for as long as the stack held. exponentiation :: Integer -> Integer -> Integer -> Integer exponentiation b e m-    | b == 1 = b+    | m == 1 = 0+    | e < 0 =+        maybe 0 (\bInv -> exponentiation bInv (negate e) m) (inverse (b `mod` m) m)+    | b == 1 = 1     | e == 0 = 1     | e == 1 = b `mod` m     | even e =@@ -240,20 +266,56 @@ -- bits it is given, and a nonce inverted that way has been taken apart before -- by watching the steps go by. ----- The moduli this is for -- the order of a group -- are prime, so the inverse--- comes from 'inverseFermat' instead.  When the modulus is not prime, that--- answer is not an inverse, and the result is checked and 'inverse' asked--- instead, so this agrees with 'inverse' on every input.  That fallback is--- reached only by parameters that are already broken.+-- The answer comes from a fixed number of division steps where the assembly+-- for them is built, and from 'inverseFermat' where it is not.  Either way it+-- is checked here by multiplying out: neither one says when the number has no+-- inverse -- the first returns something that is not one and the second+-- returns something that is not one either -- so the check is what makes this+-- agree with 'inverse' on every input, and 'inverse' is asked when it fails.+-- That fallback is reached only by parameters that are already broken. ----- It costs what an exponentiation costs: around thirty times an 'inverse'--- for a 256-bit modulus.+-- The division steps cost about a twentieth of the exponentiation: on an+-- Apple M4, inverting modulo the P-256 group order is 0.80 microseconds+-- against 6.02, and modulo the P-521 one 2.05 against 63.2. inverseSafe :: Integer -> Integer -> Maybe Integer inverseSafe g m     | m > 1 && (g * r) `mod` m == 1 = Just r     | otherwise = inverse g m   where-    r = inverseFermat g m+    r = case inverseSec g m of+        Just v -> v+        Nothing -> inverseFermat g m++-- | The inverse in a fixed number of division steps, from the vendored+-- assembly.  'Nothing' when that is not built, when the modulus is even --+-- where the routine answers without saying it cannot -- or when the numbers+-- are larger than it keeps room for.  The answer is not checked here; the+-- caller does that.+inverseSec :: Integer -> Integer -> Maybe Integer+inverseSec g m+    | m <= 1 || even m || g < 0 = Nothing+    | otherwise = unsafeDoIO $+        allocaBytes (3 * mLen) $ \out -> do+            let gp = out `plusPtr` mLen+                mp = gp `plusPtr` mLen+            _ <- Internal.i2ospOf (g `mod` m) gp mLen+            _ <- Internal.i2ospOf m mp mLen+            r <- c_modinv_sec out gp mp (fromIntegral mLen)+            if r == 0+                then do+                    !v <- Internal.os2ip out mLen+                    return (Just v)+                else return Nothing+  where+    !mLen = numBytes m++foreign import ccall unsafe "crypton_modinv_sec"+    c_modinv_sec+        :: Ptr Word8+        -> Ptr Word8+        -> Ptr Word8+        -> Word32+        -> IO CInt  -- | Raised when the assumption about the modulus is invalid. data ModulusAssertionError = ModulusAssertionError
+ Crypto/Number/ModArithmetic/Pub.hs view
@@ -0,0 +1,158 @@+{-# LANGUAGE BangPatterns #-}++-- |+-- Module      : Crypto.Number.ModArithmetic.Pub+-- License     : BSD-style+-- Stability   : experimental+-- Portability : Good+--+-- The C exponentiation for an exponent that is not a secret, and the R^2+-- it keeps per modulus.  Internal: 'expPub' is behind 'expFast', and+-- 'expPubBytes' behind RSA's public-key operation.+module Crypto.Number.ModArithmetic.Pub (+    expPub,+    expPubBytes,+) where++import Crypto.Internal.ByteArray (+    ByteArray,+    ByteArrayAccess,+    Bytes,+    allocRet,+    withByteArray,+ )+import qualified Crypto.Internal.ByteArray as B+import Crypto.Internal.Compat (unsafeDoIO)+import Crypto.Number.Basic+import qualified Crypto.Number.Serialize.Internal as Internal+import Data.IORef (IORef, atomicModifyIORef', newIORef)+import Data.Word (Word32, Word8)+import Foreign.C.Types (CInt (..))+import Foreign.Marshal.Alloc (allocaBytes)+import Foreign.Ptr (Ptr, plusPtr)+import System.IO.Unsafe (unsafePerformIO)++-- | The windowless exponentiation in C, for an exponent that is not a+--   secret.  'Nothing' when the C declines, which it does on every machine+--   where what the caller already has is faster -- so this is an+--   optimisation and never the only path.  The base has to be reduced and+--   the exponent positive, which the caller sees to.+expPub :: Integer -> Integer -> Integer -> Maybe Integer+expPub b e m = unsafeDoIO $ do+    known <- r2For m+    case known of+        Nothing -> return Nothing+        Just (r2, modu) ->+            allocaBytes (sum widths) $ \start -> case scanl plusPtr start widths of+                (out : base : expo : _) -> do+                    _ <- Internal.i2ospOf b base mLen+                    _ <- Internal.i2ospOf e expo eLen+                    r <- withByteArray modu $ \pm -> withByteArray r2 $ \pr2 ->+                        c_powm_pub_r2+                            out+                            base+                            (fromIntegral mLen)+                            expo+                            (fromIntegral eLen)+                            pm+                            (fromIntegral mLen)+                            pr2+                    if r == 0+                        then do+                            !v <- Internal.os2ip out mLen+                            return (Just v)+                        else return Nothing+                _ -> return Nothing+  where+    !mLen = numBytes m+    !eLen = numBytes e+    widths = [mLen, mLen, eLen]++-- | The same with the base and the answer as big-endian bytes, as long as+--   the modulus: what RSA verification has in hand, so neither goes through+--   an 'Integer'.  'Nothing' also when the base is not below the modulus.+expPubBytes+    :: (ByteArrayAccess bi, ByteArray bo) => Integer -> Integer -> bi -> Maybe bo+expPubBytes e m b+    | e <= 0 || m <= 1 || even m || B.length b /= mLen = Nothing+    | otherwise = unsafeDoIO $ do+        known <- r2For m+        case known of+            Nothing -> return Nothing+            Just (r2, modu)+                | not (below b modu) -> return Nothing+                | otherwise -> allocaBytes eLen $ \expo -> do+                    _ <- Internal.i2ospOf e expo eLen+                    (r, out) <- allocRet mLen $ \pout ->+                        withByteArray b $ \pb -> withByteArray modu $ \pm ->+                            withByteArray r2 $ \pr2 ->+                                c_powm_pub_r2+                                    pout+                                    pb+                                    (fromIntegral mLen)+                                    expo+                                    (fromIntegral eLen)+                                    pm+                                    (fromIntegral mLen)+                                    pr2+                    return (if r == 0 then Just out else Nothing)+  where+    !mLen = numBytes m+    !eLen = numBytes e++-- big-endian numbers of one length: the first byte that differs decides+below :: (ByteArrayAccess a, ByteArrayAccess b) => a -> b -> Bool+below x y = go 0+  where+    n = B.length x+    go i+        | i >= n = False+        | otherwise = case compare (B.index x i) (B.index y i) of+            EQ -> go (i + 1)+            o -> o == LT++-- | R^2 mod m for the public path, and m in bytes, kept for the last few+-- moduli, since a program verifies under the same keys again and again.+-- 'Nothing' is kept too: it means the C declines this modulus, which is+-- then not asked again.+r2For :: Integer -> IO (Maybe (Bytes, Bytes))+r2For m = do+    found <- atomicModifyIORef' r2Table $ \ks -> case break ((== m) . fst) ks of+        (before, hit : after) -> (hit : before ++ after, Just (snd hit))+        _ -> (ks, Nothing)+    case found of+        Just r2 -> return r2+        Nothing -> do+            let mLen = numBytes m+            (r2, modu) <- allocRet mLen $ \modu -> do+                _ <- Internal.i2ospOf m modu mLen+                (r, out) <- allocRet mLen $ \pout -> c_powm_r2 pout modu (fromIntegral mLen)+                return (if r == 0 then Just out else Nothing)+            let kept = fmap (\x -> (x, modu)) r2+            atomicModifyIORef' r2Table $ \ks -> (take r2TableSize ((m, kept) : ks), ())+            return kept++r2TableSize :: Int+r2TableSize = 8++r2Table :: IORef [(Integer, Maybe (Bytes, Bytes))]+r2Table = unsafePerformIO (newIORef [])+{-# NOINLINE r2Table #-}++-- safe, as the secure one is: the call is tens of microseconds at RSA+-- sizes, and the hundred nanoseconds this costs over an unsafe call is+-- nothing beside holding a capability for that long in a server+foreign import ccall safe "crypton_powm_pub_r2"+    c_powm_pub_r2+        :: Ptr Word8+        -> Ptr Word8+        -> Word32+        -> Ptr Word8+        -> Word32+        -> Ptr Word8+        -> Word32+        -> Ptr Word8+        -> IO CInt++foreign import ccall unsafe "crypton_powm_r2"+    c_powm_r2 :: Ptr Word8 -> Ptr Word8 -> Word32 -> IO CInt
Crypto/Number/Serialize/Internal.hs view
@@ -53,7 +53,12 @@     !padSz = ptrSz - sz  fillPtr :: Ptr Word8 -> Int -> Integer -> IO ()-fillPtr p sz m = gmpExportInteger m p `onGmpUnsupported` export (sz - 1) m+fillPtr p sz m+    -- zero is no bytes wide, and the callers above have already written the+    -- room out as zeros.  Without this the loop below starts at offset -1,+    -- never meets the 0 it stops at, and walks backwards out of the buffer.+    | sz <= 0 = return ()+    | otherwise = gmpExportInteger m p `onGmpUnsupported` export (sz - 1) m   where     export ofs i         | ofs == 0 = pokeByteOff p ofs (fromIntegral i :: Word8)
Crypto/OTP.hs view
@@ -49,9 +49,8 @@ import Crypto.MAC.HMAC import Data.Bits (complement, shiftL, shiftR, xor, (.&.), (.|.)) import Data.ByteArray.Mapping (fromW64BE)-import Data.List (foldl')+import qualified Data.List as L import Data.Word-import Prelude hiding (foldl')  -- | A one-time password which is a sequence of 4 to 9 digits. type OTP = Word32@@ -150,7 +149,7 @@     range = map (hotp h d k) [c .. c + fromIntegral s]      -- the offset of the first match, accumulated without stopping there-    (matched, offset) = foldl' pick (0, 0) (zip [0 ..] range)+    (matched, offset) = L.foldl' pick (0, 0) (zip [0 ..] range)     pick (!m, !off) (i, candidate) = (m .|. hit, off .|. (hit .&. i))       where         -- zero once something has matched, so only the first match counts@@ -162,7 +161,7 @@     -- the counters continue past the window, and wrap where the old     -- 'checkExtraOtps' wrapped     extrasMatched =-        foldl' step (complement 0) (zip (iterate (+ 1) afterFirst) extras)+        L.foldl' step (complement 0) (zip (iterate (+ 1) afterFirst) extras)     step acc (ctr, p) = acc .&. eqMask (hotp h d k ctr) p  -- | All ones when the two values are equal, zero otherwise, without branching@@ -250,7 +249,7 @@     -- every candidate is compared, and none of the comparisons stops early, so     -- neither which step matched nor how far a mismatch got is visible in how     -- long this takes-    matched = foldl' step 0 (map (hotp h d k) (range window []))+    matched = L.foldl' step 0 (map (hotp h d k) (range window []))     step acc candidate = acc .|. eqMask candidate otp  timeToCounter :: Word64 -> Word64 -> Word16 -> Word64
Crypto/PubKey/Curve25519.hs view
@@ -1,5 +1,4 @@ {-# LANGUAGE GeneralizedNewtypeDeriving #-}-{-# LANGUAGE MagicHash #-} {-# LANGUAGE ScopedTypeVariables #-}  -- |@@ -31,7 +30,6 @@ import Data.Word import Foreign.Ptr import Foreign.Storable-import GHC.Ptr  import Crypto.Error import Crypto.Internal.ByteArray (@@ -114,20 +112,19 @@     $ \result ->         withByteArray sec $ \psec ->             withByteArray pub $ \ppub ->-                ccrypton_curve25519 result psec ppub+                ccrypton_x25519 result psec ppub {-# NOINLINE dh #-}  -- | Create a public key from a secret key+-- The base point does not go in: where the assembly is built there is a table+-- for this, and it is four to five times less work than multiplying the point+-- 9 the general way. toPublic :: SecretKey -> PublicKey toPublic (SecretKey sec) = PublicKey     <$> B.allocAndFreeze 32     $ \result ->         withByteArray sec $ \psec ->-            ccrypton_curve25519 result psec basePoint-  where-    basePoint =-        Ptr-            "\x09\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"#+            ccrypton_x25519_base result psec {-# NOINLINE toPublic #-}  -- | Generate a secret key.@@ -142,12 +139,20 @@     modifyByte :: Ptr Word8 -> Int -> (Word8 -> Word8) -> IO ()     modifyByte p n f = peekByteOff p n >>= pokeByteOff p n . f -foreign import ccall "crypton_curve25519_donna"-    ccrypton_curve25519+foreign import ccall "crypton_x25519"+    ccrypton_x25519         :: Ptr Word8         -- ^ public         -> Ptr Word8         -- ^ secret         -> Ptr Word8         -- ^ basepoint+        -> IO ()++foreign import ccall "crypton_x25519_base"+    ccrypton_x25519_base+        :: Ptr Word8+        -- ^ public+        -> Ptr Word8+        -- ^ secret         -> IO ()
Crypto/PubKey/DSA.hs view
@@ -41,9 +41,12 @@     -- * Signature primitive     sign,     signWith,+    signDigest,+    signDigestWith,      -- * Verification primitive     verify,+    verifyDigest,      -- * Key pair     KeyPair (..),@@ -59,7 +62,7 @@ import Crypto.Internal.Imports import Crypto.Number.Generate import Crypto.Number.ModArithmetic (expFast, expSafe, inverse, inverseSafe)-import Crypto.PubKey.Internal (dsaTruncHash)+import Crypto.PubKey.Internal (dsaTruncHashDigest) import Crypto.Random.Types  -- | DSA Public Number, usually embedded in DSA Public Key@@ -194,14 +197,31 @@     -> msg     -- ^ message to sign     -> Maybe Signature-signWith k pk hashAlg msg = do+signWith k pk hashAlg msg = signDigestWith k pk (hashWith hashAlg msg)++-- | Sign a digest using the private key and an explicit k number.+--+-- The digest's type says which algorithm made it, so this needs nothing else+-- to name one.  'signWith' takes a @hash@ value and never reads it -- it is+-- there to fix the type -- which leaves a caller that is itself polymorphic+-- in the algorithm with nothing to pass.+signDigestWith+    :: HashAlgorithm hash+    => Integer+    -- ^ k random number+    -> PrivateKey+    -- ^ private key+    -> Digest hash+    -- ^ digest of the message to sign+    -> Maybe Signature+signDigestWith k pk digest = do     -- k comes from the caller and is only invertible when it is coprime with     -- q, which the caller cannot check without knowing q is prime.  It is also     -- a secret worth as much as the private key, so it is inverted without     -- the extended Euclidean algorithm, whose steps follow the bits of what     -- it is given     kInv <- inverseSafe k q-    let hm = dsaTruncHash hashAlg msg q+    let hm = dsaTruncHashDigest digest q         r = expSafe g k p `mod` q         s = (kInv * (hm + x * r)) `mod` q     if r == 0 || s == 0 then Nothing else Just $ Signature r s@@ -214,10 +234,17 @@ sign     :: (ByteArrayAccess msg, HashAlgorithm hash, MonadRandom m)     => PrivateKey -> hash -> msg -> m Signature-sign pk hashAlg msg = do+sign pk hashAlg msg = signDigest pk (hashWith hashAlg msg)++-- | Sign a digest using the private key.  See 'signDigestWith' for why a+-- digest rather than a @hash@ value.+signDigest+    :: (HashAlgorithm hash, MonadRandom m)+    => PrivateKey -> Digest hash -> m Signature+signDigest pk digest = do     k <- generateMax q-    case signWith k pk hashAlg msg of-        Nothing -> sign pk hashAlg msg+    case signDigestWith k pk digest of+        Nothing -> signDigest pk digest         Just sig -> return sig   where     (Params _ _ q) = private_params pk@@ -226,7 +253,13 @@ verify     :: (ByteArrayAccess msg, HashAlgorithm hash)     => hash -> PublicKey -> Signature -> msg -> Bool-verify hashAlg pk (Signature r s) m+verify hashAlg pk sig m = verifyDigest pk sig (hashWith hashAlg m)++-- | Verify a signature over a digest.  See 'signDigestWith' for why a digest+-- rather than a @hash@ value.+verifyDigest+    :: HashAlgorithm hash => PublicKey -> Signature -> Digest hash -> Bool+verifyDigest pk (Signature r s) digest     -- Reject the signature if either 0 < r < q or 0 < s < q is not satisfied.     | r <= 0 || r >= q || s <= 0 || s >= q = False     -- s is invertible for every 0 < s < q when q is prime, but the parameters@@ -235,7 +268,7 @@   where     (Params p g q) = public_params pk     y = public_y pk-    hm = dsaTruncHash hashAlg m q+    hm = dsaTruncHashDigest digest q     v = do         w <- inverse s q         let u1 = (hm * w) `mod` q
Crypto/PubKey/ECC/P256.hs view
@@ -413,8 +413,6 @@ foreign import ccall "crypton_p256e_scalar_invert"     ccrypton_p256e_scalar_invert :: Ptr P256Scalar -> Ptr P256Scalar -> IO () --- foreign import ccall "crypton_p256_modinv"---    ccrypton_p256_modinv :: Ptr P256Scalar -> Ptr P256Scalar -> Ptr P256Scalar -> IO () foreign import ccall "crypton_p256_modinv_vartime"     ccrypton_p256_modinv_vartime         :: Ptr P256Scalar -> Ptr P256Scalar -> Ptr P256Scalar -> IO ()
Crypto/PubKey/ECDSA.hs view
@@ -48,6 +48,11 @@     signDigest,     verify,     verifyDigest,++    -- * Deterministic nonces+    deterministicNonce,+    signDeterministic,+    signDigestDeterministic, ) where  import Control.Monad@@ -58,8 +63,10 @@ import Crypto.Hash import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess) import Crypto.Internal.Imports-import Crypto.Number.ModArithmetic (inverseFermat)+import Crypto.Number.Generate (generatePrefix)+import Crypto.Number.ModArithmetic (inverseSafe) import qualified Crypto.PubKey.ECC.P256 as P256+import Crypto.Random.HmacDRG (initial, update) import Crypto.Random.Types  import Data.Bits@@ -257,6 +264,75 @@ verify prx hashAlg q sig msg = verifyDigest prx q sig (hashWith hashAlg msg)  -- | Truncate a digest based on curve order size.+-- | Deterministic nonce generation according to RFC 6979.+--+-- The nonce is derived from the private key and the message alone, so a+-- signature made this way needs no random number generator and cannot be the+-- one that repeats a nonce -- which, for ECDSA, hands over the private key.+--+-- The hash used to seed the generator is given separately from the one the+-- message was digested with, as RFC 6979 allows.+--+-- The last argument is what to do with a candidate nonce.  It may answer+-- 'Nothing', in which case another candidate is drawn, which is what+-- 'signDigestDeterministic' does for the r or s that comes out zero:+--+-- > deterministicNonce prx SHA256 priv digest (\k -> signDigestWith prx k priv digest)+deterministicNonce+    :: (EllipticCurveECDSA curve, HashAlgorithm hashDRG, HashAlgorithm hashDigest)+    => proxy curve+    -> hashDRG+    -> PrivateKey curve+    -> Digest hashDigest+    -> (Scalar curve -> Maybe a)+    -> a+deterministicNonce prx alg d digest go = fst $ withDRG state run+  where+    state = update seed $ initial alg+    -- RFC 6979 section 3.2 step d: int2octets(x) || bits2octets(h1).  The+    -- second is the truncated digest taken modulo the order, which is what+    -- scalarAdd with zero does, its contract being to reduce there.+    seed =+        B.append (encodeScalar prx d) (encodeScalar prx z)+            :: B.ScrubbedBytes+    z = scalarAdd prx (tHashDigest prx digest) zeroScalar+    zeroScalar = throwCryptoError $ scalarFromInteger prx 0+    run = do+        k <- generatePrefix (curveOrderBits prx)+        case scalarFromInteger prx k of+            CryptoPassed s+                | scalarIsValid prx s -> maybe run pure (go s)+            _ -> run++-- | Sign a digest with a nonce derived from the private key and the digest,+-- as RFC 6979 says, rather than from a random number generator.+signDigestDeterministic+    :: (EllipticCurveECDSA curve, HashAlgorithm hashDRG, HashAlgorithm hashDigest)+    => proxy curve+    -> hashDRG+    -> PrivateKey curve+    -> Digest hashDigest+    -> Signature curve+signDigestDeterministic prx alg d digest =+    deterministicNonce prx alg d digest $ \k -> signDigestWith prx k d digest++-- | Sign a message with a nonce derived from the private key and the message,+-- as RFC 6979 says, rather than from a random number generator.+signDeterministic+    :: ( EllipticCurveECDSA curve+       , HashAlgorithm hashDRG+       , HashAlgorithm hash+       , ByteArrayAccess msg+       )+    => proxy curve+    -> hashDRG+    -> PrivateKey curve+    -> hash+    -> msg+    -> Signature curve+signDeterministic prx alg d hashAlg msg =+    signDigestDeterministic prx alg d (hashWith hashAlg msg)+ tHashDigest     :: (EllipticCurveECDSA curve, HashAlgorithm hash)     => proxy curve -> Digest hash -> Scalar curve@@ -296,15 +372,16 @@     => Simple.Scalar curve -> Bool ecScalarIsZero (Simple.Scalar a) = a == 0 +-- | 'inverseSafe' is the one that takes a fixed number of division steps+-- where the assembly for them is built, and checks whatever it gets by+-- multiplying out.  It answers 'Nothing' exactly where the exponentiation+-- this used to do answered zero. ecScalarInv     :: Simple.Curve c     => proxy c -> Simple.Scalar c -> Maybe (Simple.Scalar c)-ecScalarInv prx (Simple.Scalar s)-    | i == 0 = Nothing-    | otherwise = Just $ Simple.Scalar i+ecScalarInv prx (Simple.Scalar s) = Simple.Scalar <$> inverseSafe s n   where     n = Simple.curveEccN $ Simple.curveParameters prx-    i = inverseFermat s n  ecPointX     :: Simple.Curve c
Crypto/PubKey/ElGamal.hs view
@@ -57,10 +57,13 @@      -- * Signature primitives     signWith,+    signDigestWith,     sign,+    signDigest,      -- * Verification primitives     verify,+    verifyDigest, ) where  import Crypto.Error@@ -194,8 +197,28 @@     -> msg     -- ^ message to sign     -> Maybe Signature-signWith = signWithBlinder 1+signWith k params priv hashAlg msg =+    signDigestWith k params priv (hashWith hashAlg msg) +-- | Sign a digest with an explicit ephemeral value.+--+-- The digest's type says which algorithm made it, so this needs nothing else+-- to name one.  'signWith' takes a @hash@ value and never reads it -- it is+-- there to fix the type -- which leaves a caller that is itself polymorphic+-- in the algorithm with nothing to pass.+signDigestWith+    :: HashAlgorithm hash+    => Integer+    -- ^ ephemeral value k, in [1, p-2] and coprime with p-1+    -> Params+    -- ^ DH params (p,g)+    -> PrivateNumber+    -- ^ DH private key+    -> Digest hash+    -- ^ digest of the message to sign+    -> Maybe Signature+signDigestWith = signWithBlinder 1+ -- | The same with a blinder for the inversion of @k@. -- -- @k@ is inverted modulo @p-1@, which is even, so Fermat's little theorem@@ -210,15 +233,15 @@ -- When @b@ shares a factor with @p-1@ the algorithm reports it the same way -- it reports one in @k@, and the answer is the same: draw again. signWithBlinder-    :: (ByteArrayAccess msg, HashAlgorithm hash)-    => Integer -> Integer -> Params -> PrivateNumber -> hash -> msg -> Maybe Signature-signWithBlinder b k (Params p g _) (PrivateNumber x) hashAlg msg+    :: HashAlgorithm hash+    => Integer -> Integer -> Params -> PrivateNumber -> Digest hash -> Maybe Signature+signWithBlinder b k (Params p g _) (PrivateNumber x) digest     | k <= 0 || k >= p - 1 || b <= 0 || d > 1 = Nothing     | s == 0 = Nothing     | otherwise = Just $ Signature r s   where     r = expSafe g k p-    h = os2ip $ hashWith hashAlg msg+    h = os2ip digest     s = ((h - x * r) * kInv) `mod` (p - 1)     kInv = (kbInv * b) `mod` (p - 1)     (kbInv, _, d) = gcde ((k * b) `mod` (p - 1)) (p - 1)@@ -239,13 +262,20 @@     -> msg     -- ^ message to sign     -> m Signature-sign params@(Params p _ _) priv hashAlg msg = do+sign params priv hashAlg msg = signDigest params priv (hashWith hashAlg msg)++-- | Sign a digest, drawing the ephemeral value.  See 'signDigestWith' for+-- why a digest rather than a @hash@ value.+signDigest+    :: (HashAlgorithm hash, MonadRandom m)+    => Params -> PrivateNumber -> Digest hash -> m Signature+signDigest params@(Params p _ _) priv digest = do     k <- generateMax (p - 1)     -- and a blinder for the inversion of k, which is the one step here that     -- the extended Euclidean algorithm has to do     b <- generateMax (p - 1)-    case signWithBlinder b k params priv hashAlg msg of-        Nothing -> sign params priv hashAlg msg+    case signWithBlinder b k params priv digest of+        Nothing -> signDigest params priv digest         Just sig -> return sig  -- | verify a signature@@ -257,10 +287,18 @@     -> msg     -> Signature     -> Bool-verify (Params p g _) (PublicNumber y) hashAlg msg (Signature r s)+verify params pub hashAlg msg sig =+    verifyDigest params pub (hashWith hashAlg msg) sig++-- | Verify a signature over a digest.  See 'signDigestWith' for why a digest+-- rather than a @hash@ value.+verifyDigest+    :: HashAlgorithm hash+    => Params -> PublicNumber -> Digest hash -> Signature -> Bool+verifyDigest (Params p g _) (PublicNumber y) digest (Signature r s)     | or [r <= 0, r >= p, s <= 0, s >= (p - 1)] = False     | otherwise = lhs == rhs   where-    h = os2ip $ hashWith hashAlg msg+    h = os2ip digest     lhs = expFast g h p     rhs = (expFast y r p * expFast r s p) `mod` p
Crypto/PubKey/Internal.hs view
@@ -12,8 +12,7 @@ ) where  import Data.Bits (shiftR)-import Data.List (foldl')-import Prelude hiding (foldl')+import qualified Data.List as L  import Crypto.Hash import Crypto.Internal.ByteArray (ByteArrayAccess)@@ -22,7 +21,7 @@  -- | This is a strict version of and and' :: [Bool] -> Bool-and' l = foldl' (&&!) True l+and' l = L.foldl' (&&!) True l  -- | This is a strict version of &&. (&&!) :: Bool -> Bool -> Bool
+ Crypto/PubKey/MLDSA.hs view
@@ -0,0 +1,656 @@+-- |+-- Module      : Crypto.PubKey.MLDSA+-- License     : BSD-style+-- Maintainer  : Kazu Yamamoto <kazu@iij.ad.jp>+-- Stability   : experimental+-- Portability : unknown+--+-- ML-DSA, the Module-Lattice-Based Digital Signature Algorithm of+-- <https://csrc.nist.gov/pubs/fips/204/final FIPS 204>, in all three+-- parameter sets.+--+-- > (vk, sk) <- generateKeyPair MLDSA65+-- > sig      <- sign sk emptyContext message+-- > verify vk emptyContext message sig+--+-- What 'generateKeyPair' and 'sign' draw their randomness from is the+-- 'Crypto.Random.MonadRandom' instance in use.  Its documentation says what+-- an instance of your own has to be.+--+-- The parameter set is a type, so an ML-DSA-65 key cannot be passed where+-- an ML-DSA-87 one is expected.  The three are fixed by FIPS 204 and the+-- class has no other instances.+--+-- This is pure ML-DSA: the message goes in whole.  The pre-hash variant+-- (HashML-DSA) is a different algorithm with a different domain separator+-- and is not offered here.+{-# LANGUAGE DataKinds #-}+{-# LANGUAGE GeneralizedNewtypeDeriving #-}+{-# LANGUAGE ScopedTypeVariables #-}++module Crypto.PubKey.MLDSA (+    -- * Parameter sets+    MLDSA44 (..),+    MLDSA65 (..),+    MLDSA87 (..),+    MLDSA (verificationKeySize, signingKeySize, signatureSize),++    -- * Keys and signatures+    VerificationKey,+    SigningKey,+    Signature,++    -- * Smart constructors+    verificationKey,+    signingKey,+    signature,++    -- * Generating a key pair+    generateKeyPair,+    generateKeyPairAndSeed,+    keyPairFromSeed,+    toPublic,++    -- * The context string+    Context,+    context,+    emptyContext,++    -- * The message representative+    Mu,+    mu,+    messageRepresentative,++    -- ** A message that does not arrive in one piece+    MuContext,+    muInit,+    muUpdate,+    muUpdates,+    muFinalize,++    -- * Signing and verifying+    sign,+    signWith,+    signDeterministic,+    verify,++    -- * Signing and verifying a message representative+    signExternalMu,+    signExternalMuWith,+    signExternalMuDeterministic,+    verifyExternalMu,++    -- * Sizes+    seedSize,+    signingRandomnessSize,+    maxContextLength,+    muSize,+) where++import Data.Proxy (Proxy (..))+import Foreign.C.Types (CInt (..), CSize (..))+import Foreign.Ptr (Ptr, nullPtr)++import Crypto.Debug (DebugShow (..), debugShowBytes)+import Crypto.Hash (Digest, hash, hashFinalize, hashInit, hashUpdate, hashUpdates)+import qualified Crypto.Hash as Hash (Context)+import Crypto.Hash.Algorithms (SHAKE256 (..))+import Crypto.Error+import Crypto.Internal.ByteArray (+    ByteArrayAccess,+    Bytes,+    ScrubbedBytes,+    withByteArray,+ )+import qualified Crypto.Internal.ByteArray as B+import Crypto.Internal.Compat (unsafeDoIO)+import Crypto.Internal.Imports+import Crypto.Random (MonadRandom, getRandomBytes)++-- | ML-DSA-44.+data MLDSA44 = MLDSA44 deriving (Show, Eq)++-- | ML-DSA-65.+data MLDSA65 = MLDSA65 deriving (Show, Eq)++-- | ML-DSA-87.+data MLDSA87 = MLDSA87 deriving (Show, Eq)++-- | The three parameter sets of FIPS 204.+--+-- Named for the algorithm rather than \"DSA\", which is a different one that+-- crypton also has, in "Crypto.PubKey.DSA".  It is the three sets FIPS 204+-- defines, closed, carrying their sizes and the calls into the+-- implementation; only the sizes are exported.+class MLDSA p where+    -- | Size in bytes of a 'VerificationKey' of this parameter set.+    verificationKeySize :: proxy p -> Int++    -- | Size in bytes of a 'SigningKey' of this parameter set.+    signingKeySize :: proxy p -> Int++    -- | Size in bytes of a 'Signature' of this parameter set.+    signatureSize :: proxy p -> Int++    c_keypair :: proxy p -> Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt+    c_sign+        :: proxy p+        -> Ptr Word8 -> Ptr Word8 -> CSize -> Ptr Word8 -> CSize+        -> Ptr Word8 -> Ptr Word8 -> CInt -> IO CInt+    c_verify+        :: proxy p+        -> Ptr Word8 -> Ptr Word8 -> CSize -> Ptr Word8 -> CSize+        -> Ptr Word8 -> CInt -> IO CInt+    c_pkFromSk :: proxy p -> Ptr Word8 -> Ptr Word8 -> IO CInt++-- | A public verification key.+newtype VerificationKey p = VerificationKey Bytes+    deriving (Show, Eq, ByteArrayAccess, NFData)++-- | A private signing key.+newtype SigningKey p = SigningKey ScrubbedBytes+    deriving (Eq, ByteArrayAccess, NFData)++instance Show (SigningKey p) where+    show _ = "SigningKey <redacted>"++instance DebugShow (SigningKey p) where+    debugShow = debugShowBytes "SigningKey"++-- | A signature.+newtype Signature p = Signature Bytes+    deriving (Show, Eq, ByteArrayAccess, NFData)++-- | The context string a signature is bound to, at most+-- 'maxContextLength' bytes.+--+-- FIPS 204 mixes it into what is signed, so a signature made under one+-- context does not verify under another.  Two uses of one key that don't+-- agree on a context string cannot be made to accept each other's+-- signatures.  Use 'emptyContext' where there is nothing to separate -- TLS,+-- for one, signs with an empty context.+newtype Context = Context Bytes+    deriving (Show, Eq, ByteArrayAccess, NFData)++-- | The context string of length zero, which is what to sign under when+-- there is nothing to separate.+--+-- It is a context and not the absence of one: 'sign' always takes one, and+-- what this separates from is every non-empty context there is.+emptyContext :: Context+emptyContext = Context B.empty++-- | Try to build a context string.+context :: ByteArrayAccess ba => ba -> CryptoFailable Context+context bs+    | B.length bs <= maxContextLength =+        CryptoPassed $ Context $ B.copyAndFreeze bs (\_ -> return ())+    | otherwise = CryptoFailed CryptoError_ParameterInvalid++-- | The longest context string FIPS 204 allows, which is 255 bytes because+-- its length is encoded in one byte.+maxContextLength :: Int+maxContextLength = 255++-- | The message representative, @mu@ in FIPS 204: a 64-byte commitment to+-- the verification key, the context string and the message, and the only+-- part of them that signing and verification actually read.+--+-- Signing it directly is the "external mu" interface.  It is for a caller+-- that has the representative without having the message in one piece: a+-- message arriving as a stream, or hashed on another machine, or by a+-- device that holds the key and is handed only this.  TLS does not need it.+newtype Mu = Mu Bytes+    deriving (Show, Eq, ByteArrayAccess, NFData)++-- | Size in bytes of a 'Mu'.+muSize :: Int+muSize = 64++-- | Try to read a message representative.+mu :: ByteArrayAccess ba => ba -> CryptoFailable Mu+mu bs+    | B.length bs == muSize = CryptoPassed $ Mu $ B.copyAndFreeze bs (\_ -> return ())+    | otherwise = CryptoFailed CryptoError_ParameterInvalid++-- | Compute the message representative, for a caller that wants to make it+-- here and sign it later, or sign it elsewhere.+--+-- @'signExternalMuDeterministic' sk ('messageRepresentative' ('toPublic' sk) ctx msg)@+-- and @'signDeterministic' sk ctx msg@ are the same signature.+messageRepresentative+    :: (MLDSA p, ByteArrayAccess msg)+    => VerificationKey p -> Context -> msg -> Mu+messageRepresentative vk ctx msg = muFinalize (muUpdate (muInit vk ctx) msg)++-- | A 'Mu' being computed, with the message going in a piece at a time.+--+-- The name is not 'Context': that is ML-DSA's context string, which this+-- is built from and is not.+newtype MuContext = MuContext (Hash.Context (SHAKE256 512))++-- | Begin a message representative.  The key and the context string are+-- what it is bound to, and they are all that is needed before the message.+--+-- > muFinalize (muUpdates (muInit vk ctx) chunks)+--+-- is 'messageRepresentative' of the chunks joined, so a message too large+-- to hold at once never has to be.+muInit :: MLDSA p => VerificationKey p -> Context -> MuContext+muInit vk ctx =+    -- FIPS 204: tr <- H(pk, 64) at key generation, and mu <- H(tr || M', 64)+    -- when signing, with M' the domain-separated message.  Everything up to+    -- the message itself is absorbed here.+    MuContext $ hashUpdates hashInit [tr, domainPrefix ctx]+  where+    tr = B.convert (shake64 (B.convert vk :: Bytes)) :: Bytes++-- | Absorb a piece of the message.+muUpdate :: ByteArrayAccess msg => MuContext -> msg -> MuContext+muUpdate (MuContext c) msg = MuContext (hashUpdate c msg)++-- | Absorb several pieces, which is 'muUpdate' one after the other.+muUpdates :: ByteArrayAccess msg => MuContext -> [msg] -> MuContext+muUpdates (MuContext c) msgs = MuContext (hashUpdates c msgs)++-- | The message representative of everything absorbed so far.+muFinalize :: MuContext -> Mu+muFinalize (MuContext c) = Mu (B.convert (hashFinalize c :: Digest (SHAKE256 512)))++shake64 :: ByteArrayAccess ba => ba -> Digest (SHAKE256 512)+shake64 = hash++-- | Size in bytes of the seed 'keyPairFromSeed' takes, @xi@ in FIPS 204.+seedSize :: Int+seedSize = 32++-- | Size in bytes of the randomness 'signWith' takes.+signingRandomnessSize :: Int+signingRandomnessSize = 32++-- | Try to read a verification key.  Only the length is checked: a+-- verification key is a packed encoding with no redundancy to test, and one+-- that is not a real key simply verifies nothing.+verificationKey+    :: forall p ba+     . (MLDSA p, ByteArrayAccess ba)+    => ba -> CryptoFailable (VerificationKey p)+verificationKey bs+    | B.length bs == verificationKeySize (Proxy :: Proxy p) =+        CryptoPassed $ VerificationKey $ B.copyAndFreeze bs (\_ -> return ())+    | otherwise = CryptoFailed CryptoError_PublicKeySizeInvalid++-- | Try to read a signing key.+--+-- Beyond the length this runs the validity checks of the implementation:+-- the secret polynomials must have coefficients in range, and the+-- commitment and the public-key hash the key carries must match what is+-- recomputed from the rest of it.  A key that fails has been damaged or was+-- never a key, and signing with it would produce signatures nothing+-- verifies.+signingKey+    :: forall p ba+     . (MLDSA p, ByteArrayAccess ba)+    => ba -> CryptoFailable (SigningKey p)+signingKey bs+    | B.length bs /= signingKeySize p = CryptoFailed CryptoError_SecretKeySizeInvalid+    | otherwise = unsafeDoIO $ do+        (r, _ :: Bytes) <- B.allocRet (verificationKeySize p) $ \ppk ->+            withByteArray bs $ \psk -> c_pkFromSk p ppk psk+        return $+            if r == 0+                then CryptoPassed $ SigningKey $ B.copyAndFreeze bs (\_ -> return ())+                else CryptoFailed CryptoError_SecretKeyStructureInvalid+  where+    p = Proxy :: Proxy p+{-# NOINLINE signingKey #-}++-- | Try to read a signature.  Only the length is checked; whether it is a+-- signature of anything is what 'verify' answers.+signature+    :: forall p ba+     . (MLDSA p, ByteArrayAccess ba)+    => ba -> CryptoFailable (Signature p)+signature bs+    | B.length bs == signatureSize (Proxy :: Proxy p) =+        CryptoPassed $ Signature $ B.copyAndFreeze bs (\_ -> return ())+    | otherwise = CryptoFailed CryptoError_ParameterInvalid++-- | Recover the verification key a signing key was made with.+toPublic :: forall p. MLDSA p => SigningKey p -> VerificationKey p+toPublic sk = VerificationKey $ unsafeDoIO $ do+    (_ :: CInt, pk) <- B.allocRet (verificationKeySize p) $ \ppk ->+        withByteArray sk $ \psk -> c_pkFromSk p ppk psk+    return pk+  where+    p = Proxy :: Proxy p+{-# NOINLINE toPublic #-}++-- | Generate a key pair.+--+-- The seed it is derived from is drawn here and thrown away.  Use+-- 'generateKeyPairAndSeed' where it has to be kept.+generateKeyPair+    :: forall p proxy m+     . (MLDSA p, MonadRandom m)+    => proxy p -> m (VerificationKey p, SigningKey p)+generateKeyPair p = do+    (vk, sk, _) <- generateKeyPairAndSeed p+    return (vk, sk)++-- | Generate a key pair and hand back the seed it was derived from, @xi@+-- in FIPS 204.+--+-- A 'SigningKey' is the expanded key and nothing else, so the seed cannot+-- be recovered from a pair afterwards.  An application that has to write+-- the key out in a form that keeps the seed -- RFC 9881 lets an ML-DSA+-- private key be the seed, the expanded key, or both -- has to generate it+-- here:+--+-- > (vk, sk, seed) <- generateKeyPairAndSeed MLDSA65+--+-- The seed is as secret as the signing key: 'keyPairFromSeed' turns it+-- back into the same pair.+generateKeyPairAndSeed+    :: forall p proxy m+     . (MLDSA p, MonadRandom m)+    => proxy p -> m (VerificationKey p, SigningKey p, ScrubbedBytes)+generateKeyPairAndSeed p = do+    seed <- getRandomBytes seedSize :: m ScrubbedBytes+    case keyPairFromSeed p seed of+        CryptoPassed (vk, sk) -> return (vk, sk, seed)+        CryptoFailed e ->+            error ("Crypto.PubKey.MLDSA.generateKeyPairAndSeed: " ++ show e)++-- | Derive a key pair from a seed, @xi@ in FIPS 204, which must be+-- 'seedSize' bytes.+keyPairFromSeed+    :: forall p proxy ba+     . (MLDSA p, ByteArrayAccess ba)+    => proxy p -> ba -> CryptoFailable (VerificationKey p, SigningKey p)+keyPairFromSeed p seed+    | B.length seed /= seedSize = CryptoFailed CryptoError_SeedSizeInvalid+    | otherwise = unsafeDoIO $ do+        -- Not zeroed, and does not need to be: the C writes the whole+        -- buffer, and on a non-zero return the result is discarded without+        -- being read.  Anything that is *read* before being written has to+        -- use B.zero instead -- see signInternal in Crypto.PubKey.MLDSA.+        sk <- B.alloc (signingKeySize p) (\_ -> return ()) :: IO ScrubbedBytes+        (r, vk) <- B.allocRet (verificationKeySize p) $ \pvk ->+            withByteArray sk $ \psk ->+                withByteArray seed $ \pseed ->+                    c_keypair p pvk psk pseed+        return $+            if r == 0+                then CryptoPassed (VerificationKey vk, SigningKey sk)+                else CryptoFailed CryptoError_ParameterInvalid+{-# NOINLINE keyPairFromSeed #-}++-- | Sign a message.+--+-- This is the hedged signing FIPS 204 recommends: fresh randomness goes in+-- alongside the key and the message, so two signatures of one message+-- differ and a fault in one reveals less.  Verification does not care which+-- of the three entry points made the signature.+sign+    :: forall p m msg+     . (MLDSA p, MonadRandom m, ByteArrayAccess msg)+    => SigningKey p -> Context -> msg -> m (Signature p)+sign sk ctx msg = do+    rnd <- getRandomBytes signingRandomnessSize :: m ScrubbedBytes+    case signWith sk ctx msg rnd of+        CryptoPassed s -> return s+        CryptoFailed e -> error ("Crypto.PubKey.MLDSA.sign: " ++ show e)++-- | Sign with the randomness supplied, which must be+-- 'signingRandomnessSize' bytes.+--+-- For test vectors, and for callers who draw their own randomness.  Ordinary+-- use wants 'sign'.+signWith+    :: forall p msg rnd+     . (MLDSA p, ByteArrayAccess msg, ByteArrayAccess rnd)+    => SigningKey p -> Context -> msg -> rnd -> CryptoFailable (Signature p)+signWith sk ctx msg rnd+    | B.length rnd /= signingRandomnessSize = CryptoFailed CryptoError_SeedSizeInvalid+    | otherwise = signInternal sk ctx msg (Just rnd)++-- | Sign deterministically, as FIPS 204 section 3.4 allows: the randomness+-- is replaced by zeroes, so one key and one message always give one+-- signature.+--+-- This is what test vectors are written against, and what to use where the+-- signature must be reproducible.  It gives up what hedging buys, so where+-- there is a usable random source 'sign' is the better default.+signDeterministic+    :: forall p msg+     . (MLDSA p, ByteArrayAccess msg)+    => SigningKey p -> Context -> msg -> Signature p+signDeterministic sk ctx msg =+    case signInternal sk ctx msg (Nothing :: Maybe Bytes) of+        CryptoPassed s -> s+        CryptoFailed e -> error ("Crypto.PubKey.MLDSA.signDeterministic: " ++ show e)++signInternal+    :: forall p msg rnd+     . (MLDSA p, ByteArrayAccess msg, ByteArrayAccess rnd)+    => SigningKey p -> Context -> msg -> Maybe rnd -> CryptoFailable (Signature p)+signInternal sk ctx msg mrnd = unsafeDoIO $ do+    -- B.zero, not B.alloc with an empty action: alloc hands back whatever+    -- was in the memory.  That made signDeterministic sign with the last+    -- caller's bytes and produce a different signature every time, which the+    -- ACVP vectors caught only once the whole suite ran and the allocator+    -- stopped handing out fresh zeroed pages.+    let zeroes = B.zero signingRandomnessSize :: ScrubbedBytes+        withRnd f = case mrnd of+            Just r -> withByteArray r f+            Nothing -> withByteArray zeroes f+    (r, sig) <- B.allocRet (signatureSize p) $ \psig ->+        withByteArray msg $ \pmsg ->+            withByteArray pre $ \ppre ->+                withRnd $ \prnd ->+                    withByteArray sk $ \psk ->+                        c_sign+                            p+                            psig+                            pmsg+                            (fromIntegral (B.length msg))+                            ppre+                            (fromIntegral (B.length pre))+                            prnd+                            psk+                            0+    return $+        if r == 0+            then CryptoPassed (Signature sig)+            else CryptoFailed CryptoError_ParameterInvalid+  where+    p = Proxy :: Proxy p+    pre = domainPrefix ctx+{-# NOINLINE signInternal #-}++-- | Sign a message representative, drawing the randomness.+--+-- The context string is already inside the representative, which is why+-- this does not take one.+signExternalMu+    :: forall p m+     . (MLDSA p, MonadRandom m)+    => SigningKey p -> Mu -> m (Signature p)+signExternalMu sk m = do+    rnd <- getRandomBytes signingRandomnessSize :: m ScrubbedBytes+    case signExternalMuWith sk m rnd of+        CryptoPassed s -> return s+        CryptoFailed e -> error ("Crypto.PubKey.MLDSA.signExternalMu: " ++ show e)++-- | Sign a message representative with the randomness supplied.+signExternalMuWith+    :: (MLDSA p, ByteArrayAccess rnd)+    => SigningKey p -> Mu -> rnd -> CryptoFailable (Signature p)+signExternalMuWith sk m rnd+    | B.length rnd /= signingRandomnessSize = CryptoFailed CryptoError_SeedSizeInvalid+    | otherwise = signMu sk m (Just rnd)++-- | Sign a message representative deterministically.+signExternalMuDeterministic+    :: MLDSA p => SigningKey p -> Mu -> Signature p+signExternalMuDeterministic sk m =+    case signMu sk m (Nothing :: Maybe Bytes) of+        CryptoPassed s -> s+        CryptoFailed e ->+            error ("Crypto.PubKey.MLDSA.signExternalMuDeterministic: " ++ show e)++-- | Verify a signature of a message representative.+verifyExternalMu+    :: forall p. MLDSA p => VerificationKey p -> Mu -> Signature p -> Bool+verifyExternalMu vk m sig+    | B.length sig /= signatureSize p = False+    | otherwise = unsafeDoIO $+        withByteArray sig $ \psig ->+            withByteArray m $ \pmu ->+                withByteArray vk $ \pvk -> do+                    r <-+                        c_verify+                            p+                            psig+                            pmu+                            (fromIntegral muSize)+                            nullPtr+                            0+                            pvk+                            1+                    return (r == 0)+  where+    p = Proxy :: Proxy p+{-# NOINLINE verifyExternalMu #-}++-- The external-mu entry points are the ordinary ones with the last argument+-- set: the representative goes in where the message would, there is no+-- domain separation prefix to prepend because it is already inside, and the+-- implementation is told so.+signMu+    :: forall p rnd+     . (MLDSA p, ByteArrayAccess rnd)+    => SigningKey p -> Mu -> Maybe rnd -> CryptoFailable (Signature p)+signMu sk m mrnd = unsafeDoIO $ do+    let zeroes = B.zero signingRandomnessSize :: ScrubbedBytes+        withRnd f = case mrnd of+            Just r -> withByteArray r f+            Nothing -> withByteArray zeroes f+    (r, sig) <- B.allocRet (signatureSize p) $ \psig ->+        withByteArray m $ \pmu ->+            withRnd $ \prnd ->+                withByteArray sk $ \psk ->+                    c_sign p psig pmu (fromIntegral muSize) nullPtr 0 prnd psk 1+    return $+        if r == 0+            then CryptoPassed (Signature sig)+            else CryptoFailed CryptoError_ParameterInvalid+  where+    p = Proxy :: Proxy p+{-# NOINLINE signMu #-}++-- | Verify a signature.+--+-- The context must be the one it was signed under; anything else is a+-- rejection, which is what the context is for.+verify+    :: forall p msg+     . (MLDSA p, ByteArrayAccess msg)+    => VerificationKey p -> Context -> msg -> Signature p -> Bool+verify vk ctx msg sig+    | B.length sig /= signatureSize p = False+    | otherwise = unsafeDoIO $+        withByteArray sig $ \psig ->+            withByteArray msg $ \pmsg ->+                withByteArray pre $ \ppre ->+                    withByteArray vk $ \pvk -> do+                        r <-+                            c_verify+                                p+                                psig+                                pmsg+                                (fromIntegral (B.length msg))+                                ppre+                                (fromIntegral (B.length pre))+                                pvk+                                0+                        return (r == 0)+  where+    p = Proxy :: Proxy p+    pre = domainPrefix ctx+{-# NOINLINE verify #-}++-- | The domain separation prefix of FIPS 204 for pure ML-DSA, which is a+-- zero byte, the context's length and the context itself.  It is built here+-- rather than taken from the implementation because it is three bytes of+-- concatenation and doing it here keeps one fewer foreign call.+domainPrefix :: Context -> Bytes+domainPrefix (Context ctx) =+    B.concat [B.pack [0, fromIntegral (B.length ctx)] :: Bytes, B.convert ctx]++instance MLDSA MLDSA44 where+    verificationKeySize _ = 1312+    signingKeySize _ = 2560+    signatureSize _ = 2420+    c_keypair _ = c_mldsa44_keypair+    c_sign _ = c_mldsa44_sign+    c_verify _ = c_mldsa44_verify+    c_pkFromSk _ = c_mldsa44_pk_from_sk++instance MLDSA MLDSA65 where+    verificationKeySize _ = 1952+    signingKeySize _ = 4032+    signatureSize _ = 3309+    c_keypair _ = c_mldsa65_keypair+    c_sign _ = c_mldsa65_sign+    c_verify _ = c_mldsa65_verify+    c_pkFromSk _ = c_mldsa65_pk_from_sk++instance MLDSA MLDSA87 where+    verificationKeySize _ = 2592+    signingKeySize _ = 4896+    signatureSize _ = 4627+    c_keypair _ = c_mldsa87_keypair+    c_sign _ = c_mldsa87_sign+    c_verify _ = c_mldsa87_verify+    c_pkFromSk _ = c_mldsa87_pk_from_sk++foreign import ccall unsafe "crypton_mldsa44_keypair_internal"+    c_mldsa44_keypair :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt+foreign import ccall unsafe "crypton_mldsa44_signature_internal"+    c_mldsa44_sign+        :: Ptr Word8 -> Ptr Word8 -> CSize -> Ptr Word8 -> CSize+        -> Ptr Word8 -> Ptr Word8 -> CInt -> IO CInt+foreign import ccall unsafe "crypton_mldsa44_verify_internal"+    c_mldsa44_verify+        :: Ptr Word8 -> Ptr Word8 -> CSize -> Ptr Word8 -> CSize+        -> Ptr Word8 -> CInt -> IO CInt+foreign import ccall unsafe "crypton_mldsa44_pk_from_sk"+    c_mldsa44_pk_from_sk :: Ptr Word8 -> Ptr Word8 -> IO CInt++foreign import ccall unsafe "crypton_mldsa65_keypair_internal"+    c_mldsa65_keypair :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt+foreign import ccall unsafe "crypton_mldsa65_signature_internal"+    c_mldsa65_sign+        :: Ptr Word8 -> Ptr Word8 -> CSize -> Ptr Word8 -> CSize+        -> Ptr Word8 -> Ptr Word8 -> CInt -> IO CInt+foreign import ccall unsafe "crypton_mldsa65_verify_internal"+    c_mldsa65_verify+        :: Ptr Word8 -> Ptr Word8 -> CSize -> Ptr Word8 -> CSize+        -> Ptr Word8 -> CInt -> IO CInt+foreign import ccall unsafe "crypton_mldsa65_pk_from_sk"+    c_mldsa65_pk_from_sk :: Ptr Word8 -> Ptr Word8 -> IO CInt++foreign import ccall unsafe "crypton_mldsa87_keypair_internal"+    c_mldsa87_keypair :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt+foreign import ccall unsafe "crypton_mldsa87_signature_internal"+    c_mldsa87_sign+        :: Ptr Word8 -> Ptr Word8 -> CSize -> Ptr Word8 -> CSize+        -> Ptr Word8 -> Ptr Word8 -> CInt -> IO CInt+foreign import ccall unsafe "crypton_mldsa87_verify_internal"+    c_mldsa87_verify+        :: Ptr Word8 -> Ptr Word8 -> CSize -> Ptr Word8 -> CSize+        -> Ptr Word8 -> CInt -> IO CInt+foreign import ccall unsafe "crypton_mldsa87_pk_from_sk"+    c_mldsa87_pk_from_sk :: Ptr Word8 -> Ptr Word8 -> IO CInt
+ Crypto/PubKey/MLKEM.hs view
@@ -0,0 +1,448 @@+-- |+-- Module      : Crypto.PubKey.MLKEM+-- License     : BSD-style+-- Maintainer  : Kazu Yamamoto <kazu@iij.ad.jp>+-- Stability   : experimental+-- Portability : unknown+--+-- ML-KEM, the Module-Lattice-Based Key-Encapsulation Mechanism of+-- <https://csrc.nist.gov/pubs/fips/203/final FIPS 203>, in all three+-- parameter sets.+--+-- A key encapsulation mechanism is not a Diffie-Hellman: there is no shared+-- secret to be computed from two key pairs.  One side publishes an+-- 'EncapsulationKey'; the other calls 'encapsulate' on it, which draws a+-- fresh secret and returns it along with a 'Ciphertext' that only the holder+-- of the matching 'DecapsulationKey' can turn back into that secret.+--+-- > (ek, dk)  <- generateKeyPair MLKEM768        -- the receiver+-- > (ct, ss)  <- encapsulate ek                  -- the sender+-- > let ss'   =  decapsulate dk ct               -- the receiver, again+-- > ss == ss'+--+-- What 'generateKeyPair' and 'encapsulate' draw their randomness from is+-- the 'Crypto.Random.MonadRandom' instance in use.  Its documentation says+-- what an instance of your own has to be.+--+-- The parameter set is a type, so an ML-KEM-768 key cannot be passed where+-- an ML-KEM-1024 one is expected.  The three are fixed by FIPS 203 and the+-- class has no other instances.+{-# LANGUAGE GeneralizedNewtypeDeriving #-}+{-# LANGUAGE TypeFamilies #-}+{-# LANGUAGE TypeOperators #-}+{-# LANGUAGE ScopedTypeVariables #-}++module Crypto.PubKey.MLKEM (+    -- * Parameter sets+    MLKEM512 (..),+    MLKEM768 (..),+    MLKEM1024 (..),+    MLKEM (encapsulationKeySize, decapsulationKeySize, ciphertextSize),++    -- * Keys, ciphertexts and shared secrets+    --+    -- | These are the associated types of 'KEM', re-exported so that a+    -- caller of this module alone has them.+    KEM (..),+    SharedSecret (..),++    -- * Smart constructors+    encapsulationKey,+    decapsulationKey,+    ciphertext,++    -- * What ML-KEM has beyond the class+    generateKeyPairAndSeed,+    keyPairFromSeed,++    -- * Sizes+    seedSize,+    encapsulationCoinsSize,+    sharedSecretSize,+) where++import Data.Proxy (Proxy (..))+import Foreign.C.Types (CInt (..))+import Foreign.Ptr (Ptr)++import Crypto.Debug (DebugShow (..), debugShowBytes)+import Crypto.Error+import Crypto.KEM+import Crypto.Internal.ByteArray (+    ByteArrayAccess,+    Bytes,+    ScrubbedBytes,+    withByteArray,+ )+import qualified Crypto.Internal.ByteArray as B+import Crypto.Internal.Compat (unsafeDoIO)+import Crypto.Internal.Imports+import Crypto.Random (MonadRandom, getRandomBytes)++-- | ML-KEM-512.+data MLKEM512 = MLKEM512 deriving (Show, Eq)++-- | ML-KEM-768.  This is the set TLS uses, on its own and as the+-- lattice half of the hybrid groups.+data MLKEM768 = MLKEM768 deriving (Show, Eq)++-- | ML-KEM-1024.+data MLKEM1024 = MLKEM1024 deriving (Show, Eq)++-- | The three parameter sets of FIPS 203.+--+-- This is not an abstract KEM interface and does not try to be: it is the+-- three sets FIPS 203 defines, closed, carrying their sizes and the calls+-- into the implementation.  Only the sizes are exported.  If crypton grows+-- a second KEM and an interface common to both is wanted, that belongs in+-- a module of its own, with this as one of its instances.+class+    ( KEM p+    , EncapsulationKey p ~ MLKEMEncapsulationKey p+    , DecapsulationKey p ~ MLKEMDecapsulationKey p+    , Ciphertext p ~ MLKEMCiphertext p+    , Coins p ~ ScrubbedBytes+    ) =>+    MLKEM p+    where+    -- | Size in bytes of an 'EncapsulationKey' of this parameter set.+    encapsulationKeySize :: proxy p -> Int++    -- | Size in bytes of a 'DecapsulationKey' of this parameter set.+    decapsulationKeySize :: proxy p -> Int++    -- | Size in bytes of a 'Ciphertext' of this parameter set.+    ciphertextSize :: proxy p -> Int++    c_keypair :: proxy p -> Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt+    c_enc :: proxy p -> Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt+    c_dec :: proxy p -> Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt+    c_checkPk :: proxy p -> Ptr Word8 -> IO CInt+    c_checkSk :: proxy p -> Ptr Word8 -> IO CInt++-- | A public encapsulation key, @ek@ in FIPS 203.+newtype MLKEMEncapsulationKey p = MLKEMEncapsulationKey Bytes+    deriving (Show, Eq, ByteArrayAccess, NFData)++-- | A private decapsulation key, @dk@ in FIPS 203.  It embeds the matching+-- encapsulation key, which is why it is the larger of the two.+newtype MLKEMDecapsulationKey p = MLKEMDecapsulationKey ScrubbedBytes+    deriving (Eq, ByteArrayAccess, NFData)++instance Show (MLKEMDecapsulationKey p) where+    show _ = "DecapsulationKey <redacted>"++instance DebugShow (MLKEMDecapsulationKey p) where+    debugShow = debugShowBytes "DecapsulationKey"++-- | The value 'encapsulate' produces and 'decapsulate' consumes.+newtype MLKEMCiphertext p = MLKEMCiphertext Bytes+    deriving (Show, Eq, ByteArrayAccess, NFData)++-- | Size in bytes of the seed 'keyPairFromSeed' takes, which is @d@ and @z@+-- of FIPS 203 one after the other.+seedSize :: Int+seedSize = 64++-- | Size in bytes of the randomness 'encapsulateWith' takes, @m@ in+-- FIPS 203.+encapsulationCoinsSize :: Int+encapsulationCoinsSize = 32++-- | Size in bytes of a 'SharedSecret'.+sharedSecretSize :: Int+sharedSecretSize = 32++-- | Try to read an encapsulation key.+--+-- Beyond the length this runs the check of FIPS 203 section 7.2: the key+-- must be the encoding of coefficients that are all in range, which is to+-- say it must survive a decode and re-encode unchanged.  A key that fails+-- it is not one any honest party produced.+encapsulationKey+    :: forall p ba+     . (MLKEM p, ByteArrayAccess ba)+    => ba -> CryptoFailable (EncapsulationKey p)+encapsulationKey bs+    | B.length bs /= encapsulationKeySize p = CryptoFailed CryptoError_PublicKeySizeInvalid+    | otherwise = unsafeDoIO $ withByteArray bs $ \inp -> do+        r <- c_checkPk p inp+        return $+            if r == 0+                then CryptoPassed $ MLKEMEncapsulationKey $ B.copyAndFreeze bs (\_ -> return ())+                else CryptoFailed CryptoError_PublicKeyStructureInvalid+  where+    p = Proxy :: Proxy p+{-# NOINLINE encapsulationKey #-}++-- | Try to read a decapsulation key.+--+-- Beyond the length this runs the check of FIPS 203 section 7.3: the hash+-- of the encapsulation key the private key embeds must match the copy of+-- that hash it also embeds.  The two disagreeing means the key was not+-- produced as a pair, and decapsulating with it would silently answer with+-- the implicit rejection every time.+decapsulationKey+    :: forall p ba+     . (MLKEM p, ByteArrayAccess ba)+    => ba -> CryptoFailable (DecapsulationKey p)+decapsulationKey bs+    | B.length bs /= decapsulationKeySize p = CryptoFailed CryptoError_SecretKeySizeInvalid+    | otherwise = unsafeDoIO $ withByteArray bs $ \inp -> do+        r <- c_checkSk p inp+        return $+            if r == 0+                then CryptoPassed $ MLKEMDecapsulationKey $ B.copyAndFreeze bs (\_ -> return ())+                else CryptoFailed CryptoError_SecretKeyStructureInvalid+  where+    p = Proxy :: Proxy p+{-# NOINLINE decapsulationKey #-}++-- | Try to read a ciphertext.  Only the length is checked; every string of+-- the right length is a ciphertext that 'decapsulate' will answer.+ciphertext+    :: forall p ba+     . (MLKEM p, ByteArrayAccess ba)+    => ba -> CryptoFailable (Ciphertext p)+ciphertext bs+    | B.length bs == ciphertextSize (Proxy :: Proxy p) =+        CryptoPassed $ MLKEMCiphertext $ B.copyAndFreeze bs (\_ -> return ())+    | otherwise = CryptoFailed CryptoError_PointSizeInvalid++-- | Generate a key pair.+--+-- The seed it is derived from is drawn here and thrown away.  Use+-- 'generateKeyPairAndSeed' where it has to be kept.+mlkemGenerateKeyPair+    :: forall p proxy m+     . (MLKEM p, MonadRandom m)+    => proxy p -> m (MLKEMEncapsulationKey p, DecapsulationKey p)+mlkemGenerateKeyPair p = do+    (ek, dk, _) <- generateKeyPairAndSeed p+    return (ek, dk)++-- | Generate a key pair and hand back the seed it was derived from, @d@+-- and @z@ of FIPS 203 one after the other.+--+-- A 'DecapsulationKey' is the expanded key and nothing else, so the seed+-- cannot be recovered from a pair afterwards.  An application that has to+-- write the key out in a form that keeps the seed has to generate it here:+--+-- > (ek, dk, seed) <- generateKeyPairAndSeed MLKEM768+--+-- The seed is as secret as the decapsulation key: 'keyPairFromSeed' turns+-- it back into the same pair.+generateKeyPairAndSeed+    :: forall p proxy m+     . (MLKEM p, MonadRandom m)+    => proxy p+    -> m (MLKEMEncapsulationKey p, DecapsulationKey p, ScrubbedBytes)+generateKeyPairAndSeed p = do+    seed <- getRandomBytes seedSize :: m ScrubbedBytes+    case keyPairFromSeed p seed of+        CryptoPassed (ek, dk) -> return (ek, dk, seed)+        CryptoFailed e ->+            error ("Crypto.PubKey.MLKEM.generateKeyPairAndSeed: " ++ show e)++-- | Derive a key pair from a seed, which is @d@ and @z@ of FIPS 203 one+-- after the other and must be 'seedSize' bytes.+--+-- This is the entry point to use when the seed comes from somewhere+-- particular -- a test vector, or a store that keeps seeds rather than+-- expanded keys.  For an ordinary key, 'generateKeyPair' draws the seed+-- itself.+keyPairFromSeed+    :: forall p proxy ba+     . (MLKEM p, ByteArrayAccess ba)+    => proxy p+    -> ba+    -> CryptoFailable (MLKEMEncapsulationKey p, DecapsulationKey p)+keyPairFromSeed p seed+    | B.length seed /= seedSize = CryptoFailed CryptoError_SeedSizeInvalid+    | otherwise = unsafeDoIO $ do+        -- Not zeroed, and does not need to be: the C writes the whole+        -- buffer, and on a non-zero return the result is discarded without+        -- being read.  Anything that is *read* before being written has to+        -- use B.zero instead -- see signInternal in Crypto.PubKey.MLDSA.+        dk <- B.alloc (decapsulationKeySize p) (\_ -> return ()) :: IO ScrubbedBytes+        (r, ek) <- B.allocRet (encapsulationKeySize p) $ \pek ->+            withByteArray dk $ \pdk ->+                withByteArray seed $ \pseed ->+                    c_keypair p pek pdk pseed+        return $+            if r == 0+                then CryptoPassed (MLKEMEncapsulationKey ek, MLKEMDecapsulationKey dk)+                else CryptoFailed CryptoError_ParameterInvalid+{-# NOINLINE keyPairFromSeed #-}++-- | Encapsulate against a public key, drawing the randomness.+mlkemEncapsulate+    :: forall p m+     . (MLKEM p, MonadRandom m)+    => MLKEMEncapsulationKey p+    -> m (CryptoFailable (Ciphertext p, SharedSecret))+mlkemEncapsulate ek = do+    coins <- getRandomBytes encapsulationCoinsSize :: m ScrubbedBytes+    return (mlkemEncapsulateWith ek coins)++-- The class's 'encapsulateWith' for ML-KEM, where the coins are @m@ of+-- FIPS 203 and must be 'encapsulationCoinsSize' bytes.+mlkemEncapsulateWith+    :: forall p+     . MLKEM p+    => MLKEMEncapsulationKey p+    -> ScrubbedBytes+    -> CryptoFailable (Ciphertext p, SharedSecret)+mlkemEncapsulateWith ek coins+    | B.length coins /= encapsulationCoinsSize = CryptoFailed CryptoError_SeedSizeInvalid+    | otherwise = unsafeDoIO $ do+        ss <- B.alloc sharedSecretSize (\_ -> return ()) :: IO ScrubbedBytes+        (r, ct) <- B.allocRet (ciphertextSize p) $ \pct ->+            withByteArray ss $ \pss ->+                withByteArray ek $ \pek ->+                    withByteArray coins $ \pcoins ->+                        c_enc p pct pss pek pcoins+        return $+            if r == 0+                then CryptoPassed (MLKEMCiphertext ct, SharedSecret ss)+                else CryptoFailed CryptoError_ParameterInvalid+  where+    p = Proxy :: Proxy p+{-# NOINLINE mlkemEncapsulateWith #-}++-- | Recover the shared secret from a ciphertext.+--+-- A ciphertext that was not produced by encapsulating against the matching+-- key is not an error.  ML-KEM rejects implicitly: it yields a secret+-- derived from the private key and the ciphertext, and the caller cannot+-- tell that case from the other one, which is the point -- telling them+-- apart is what a chosen-ciphertext attack needs.  A ciphertext that does+-- not belong here shows up later, as the two sides failing to agree on+-- anything.+--+-- The checks FIPS 203 does require are at the point where bytes become a+-- value of these types, which is where they can be reported:+--+-- * The ciphertext type check of section 7.3 is its length, and+--   'ciphertext' is the only way to build a 'Ciphertext' from bytes.  There+--   is nothing else to check: a ciphertext's coefficients are compressed to+--   fewer than twelve bits, so every bit pattern decodes to a value in+--   range.+-- * The hash check of section 7.3 is on the decapsulation key, and+--   'decapsulationKey' runs it; a key from 'generateKeyPair' or+--   'keyPairFromSeed' satisfies it by construction.+--+-- So the result is 'CryptoPassed' for every key and ciphertext this module+-- can produce.  It is 'CryptoFailable' rather than a bare 'SharedSecret'+-- because the implementation checks the key again on its way through, and+-- what it finds is better reported than turned into an exception.+mlkemDecapsulate+    :: forall p+     . MLKEM p+    => DecapsulationKey p -> Ciphertext p -> CryptoFailable SharedSecret+mlkemDecapsulate dk ct = unsafeDoIO $ do+    (r, ss) <- B.allocRet sharedSecretSize $ \pss ->+        withByteArray ct $ \pct ->+            withByteArray dk $ \pdk ->+                c_dec (Proxy :: Proxy p) pss pct pdk+    return $+        if r == (0 :: CInt)+            then CryptoPassed (SharedSecret ss)+            else CryptoFailed CryptoError_SecretKeyStructureInvalid+{-# NOINLINE mlkemDecapsulate #-}++-- The class's view of the three sets.  The operations are the ones above;+-- only the shape of the arguments differs, because the class takes the+-- mechanism as a proxy.+instance KEM MLKEM512 where+    type EncapsulationKey MLKEM512 = MLKEMEncapsulationKey MLKEM512+    type DecapsulationKey MLKEM512 = MLKEMDecapsulationKey MLKEM512+    type Ciphertext MLKEM512 = MLKEMCiphertext MLKEM512+    type Coins MLKEM512 = ScrubbedBytes+    generateKeyPair = mlkemGenerateKeyPair+    encapsulate _ = mlkemEncapsulate+    encapsulateWith _ = mlkemEncapsulateWith+    decapsulate _ = mlkemDecapsulate++instance KEM MLKEM768 where+    type EncapsulationKey MLKEM768 = MLKEMEncapsulationKey MLKEM768+    type DecapsulationKey MLKEM768 = MLKEMDecapsulationKey MLKEM768+    type Ciphertext MLKEM768 = MLKEMCiphertext MLKEM768+    type Coins MLKEM768 = ScrubbedBytes+    generateKeyPair = mlkemGenerateKeyPair+    encapsulate _ = mlkemEncapsulate+    encapsulateWith _ = mlkemEncapsulateWith+    decapsulate _ = mlkemDecapsulate++instance KEM MLKEM1024 where+    type EncapsulationKey MLKEM1024 = MLKEMEncapsulationKey MLKEM1024+    type DecapsulationKey MLKEM1024 = MLKEMDecapsulationKey MLKEM1024+    type Ciphertext MLKEM1024 = MLKEMCiphertext MLKEM1024+    type Coins MLKEM1024 = ScrubbedBytes+    generateKeyPair = mlkemGenerateKeyPair+    encapsulate _ = mlkemEncapsulate+    encapsulateWith _ = mlkemEncapsulateWith+    decapsulate _ = mlkemDecapsulate++instance MLKEM MLKEM512 where+    encapsulationKeySize _ = 800+    decapsulationKeySize _ = 1632+    ciphertextSize _ = 768+    c_keypair _ = c_mlkem512_keypair+    c_enc _ = c_mlkem512_enc+    c_dec _ = c_mlkem512_dec+    c_checkPk _ = c_mlkem512_check_pk+    c_checkSk _ = c_mlkem512_check_sk++instance MLKEM MLKEM768 where+    encapsulationKeySize _ = 1184+    decapsulationKeySize _ = 2400+    ciphertextSize _ = 1088+    c_keypair _ = c_mlkem768_keypair+    c_enc _ = c_mlkem768_enc+    c_dec _ = c_mlkem768_dec+    c_checkPk _ = c_mlkem768_check_pk+    c_checkSk _ = c_mlkem768_check_sk++instance MLKEM MLKEM1024 where+    encapsulationKeySize _ = 1568+    decapsulationKeySize _ = 3168+    ciphertextSize _ = 1568+    c_keypair _ = c_mlkem1024_keypair+    c_enc _ = c_mlkem1024_enc+    c_dec _ = c_mlkem1024_dec+    c_checkPk _ = c_mlkem1024_check_pk+    c_checkSk _ = c_mlkem1024_check_sk++foreign import ccall unsafe "crypton_mlkem512_keypair_derand"+    c_mlkem512_keypair :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt+foreign import ccall unsafe "crypton_mlkem512_enc_derand"+    c_mlkem512_enc :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt+foreign import ccall unsafe "crypton_mlkem512_dec"+    c_mlkem512_dec :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt+foreign import ccall unsafe "crypton_mlkem512_check_pk"+    c_mlkem512_check_pk :: Ptr Word8 -> IO CInt+foreign import ccall unsafe "crypton_mlkem512_check_sk"+    c_mlkem512_check_sk :: Ptr Word8 -> IO CInt++foreign import ccall unsafe "crypton_mlkem768_keypair_derand"+    c_mlkem768_keypair :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt+foreign import ccall unsafe "crypton_mlkem768_enc_derand"+    c_mlkem768_enc :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt+foreign import ccall unsafe "crypton_mlkem768_dec"+    c_mlkem768_dec :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt+foreign import ccall unsafe "crypton_mlkem768_check_pk"+    c_mlkem768_check_pk :: Ptr Word8 -> IO CInt+foreign import ccall unsafe "crypton_mlkem768_check_sk"+    c_mlkem768_check_sk :: Ptr Word8 -> IO CInt++foreign import ccall unsafe "crypton_mlkem1024_keypair_derand"+    c_mlkem1024_keypair :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt+foreign import ccall unsafe "crypton_mlkem1024_enc_derand"+    c_mlkem1024_enc :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt+foreign import ccall unsafe "crypton_mlkem1024_dec"+    c_mlkem1024_dec :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt+foreign import ccall unsafe "crypton_mlkem1024_check_pk"+    c_mlkem1024_check_pk :: Ptr Word8 -> IO CInt+foreign import ccall unsafe "crypton_mlkem1024_check_sk"+    c_mlkem1024_check_sk :: Ptr Word8 -> IO CInt
Crypto/PubKey/MaskGenFunction.hs view
@@ -1,4 +1,4 @@-{-# LANGUAGE BangPatterns #-}+{-# LANGUAGE ScopedTypeVariables #-}  -- | -- Module      : Crypto.PubKey.MaskGenFunction@@ -11,10 +11,17 @@     mgf1, ) where -import Crypto.Hash-import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess, Bytes)+import Control.Monad (forM_)+import Crypto.Hash (Context, Digest)+import Crypto.Hash.IO (HashAlgorithm (..))+import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess) import qualified Crypto.Internal.ByteArray as B-import Crypto.Number.Serialize (i2ospOf_)+import Data.Bits (shiftR)+import Data.Word (Word8)+import Foreign.Marshal.Alloc (allocaBytes)+import Foreign.Marshal.Utils (copyBytes)+import Foreign.Ptr (Ptr, castPtr, plusPtr)+import Foreign.Storable (pokeByteOff)  -- | Represent a mask generation algorithm type MaskGenAlgorithm seed output =@@ -26,20 +33,33 @@  -- | Mask generation algorithm MGF1 mgf1-    :: (ByteArrayAccess seed, ByteArray output, HashAlgorithm hashAlg)+    :: forall seed output hashAlg+     . (ByteArrayAccess seed, ByteArray output, HashAlgorithm hashAlg)     => hashAlg     -> seed     -> Int     -> output-mgf1 hashAlg seed len =-    let !seededCtx = hashUpdate (hashInitWith hashAlg) seed-     in B.take len $-            B.concat $-                map (hashCounter seededCtx) [0 .. fromIntegral (maxCounter - 1)]+mgf1 hashAlg seed len+    | len <= 0 = B.empty+    | otherwise = B.allocAndFreeze len $ \out ->+        -- the seeded context is kept, and copied for each block rather than+        -- rebuilt; each block's digest is written straight into the output+        allocaBytes ctxSize $ \(seeded :: Ptr (Context hashAlg)) ->+            allocaBytes ctxSize $ \(ctx :: Ptr (Context hashAlg)) ->+                allocaBytes digestLen $ \(dig :: Ptr (Digest hashAlg)) ->+                    allocaBytes 4 $ \(cnt :: Ptr Word8) -> do+                        hashInternalInit seeded+                        B.withByteArray seed $ \p ->+                            hashInternalUpdate seeded p (fromIntegral (B.length seed))+                        forM_ [0 .. maxCounter - 1] $ \i -> do+                            copyBytes ctx seeded ctxSize+                            forM_ [0 .. 3] $ \k ->+                                pokeByteOff cnt k (fromIntegral (i `shiftR` (8 * (3 - k))) :: Word8)+                            hashInternalUpdate ctx cnt 4+                            hashInternalFinalize ctx dig+                            let off = i * digestLen+                            copyBytes (out `plusPtr` off) (castPtr dig) (min digestLen (len - off))   where+    ctxSize = hashInternalContextSize hashAlg     digestLen = hashDigestSize hashAlg-    (chunks, left) = len `divMod` digestLen-    maxCounter = if left > 0 then chunks + 1 else chunks--    hashCounter :: HashAlgorithm a => Context a -> Integer -> Digest a-    hashCounter ctx counter = hashFinalize $ hashUpdate ctx (i2ospOf_ 4 counter :: Bytes)+    maxCounter = (len + digestLen - 1) `div` digestLen
+ Crypto/PubKey/RSA/Blinded.hs view
@@ -0,0 +1,29 @@+-- |+-- Module      : Crypto.PubKey.RSA.Blinded+-- License     : BSD-style+-- Stability   : experimental+-- Portability : Good+--+-- An RSA private key that keeps its blinding between operations.+--+-- Blinding hides the private-key operation's input from timing: it is+-- multiplied by @r^e@ for a random @r@ first, and the result by @r^-1@+-- after.  Drawing a new @r@ costs a modular inverse and an exponentiation,+-- about a tenth of an RSA-2048 signature.  A t'BlindedKey' draws one and+-- squares the pair for each use after it, which is what OpenSSL does, and+-- draws a new one every 32 uses, from the 'Crypto.Random.Types.MonadRandom'+-- the operation runs in.+--+-- The key is threaded through the operations as a hash context is: each+-- function in "Crypto.PubKey.RSA.PKCS15", "Crypto.PubKey.RSA.PSS" and+-- "Crypto.PubKey.RSA.OAEP" whose name ends in @Blinded@ returns the key for+-- the next one.  The @Safer@ functions there keep a t'BlindedKey' per key+-- themselves, for the last few keys used, so they get the same saving+-- without one being passed.+module Crypto.PubKey.RSA.Blinded (+    BlindedKey,+    blindedKey,+    blindedPrivateKey,+) where++import Crypto.PubKey.RSA.Blinded.Internal
+ Crypto/PubKey/RSA/Blinded/Internal.hs view
@@ -0,0 +1,136 @@+{-# LANGUAGE BangPatterns #-}+{-# LANGUAGE ScopedTypeVariables #-}+{-# OPTIONS_GHC -fno-cse -fno-full-laziness #-}++-- |+-- Module      : Crypto.PubKey.RSA.Blinded.Internal+-- License     : BSD-style+-- Stability   : experimental+-- Portability : Good+--+-- A private key with a blinding pair that is reused by squaring, as OpenSSL+-- does, rather than generated afresh for every operation.  The key is a value+-- threaded through the operations, as a hash 'Crypto.Hash.Context' is.+module Crypto.PubKey.RSA.Blinded.Internal (+    BlindedKey (..),+    blindedKey,+    dpBlinded,+    Draw,+    drawFor,+    withBlindingFor,+    dpWithPair,+) where++import Data.IORef+import System.IO.Unsafe (unsafePerformIO)++import Crypto.Internal.ByteArray (ByteArray, ScrubbedBytes)+import Crypto.Number.Generate (generateMax)+import Crypto.Number.ModArithmetic (expFast, inverseCoprimes)+import Crypto.Number.Serialize (i2ospOf_, os2ip)+import Crypto.PubKey.RSA.Prim (dp)+import Crypto.PubKey.RSA.Types+import Crypto.Random.Types++-- | A private key together with the blinding its next operation uses.+--+-- Each operation multiplies the input by @r^e@ and the output by @r^-1@, and+-- returns the key with both squared for the next one, so a pair costs two+-- multiplications after the first.  A new @r@ is drawn every 32 operations.+-- Use the key an operation returns: handing the same one to two operations+-- gives both the same pair.+data BlindedKey = BlindedKey+    { blindedPrivateKey :: PrivateKey+    -- ^ the key it was made from+    , blindedPair :: Pair+    }++-- | @r^e@ and @r^-1@ modulo n, and how many more uses they have after this.+data Pair = Pair !Int !Integer !Integer | Spent++-- | Uses of one @r@ before another is drawn, OpenSSL's number.+refresh :: Int+refresh = 32++-- | A t'BlindedKey' for a private key.  The first pair is drawn when it is+-- first used.+blindedKey :: PrivateKey -> BlindedKey+blindedKey pk = BlindedKey pk Spent++-- | The random numbers a new pair is made from: drawing them is cheap, and+-- making the pair, an inverse and an exponentiation, is done only if needed.+data Draw = Draw !Integer !Integer++-- | Draw for a key, as 'Crypto.PubKey.RSA.generateBlinder' does.+drawFor :: forall m. MonadRandom m => PrivateKey -> m Draw+drawFor pk = do+    r <- generateMax (private_n pk)+    u <- os2ip <$> (getRandomBytes 16 :: m ScrubbedBytes)+    return (Draw r u)++-- | @r^e@ and @r^-1@.  The inverse is taken of @r*u@ and multiplied by @u@,+-- so the extended Euclidean algorithm never sees @r@, as in+-- 'Crypto.PubKey.RSA.generateBlinder'.+makePair :: PrivateKey -> Draw -> (Integer, Integer)+makePair pk (Draw r u) =+    (expFast r (private_e pk) n, (inverseCoprimes ((r * u) `mod` n) n * u) `mod` n)+  where+    n = private_n pk++-- | The pair for this operation, and the key holding the next one.  The+-- draw is used only when a new pair is due.+takePair :: Draw -> BlindedKey -> ((Integer, Integer), BlindedKey)+takePair d (BlindedKey pk pair) = case pair of+    Pair k a ai -> ((a, ai), BlindedKey pk (next k a ai))+    Spent -> case makePair pk d of+        (a, ai) -> ((a, ai), BlindedKey pk (next refresh a ai))+  where+    n = private_n pk+    next k a ai+        | k > 1 = Pair (k - 1) ((a * a) `mod` n) ((ai * ai) `mod` n)+        | otherwise = Spent++-- | The decryption primitive, blinded with a given pair.+dpWithPair :: ByteArray ba => (Integer, Integer) -> PrivateKey -> ba -> ba+dpWithPair (a, ai) pk c = i2ospOf_ k ((ai * os2ip m) `mod` n)+  where+    n = private_n pk+    k = private_size pk+    m = dp Nothing pk (i2ospOf_ k ((a * os2ip c) `mod` n) `asTypeOf` c)++-- | The decryption primitive, blinded with the key's pair, and the key for+-- the next operation.  Randomness is drawn only when a new pair is due.+dpBlinded+    :: (MonadRandom m, ByteArray ba) => BlindedKey -> ba -> m (ba, BlindedKey)+dpBlinded bk c = do+    d <- case blindedPair bk of+        Pair {} -> return (Draw 0 0)+        Spent -> drawFor (blindedPrivateKey bk)+    let (pair, bk') = takePair d bk+    return (dpWithPair pair (blindedPrivateKey bk) c, bk')++-- | Run an operation with the pair kept for a key, for the functions that+-- take a bare t'PrivateKey'.  One t'BlindedKey' is kept per public key, for+-- the last few keys used; taking its pair and leaving the next is atomic, so+-- no two operations get the same pair.  The caller draws every time, since+-- whether a new pair is due is known only in here.+withBlindingFor :: PrivateKey -> Draw -> ((Integer, Integer) -> a) -> a+withBlindingFor pk d f = unsafePerformIO $ do+    pair <- atomicModifyIORef' table step+    return (f pair)+  where+    pub = private_pub pk+    step ks = case break ((== pub) . private_pub . blindedPrivateKey) ks of+        (before, bk : after) -> found bk (before ++ after)+        _ -> found (blindedKey pk) ks+    found bk rest = case takePair d bk of+        (pair@(!_, !_), bk') -> (take tableSize (bk' : rest), pair)+{-# NOINLINE withBlindingFor #-}++-- | How many keys 'withBlindingFor' remembers.+tableSize :: Int+tableSize = 4++table :: IORef [BlindedKey]+table = unsafePerformIO (newIORef [])+{-# NOINLINE table #-}
Crypto/PubKey/RSA/OAEP.hs view
@@ -18,22 +18,22 @@     -- * OAEP decryption     decrypt,     decryptSafer,+    decryptBlinded, ) where  import Crypto.Hash import Crypto.Number.Serialize (os2ip) import Crypto.PubKey.Internal (and') import Crypto.PubKey.MaskGenFunction-import Crypto.PubKey.RSA (generateBlinder)+import Crypto.PubKey.RSA.Blinded.Internal import Crypto.PubKey.RSA.Prim import Crypto.PubKey.RSA.Types import Crypto.Random.Types import Data.Bits (complement, shiftR, xor, (.&.), (.|.)) import Data.ByteString (ByteString) import qualified Data.ByteString as B-import Data.List (foldl')+import qualified Data.List as L import Data.Word (Word32)-import Prelude hiding (foldl')  import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess) import qualified Crypto.Internal.ByteArray as B (constEq, convert)@@ -156,7 +156,7 @@     -- is zero; all of them are looked at either way     oneIndex =         fst $-            foldl'+            L.foldl'                 step                 (fromIntegral (B.length db1) :: Word32, 1 :: Word32)                 (zip [0 ..] (B.unpack db1))@@ -181,10 +181,9 @@  -- | Decrypt a ciphertext using OAEP ----- When the signature is not in a context where an attacker could gain--- information from the timing of the operation, the blinder can be set to None.------ If unsure always set a blinder or use decryptSafer+-- The blinder is optional and 'Nothing' is accepted, but see t'Blinder' for+-- what it covers and when leaving it out is a decision rather than a default.+-- 'decryptSafer' generates one for you. -- -- Following RFC 8017, the ciphertext is rejected unless it is exactly as long -- as the modulus (section 7.1.2, step 1) and its integer representative is@@ -214,7 +213,8 @@     k = private_size pk     hashLen = hashDigestSize (oaepHash oaep) --- | Decrypt a ciphertext using OAEP and by automatically generating a blinder.+-- | Decrypt a ciphertext using OAEP, blinded with the pair kept for this key;+-- see "Crypto.PubKey.RSA.Blinded". decryptSafer     :: (HashAlgorithm hash, MonadRandom m)     => OAEPParams hash ByteString ByteString@@ -224,6 +224,34 @@     -> ByteString     -- ^ Cipher text     -> m (Either Error ByteString)-decryptSafer oaep pk cipher = do-    blinder <- generateBlinder (private_n pk)-    return (decrypt (Just blinder) oaep pk cipher)+decryptSafer oaep pk cipher+    | B.length cipher /= k = return (Left MessageSizeIncorrect)+    | os2ip cipher >= private_n pk = return (Left MessageSizeIncorrect)+    | k < 2 * hashLen + 2 = return (Left InvalidParameters)+    | otherwise = do+        d <- drawFor pk+        return $ withBlindingFor pk d $ \pair -> unpad oaep k (dpWithPair pair pk cipher)+  where+    k = private_size pk+    hashLen = hashDigestSize (oaepHash oaep)++-- | 'decrypt' with the blinding a t'BlindedKey' keeps, and the key for the+-- next operation.+decryptBlinded+    :: (HashAlgorithm hash, MonadRandom m)+    => OAEPParams hash ByteString ByteString+    -> BlindedKey+    -> ByteString+    -- ^ Cipher text+    -> m (Either Error ByteString, BlindedKey)+decryptBlinded oaep bk cipher+    | B.length cipher /= k = return (Left MessageSizeIncorrect, bk)+    | os2ip cipher >= private_n pk = return (Left MessageSizeIncorrect, bk)+    | k < 2 * hashLen + 2 = return (Left InvalidParameters, bk)+    | otherwise = do+        (m, bk') <- dpBlinded bk cipher+        return (unpad oaep k m, bk')+  where+    pk = blindedPrivateKey bk+    k = private_size pk+    hashLen = hashDigestSize (oaepHash oaep)
Crypto/PubKey/RSA/PKCS15.hs view
@@ -15,10 +15,21 @@     decryptSafer,     sign,     signSafer,+    signDigest,+    signSaferDigest,+    signDigestInfo,+    signSaferDigestInfo, +    -- * Private key operations with a kept blinding+    decryptBlinded,+    signDigestBlinded,+    signDigestInfoBlinded,+     -- * Public key operations     encrypt,     verify,+    verifyDigest,+    verifyDigestInfo,      -- * Hash ASN1 description     HashAlgorithmASN1,@@ -27,7 +38,7 @@ import Crypto.Hash import Crypto.Number.Serialize (os2ip) import Crypto.PubKey.Internal (and')-import Crypto.PubKey.RSA (generateBlinder)+import Crypto.PubKey.RSA.Blinded.Internal import Crypto.PubKey.RSA.Prim import Crypto.PubKey.RSA.Types import Crypto.Random.Types@@ -38,8 +49,7 @@  import Crypto.Internal.ByteArray (ByteArray, Bytes) import qualified Crypto.Internal.ByteArray as B-import Data.List (foldl')-import Prelude hiding (foldl')+import qualified Data.List as L  -- | A specialized class for hash algorithm that can product -- a ASN1 wrapped description the algorithm plus the content@@ -405,10 +415,10 @@     :: ByteArray signature => Int -> signature -> Either Error signature padSignature klen signature     | klen < siglen + 11 = Left SignatureTooLong-    | otherwise = Right (B.pack padding `B.append` signature)+    | otherwise =+        Right (B.concat [B.pack [0, 1], B.replicate (klen - siglen - 3) 0xff, B.pack [0], signature])   where     siglen = B.length signature-    padding = 0 : 1 : (replicate (klen - siglen - 3) 0xff ++ [0])  -- | Try to remove a standard PKCS1.5 encryption padding. --@@ -431,7 +441,7 @@      -- index of the first zero octet in ps0m, counted from the start of packed,     -- or len when there is none; every octet is looked at either way-    zeroIndex = fst $ foldl' step (fromIntegral len :: Word32, 1 :: Word32) indexed+    zeroIndex = fst $ L.foldl' step (fromIntegral len :: Word32, 1 :: Word32) indexed     indexed = zip [2 ..] (B.unpack ps0m)     step (idx, unseen) (i, b) = (select found i idx, unseen .&. complement found)       where@@ -453,10 +463,9 @@  -- | decrypt message using the private key. ----- When the decryption is not in a context where an attacker could gain--- information from the timing of the operation, the blinder can be set to None.------ If unsure always set a blinder or use decryptSafer+-- The blinder is optional and 'Nothing' is accepted, but see t'Blinder' for+-- what it covers and when leaving it out is a decision rather than a default.+-- 'decryptSafer' generates one for you. -- -- The message is returned un-padded. --@@ -482,7 +491,8 @@     -- "convert" must be apply to "c".     | otherwise = unpad $ dp blinder pk $ B.convert c --- | decrypt message using the private key and by automatically generating a blinder.+-- | decrypt message using the private key, blinded with the pair kept for+-- this key; see "Crypto.PubKey.RSA.Blinded". decryptSafer     :: (MonadRandom m, ByteArray ba)     => PrivateKey@@ -490,10 +500,30 @@     -> ByteString     -- ^ cipher text     -> m (Either Error ba)-decryptSafer pk b = do-    blinder <- generateBlinder (private_n pk)-    return (decrypt (Just blinder) pk b)+decryptSafer pk b+    | B.length b /= private_size pk = return (Left MessageSizeIncorrect)+    | os2ip b >= private_n pk = return (Left MessageSizeIncorrect)+    | otherwise = do+        d <- drawFor pk+        return $ withBlindingFor pk d $ \pair -> unpad (dpWithPair pair pk (B.convert b)) +-- | 'decrypt' with the blinding a t'BlindedKey' keeps, and the key for the+-- next operation.+decryptBlinded+    :: (MonadRandom m, ByteArray ba)+    => BlindedKey+    -> ByteString+    -- ^ cipher text+    -> m (Either Error ba, BlindedKey)+decryptBlinded bk c+    | B.length c /= private_size pk = return (Left MessageSizeIncorrect, bk)+    | os2ip c >= private_n pk = return (Left MessageSizeIncorrect, bk)+    | otherwise = do+        (m, bk') <- dpBlinded bk (B.convert c)+        return (unpad m, bk')+  where+    pk = blindedPrivateKey bk+ -- | encrypt a bytestring using the public key. -- -- The message needs to be smaller than the key size - 11.@@ -508,10 +538,15 @@  -- | sign message using private key, a hash and its ASN1 description ----- When the signature is not in a context where an attacker could gain--- information from the timing of the operation, the blinder can be set to None.+-- __Deprecated.__  Use 'signDigest', or 'signDigestInfo' where this would+-- have been given 'Nothing'.  The @Maybe hashAlg@ argument says both \"hash+-- it with this\" and \"do not hash it\", and down the first path the value+-- inside the 'Just' is never read: it is there to fix the type, which+-- leaves a caller that has only a type with nothing to pass. ----- If unsure always set a blinder or use signSafer+-- The blinder is optional and 'Nothing' is accepted, but see t'Blinder' for+-- what it covers and when leaving it out is a decision rather than a default.+-- 'signSafer' generates one for you. sign     :: HashAlgorithmASN1 hashAlg     => Maybe Blinder@@ -524,8 +559,15 @@     -- ^ message to sign     -> Either Error ByteString sign blinder hashDescr pk m = dp blinder pk `fmap` makeSignature hashDescr (private_size pk) m+{-# DEPRECATED+    sign+    "Use signDigest, or signDigestInfo when the message is already a DigestInfo"+    #-} --- | sign message using the private key and by automatically generating a blinder.+-- | sign message using the private key, blinded with the pair kept for this+-- key; see "Crypto.PubKey.RSA.Blinded".+--+-- __Deprecated.__  Use 'signSaferDigest', or 'signSaferDigestInfo'. signSafer     :: (HashAlgorithmASN1 hashAlg, MonadRandom m)     => Maybe hashAlg@@ -536,11 +578,18 @@     -- ^ message to sign     -> m (Either Error ByteString) signSafer hashAlg pk m = do-    blinder <- generateBlinder (private_n pk)-    return (sign (Just blinder) hashAlg pk m)+    d <- drawFor pk+    return $ signWithBlindingFor pk d (makeSignature hashAlg (private_size pk) m)+{-# DEPRECATED+    signSafer+    "Use signSaferDigest, or signSaferDigestInfo when the message is already a DigestInfo"+    #-}  -- | verify message with the signed message --+-- __Deprecated.__  Use 'verifyDigest', or 'verifyDigestInfo' where this+-- would have been given 'Nothing'.+-- -- Following RFC 8017, the signature is rejected unless it is exactly as long -- as the modulus (section 8.2.2, step 1) and its integer representative is -- below the modulus (section 5.2.2, step 1).  Verification works by@@ -557,13 +606,170 @@     -> ByteString     -- ^ Signature     -> Bool-verify hashAlg pk m sm+verify hashAlg pk m sm =+    verifyEncoded pk (makeSignature hashAlg (public_size pk) m) sm+{-# DEPRECATED+    verify+    "Use verifyDigest, or verifyDigestInfo when the message is already a DigestInfo"+    #-}++-- | The two checks of RFC 8017 and the comparison, shared by the three+-- verification entry points.  The expected encoding is a thunk and is+-- forced only once the checks have passed, as it was when this was written+-- out inside 'verify'.+verifyEncoded+    :: PublicKey+    -> Either Error ByteString+    -- ^ the encoding a signature of this message would have+    -> ByteString+    -- ^ signature+    -> Bool+verifyEncoded pk expected sm     | B.length sm /= public_size pk = False     | os2ip sm >= public_n pk = False     | otherwise =-        case makeSignature hashAlg (public_size pk) m of+        case expected of             Left _ -> False-            Right s -> s == (ep pk sm)+            Right s -> s == ep pk sm++-- | Sign a digest.+--+-- The digest's type says which algorithm made it, so this needs nothing+-- else to name one: the ASN.1 DigestInfo prefix comes from the+-- 'HashAlgorithmASN1' instance that type selects.  'sign' takes @Maybe+-- hashAlg@ and never reads the value inside the @Just@ -- it is there only+-- to fix the type -- which is no burden when the caller has a value and+-- leaves nothing to pass when the caller is itself polymorphic in the+-- algorithm.+--+-- > signDigest blinder key (hashWith SHA256 message)+--+-- The blinder is optional and 'Nothing' is accepted, but see t'Blinder' for+-- what it covers and when leaving it out is a decision rather than a+-- default.  'signSaferDigest' generates one for you.+signDigest+    :: HashAlgorithmASN1 hashAlg+    => Maybe Blinder+    -- ^ optional blinder+    -> PrivateKey+    -- ^ private key+    -> Digest hashAlg+    -- ^ digest of the message to sign+    -> Either Error ByteString+signDigest blinder pk digest =+    dp blinder pk `fmap` padSignature (private_size pk) (hashDigestASN1 digest)++-- | 'signDigest' blinded with the pair kept for this key, as 'signSafer' is+-- to 'sign'; see "Crypto.PubKey.RSA.Blinded".+signSaferDigest+    :: (HashAlgorithmASN1 hashAlg, MonadRandom m)+    => PrivateKey+    -- ^ private key+    -> Digest hashAlg+    -- ^ digest of the message to sign+    -> m (Either Error ByteString)+signSaferDigest pk digest = do+    d <- drawFor pk+    return $+        signWithBlindingFor pk d (padSignature (private_size pk) (hashDigestASN1 digest))++-- | 'signDigest' with the blinding a t'BlindedKey' keeps, and the key for the+-- next operation.+signDigestBlinded+    :: (HashAlgorithmASN1 hashAlg, MonadRandom m)+    => BlindedKey+    -> Digest hashAlg+    -- ^ digest of the message to sign+    -> m (Either Error ByteString, BlindedKey)+signDigestBlinded bk digest =+    signBlindedEncoded bk $+        padSignature (private_size (blindedPrivateKey bk)) (hashDigestASN1 digest)++-- | Sign something that is already a DigestInfo, the ASN.1 structure+-- naming a hash algorithm and carrying a digest under it.+--+-- This is what @'sign' blinder 'Nothing'@ does.  It needs no+-- 'HashAlgorithmASN1' constraint, because nothing here hashes or encodes:+-- the caller has done both.  @'sign' blinder 'Nothing'@ carries the+-- constraint anyway, and since the type variable then appears nowhere else+-- the caller has to name an algorithm that is never used --+-- @'sign' blinder ('Nothing' :: 'Maybe' 'Crypto.Hash.SHA256')@ -- to say+-- which one it is not using.+signDigestInfo+    :: Maybe Blinder+    -- ^ optional blinder+    -> PrivateKey+    -- ^ private key+    -> ByteString+    -- ^ a DigestInfo, encoded+    -> Either Error ByteString+signDigestInfo blinder pk di =+    dp blinder pk `fmap` padSignature (private_size pk) di++-- | 'signDigestInfo' blinded with the pair kept for this key.+signSaferDigestInfo+    :: MonadRandom m+    => PrivateKey+    -- ^ private key+    -> ByteString+    -- ^ a DigestInfo, encoded+    -> m (Either Error ByteString)+signSaferDigestInfo pk di = do+    d <- drawFor pk+    return $ signWithBlindingFor pk d (padSignature (private_size pk) di)++-- | 'signDigestInfo' with the blinding a t'BlindedKey' keeps, and the key for+-- the next operation.+signDigestInfoBlinded+    :: MonadRandom m+    => BlindedKey+    -> ByteString+    -- ^ a DigestInfo, encoded+    -> m (Either Error ByteString, BlindedKey)+signDigestInfoBlinded bk di =+    signBlindedEncoded bk $ padSignature (private_size (blindedPrivateKey bk)) di++-- | The private-key operation on an encoded signature, if there is one; the+-- key is used up only if there is.+signBlindedEncoded+    :: MonadRandom m+    => BlindedKey+    -> Either Error ByteString+    -> m (Either Error ByteString, BlindedKey)+signBlindedEncoded bk (Left err) = return (Left err, bk)+signBlindedEncoded bk (Right em) = do+    (s, bk') <- dpBlinded bk em+    return (Right s, bk')++-- | The same, with the pair the @Safer@ functions keep for this key.+signWithBlindingFor+    :: PrivateKey -> Draw -> Either Error ByteString -> Either Error ByteString+signWithBlindingFor _ _ (Left err) = Left err+signWithBlindingFor pk d (Right em) = Right (withBlindingFor pk d (\pair -> dpWithPair pair pk em))++-- | Verify a signature over a digest.  The checks are 'verify's.+verifyDigest+    :: HashAlgorithmASN1 hashAlg+    => PublicKey+    -> Digest hashAlg+    -- ^ digest of the message+    -> ByteString+    -- ^ signature+    -> Bool+verifyDigest pk digest sm =+    verifyEncoded pk (padSignature (public_size pk) (hashDigestASN1 digest)) sm++-- | Verify a signature over something that is already a DigestInfo, which+-- is what @'verify' 'Nothing'@ does.+verifyDigestInfo+    :: PublicKey+    -> ByteString+    -- ^ a DigestInfo, encoded+    -> ByteString+    -- ^ signature+    -> Bool+verifyDigestInfo pk di sm =+    verifyEncoded pk (padSignature (public_size pk) di) sm  -- | make signature digest, used in 'sign' and 'verify' makeSignature
Crypto/PubKey/RSA/PSS.hs view
@@ -16,6 +16,8 @@     signDigest,     signSafer,     signDigestSafer,+    signBlinded,+    signDigestBlinded,     verify,     verifyDigest, ) where@@ -24,15 +26,15 @@ import Crypto.Number.Basic (numBits) import Crypto.Number.Serialize (os2ip) import Crypto.PubKey.MaskGenFunction-import Crypto.PubKey.RSA (generateBlinder)+import Crypto.PubKey.RSA.Blinded.Internal import Crypto.PubKey.RSA.Prim import Crypto.PubKey.RSA.Types import Crypto.Random.Types-import Data.Bits (shiftR, xor, (.&.))+import Data.Bits (complement, shiftR, (.&.)) import Data.Word  import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess)-import qualified Crypto.Internal.ByteArray as B (convert, eq)+import qualified Crypto.Internal.ByteArray as B (bxor, convert, eq)  import Data.ByteString (ByteString) import qualified Data.ByteString as B@@ -69,6 +71,9 @@ -- | Sign using the PSS parameters and the salt explicitely passed as parameters. -- -- the function ignore SaltLength from the PSS Parameters+--+-- See t'Blinder' for what the optional blinder covers and when leaving it out+-- is a decision rather than a default.  'signSafer' generates one for you. signDigestWithSalt     :: HashAlgorithm hash     => ByteString@@ -82,9 +87,20 @@     -> Digest hash     -- ^ Message digest     -> Either Error ByteString-signDigestWithSalt salt blinder params pk digest+signDigestWithSalt salt blinder params pk digest =+    dp blinder pk <$> encodeDigest salt params pk digest++-- | The encoded message the private-key operation is applied to.+encodeDigest+    :: HashAlgorithm hash+    => ByteString+    -> PSSParams hash ByteString ByteString+    -> PrivateKey+    -> Digest hash+    -> Either Error ByteString+encodeDigest salt params pk digest     | emLen < hashLen + saltLen + 2 = Left InvalidParameters-    | otherwise = Right $ dp blinder pk em+    | otherwise = Right em   where     k = private_size pk     emLen = if emTruncate pubBits then k - 1 else k@@ -97,12 +113,15 @@     h = B.convert $ hashWith (pssHash params) m'     db = B.concat [B.replicate (dbLen - saltLen - 1) 0, B.singleton 1, salt]     dbmask = pssMaskGenAlg params h dbLen-    maskedDB = B.pack $ normalizeToKeySize pubBits $ B.zipWith xor db dbmask+    maskedDB = normalizeToKeySize pubBits (B.bxor db dbmask :: ByteString)     em = B.concat [maskedDB, h, B.singleton (pssTrailerField params)]  -- | Sign using the PSS parameters and the salt explicitely passed as parameters. -- -- the function ignore SaltLength from the PSS Parameters+--+-- See t'Blinder' for what the optional blinder covers and when leaving it out+-- is a decision rather than a default.  'signSafer' generates one for you. signWithSalt     :: HashAlgorithm hash     => ByteString@@ -121,6 +140,9 @@     mHash = hashWith (pssHash params) m  -- | Sign using the PSS Parameters+--+-- See t'Blinder' for what the optional blinder covers and when leaving it out+-- is a decision rather than a default.  'signSafer' generates one for you. sign     :: (HashAlgorithm hash, MonadRandom m)     => Maybe Blinder@@ -137,6 +159,9 @@     return (signWithSalt salt blinder params pk m)  -- | Sign using the PSS Parameters+--+-- See t'Blinder' for what the optional blinder covers and when leaving it out+-- is a decision rather than a default.  'signSafer' generates one for you. signDigest     :: (HashAlgorithm hash, MonadRandom m)     => Maybe Blinder@@ -152,7 +177,8 @@     salt <- getRandomBytes (pssSaltLength params)     return (signDigestWithSalt salt blinder params pk digest) --- | Sign using the PSS Parameters and an automatically generated blinder.+-- | Sign using the PSS Parameters, blinded with the pair kept for this key;+-- see "Crypto.PubKey.RSA.Blinded". signSafer     :: (HashAlgorithm hash, MonadRandom m)     => PSSParams hash ByteString ByteString@@ -162,11 +188,10 @@     -> ByteString     -- ^ message to sign     -> m (Either Error ByteString)-signSafer params pk m = do-    blinder <- generateBlinder (private_n pk)-    sign (Just blinder) params pk m+signSafer params pk m = signDigestSafer params pk (hashWith (pssHash params) m) --- | Sign using the PSS Parameters and an automatically generated blinder.+-- | Sign using the PSS Parameters, blinded with the pair kept for this key;+-- see "Crypto.PubKey.RSA.Blinded". signDigestSafer     :: (HashAlgorithm hash, MonadRandom m)     => PSSParams hash ByteString ByteString@@ -177,9 +202,40 @@     -- ^ message digst     -> m (Either Error ByteString) signDigestSafer params pk digest = do-    blinder <- generateBlinder (private_n pk)-    signDigest (Just blinder) params pk digest+    salt <- getRandomBytes (pssSaltLength params)+    d <- drawFor pk+    return $ case encodeDigest salt params pk digest of+        Left err -> Left err+        Right em -> Right (withBlindingFor pk d (\pair -> dpWithPair pair pk em)) +-- | 'signSafer' with the blinding a t'BlindedKey' keeps, and the key for the+-- next operation.+signBlinded+    :: (HashAlgorithm hash, MonadRandom m)+    => PSSParams hash ByteString ByteString+    -> BlindedKey+    -> ByteString+    -- ^ message to sign+    -> m (Either Error ByteString, BlindedKey)+signBlinded params bk m = signDigestBlinded params bk (hashWith (pssHash params) m)++-- | 'signDigestSafer' with the blinding a t'BlindedKey' keeps, and the key+-- for the next operation.+signDigestBlinded+    :: (HashAlgorithm hash, MonadRandom m)+    => PSSParams hash ByteString ByteString+    -> BlindedKey+    -> Digest hash+    -- ^ message digest+    -> m (Either Error ByteString, BlindedKey)+signDigestBlinded params bk digest = do+    salt <- getRandomBytes (pssSaltLength params)+    case encodeDigest salt params (blindedPrivateKey bk) digest of+        Left err -> return (Left err, bk)+        Right em -> do+            (s, bk') <- dpBlinded bk em+            return (Right s, bk')+ -- | Verify a signature using the PSS Parameters verify     :: HashAlgorithm hash@@ -221,6 +277,7 @@     | B.length s /= k = False     | os2ip s >= public_n pk = False     | B.any (/= 0) pre = False+    | B.any (\x -> x .&. topBits /= 0) (B.take 1 maskedDB) = False     | B.last em /= pssTrailerField params = False     | B.any (/= 0) ps0 = False     | b1 /= B.singleton 1 = False@@ -234,12 +291,19 @@     emLen = if emTruncate pubBits then k - 1 else k     dbLen = emLen - hashLen - 1     pubBits = numBits (public_n pk)+    -- RFC 8017 9.1.2 step 6: the leftmost 8*emLen - emBits bits of the+    -- leftmost octet of maskedDB have to be zero already.  Step 9 clears+    -- them in DB, which is what normalizeToKeySize does below, and clearing+    -- is not checking: without this an encoding with the top bit set -- one+    -- the standard calls inconsistent -- verifies as though it were sound,+    -- because the bit that made it wrong is thrown away before it is read.+    topBits = complement (normalizeMask pubBits)     -- unmarshall fields     (pre, em) = B.splitAt (k - emLen) (ep pk s) -- drop 0..1 byte     maskedDB = B.take dbLen em     h = B.take hashLen $ B.drop (B.length maskedDB) em     dbmask = pssMaskGenAlg params h dbLen-    db = B.pack $ normalizeToKeySize pubBits $ B.zipWith xor maskedDB dbmask+    db = normalizeToKeySize pubBits (B.bxor maskedDB dbmask :: ByteString)     (ps0, z) = B.break (== 1) db     (b1, salt) = B.splitAt 1 z     m' = B.concat [B.replicate 8 0, mHash, salt]@@ -249,9 +313,15 @@ emTruncate :: Int -> Bool emTruncate bits = ((bits - 1) .&. 0x7) == 0 -normalizeToKeySize :: Int -> [Word8] -> [Word8]-normalizeToKeySize _ [] = [] -- very unlikely-normalizeToKeySize bits (x : xs) = x .&. mask : xs+normalizeToKeySize :: Int -> ByteString -> ByteString+normalizeToKeySize bits x = case B.uncons x of+    Nothing -> x -- very unlikely+    Just (h, t) -> B.cons (h .&. normalizeMask bits) t++-- | The bits of the leftmost octet that belong to the encoding: the low+-- @emBits `mod` 8@ of them, or all eight when that is zero.  Its complement+-- is the bits RFC 8017 requires to be zero.+normalizeMask :: Int -> Word8+normalizeMask bits = if sh > 0 then 0xff `shiftR` (8 - sh) else 0xff   where-    mask = if sh > 0 then 0xff `shiftR` (8 - sh) else 0xff     sh = (bits - 1) .&. 0x7
Crypto/PubKey/RSA/Prim.hs view
@@ -1,3 +1,5 @@+{-# LANGUAGE BangPatterns #-}+ -- | -- Module      : Crypto.PubKey.RSA.Prim -- License     : BSD-style@@ -12,9 +14,19 @@     ep, ) where +import Data.Memory.PtrMethods (memSet)+import Data.Word (Word32, Word8)+import Foreign.C.Types (CInt (..))+import Foreign.Marshal.Alloc (allocaBytes)+import Foreign.Ptr (Ptr, plusPtr)+ import Crypto.Internal.ByteArray (ByteArray)+import Crypto.Internal.Compat (unsafeDoIO)+import Crypto.Number.Basic (numBytes) import Crypto.Number.ModArithmetic (expFast, expSafe)+import Crypto.Number.ModArithmetic.Pub (expPubBytes) import Crypto.Number.Serialize (i2ospOf_, os2ip)+import qualified Crypto.Number.Serialize.Internal as Internal import Crypto.PubKey.RSA.Types  {- dpSlow computes the decrypted message not using any precomputed cache value.@@ -29,22 +41,78 @@ dpFast (Blinder r rm1) pk c =     i2ospOf_         (private_size pk)-        (multiplication rm1 (m2 + h * (private_q pk)) (private_n pk))+        (multiplication rm1 (crtChecked pk iC (m2 + h * (private_q pk))) (private_n pk))   where     re = expFast r (public_e $ private_pub pk) (private_n pk)     iC = multiplication re (os2ip c) (private_n pk)-    m1 = expSafe iC (private_dP pk) (private_p pk)-    m2 = expSafe iC (private_dQ pk) (private_q pk)+    (m1, m2) = crtHalves iC pk     h = ((private_qinv pk) * (m1 - m2)) `mod` (private_p pk)  dpFastNoBlinder :: ByteArray ba => PrivateKey -> ba -> ba-dpFastNoBlinder pk c = i2ospOf_ (private_size pk) (m2 + h * (private_q pk))+dpFastNoBlinder pk c = i2ospOf_ (private_size pk) (crtChecked pk iC (m2 + h * (private_q pk)))   where     iC = os2ip c-    m1 = expSafe iC (private_dP pk) (private_p pk)-    m2 = expSafe iC (private_dQ pk) (private_q pk)+    (m1, m2) = crtHalves iC pk     h = ((private_qinv pk) * (m1 - m2)) `mod` (private_p pk) +-- | The CRT answer m for c, raised back to e before it is let out: a fault+-- in one half gives an m whose gcd with n is the other prime (Boneh, DeMillo+-- and Lipton).  On a mismatch it is computed again without CRT, as OpenSSL does.+crtChecked :: PrivateKey -> Integer -> Integer -> Integer+crtChecked pk c m+    | expFast m (public_e (private_pub pk)) n == c `mod` n = m+    | private_d pk > 0 = expSafe c (private_d pk) n+    | otherwise = m+  where+    n = private_n pk++-- | c^dP mod p and c^dQ mod q, the halves of the private-key operation: both+-- at once where the C can (AVX-512 IFMA, and the sizes of RSA-2048, -3072 and+-- -4096), one after the other where it declines.+crtHalves :: Integer -> PrivateKey -> (Integer, Integer)+crtHalves c pk = case expSafeX2 (c `mod` p) dP p (c `mod` q) dQ q of+    Just r -> r+    Nothing -> (expSafe c dP p, expSafe c dQ q)+  where+    p = private_p pk+    q = private_q pk+    dP = private_dP pk+    dQ = private_dQ pk++-- | Two secure exponentiations in one call.  The bases are reduced already+-- and the moduli are the same length; both exponents go in at that length,+-- so the length the C sees is the modulus', as 'expSafe' pads to whole words.+expSafeX2 :: Integer -> Integer -> Integer -> Integer -> Integer -> Integer -> Maybe (Integer, Integer)+expSafeX2 b1 e1 m1 b2 e2 m2+    | numBytes m2 /= len || not (odd m1 && odd m2) = Nothing+    | e1 < 0 || e2 < 0 || numBytes e1 > len || numBytes e2 > len = Nothing+    | otherwise = unsafeDoIO $+        allocaBytes (8 * len) $ \start -> case take 8 (iterate (`plusPtr` len) start) of+            [o1, pb1, pe1, pm1, o2, pb2, pe2, pm2] -> do+                mapM_+                    (\(v, ptr) -> Internal.i2ospOf v ptr len)+                    [(b1, pb1), (e1, pe1), (m1, pm1), (b2, pb2), (e2, pe2), (m2, pm2)]+                r <- c_powm_sec_x2 o1 pb1 l pe1 l pm1 l o2 pb2 l pe2 l pm2 l+                -- the exponents are the caller's secrets+                memSet pe1 0 len+                memSet pe2 0 len+                if r /= 0+                    then return Nothing+                    else do+                        !v1 <- Internal.os2ip o1 len+                        !v2 <- Internal.os2ip o2 len+                        return (Just (v1, v2))+            _ -> return Nothing+  where+    !len = numBytes m1+    l = fromIntegral len :: Word32++foreign import ccall safe "crypton_powm_sec_x2"+    c_powm_sec_x2+        :: Ptr Word8 -> Ptr Word8 -> Word32 -> Ptr Word8 -> Word32 -> Ptr Word8 -> Word32+        -> Ptr Word8 -> Ptr Word8 -> Word32 -> Ptr Word8 -> Word32 -> Ptr Word8 -> Word32+        -> IO CInt+ -- | Compute the RSA decrypt primitive. -- if the p and q numbers are available, then dpFast is used -- otherwise, we use dpSlow which only need d and n.@@ -56,7 +124,9 @@  -- | Compute the RSA encrypt primitive ep :: ByteArray ba => PublicKey -> ba -> ba-ep pk m = i2ospOf_ (public_size pk) $ expFast (os2ip m) (public_e pk) (public_n pk)+ep pk m = case expPubBytes (public_e pk) (public_n pk) m of+    Just r -> r+    Nothing -> i2ospOf_ (public_size pk) $ expFast (os2ip m) (public_e pk) (public_n pk)  -- | multiply 2 integers in Zm only performing the modulo operation if necessary multiplication :: Integer -> Integer -> Integer -> Integer
Crypto/PubKey/RSA/Types.hs view
@@ -27,8 +27,32 @@  import GHC.Generics --- | Blinder which is used to obfuscate the timing--- of the decryption primitive (used by decryption and signing).+-- | A blinder, which keeps the timing of the private key operation from+-- saying anything about the number it was given.+--+-- The private exponent is not what is at risk.  'Crypto.Number.ModArithmetic.expSafe',+-- which the exponentiation goes through, keeps the /value/ of an exponent out+-- of the work it does.+--+-- What a blinder covers is the other side.  Without one, the operation runs+-- on the ciphertext as it arrived, so how long it takes depends on a number+-- an attacker may have chosen and can vary -- which is what a remote timing+-- attack on RSA needs.  With one, the input is multiplied by a random value+-- first and that value divided out afterwards, so the timing carries nothing+-- an attacker can steer.+--+-- Every private key operation here takes a @'Maybe' t'Blinder'@.  The+-- @Safer@ form of each -- 'Crypto.PubKey.RSA.PKCS15.decryptSafer',+-- 'Crypto.PubKey.RSA.PKCS15.signSafer' and their kind -- generates one and is+-- the one to reach for.  Pass 'Nothing' only where the input is not attacker+-- controlled and you have decided that it is not.+--+-- A blinder costs one more exponentiation, by the public exponent, which is+-- the cheap direction: measured on an Apple M4, PKCS#1 v1.5 signing goes from+-- about 601 to about 620 microseconds.+--+-- Use a blinder once.  'Crypto.PubKey.RSA.generateBlinder' makes a fresh one;+-- carrying one across operations is not what it is for. data Blinder = Blinder !Integer !Integer     deriving (Show, Eq)
Crypto/PubKey/Rabin/Basic.hs view
@@ -25,6 +25,38 @@ -- -- Around all of that is 'Integer' arithmetic, whose cost follows the size of -- the numbers; see "Crypto.PubKey.DSA" for that note at more length.+--+-- == The hash algorithm is passed as a value, and that is not going to change+--+-- 'sign', 'signWith' and 'verify' take a @hash@ argument whose value they+-- never read.  Every 'Crypto.Hash.HashAlgorithm' instance is a nullary+-- constructor, and the algorithm comes from the type; the value is there to+-- carry the type and nothing else.  A caller holding a value loses nothing+-- by it, and a caller that is itself polymorphic in the algorithm has no+-- value to pass.+--+-- Elsewhere in crypton that is answered by taking a+-- 'Crypto.Hash.Digest' instead: the digest carries the algorithm in its+-- type and the bytes in its value, so nothing is passed only to name a+-- type.  "Crypto.PubKey.RSA.PKCS15", "Crypto.PubKey.DSA",+-- "Crypto.PubKey.ElGamal", "Crypto.PubKey.Rabin.RW" and+-- "Crypto.PubKey.Rabin.Modified" all do that.+--+-- This module cannot.  What is hashed here is not the message:+--+-- > h = os2ip $ hashWith hashAlg $ B.append padding m+--+-- and the padding is not the caller's either.  'sign' searches for one,+-- drawing eight bytes at a time until the first octet is non-zero and the+-- Jacobi symbols of the hash modulo each private prime are both 1.  Which+-- bytes get hashed is therefore decided inside the signing operation, after+-- the caller has handed over the message, so there is no digest for a+-- caller to compute in advance.+--+-- A proxy argument would work where a digest cannot.  It is deliberately+-- not added: it would be the one exception to a rule the rest of the+-- library now follows, and this module has no callers asking for it.  See+-- the survey in <https://github.com/kazu-yamamoto/crypton/issues/304>. module Crypto.PubKey.Rabin.Basic (     PublicKey (..),     PrivateKey (..),
Crypto/PubKey/Rabin/Modified.hs view
@@ -17,7 +17,9 @@     PrivateKey (..),     generate,     sign,+    signDigest,     verify,+    verifyDigest, ) where  import Crypto.Debug (DebugShow (..))@@ -109,10 +111,25 @@     -> ByteString     -- ^ message to sign     -> Either Error Integer-sign pk hashAlg m =+sign pk hashAlg m = signDigest pk (hashWith hashAlg m)++-- | Sign a digest using the private key.+--+-- The digest's type says which algorithm made it, so this needs nothing else+-- to name one.  'sign' takes a @hash@ value and never reads it -- it is there+-- to fix the type -- which leaves a caller that is itself polymorphic in the+-- algorithm with nothing to pass.+signDigest+    :: HashAlgorithm hash+    => PrivateKey+    -- ^ private key+    -> Digest hash+    -- ^ digest of the message to sign+    -> Either Error Integer+signDigest pk digest =     let d = private_d pk         n = public_n $ private_pub pk-        h = os2ip $ hashWith hashAlg m+        h = os2ip digest         limit = (n - 6) `div` 16      in if h > limit             then Left MessageTooLong@@ -135,14 +152,27 @@     -> Integer     -- ^ signature     -> Bool-verify pk hashAlg m s+verify pk hashAlg m s = verifyDigest pk (hashWith hashAlg m) s++-- | Verify a signature over a digest.  See 'signDigest' for why a digest+-- rather than a @hash@ value.+verifyDigest+    :: HashAlgorithm hash+    => PublicKey+    -- ^ public key+    -> Digest hash+    -- ^ digest of the message+    -> Integer+    -- ^ signature+    -> Bool+verifyDigest pk digest s     -- squaring works modulo n, so s + n and -s would verify wherever s does     | s < 0 || s >= n = False     | otherwise = go   where     n = public_n pk     go =-        let h = os2ip $ hashWith hashAlg m+        let h = os2ip digest             s' = expSafe s 2 n             s'' = case s' `mod` 8 of                 6 -> s'
Crypto/PubKey/Rabin/OAEP.hs view
@@ -17,9 +17,8 @@ import Data.Bits (complement, shiftR, xor, (.&.), (.|.)) import Data.ByteString (ByteString) import qualified Data.ByteString as B-import Data.List (foldl')+import qualified Data.List as L import Data.Word (Word32)-import Prelude hiding (foldl')  import Crypto.Hash import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess)@@ -127,7 +126,7 @@     -- is zero; all of them are looked at either way     oneIndex =         fst $-            foldl'+            L.foldl'                 step                 (fromIntegral (B.length db1) :: Word32, 1 :: Word32)                 (zip [0 ..] (B.unpack db1))
Crypto/PubKey/Rabin/RW.hs view
@@ -21,7 +21,9 @@     encryptWithSeed,     decrypt,     sign,+    signDigest,     verify,+    verifyDigest, ) where  import Crypto.Debug (DebugShow (..))@@ -181,11 +183,26 @@     -> ByteString     -- ^ message to sign     -> Either Error Integer-sign pk hashAlg m =+sign pk hashAlg m = signDigest pk (hashWith hashAlg m)++-- | Sign a digest using the private key.+--+-- The digest's type says which algorithm made it, so this needs nothing else+-- to name one.  'sign' takes a @hash@ value and never reads it -- it is there+-- to fix the type -- which leaves a caller that is itself polymorphic in the+-- algorithm with nothing to pass.+signDigest+    :: HashAlgorithm hash+    => PrivateKey+    -- ^ private key+    -> Digest hash+    -- ^ digest of the message to sign+    -> Either Error Integer+signDigest pk digest =     let d = private_d pk         n = public_n $ private_pub pk      in do-            m' <- ep1 n $ os2ip $ hashWith hashAlg m+            m' <- ep1 n $ os2ip digest             return $ dp1 d n m'  -- | Verify signature using hash algorithm and public key.@@ -200,11 +217,24 @@     -> Integer     -- ^ signature     -> Bool-verify pk hashAlg m s+verify pk hashAlg m s = verifyDigest pk (hashWith hashAlg m) s++-- | Verify a signature over a digest.  See 'signDigest' for why a digest+-- rather than a @hash@ value.+verifyDigest+    :: HashAlgorithm hash+    => PublicKey+    -- ^ public key+    -> Digest hash+    -- ^ digest of the message+    -> Integer+    -- ^ signature+    -> Bool+verifyDigest pk digest s     -- squaring works modulo n, so s + n and -s would verify wherever s does     | s < 0 || s >= n = False     | otherwise =-        let h = os2ip $ hashWith hashAlg m+        let h = os2ip digest             h' = dp2 n $ ep2 n s          in h' == h   where
Crypto/Random.hs view
@@ -7,35 +7,159 @@ -- Maintainer  : Vincent Hanquez <vincent@snarc.org> -- Stability   : stable -- Portability : good+--+-- Random bytes, drawn either from the system or from a generator whose+-- seed you hold.+--+-- == Drawing from the system+--+-- 'getRandomBytes' in 'IO' is the ordinary way to get bytes nobody can+-- predict.  The length is in bytes and the type is any 'ByteArray', so the+-- result is usually pinned down by where it goes:+--+-- > import Crypto.Random (getRandomBytes)+-- > import Data.ByteString (ByteString)+-- >+-- > nonce <- getRandomBytes 12 :: IO ByteString+--+-- Everything in this library that needs randomness takes it the same way,+-- through a @MonadRandom m =>@ constraint, so running it in 'IO' is the+-- whole of choosing this source:+--+-- > import qualified Crypto.PubKey.RSA as RSA+-- >+-- > (pub, priv) <- RSA.generate 256 0x10001+--+-- Where the operating system offers @getrandom(2)@ or @getentropy(3)@ the+-- bytes come from a ChaCha20 generator belonging to the operating system+-- thread the call runs on, seeded from the system and reseeded as it goes;+-- see "Crypto.Random.SysDRG" for what that is and what it is not.  Where+-- it does not -- Windows, for now -- they come from the system on every+-- call, as they always did.  Either way the caller writes the same line.+--+-- Any Haskell thread may draw, and none has to say which generator it+-- wants:+--+-- > import Control.Concurrent (forkIO)+-- >+-- > mapM_ (\_ -> forkIO (getRandomBytes 32 >>= use)) [1 .. 64 :: Int]+--+-- The generator belongs to the operating system thread that happens to be+-- carrying the Haskell thread when the call is made, and is found through+-- that thread's own storage rather than by anything the caller passes.  A+-- @forkIO@ thread moves between capabilities, so two draws from one Haskell+-- thread may be answered by two generators; they are seeded independently+-- of each other, so it makes no difference which one answers.+--+-- == Drawing from a generator you hold+--+-- A generator of your own is the way to draw many times from one seed.+-- The system is asked once, when the generator is made, and not again.+-- That is how @tls@ does a connection: 'seedNew' when it opens, and+-- everything the handshake needs afterwards from the generator, whose+-- state the connection carries from one draw to the next.+--+-- > import Crypto.Random (ChaChaDRG, drgNewSeed, seedNew, withDRG)+-- >+-- > data Connection = Connection { connRNG :: ChaChaDRG }+-- >+-- > newConnection :: IO Connection+-- > newConnection = do+-- >     seed <- seedNew                   -- the only draw from the system+-- >     return (Connection (drgNewSeed seed))+-- >+-- > -- every later draw advances the connection's own generator+-- > connRandom :: Int -> Connection -> (ByteString, Connection)+-- > connRandom n conn =+-- >     let (bytes, rng') = withDRG (connRNG conn) (getRandomBytes n)+-- >      in (bytes, conn{connRNG = rng'})+--+-- Inside 'withDRG' the same 'getRandomBytes' resolves to the instance for+-- 'MonadPseudoRandom' rather than the one for 'IO', so it touches neither+-- the system nor the per-thread generator: it advances the generator it was+-- given and hands it back.  Keep that generator and the bytes are+-- reproducible from the seed, which is what makes a test repeatable -- and+-- what makes a generator unfit for keys unless its seed came from the+-- system.+--+-- 'drgNew' is 'seedNew' and 'drgNewSeed' in one step; 'drgNewTest' takes+-- four numbers instead of a seed, for a test that must give the same answer+-- twice.+--+-- == Drawing from a monad stacked on IO+--+-- There are two instances, one for 'IO' and one for 'MonadPseudoRandom',+-- and none for the transformers, so a @ReaderT env IO@ or a @StateT s IO@+-- reaches the system through 'liftIO':+--+-- > import Control.Monad.IO.Class (liftIO)+-- > import Control.Monad.Trans.Reader (ReaderT)+-- >+-- > newKey :: ReaderT env IO ByteString+-- > newKey = liftIO (getRandomBytes 32)+--+-- What that draws is what the first section describes, no more and no less:+-- the 'IO' instance, the generator of the operating system thread the call+-- lands on, seeded from the system and reseeded as it goes.  'liftIO' says+-- which monad the draw happens in and nothing about where the bytes come+-- from.+--+-- Writing an instance for the stack instead would make the choice invisible+-- at the call site, and the class cannot tell a strong source from a weak+-- one -- see 'MonadRandom'.+--+-- == When the system will not give any+--+-- Drawing randomness is the one thing here with nothing to fall back on,+-- so the failure is an exception rather than a value: there is no sensible+-- 'Maybe' to return and no partial answer worth having.  Since 2.2 it is an+-- 'EntropyError' and not a @Control.Exception.ErrorCall@ or an+-- @Control.Exception.IOException@, which is worth knowing if you were+-- catching one of those:+--+-- > import Control.Exception (catch)+-- > import Crypto.Random (EntropyError (..), getRandomBytes)+-- >+-- > key <- getRandomBytes 32 `catch` \e -> case e of+-- >     NoEntropySource   -> fail "this system offers no randomness at all"+-- >     EntropyShort w g  -> fail (show w ++ " bytes wanted, " ++ show g ++ " arrived")+-- >     EntropySourceLost s -> fail ("the source " ++ s ++ " went away")+--+-- Catching it at all is a decision rather than a default.  A program that+-- cannot get randomness cannot make a key, and stopping is usually the+-- honest thing; the reason to catch is to say so in the program's own+-- terms rather than in crypton's. module Crypto.Random (-    -- * Deterministic instances-    ChaChaDRG,-    SystemDRG,-    Seed,+    -- * Drawing from the system+    MonadRandom (..), -    -- * Seed+    -- * Drawing from a generator you hold+    Seed,     seedNew,     seedFromInteger,     seedToInteger,     seedFromBinary,--    -- * Deterministic Random class-    getSystemDRG,-    drgNew,     drgNewSeed,+    drgNew,     drgNewTest,     withDRG,     withRandomBytes,+    MonadPseudoRandom,     DRG (..), -    -- * Random abstraction-    MonadRandom (..),-    MonadPseudoRandom,+    -- * The generators+    ChaChaDRG,+    SystemDRG,+    getSystemDRG,++    -- * When the system will not give any+    EntropyError (..), ) where  import Crypto.Error import Crypto.Internal.Imports import Crypto.Random.ChaChaDRG+import Crypto.Random.Entropy (EntropyError (..)) import Crypto.Random.SystemDRG import Crypto.Random.Types import Data.ByteArray (ByteArray, ByteArrayAccess, ScrubbedBytes)@@ -50,6 +174,9 @@ import Foreign.Ptr (Ptr, castPtr) #endif +-- | The material a deterministic generator is built from.  Two generators+-- made from one seed produce the same bytes, which is what 'drgNewSeed' is+-- for and why a seed kept anywhere is as good as the keys drawn from it. newtype Seed = Seed ScrubbedBytes     deriving (ByteArrayAccess)
Crypto/Random/Entropy.hs view
@@ -6,16 +6,37 @@ -- Portability : Good module Crypto.Random.Entropy (     getEntropy,+    EntropyError (..), ) where  import Crypto.Internal.ByteArray (ByteArray) import qualified Crypto.Internal.ByteArray as B-import Data.Maybe (catMaybes)+import System.IO.Unsafe (unsafeInterleaveIO, unsafePerformIO)  import Crypto.Random.Entropy.Unsafe +-- | The backends this system has, worked out once and no further than+-- needed.+--+-- Opening one is not free: for a device file it means opening and closing+-- @\/dev\/random@ or @\/dev\/urandom@ just to learn that it is there.  This+-- used to be done on every call, and for the whole list before any backend+-- was asked for a byte, so a program paid for both devices even when the+-- first backend answered everything.+--+-- Once, now, and lazily: 'replenish' stops as soon as the buffer is full,+-- which leaves the tail of this list unforced, so a system where the first+-- backend answers never opens a device at all.+{-# NOINLINE openedBackends #-}+openedBackends :: [EntropyBackend]+openedBackends = unsafePerformIO (openAsNeeded supportedBackends)+  where+    openAsNeeded [] = return []+    openAsNeeded (o : os) = do+        m <- o+        rest <- unsafeInterleaveIO (openAsNeeded os)+        return $ maybe rest (: rest) m+ -- | Get some entropy from the system source of entropy getEntropy :: ByteArray byteArray => Int -> IO byteArray-getEntropy n = do-    backends <- catMaybes `fmap` sequence supportedBackends-    B.alloc n (replenish n backends)+getEntropy n = B.alloc n (replenish n openedBackends)
Crypto/Random/Entropy/Backend.hs view
@@ -11,27 +11,39 @@     ( EntropyBackend     , supportedBackends     , gatherBackend+    , EntropyError(..)     ) where  import Foreign.Ptr import Data.Proxy import Data.Word (Word8) import Crypto.Random.Entropy.Source-#ifdef SUPPORT_RDRAND-import Crypto.Random.Entropy.RDRand-#endif #ifdef WINDOWS import Crypto.Random.Entropy.Windows #else+import Crypto.Random.Entropy.SysRandom import Crypto.Random.Entropy.Unix #endif --- | All supported backends +-- | All supported backends, best first.+--+-- The system call comes before everything else: it is the kernel's own+-- generator, it needs no descriptor, and it is what the rest of the world+-- reaches for now.  The device files are what is left when the system has+-- no such call.+--+-- RDRAND is deliberately not here, though it used to be first on x86.  A+-- list like this one is a list of alternatives, and whichever answers+-- first decides the bytes on its own -- which is the one thing #298 says+-- RDRAND should not do.  Nor does it contribute anywhere else: every+-- system this builds for already feeds the instruction into the pool the+-- call above draws from, so asking it again adds nothing.  See the note in+-- @cbits\/crypton_sysdrg.c@. supportedBackends :: [IO (Maybe EntropyBackend)] supportedBackends =     [-#ifdef SUPPORT_RDRAND-    openBackend (Proxy :: Proxy RDRand),+#ifndef WINDOWS+    openBackend (Proxy :: Proxy SysRandom), #endif #ifdef WINDOWS     openBackend (Proxy :: Proxy WinCryptoAPI)
− Crypto/Random/Entropy/RDRand.hs 
@@ -1,39 +0,0 @@-{-# LANGUAGE ForeignFunctionInterface #-}---- |--- Module      : Crypto.Random.Entropy.RDRand--- License     : BSD-style--- Maintainer  : Vincent Hanquez <vincent@snarc.org>--- Stability   : experimental--- Portability : Good-module Crypto.Random.Entropy.RDRand (-    RDRand,-) where--import Crypto.Random.Entropy.Source-import Data.Word (Word8)-import Foreign.C.Types-import Foreign.Ptr--foreign import ccall unsafe "crypton_cpu_has_rdrand"-    c_cpu_has_rdrand :: IO CInt--foreign import ccall unsafe "crypton_get_rand_bytes"-    c_get_rand_bytes :: Ptr Word8 -> CInt -> IO CInt---- | Fake handle to Intel RDRand entropy CPU instruction-data RDRand = RDRand--instance EntropySource RDRand where-    entropyOpen = rdrandGrab-    entropyGather _ = rdrandGetBytes-    entropyClose _ = return ()--rdrandGrab :: IO (Maybe RDRand)-rdrandGrab = supported `fmap` c_cpu_has_rdrand-  where-    supported 0 = Nothing-    supported _ = Just RDRand--rdrandGetBytes :: Ptr Word8 -> Int -> IO Int-rdrandGetBytes ptr sz = fromIntegral `fmap` c_get_rand_bytes ptr (fromIntegral sz)
Crypto/Random/Entropy/Source.hs view
@@ -6,8 +6,32 @@ -- Portability : Good module Crypto.Random.Entropy.Source where +import Control.Exception (Exception) import Data.Word (Word8) import Foreign.Ptr++-- | The system would not give the entropy it was asked for.+--+-- This is the one failure in the library with nothing to fall back on and+-- nothing sensible to return: a key drawn from bytes that are not random+-- is worse than no key.  It used to be reported with 'error' and 'fail',+-- which left a caller no way to tell it from a bug in the library, and no+-- way to say anything useful about it.+data EntropyError+    = -- | The system offers no source of entropy at all.  On Unix that+      -- means no @getrandom(2)@, no @getentropy(3)@ and no @\/dev@ to read+      -- from; a container built from nothing would look like this.+      NoEntropySource+    | -- | The sources between them gave fewer bytes than were asked for,+      -- three times over.  The two numbers are how many were wanted and+      -- how many arrived.+      EntropyShort Int Int+    | -- | A source that could be opened once could not be opened again.+      -- The string names it.+      EntropySourceLost String+    deriving (Show, Eq)++instance Exception EntropyError  -- | A handle to an entropy maker, either a system capability -- or a hardware generator.
+ Crypto/Random/Entropy/SysRandom.hs view
@@ -0,0 +1,50 @@+{-# LANGUAGE ForeignFunctionInterface #-}++-- |+-- Module      : Crypto.Random.Entropy.SysRandom+-- License     : BSD-style+-- Maintainer  : Kazu Yamamoto <kazu@iij.ad.jp>+-- Stability   : experimental+-- Portability : Unix+--+-- The kernel's own generator, reached without a file descriptor:+-- @getrandom(2)@ on Linux and FreeBSD, @getentropy(3)@ where that is what+-- the system has.+--+-- This is the source to prefer over reading @\/dev\/urandom@.  It needs no+-- path and no descriptor, so it still answers where @\/dev@ is not mounted+-- or not populated, and it cannot be defeated by a full descriptor table.+module Crypto.Random.Entropy.SysRandom (+    SysRandom,+) where++import Crypto.Random.Entropy.Source+import Data.Word (Word8)+import Foreign.C.Types+import Foreign.Ptr++-- Both are @safe@ rather than @unsafe@: at early boot, before the kernel+-- pool is initialised, these calls block, and an unsafe call that blocks+-- holds the capability it runs on.+foreign import ccall safe "crypton_sysrandom_available"+    c_sysrandom_available :: IO CInt++foreign import ccall safe "crypton_sysrandom_bytes"+    c_sysrandom_bytes :: Ptr Word8 -> CInt -> IO CInt++-- | The system call, where there is one.+data SysRandom = SysRandom++instance EntropySource SysRandom where+    -- Asked at run time and not only at compile time: a binary built where+    -- the header declares the call can still run on a kernel that answers+    -- ENOSYS.+    entropyOpen = available `fmap` c_sysrandom_available+      where+        available 0 = Nothing+        available _ = Just SysRandom++    entropyGather _ ptr n =+        fromIntegral `fmap` c_sysrandom_bytes ptr (fromIntegral n)++    entropyClose _ = return ()
Crypto/Random/Entropy/Unix.hs view
@@ -61,7 +61,7 @@ withDev filepath f =     openDev filepath >>= \h ->         case h of-            Nothing -> error ("device " ++ filepath ++ " cannot be grabbed")+            Nothing -> E.throwIO (EntropySourceLost filepath)             Just fd -> f fd `E.finally` closeDev fd  closeDev :: H -> IO ()
Crypto/Random/Entropy/Unsafe.hs view
@@ -9,25 +9,27 @@     module Crypto.Random.Entropy.Backend, ) where +import Control.Exception (throwIO)+ import Crypto.Random.Entropy.Backend import Data.Word (Word8) import Foreign.Ptr (Ptr, plusPtr)  -- | Refill the entropy in a buffer ----- Call each entropy backend in turn until the buffer has--- been replenished.+-- Call each entropy backend in turn until the buffer has been replenished. ----- If the buffer cannot be refill after 3 loopings, this will raise--- an User Error exception+-- Throws 'EntropyError': 'NoEntropySource' when there is no backend at all,+-- and 'EntropyShort' when three passes over the backends still leave the+-- buffer unfilled. replenish :: Int -> [EntropyBackend] -> Ptr Word8 -> IO ()-replenish _ [] _ = fail "crypton: random: cannot get any source of entropy on this system"+replenish _ [] _ = throwIO NoEntropySource replenish poolSize backends ptr = loop 0 backends ptr poolSize   where     loop :: Int -> [EntropyBackend] -> Ptr Word8 -> Int -> IO ()     loop _ _ _ 0 = return ()     loop retry [] p n-        | retry == 3 = error "crypton: random: cannot fully replenish"+        | retry == 3 = throwIO $ EntropyShort poolSize (poolSize - n)         | otherwise = loop (retry + 1) backends p n     loop retry (b : bs) p n = do         r <- gatherBackend b p n
Crypto/Random/Entropy/Windows.hs view
@@ -23,6 +23,7 @@ import Foreign.Storable (peek) import System.Win32.Types (getLastError) +import Control.Exception (throwIO) import Crypto.Random.Entropy.Source  @@ -38,7 +39,8 @@         case mctx of             Nothing  -> do                 lastError <- getLastError-                fail $ "cannot re-grab win crypto api: error " ++ show lastError+                throwIO $ EntropySourceLost+                    ("the Windows crypto API: error " ++ show lastError)             Just ctx -> do                 r <- cryptGenRandom ctx ptr n                 cryptReleaseCtx ctx@@ -100,4 +102,5 @@         then return ()         else do             lastError <- getLastError-            fail $ "cryptReleaseCtx: error " ++ show lastError+            throwIO $ EntropySourceLost+                ("cryptReleaseCtx: error " ++ show lastError)
+ Crypto/Random/SysDRG.hs view
@@ -0,0 +1,82 @@+{-# LANGUAGE ForeignFunctionInterface #-}++-- |+-- Module      : Crypto.Random.SysDRG+-- License     : BSD-style+-- Maintainer  : Kazu Yamamoto <kazu@iij.ad.jp>+-- Stability   : experimental+-- Portability : Unix, Windows+--+-- The generator behind the 'Crypto.Random.MonadRandom' instance for 'IO'.+--+-- A ChaCha20 generator per operating system thread, seeded from a+-- process-wide generator, seeded in turn from the system entropy pool.+-- The state and the reseeding live in+-- @cbits\/crypton_sysdrg.c@, because a @forkIO@ thread is not an operating+-- system thread -- it moves between capabilities -- so state held against+-- one would be shared by threads running at the same time.+--+-- == What it is, and what it is not+--+-- It is not a DRBG of NIST SP 800-90A.  That standard names three --+-- @Hash_DRBG@, @HMAC_DRBG@ and @CTR_DRBG@ -- and none of them is built on a+-- stream cipher.  The name here is the older and looser sense of the word.+--+-- What it is is the shape @arc4random@ on OpenBSD and @get_random_bytes@ in+-- the Linux kernel both have, and the one asked for in+-- <https://github.com/kazu-yamamoto/crypton/issues/298>.  That shape is+-- common; it is not specified anywhere, so the parts a standard would have+-- fixed were chosen here instead:+--+-- * ChaCha20, rather than AES in counter mode.+-- * SHA-512 over the system's bytes, rather than a derivation function a+--   standard would have named.+-- * A mebibyte per thread, and a mebibyte of issued seed for the process+--   generator, as the points at which to reseed.  Those numbers are a+--   choice, not a result.+--+-- The pieces underneath are specified: ChaCha20 is RFC 8439, SHA-512 is+-- FIPS 180-4.  The way they are put together is not.+--+-- == What a compromised state gives away+--+-- Each draw ends by taking the next forty bytes of keystream as the key and+-- nonce and dropping the ones that made the output, so a state read after a+-- draw is not the state that produced it.  ChaCha20 does not run backwards+-- and the key that would be needed is gone, which is backtracking+-- resistance in the terms of SP 800-90A.  Without it, a key stands until+-- the next reseed and anyone holding the state can wind the counter back+-- over everything issued since -- a mebibyte of output that was meant to be+-- secret.  It is what @arc4random@ does, and for the same reason.+--+-- Prediction resistance is what the reseeding gives.  A reseed draws from+-- the system again, so a state that has been read does not determine what+-- comes after one.+module Crypto.Random.SysDRG (+    sysDRGBytes,+) where++import Data.Word (Word8)+import Foreign.C.Types (CInt (..))+import Foreign.Ptr (Ptr)++import Crypto.Internal.ByteArray (ByteArray)+import qualified Crypto.Internal.ByteArray as B++-- Safe, not unsafe: seeding can reach a getrandom(2) that blocks until the+-- kernel pool is ready, and an unsafe call that blocks holds the capability+-- it runs on.+foreign import ccall safe "crypton_sysdrg_bytes"+    c_sysdrg_bytes :: Ptr Word8 -> CInt -> IO CInt++-- | Draw bytes, or 'Nothing' if the generator cannot be seeded.+--+-- It cannot be seeded where the system has no @getrandom(2)@ or+-- @getentropy(3)@; the caller falls back to+-- 'Crypto.Random.Entropy.getEntropy' there, which is the path this system+-- had before the generator existed.+sysDRGBytes :: ByteArray byteArray => Int -> IO (Maybe byteArray)+sysDRGBytes n = do+    (got, out) <- B.allocRet n $ \ptr ->+        fromIntegral `fmap` c_sysdrg_bytes ptr (fromIntegral n)+    return $ if got == n then Just out else Nothing
Crypto/Random/Types.hs view
@@ -13,8 +13,21 @@  import Crypto.Internal.ByteArray import Crypto.Random.Entropy+import Crypto.Random.SysDRG (sysDRGBytes)  -- | A monad constraint that allows to generate random bytes+--+-- Everything in this library that draws a key, a nonce or a signature's+-- randomness draws it through this class, and cannot tell a strong source+-- from a weak one.  The two instances below are the ones to reach for:+-- @IO@ reads the system entropy source, and @MonadPseudoRandom@ runs a+-- 'DRG' seeded from it.+--+-- An instance of your own is held to the same standard.  A generator that+-- another party can predict, or that repeats, yields keys and signatures+-- that give away what they are meant to keep -- so the deliberately+-- repeatable instance that makes a test reproducible is not one to ship+-- with. class Monad m => MonadRandom m where     getRandomBytes :: ByteArray byteArray => Int -> m byteArray @@ -23,8 +36,13 @@     -- | Generate N bytes of randomness from a DRG     randomBytesGenerate :: ByteArray byteArray => Int -> gen -> (byteArray, gen) +-- | Through the generator of 'Crypto.Random.SysDRG': a ChaCha20 generator+-- per operating system thread, reseeded from the system.  Where that+-- cannot be seeded -- a system with no @getrandom(2)@ or @getentropy(3)@,+-- which includes Windows for now -- this is the system entropy source+-- directly, as it was before the generator existed. instance MonadRandom IO where-    getRandomBytes = getEntropy+    getRandomBytes n = sysDRGBytes n >>= maybe (getEntropy n) return  -- | A simple Monad class very similar to a State Monad -- with the state being a DRG.
Crypto/System/CPU.hs view
@@ -1,6 +1,6 @@ {-# LANGUAGE CPP #-}-{-# LANGUAGE DeriveDataTypeable #-} {-# LANGUAGE ForeignFunctionInterface #-}+{-# LANGUAGE PatternSynonyms #-}  -- | -- Module      : Crypto.System.CPU@@ -11,57 +11,218 @@ -- -- Gives information about crypton runtime environment. module Crypto.System.CPU (-    ProcessorOption (..),+    -- The names are bundled with the type rather than listed as+    -- `pattern' exports, so an importer writes ProcessorOption (..), or+    -- names the ones it wants, as it would for a type with constructors.+    ProcessorOption (+        -- x86+        AESNI,+        PCLMUL,+        SSSE3,+        AVX,+        AVX2,+        SHANI,+        MOVBE,+        ADX,+        VAES,+        VAES512,+        AVX512IFMA,+        -- AArch64+        NEON,+        ARMAES,+        ARMPMULL,+        ARMSHA1,+        ARMSHA2,+        ARMSHA512,+        -- PowerISA+        PPCAES,+        PPCVPMSUM+    ),     processorOptions,-) where -import Data.Data-import Data.List (findIndices)-#ifdef SUPPORT_RDRAND-import Data.Maybe (isJust)-#endif-import Data.Word (Word8)-import Foreign.Ptr-import Foreign.Storable+    -- * Questions that do not name an architecture+    hasAESAcceleration,+    hasGHASHAcceleration,+) where +import Control.Monad (filterM)+import Data.List (sort)+import Data.Word (Word16)+import Foreign.C.Types (CInt (..), CUInt (..)) import Crypto.Internal.Compat -#ifdef SUPPORT_RDRAND-import Crypto.Random.Entropy.RDRand-import Crypto.Random.Entropy.Source-#endif+-- | A processor feature crypton looked for, and dispatches on where it+-- finds it.+--+-- This is a number with names rather than a sum of constructors, and the+-- names are pattern synonyms with no @COMPLETE@ pragma, so a @case@ over+-- them needs a catch-all and a feature named in a later release breaks+-- nothing that compiled against this one.  The same reason 'Show' is+-- written out below: a program built against an older crypton still says+-- something useful about a value from a newer one.+--+-- The names are the processor's, not the operation's.  'AESNI' is x86's+-- and 'ARMAES' is AArch64's, and a machine reports only the ones it has;+-- ask 'hasAESAcceleration' if the question is whether AES is fast here.+--+-- They are bundled with the type in the export list, so @ProcessorOption+-- (..)@ brings in all of them and naming one brings in that one, as for a+-- type with constructors.  The constructor underneath is not exported:+-- these values say what the processor was found to have, and a caller has+-- nothing to build.+newtype ProcessorOption = ProcessorOption Word16+    deriving (Eq, Ord) --- | CPU options impacting cryptography implementation and library performance.-data ProcessorOption-    = -- | Support for AES instructions, with flag @support_aesni@-      AESNI-    | -- | Support for CLMUL instructions, with flag @support_pclmuldq@-      PCLMUL-    | -- | Support for RDRAND instruction, with flag @support_rdrand@-      RDRAND-    deriving (Show, Eq, Enum, Data)+-- | Support for AES instructions, with flag @support_aesni@.+pattern AESNI :: ProcessorOption+pattern AESNI = ProcessorOption 0 --- | Options which have been enabled at compile time and are supported by the--- current CPU.+-- | Support for CLMUL instructions, with flag @support_pclmuldq@.+pattern PCLMUL :: ProcessorOption+pattern PCLMUL = ProcessorOption 1++-- | Supplemental SSE3.+pattern SSSE3 :: ProcessorOption+pattern SSSE3 = ProcessorOption 3++-- | AVX, and an operating system that saves its registers.+pattern AVX :: ProcessorOption+pattern AVX = ProcessorOption 4++-- | AVX2, and an operating system that saves its registers.+pattern AVX2 :: ProcessorOption+pattern AVX2 = ProcessorOption 5++-- | The SHA extensions, @sha1rnds4@ and @sha256rnds2@ and their neighbours.+pattern SHANI :: ProcessorOption+pattern SHANI = ProcessorOption 6++-- | The byte-swapping load.+pattern MOVBE :: ProcessorOption+pattern MOVBE = ProcessorOption 7++-- | @MULX@, @ADCX@ and @ADOX@: the two independent carry chains.+pattern ADX :: ProcessorOption+pattern ADX = ProcessorOption 8++-- | The AES and carry-less multiply instructions in their 256-bit form.+pattern VAES :: ProcessorOption+pattern VAES = ProcessorOption 9++-- | The same pair in their 512-bit form.+pattern VAES512 :: ProcessorOption+pattern VAES512 = ProcessorOption 10++-- | The AVX-512 52-bit integer multiply-adds, with AVX-512 F and VL, and the+-- operating system saving their state: RSA's private-key operation uses them.+pattern AVX512IFMA :: ProcessorOption+pattern AVX512IFMA = ProcessorOption 19++-- | Advanced SIMD, which is not optional on AArch64.+pattern NEON :: ProcessorOption+pattern NEON = ProcessorOption 11++-- | The ARMv8 AES instructions.+pattern ARMAES :: ProcessorOption+pattern ARMAES = ProcessorOption 12++-- | @PMULL@, the ARMv8 carry-less multiply.+pattern ARMPMULL :: ProcessorOption+pattern ARMPMULL = ProcessorOption 13++-- | The ARMv8 SHA-1 instructions.+pattern ARMSHA1 :: ProcessorOption+pattern ARMSHA1 = ProcessorOption 14++-- | The ARMv8 SHA-256 instructions.+pattern ARMSHA2 :: ProcessorOption+pattern ARMSHA2 = ProcessorOption 15++-- | The ARMv8.2 SHA-512 instructions, which are optional where SHA-256's+-- are not.+pattern ARMSHA512 :: ProcessorOption+pattern ARMSHA512 = ProcessorOption 16++-- | Support for the PowerISA 2.07 vector AES instructions, which POWER8 was+-- the first to implement.+pattern PPCAES :: ProcessorOption+pattern PPCAES = ProcessorOption 17++-- | Support for @vpmsumd@, the vector carry-less multiply that came with+-- them, which is what makes GHASH fast.+pattern PPCVPMSUM :: ProcessorOption+pattern PPCVPMSUM = ProcessorOption 18++-- | Named where the name is known, numbered where it is not, so that a+-- binary built against an older crypton can still print a value a newer one+-- produced.+instance Show ProcessorOption where+    show AESNI = "AESNI"+    show PCLMUL = "PCLMUL"+    show SSSE3 = "SSSE3"+    show AVX = "AVX"+    show AVX2 = "AVX2"+    show SHANI = "SHANI"+    show MOVBE = "MOVBE"+    show ADX = "ADX"+    show VAES = "VAES"+    show VAES512 = "VAES512"+    show AVX512IFMA = "AVX512IFMA"+    show NEON = "NEON"+    show ARMAES = "ARMAES"+    show ARMPMULL = "ARMPMULL"+    show ARMSHA1 = "ARMSHA1"+    show ARMSHA2 = "ARMSHA2"+    show ARMSHA512 = "ARMSHA512"+    show PPCAES = "PPCAES"+    show PPCVPMSUM = "PPCVPMSUM"+    show (ProcessorOption n) = "ProcessorOption " ++ show n++-- | Options which have been enabled at compile time and which this machine+-- says may be used: on x86-64 what the processor declares through CPUID, and+-- for the vector extensions only where the operating system saves their+-- state as well; elsewhere what the operating system reports of the+-- processor.+--+-- That is what crypton itself decides by, and it is not always everything+-- the core underneath executes.  A guest under a hypervisor may be shown an+-- older processor model than the host's, so that it can be moved to another+-- host -- and an instruction used because it happens to run would fail+-- there.  So an option missing here means crypton will not use it, not that+-- the hardware lacks it.+--+-- Sorted, and without repeats.  A machine reports the names of its own+-- architecture only: an AArch64 processor with AES says 'ARMAES', not+-- 'AESNI', which it does not have. processorOptions :: [ProcessorOption]-processorOptions = unsafeDoIO $ do-    p <- crypton_aes_cpu_init-    options <- traverse (getOption p) aesOptions-    rdrand <- hasRDRand-    return (decodeOptions options ++ [RDRAND | rdrand])+processorOptions = unsafeDoIO (sort <$> filterM askC allOptions)   where-    aesOptions = [AESNI .. PCLMUL]-    getOption p = peekElemOff p . fromEnum-    decodeOptions = map toEnum . findIndices (> 0)+    -- 2 is not asked for and has no name: it was RDRAND, which crypton no+    -- longer dispatches on.  The number is left out rather than reused, so+    -- that the others keep the values they had.+    allOptions = [ProcessorOption n | n <- [0 .. 19], n /= 2]+    askC (ProcessorOption n) =+        (/= 0) <$> crypton_cpu_option (fromIntegral n) {-# NOINLINE processorOptions #-} -hasRDRand :: IO Bool-#ifdef SUPPORT_RDRAND-hasRDRand = fmap isJust getRDRand-  where getRDRand = entropyOpen :: IO (Maybe RDRand)-#else-hasRDRand = return False-#endif+-- | Is there hardware AES on this machine?+--+-- The instructions have different names on different architectures, and a+-- caller that wants to know whether AES-GCM will be fast wants this rather+-- than either name.+hasAESAcceleration :: Bool+hasAESAcceleration =+    any+        (`elem` processorOptions)+        [AESNI, ARMAES, PPCAES] -foreign import ccall unsafe "crypton_aes_cpu_init"-    crypton_aes_cpu_init :: IO (Ptr Word8)+-- | Is there a hardware carry-less multiply, which is what GHASH, and so+-- AES-GCM, spends its time in once AES itself is fast?+hasGHASHAcceleration :: Bool+hasGHASHAcceleration =+    any+        (`elem` processorOptions)+        [PCLMUL, ARMPMULL, PPCVPMSUM]++foreign import ccall unsafe "crypton_cpu_option"+    crypton_cpu_option :: CUInt -> IO CInt
Crypto/Tutorial.hs view
@@ -1,4 +1,7 @@ -- | Examples of how to use @crypton@.+--+-- Every code block here is extracted and compiled against this version of+-- the library by @tests\/tutorial\/run.sh@, so what is written below builds. module Crypto.Tutorial (     -- * API design     -- $api_design@@ -6,9 +9,21 @@     -- * Hash algorithms     -- $hash_algorithms -    -- * Symmetric block ciphers-    -- $symmetric_block_ciphers+    -- * Authenticated encryption+    -- $authenticated_encryption +    -- * Comparing secrets+    -- $comparing_secrets++    -- * Password storage+    -- $password_storage++    -- * Key derivation+    -- $key_derivation++    -- * Digital signatures+    -- $digital_signatures+     -- * Combining primitives     -- $combining_primitives ) where@@ -31,6 +46,14 @@ -- Error conditions are returned with data type 'Crypto.Error.CryptoFailable'. -- Functions in module "Crypto.Error" can convert those values to runtime -- exceptions, 'Maybe' or 'Either' values.+--+-- Types that hold a secret do not print it.  A private key's 'Show' renders+-- whatever is public and @\<secret\>@ or @\<scrubbed-bytes\>@ for the rest,+-- because 'Show' is what @print@, @error@, an exception and a failing test+-- all reach for, and a key arriving in a log that way is an accident nobody+-- asked for.  "Crypto.Debug" is how one is printed when printing it is what+-- was meant.  Several of those types keep their bytes in+-- 'Data.ByteArray.ScrubbedBytes', which is wiped when it is collected.  -- $hash_algorithms --@@ -90,72 +113,258 @@ -- >     hashMutableUpdate ctx ("dog"    :: ByteString) -- >     hashMutableFinalize ctx >>= print --- $symmetric_block_ciphers+-- $authenticated_encryption --+-- Encrypting hides a message; it does not stop anyone changing it.  Under a+-- counter or stream mode, flipping a bit of the ciphertext flips the same+-- bit of the plaintext, and the receiver has no way to tell.  An AEAD mode+-- binds the message, and anything else named as associated data, to a short+-- authentication tag, and decrypting something that does not match that tag+-- returns nothing at all.+--+-- That is the mode to use.  The unauthenticated ones are in this library+-- for protocols that authenticate separately, not as a starting point.+-- -- > {-# LANGUAGE OverloadedStrings #-}--- > {-# LANGUAGE ScopedTypeVariables #-}--- > {-# LANGUAGE GADTs #-} -- > -- > import           Crypto.Cipher.AES (AES256)--- > import           Crypto.Cipher.Types (BlockCipher(..), Cipher(..), nullIV, KeySizeSpecifier(..), IV, makeIV)--- > import           Crypto.Error (CryptoFailable(..), CryptoError(..))+-- > import           Crypto.Cipher.Types+-- >                      ( AEADMode (AEAD_GCM)+-- >                      , AuthTag+-- >                      , BlockCipher (aeadInit)+-- >                      , Cipher (cipherInit, cipherKeySize)+-- >                      , KeySizeSpecifier (..)+-- >                      , aeadSimpleDecrypt+-- >                      , aeadSimpleEncrypt+-- >                      )+-- > import           Crypto.Error (CryptoError, eitherCryptoError)+-- > import qualified Crypto.Random.Types as CRT -- >+-- > import           Data.ByteArray (ByteArray, ByteArrayAccess, ScrubbedBytes)+-- > import           Data.ByteString (ByteString)+-- >+-- > -- | A key of the length the cipher asks for, rather than a length+-- > -- written out here.  ScrubbedBytes rather than ByteString, so that it+-- > -- is wiped when it is collected and does not print.+-- > genSecretKey :: (Cipher c, CRT.MonadRandom m) => c -> m ScrubbedBytes+-- > genSecretKey c = CRT.getRandomBytes (longest (cipherKeySize c))+-- >   where+-- >     longest (KeySizeFixed n)   = n+-- >     longest (KeySizeRange _ n) = n+-- >     longest (KeySizeEnum ns)   = maximum ns+-- >+-- > -- | A fresh nonce for every message.  GCM must never see one twice+-- > -- under the same key: a repeat does not just expose those two messages,+-- > -- it hands over the key that authenticates all of them.  Twelve random+-- > -- bytes, sent along with the ciphertext.+-- > genNonce :: CRT.MonadRandom m => m ByteString+-- > genNonce = CRT.getRandomBytes 12+-- >+-- > -- | Encrypt and authenticate.  The associated data is authenticated but+-- > -- not encrypted: it is for what the receiver can already see and must+-- > -- not have had altered, such as a header or an address.+-- > encrypt+-- >     :: (ByteArray key, ByteArrayAccess nonce, ByteArrayAccess aad, ByteArray ba)+-- >     => key -> nonce -> aad -> ba -> Either CryptoError (AuthTag, ba)+-- > encrypt key nonce aad plaintext = do+-- >     cipher <- eitherCryptoError (cipherInit key) :: Either CryptoError AES256+-- >     aead <- eitherCryptoError (aeadInit AEAD_GCM cipher nonce)+-- >     return (aeadSimpleEncrypt aead aad plaintext 16)+-- >+-- > -- | And back, with two different failures.  Left is this code used+-- > -- wrongly -- a key of the wrong length, a mode the cipher has not got.+-- > -- Nothing is a message that is not the one that was sent; it carries no+-- > -- plaintext and says nothing about which byte was wrong, both of which+-- > -- are the point.+-- > decrypt+-- >     :: (ByteArray key, ByteArrayAccess nonce, ByteArrayAccess aad, ByteArray ba)+-- >     => key -> nonce -> aad -> AuthTag -> ba -> Either CryptoError (Maybe ba)+-- > decrypt key nonce aad tag ciphertext = do+-- >     cipher <- eitherCryptoError (cipherInit key) :: Either CryptoError AES256+-- >     aead <- eitherCryptoError (aeadInit AEAD_GCM cipher nonce)+-- >     return (aeadSimpleDecrypt aead aad ciphertext tag)+-- >+-- > exampleAES256GCM :: ByteString -> IO ()+-- > exampleAES256GCM msg = do+-- >     key <- genSecretKey (undefined :: AES256)+-- >     nonce <- genNonce+-- >     let aad = "to: alice" :: ByteString+-- >     case encrypt key nonce aad msg of+-- >         Left err -> error (show err)+-- >         Right (tag, ciphertext) -> do+-- >             putStrLn $ "ciphertext: " ++ show ciphertext+-- >             putStrLn $ "       tag: " ++ show tag+-- >             putStrLn $ " recovered: "+-- >                 ++ show (decrypt key nonce aad tag ciphertext)+-- >             -- The same bytes and the same tag, with one thing changed+-- >             -- that was never encrypted: Right Nothing.+-- >             putStrLn $ "redirected: "+-- >                 ++ show (decrypt key nonce ("to: eve" :: ByteString) tag ciphertext)+--+-- The two functions above work for any cipher that has an AEAD mode; what+-- changes is the mode given to 'Crypto.Cipher.Types.aeadInit'.+-- "Crypto.Cipher.ChaChaPoly1305" is the one to prefer on a machine with no+-- AES instructions, and "Crypto.Cipher.AESGCMSIV" is the one that survives+-- a repeated nonce, at the price of needing the whole message before it can+-- begin.++-- $comparing_secrets+--+-- Comparing two byte strings with '==' stops at the first byte that+-- differs, so how long it takes says where that byte was.  Against an+-- authentication tag that is the whole secret: someone who can send a guess+-- and time the answer finds the first byte in a few hundred tries, then the+-- second, and has a tag that was supposed to cost 2^128 in a few thousand.+--+-- crypton's own authentication types already compare in constant time, so+-- for those there is nothing to do: 'Crypto.MAC.HMAC.HMAC',+-- 'Crypto.MAC.CMAC.CMAC', 'Crypto.MAC.Poly1305.Auth' and+-- 'Crypto.Cipher.Types.AuthTag' have an 'Eq' that looks at every byte+-- whatever it finds.  What needs care is a tag that arrives as bytes, and+-- 'Data.ByteArray.constEq' is the comparison for it.+--+-- > {-# LANGUAGE OverloadedStrings #-}+-- >+-- > import           Crypto.Hash.Algorithms (SHA256)+-- > import           Crypto.MAC.HMAC (HMAC, hmac)+-- >+-- > import qualified Data.ByteArray as BA+-- > import           Data.ByteString (ByteString)+-- >+-- > -- | The tag came off the wire as bytes, so it is compared as bytes.+-- > authentic :: ByteString -> ByteString -> ByteString -> Bool+-- > authentic key message tag =+-- >     BA.constEq tag (hmac key message :: HMAC SHA256)+-- >+-- > -- | Once it has been parsed into the library's own type, (==) is+-- > -- already the constant-time comparison.+-- > authentic' :: ByteString -> ByteString -> HMAC SHA256 -> Bool+-- > authentic' key message tag = tag == hmac key message++-- $password_storage+--+-- A password is not a key.  It is short and it is guessable, and whoever+-- takes the database can try every likely one without being watched.  What+-- answers that is a function that is deliberately expensive to compute, and+-- crypton has four: "Crypto.KDF.BCrypt", "Crypto.KDF.Scrypt",+-- "Crypto.KDF.Argon2" and "Crypto.KDF.PBKDF2".  A plain hash is not one of+-- them, however many times it is applied by hand.+--+-- bcrypt leaves the least to get wrong, because the record it returns+-- carries the salt and the cost inside it:+--+-- > import Crypto.KDF.BCrypt (hashPassword, validatePassword)+-- >+-- > import Data.ByteString (ByteString)+-- >+-- > -- | What goes in the database.  The salt is drawn inside and ends up in+-- > -- the result, so two accounts with the same password do not look alike.+-- > register :: ByteString -> IO ByteString+-- > register password = hashPassword 12 password+-- >+-- > -- | And what is checked against it.  The cost comes out of the stored+-- > -- record, so raising it for new accounts leaves the old ones working.+-- > login :: ByteString -> ByteString -> Bool+-- > login password stored = validatePassword password stored+--+-- Argon2 is the stronger choice and the one to pick for something new: it+-- asks for memory as well as time, which is what takes the advantage away+-- from the hardware that bcrypt's small working set leaves room for.+-- Nothing is encoded for the caller, though -- the salt and the options are+-- theirs to store, and without all three the hash cannot be recomputed when+-- the user comes back.+--+-- > import           Crypto.Error (CryptoFailable)+-- > import qualified Crypto.KDF.Argon2 as Argon2 -- > import qualified Crypto.Random.Types as CRT -- >--- > import           Data.ByteArray (ByteArray) -- > import           Data.ByteString (ByteString) -- >--- > -- | Not required, but most general implementation--- > data Key c a where--- >   Key :: (BlockCipher c, ByteArray a) => a -> Key c a+-- > -- | Argon2id, which is the variant to prefer: it resists both a machine+-- > -- built to guess and a process watching the cache.+-- > options :: Argon2.Options+-- > options = Argon2.defaultOptions{Argon2.variant = Argon2.Argon2id} -- >--- > -- | Generates a string of bytes (key) of a specific length for a given block cipher--- > genSecretKey :: forall m c a. (CRT.MonadRandom m, BlockCipher c, ByteArray a) => c -> Int -> m (Key c a)--- > genSecretKey _ = fmap Key . CRT.getRandomBytes+-- > newSalt :: CRT.MonadRandom m => m ByteString+-- > newSalt = CRT.getRandomBytes 16 -- >--- > -- | Generate a random initialization vector for a given block cipher--- > genRandomIV :: forall m c. (CRT.MonadRandom m, BlockCipher c) => c -> m (Maybe (IV c))--- > genRandomIV _ = do--- >   bytes :: ByteString <- CRT.getRandomBytes $ blockSize (undefined :: c)--- >   return $ makeIV bytes+-- > derive :: ByteString -> ByteString -> CryptoFailable ByteString+-- > derive salt password = Argon2.hash options password salt 32++-- $key_derivation+--+-- HKDF turns one secret into as many keys as a protocol needs.  It is for+-- material that is already unguessable -- what comes out of a+-- Diffie-Hellman, or a key already agreed -- and it is deliberately cheap,+-- which is exactly what makes it the wrong thing for a password.  Those go+-- to the section above.+--+-- The info string is what keeps the outputs independent: the same secret+-- with a different info gives an unrelated key, so each use of a secret+-- names itself there.+--+-- > {-# LANGUAGE OverloadedStrings #-} -- >--- > -- | Initialize a block cipher--- > initCipher :: (BlockCipher c, ByteArray a) => Key c a -> Either CryptoError c--- > initCipher (Key k) = case cipherInit k of--- >   CryptoFailed e -> Left e--- >   CryptoPassed a -> Right a+-- > import           Crypto.Hash.Algorithms (SHA256)+-- > import qualified Crypto.KDF.HKDF as HKDF -- >--- > encrypt :: (BlockCipher c, ByteArray a) => Key c a -> IV c -> a -> Either CryptoError a--- > encrypt secretKey initIV msg =--- >   case initCipher secretKey of--- >     Left e -> Left e--- >     Right c -> Right $ ctrCombine c initIV msg+-- > import           Data.ByteArray (ScrubbedBytes)+-- > import           Data.ByteString (ByteString) -- >--- > decrypt :: (BlockCipher c, ByteArray a) => Key c a -> IV c -> a -> Either CryptoError a--- > decrypt = encrypt+-- > -- | One shared secret in, two unrelated keys out.+-- > directionKeys :: ByteString -> ByteString -> (ScrubbedBytes, ScrubbedBytes)+-- > directionKeys salt shared = (keyFor "client write", keyFor "server write")+-- >   where+-- >     prk = HKDF.extract salt shared :: HKDF.PRK SHA256+-- >     keyFor info = HKDF.expand prk (info :: ByteString) 32++-- $digital_signatures+--+-- Ed25519 is the one to reach for.  The keys are thirty-two bytes, there is+-- nothing to choose and nothing to encode, and signing needs no randomness,+-- so it cannot be ruined by a bad source of it.  "Crypto.PubKey.Ed448" is+-- the same shape at a larger size, "Crypto.PubKey.ECDSA" and+-- "Crypto.PubKey.RSA.PSS" are there for protocols that ask for them, and+-- "Crypto.PubKey.MLDSA" is the post-quantum one.+--+-- > {-# LANGUAGE OverloadedStrings #-} -- >--- > exampleAES256 :: ByteString -> IO ()--- > exampleAES256 msg = do--- >   -- secret key needs 256 bits (32 * 8)--- >   secretKey <- genSecretKey (undefined :: AES256) 32--- >   mInitIV <- genRandomIV (undefined :: AES256)--- >   case mInitIV of--- >     Nothing -> error "Failed to generate and initialization vector."--- >     Just initIV -> do--- >       let encryptedMsg = encrypt secretKey initIV msg--- >           decryptedMsg = decrypt secretKey initIV =<< encryptedMsg--- >       case (,) <$> encryptedMsg <*> decryptedMsg of--- >         Left err -> error $ show err--- >         Right (eMsg, dMsg) -> do--- >           putStrLn $ "Original Message: " ++ show msg--- >           putStrLn $ "Message after encryption: " ++ show eMsg--- >           putStrLn $ "Message after decryption: " ++ show dMsg+-- > import           Crypto.Error (throwCryptoError)+-- > import qualified Crypto.PubKey.Ed25519 as Ed25519+-- >+-- > import qualified Data.ByteArray as BA+-- > import           Data.ByteString (ByteString)+-- >+-- > exampleEd25519 :: ByteString -> IO ()+-- > exampleEd25519 msg = do+-- >     sk <- Ed25519.generateSecretKey+-- >     let pk = Ed25519.toPublic sk+-- >         sig = Ed25519.sign sk pk msg+-- >     print (Ed25519.verify pk msg sig)+-- >     -- The same signature against a message one byte longer: False.+-- >     print (Ed25519.verify pk (msg <> "!") sig)+-- >+-- > -- | A secret key on its way to storage.  Printing one does not reveal+-- > -- it -- see the first section -- so this is the way out.+-- > store :: Ed25519.SecretKey -> ByteString+-- > store = BA.convert+-- >+-- > -- | And the way back in, which is checked, because the bytes read from+-- > -- a file may be anything at all.+-- > load :: ByteString -> Ed25519.SecretKey+-- > load = throwCryptoError . Ed25519.secretKey  -- $combining_primitives -- -- This example shows how to use Curve25519, XSalsa and Poly1305 primitives to -- emulate NaCl's @crypto_box@ construct. --+-- It is here to show how the pieces fit together, not as something to+-- deploy.  An authenticated encryption scheme assembled by hand is the kind+-- of code that is wrong in ways no test notices; for actual use,+-- "Crypto.Cipher.ChaChaPoly1305" does this job with the mistakes already+-- made.+-- -- > import qualified Data.ByteArray as BA -- > import           Data.ByteString (ByteString) -- > import qualified Data.ByteString as B@@ -167,6 +376,9 @@ -- > -- > -- | Build a @crypto_box@ packet encrypting the specified content with a -- > -- 192-bit nonce, receiver public key and sender private key.+-- > crypto_box+-- >     :: ByteString -> ByteString -> X25519.PublicKey -> X25519.SecretKey+-- >     -> ByteString -- > crypto_box content nonce pk sk = BA.convert tag `B.append` c -- >   where -- >     zero         = B.replicate 16 0@@ -181,6 +393,9 @@ -- > -- > -- | Try to open a @crypto_box@ packet and recover the content using the -- > -- 192-bit nonce, sender public key and receiver private key.+-- > crypto_box_open+-- >     :: ByteString -> ByteString -> X25519.PublicKey -> X25519.SecretKey+-- >     -> Maybe ByteString -- > crypto_box_open packet nonce pk sk -- >     | B.length packet < 16 = Nothing -- >     | BA.constEq tag' tag  = Just content
LICENSE view
@@ -1,4 +1,5 @@ Copyright (c) 2006-2015 Vincent Hanquez <vincent@snarc.org>+Copyright (c) 2023-2026 Kazu Yamamoto <kazu@iij.ad.jp>  All rights reserved.
README.md view
@@ -3,24 +3,98 @@ crypton ========== -Crypton is a fork from cryptonite with the original author's permission.+`crypton` is a fork from `cryptonite` with the original author's permission. -Crypton is a haskell repository of cryptographic primitives. Each crypto-algorithm has specificities that are hard to wrap in common APIs and types,-so instead of trying to provide a common ground for algorithms, this package-provides a non-consistent low-level API. -If you have no idea what you're doing, please do not use this directly.-Instead, rely on higher level protocols or implementations.+`crypton` is a low-level cryptography library. To achieve high+performance, it utilizes C and assembly language to define FFI+bindings, structuring them in a way that makes them easy to use. ++Side channels+-------------++### AES++AES is where this matters most, and which implementation runs is decided at+runtime from what the processor has.++On x86-64 with AES-NI and carry-less multiply, on AArch64 and on 32-bit ARM+with the ARMv8 cryptographic extension, and on ppc64le with the vector+instructions PowerISA 2.07 brought to POWER8, AES and GHASH are instructions+rather than software.  crypton's AES and AES-GCM then make no branch and no+memory access that depends on the key or on the data: the secrets stay in+vector registers and never reach one a branch can test.  Every x86-64 part+since about 2010 and every AArch64 part in ordinary use has these, and the+ppc64le path is little-endian only.++Where none of them is there, crypton computes AES and GHASH without a table.+The portable AES is bitsliced -- the S-box is boolean algebra over four+blocks held across eight 64-bit words -- and the GF(2^128) multiply is built+from shifts, masks and integer multiplies.  Neither looks anything up, so+neither derives an address from a secret, and **the portable path is constant+time as well**.  The code is BearSSL's, under MIT, in `cbits/bearssl`.++The constant-time harness in `cbits/tests/ct` runs the same driver against+the portable implementation and against each architecture's instructions, and+every one of them is required to report nothing at all.++So the answer does not depend on the machine any more, and what is left to+ask is whether AES is fast on it.  `hasAESAcceleration` and+`hasGHASHAcceleration` answer that without naming an architecture.+`processorOptions` says what was found, in the processor's own names --+`AESNI` and `PCLMUL` on x86, `ARMAES` and `ARMPMULL` on AArch64 and 32-bit+ARM, `PPCAES` and `PPCVPMSUM` on ppc64le -- along with everything else it was+asked about, which is unrelated to this.++    ghci> import Crypto.System.CPU+    ghci> hasAESAcceleration+    True+    ghci> processorOptions           -- on an x86-64 machine+    [AESNI,PCLMUL,SSSE3,AVX,AVX2,SHANI,MOVBE,ADX,VAES]+    ghci> processorOptions           -- and on an AArch64 one+    [NEON,ARMAES,ARMPMULL,ARMSHA1,ARMSHA2,ARMSHA512]++### RSA++RSA is the other place to know about, and there the choice is the caller's.+The private key operations in `Crypto.PubKey.RSA.PKCS15`, `.OAEP` and `.PSS`+take a `Maybe Blinder`, and `Nothing` is no harder to write than the safe+form:++    decrypt     :: Maybe Blinder -> PrivateKey -> ByteString -> ...+    decryptSafer :: MonadRandom m => PrivateKey -> ByteString -> m ...++The exponent itself is not what is at risk.  `expSafe` keeps the *value* of an+exponent out of the work it does, so the private exponent does not leak+through the exponentiation.  What a blinder covers is the other side: without+one, the operation runs on the ciphertext the caller was handed, so how long+it takes depends on a number an attacker may have chosen and can vary.  That+is what a remote timing attack on RSA needs.  With a blinder the input is+multiplied by a random value first and the result divided out afterwards, so+the timing carries nothing an attacker can steer.++`decryptSafer` and `signSafer` generate the blinder themselves and are the+ones to reach for.  Pass `Nothing` only where the input is not attacker+controlled and you have decided that it is not.++The RSA rows in the tables below are the unblinded path.  A fresh blinder+costs one more exponentiation, by the public exponent, which is the cheap+direction: measured on the M4, signing goes from about 316 to about 331+microseconds, under five per cent.  `signSafer` and `decryptSafer` pay less+than that from 2.2.4: they keep the blinder of a key they have seen and+square it for the next use, as OpenSSL does, drawing a fresh one every 32+uses.  `Crypto.PubKey.RSA.Blinded` gives a caller the same thing to hold, a+`BlindedKey` passed along from one operation to the next.+ Performance ----------- -The algorithms a TLS connection uses, measured against the previous release-and against OpenSSL on the same machine.  Throughput is over 16 KiB messages;-the public key operations are one operation each; every figure is the best of-several runs, and crypton and OpenSSL are run alternately so that neither gets-the quieter machine.+The algorithms a TLS connection uses, measured against the last release+before the rewrite and against OpenSSL on the same machine.  Throughput is+over 16 KiB messages; the public key operations are one operation each; every+figure is the best of several runs, and crypton and OpenSSL are run+alternately so that neither gets the quieter machine.  Bulk encryption and hashing are measured through crypton's C layer, as `openssl speed` measures OpenSSL's.  The public key operations are measured@@ -28,125 +102,209 @@ is what a program actually calls; the Haskell layer adds well under a microsecond, which the X25519 and ECDH P-256 rows confirm by agreeing with a C-level measurement to within a percent.  Both releases of crypton are built-the same way -- `-optc-O3`, which is what both of them ask for -- and each-column of a table comes from one run on the machine named above it.+the same way -- `-optc-O3`, which is what each asks for -- and by+`cabal build`, since a copy of the sources compiled by hand does not measure+what a program linking the library gets, and leaves out whole implementations+without saying so.  Each column of a table comes from one run on the machine+named above it.  ### x86-64 -An AMD EPYC 7763, which has AES-NI, PCLMULQDQ, AVX2, ADX and the SHA-extensions, against OpenSSL 3.0.13.+An AMD EPYC 7763, which has AES-NI, PCLMULQDQ, AVX2, ADX, VAES, VPCLMULQDQ+and the SHA extensions, against OpenSSL 4.0.3.  Throughput in MB/s, **higher is better**: -| | crypton 1.1.5 | crypton 2.0.0 | OpenSSL | 2.0.0 / OpenSSL |+| | crypton 1.1.5 | crypton 2.2.4 | OpenSSL | 2.2.4 / OpenSSL | | --- | ---: | ---: | ---: | ---: |-| AES-128-GCM | 1331 | 4118 | 4264 | 0.97 |-| AES-256-GCM | 1090 | 3810 | 3951 | 0.96 |-| ChaCha20-Poly1305 | 398 | 2195 | 2191 | 1.00 |-| SHA-1 | 738 | 1678 | 1672 | 1.00 |-| SHA-256 | 286 | 1585 | 1570 | 1.01 |-| SHA-512 | 448 | 769 | 746 | 1.03 |-| SHA3-256 | 109 | 421 | 426 | 0.99 |+| AES-128-GCM | 1335 | **6041** | 4066 | 1.49 |+| AES-256-GCM | 1093 | **5492** | 3766 | 1.46 |+| ChaCha20-Poly1305 | 399 | 2195 | 2222 | 0.99 |+| SHA-1 | 727 | 1663 | 1672 | 0.99 |+| SHA-256 | 289 | 1585 | 1576 | 1.01 |+| SHA-512 | 448 | 770 | 749 | 1.03 |+| SHA3-256 | 109 | 421 | 429 | 0.98 | -Time per operation in microseconds, **lower is better** -- so the last column-divides OpenSSL's time by crypton's, and is again better the larger it is:+Time per operation in microseconds, **lower is better**: -| | crypton 1.1.5 | crypton 2.0.0 | OpenSSL | OpenSSL / 2.0.0 |+| | crypton 1.1.5 | crypton 2.2.4 | OpenSSL | OpenSSL / 2.2.4 | | --- | ---: | ---: | ---: | ---: |-| X25519 | 43.57 | 43.52 | 36.58 | 0.84 |-| ECDH P-256 | 163.8 | 163.7 | 52.36 | 0.32 |-| ECDH P-384 | 2241 | 1101 | 857.1 | 0.78 |-| Ed25519 sign | 28.52 | 28.25 | 43.49 | 1.54 |-| Ed25519 verify | 46.06 | 46.16 | 119.3 | 2.59 |-| ECDSA P-256 sign | 76.04 | 75.02 | 22.91 | 0.31 |-| ECDSA P-256 verify | 229.2 | 228.5 | 67.98 | 0.30 |-| ECDSA P-384 sign | 2271 | 387.6 | 904.3 | 2.33 |-| ECDSA P-384 verify | 2667 | 1493 | 746.2 | 0.50 |-| RSA-2048 sign/decrypt | 759.8 | 1311 | 660.1 | 0.50 |-| RSA-2048 verify/encrypt | 31.62 | 28.59 | 18.63 | 0.65 |+| X25519 | 44.38 | 28.29 | 36.61 | 1.29 |+| ECDH P-256 | 164.0 | 51.31 | 51.70 | 1.01 |+| ECDH P-384 | 2242 | **164.3** | 836.1 | 5.09 |+| Ed25519 sign | 43.86 | 18.50 | 33.74 | 1.82 |+| Ed25519 verify | 48.30 | 48.43 | 109.8 | 2.27 |+| ECDSA P-256 sign | 81.80 | 19.72 | 21.84 | 1.11 |+| ECDSA P-256 verify | 231.8 | 70.47 | 67.55 | 0.96 |+| ECDSA P-384 sign | 2267 | **305.2** | 879.5 | 2.88 |+| ECDSA P-384 verify | 2698 | **473.6** | 725.2 | 1.53 |+| RSA-2048 sign/decrypt | 760.1 | 612.3 | 659.6 | 1.08 |+| RSA-2048 verify/encrypt | 33.61 | 23.50 | 18.88 | 0.80 |+| RSA-3072 sign/decrypt | 2499 | 2324 | 2012 | 0.87 |+| RSA-3072 verify/encrypt | 66.80 | 61.91 | 40.14 | 0.65 |  ### AArch64  An Apple M4, which has the AES, PMULL, SHA-1, SHA-2, SHA-512 and SHA-3-instructions, against OpenSSL 3.6.4.+instructions, against OpenSSL 4.0.3.  Throughput in MB/s, **higher is better**: -| | crypton 1.1.5 | crypton 2.0.0 | OpenSSL | 2.0.0 / OpenSSL |+| | crypton 1.1.5 | crypton 2.2.4 | OpenSSL | 2.2.4 / OpenSSL | | --- | ---: | ---: | ---: | ---: |-| AES-128-GCM | 126 | 8702 | 10719 | 0.81 |-| AES-256-GCM | 98 | 7648 | 9154 | 0.84 |-| ChaCha20-Poly1305 | 758 | 2323 | 2244 | 1.04 |-| SHA-1 | 1199 | 3380 | 3346 | 1.01 |-| SHA-256 | 467 | 3394 | 3352 | 1.01 |-| SHA-512 | 723 | 1868 | 1851 | 1.01 |-| SHA3-256 | 548 | 1091 | 1054 | 1.04 |+| AES-128-GCM | 131 | **12052** | 10922 | 1.10 |+| AES-256-GCM | 99 | **10192** | 9208 | 1.11 |+| ChaCha20-Poly1305 | 783 | 2342 | 2261 | 1.04 |+| SHA-1 | 1234 | 3383 | 3347 | 1.01 |+| SHA-256 | 486 | 3392 | 3354 | 1.01 |+| SHA-512 | 742 | 1869 | 1860 | 1.00 |+| SHA3-256 | 552 | 1096 | 1092 | 1.00 | -Time per operation in microseconds, **lower is better**; the last column again-divides OpenSSL's time by crypton's:+Time per operation in microseconds, **lower is better**: -| | crypton 1.1.5 | crypton 2.0.0 | OpenSSL | OpenSSL / 2.0.0 |+| | crypton 1.1.5 | crypton 2.2.4 | OpenSSL | OpenSSL / 2.2.4 | | --- | ---: | ---: | ---: | ---: |-| X25519 | 18.44 | 18.41 | 18.41 | 1.00 |-| ECDH P-256 | 69.46 | 56.24 | 24.68 | 0.44 |-| ECDH P-384 | 3252 | 511.1 | 379.7 | 0.74 |-| Ed25519 sign | 13.75 | 13.14 | 15.90 | 1.21 |-| Ed25519 verify | 18.17 | 18.04 | 39.27 | 2.18 |-| ECDSA P-256 sign | 32.56 | 27.91 | 11.05 | 0.40 |-| ECDSA P-256 verify | 96.50 | 80.19 | 32.82 | 0.41 |-| ECDSA P-384 sign | 3219 | 169.3 | 403.4 | 2.38 |-| ECDSA P-384 verify | 3807 | 688.1 | 335.1 | 0.49 |-| RSA-2048 sign/decrypt | 451.8 | 605.4 | 325.0 | 0.54 |-| RSA-2048 verify/encrypt | 18.32 | 15.28 | 8.50 | 0.56 |+| X25519 | 17.92 | **12.02** | 15.32 | 1.28 |+| ECDH P-256 | 67.50 | **19.05** | 24.36 | 1.28 |+| ECDH P-384 | 3256 | **71.78** | 373.1 | 5.20 |+| Ed25519 sign | 13.38 | **7.717** | 13.31 | 1.72 |+| Ed25519 verify | 18.32 | 18.18 | 34.77 | 1.91 |+| ECDSA P-256 sign | 31.08 | **6.500** | 10.66 | 1.64 |+| ECDSA P-256 verify | 94.27 | **26.58** | 32.25 | 1.21 |+| ECDSA P-384 sign | 3225 | **124.0** | 394.2 | 3.18 |+| ECDSA P-384 verify | 3866 | **203.4** | 332.3 | 1.63 |+| RSA-2048 sign/decrypt | 451.3 | 318.4 | 321.5 | 1.01 |+| RSA-2048 verify/encrypt | 18.15 | 11.12 | 8.405 | 0.76 |+| RSA-3072 sign/decrypt | 1473 | 937.4 | 931.1 | 0.99 |+| RSA-3072 verify/encrypt | 47.95 | 43.63 | 17.89 | 0.41 |  ### What the numbers say -1.1.5 had no AArch64 code of its own at all, which is why AES-GCM there is-sixty-nine times what it was.  On x86-64 it had AES-NI and nothing else.  The-curves over a prime field other than P-256 moved from Haskell `Integer`-arithmetic into C, which is the nineteenfold change in ECDSA P-384 signing on-the M4.  X25519 and Ed25519 are unchanged between the two releases, and the-rows say so: where they differ by half a percent, that is the measurement and-not the code.  P-256 is unchanged on x86-64 and a fifth faster on AArch64,-which is the paragraph below.+There are two changes behind the 1.1.5 column and the 2.2.4 one, not a+single steady improvement. -Where crypton is behind, it is behind for three separate reasons.+The first, in 2.0.0, was a rewrite: the bulk algorithms moved into C, the+curves other than P-256 moved out of Haskell `Integer` arithmetic, and+everything that touches a secret was made to take the same time whatever the+secret is.  1.1.5 had no AArch64 code of its own at all, which is why AES-GCM+there is close to a hundred times what it was, and on x86-64 it had AES-NI+and nothing else. -*P-256.*  crypton's field arithmetic is C where OpenSSL's is hand-written-assembly, and that is what is left of the difference: the two differ by about-the same factor on every P-256 row, and nothing above the field -- a wider-window, a different addition formula, another field representation -- recovers-a useful part of it.+The second, from 2.1.0 onwards, is assembly, for the operations where C+cannot reach.  Which of the two a row owes its gain to is not the same+everywhere: ECDSA P-384 signing took nineteenfold from the rewrite and a+further fifth from the assembly, while X25519 waited for the assembly+entirely and ECDH P-384 is almost all of it. -The AArch64 rows are better than the x86-64 ones because of where a field-multiplication's latency goes.  It ends in a carry chain the width of the-number, and the curve arithmetic has independent products that could cover-that chain -- but only if the compiler inlines the reduction instead of-calling it, since a call is a fence.  Asking it to costs code and pays where-there are registers enough to hold two chains at once: a quarter on AArch64,-where there are thirty-one, and nothing on x86-64, where there are fifteen and-the same request makes it slower.  So x86-64 is left to the compiler's own-judgement and stays at 0.3.+Most of that assembly is not crypton's.  The prime curves, the inverse modulo+a group order, X25519, and RSA's Montgomery multiplication on x86-64 go+through [s2n-bignum](https://github.com/awslabs/s2n-bignum), vendored in+`cbits/s2n`.  Every routine in it carries a machine-checked proof in+HOL-Light that it computes what it says, and is written in a constant-time+style.  It is `Apache-2.0 OR ISC OR MIT-0`, and crypton takes it under ISC. -*RSA signing.*  2.0.0 is slower than 1.1.5 here on purpose.  Its modular-exponentiation no longer indexes a table with the bits of the exponent, and-hiding the exponent is what the difference buys.  What is left of the gap-against OpenSSL is the Montgomery multiplication, which is assembly there and-C here.+That licence is why any of this was possible.  The obvious assembly to reach+for is OpenSSL's and BoringSSL's `ecp_nistz256`, and it cannot be used here:+it is Apache-2.0 only, and Intel and CloudFlare hold copyright in it besides+OpenSSL, so nobody is in a position to relicense it. -*The AVX-512 instructions.*  Neither machine above has them.  On one that does--- an EPYC 9V74, measured the same way -- OpenSSL uses them for AES-GCM and-ChaCha20 and reaches 12003 and 3789 MB/s, against 4745 and 2372 for crypton,-whose vendored assembly is generated without them.  Those ratios are 0.40 and-0.63 rather than 0.97 and 1.00.  Nothing else in either table moves by more-than a few percent between the two processors.+crypton is behind in a few rows now: RSA verification on both architectures,+and RSA-3072 signing and ECDSA P-256 verification on x86-64.  The AES-GCM rows+were the other half of this section until 2.1.5; they are ahead on both+machines now, and what the instructions do is still worth setting out. +*RSA.*  2.0.0 made signing slower than 1.1.5 on purpose: its modular+exponentiation stopped indexing a table with the bits of the exponent, and+hiding the exponent is what the difference bought.  2.2.4 has repaid that+cost on both machines, by different means.  On x86-64 s2n-bignum's+Montgomery multiplication is twice the C's, because the C cannot form the two+carry chains `ADCX` and `ADOX` give, and 2.2.4 signs in less than 1.1.5+took, at 3072 bits as well as 2048.  On AArch64 s2n-bignum's routines measure+no better than the C, so the gain there is the C's own: the multiplication+and squaring for each key size are generated and unrolled, with the+reduction folded into each column as in Koç, Acar and Kaliski's FIPS method,+and signing is level with OpenSSL.  The measurements are in+[#275](https://github.com/kazu-yamamoto/crypton/issues/275).++On processors with AVX-512 IFMA, Intel Xeons from Ice Lake and AMD from Zen 4,+2.2.4 computes the two halves of the CRT together in 52-bit digits, Gueron and+Krasnov's method and the one OpenSSL takes there.  Neither machine above has+IFMA, so neither table shows it.++Verification's exponent is 65537, seventeen bits, so there is little+exponentiation to speed up and the work around it counts.  2.2.4 keeps R^2,+the Montgomery constant of a modulus, for the moduli it has seen, which is+most of what the RSA-2048 verification rows gained.++*The wide AES instructions.*  `VAES` and `VPCLMULQDQ` do two blocks where+`AES-NI` and `PCLMULQDQ` do one, and four in their 512-bit form.  crypton uses+the 256-bit form where the processor has it, which is Zen 3 and Ice Lake+onwards, and the 512-bit form where that is worth having, which is Ice Lake and+Zen 5 onwards.  There was nothing to borrow: the wide AES-GCM in OpenSSL,+BoringSSL and AWS-LC is Apache-2.0 and s2n-bignum has no GCM, so both files are+crypton's own.++Having the 256-bit one is where the 1.49 in the x86-64 table comes from, and+it is narrower than it sounds.  The EPYC 7763 is Zen 3: VAES and VPCLMULQDQ,+no AVX-512.  OpenSSL's x86-64 AES-GCM is `aesni-gcm-x86_64.pl`, which is+128-bit -- its `vaesenc`s are the VEX encoding of `AESENC` on `xmm`, and+there is not one `ymm` in the file -- or `aes-gcm-avx512.pl`, which wants+`AVX512VAES`.  There is no rung between them, so on this processor OpenSSL+takes a block at a time where crypton takes two.  The same idea as theirs,+one step further down the feature ladder; not a better one.++The 512-bit path arrived after 2.1.2, so it is in the 2.2.4 column -- but+neither machine in the tables above has AVX-512, so neither column shows it.+On the runners that do, measured over 16 KiB in MB/s: an EPYC 9V45 (Zen 5)+goes from 9616 to 14268 with it, a Xeon 6973P-C from 8095 to 9848, a Xeon+8573C from 6983 to 8447.  OpenSSL on those machines is ahead still -- 25760+on the first of them -- because it interleaves the GHASH with the AES where+crypton does them in turn.  Zen 4 keeps the 256-bit path: its 512-bit+instructions are two passes through a 256-bit datapath, so the wider encoding+buys nothing there and costs a little.++AArch64 has no counterpart to any of these: one AES block and one GHASH+multiplication at a time is all the instruction set offers.  Its AES-GCM+rows were 0.85 and 0.87 until 2.1.5, for that reason.  What closed it was+not width but the GHASH's representation -- H is twisted once at key setup+so that GCM's bit reflection is already undone, which turns a reduction of+some twenty-five shifts and XORs into two PMULL and six EOR and makes+Karatsuba worth taking.  The scheme is ARM's, from the BSD-3-Clause part of+[AArch64cryptolib](https://github.com/ARM-software/AArch64cryptolib),+written out in crypton's own intrinsics.  The AES there is ahead of+OpenSSL's and always was; it was the GHASH beside it that was behind.+ One row wants a word of its own: crypton's `Ed25519.sign` derives the public key from the secret key every time it signs, so that a caller who passes a public key that does not match cannot be made to leak the private one.  That-costs a second scalar multiplication, which OpenSSL's signing does not pay.+costs a second scalar multiplication, which OpenSSL's signing does not pay --+and the row is still 1.72 on AArch64 and 1.82 on x86-64, so the safety is had+for nothing here rather than paid for. +Ed25519 verification wanted one too, for an unhappier reason, and 2.2.1+settles it.  On x86-64 the row was about eight per cent slower from 2.1.8+onwards: 2.1.5, 2.1.6 and 2.1.7 all measure near 48 microseconds there,+2.1.8, 2.1.10 and 2.2.0 all measure near 52, and 2.2.1 is near 48 again.+Nothing in Ed25519 ever changed -- no commit touches `cbits/ed25519` across+any of it, and AArch64 never moved, the compiled `ed25519.o` there being+byte for byte the same object either side of 2.1.8.  What moved was a line+in this package's cabal file.  The ML-KEM and ML-DSA backends that 2.1.8+vendored want `-mavx2 -mbmi2`, and they were given as `cc-options` inside an+`if arch(x86_64)` block -- but cc-options in a conditional belong to the+component and not to the block, so every C file in the library was compiled+with AVX2 allowed, `cbits/ed25519` among them, and what the compiler made of+it that way is slower on this machine than what it replaced.  Signing, which+is the same file and a different routine, is where it always was.  2.2.1+narrowed the flags to the assembly that asked for them, which is also what+stopped a binary built on one x86-64 machine from dying on another, and the+row came back.+ SHA-1 is in the tables because a number of protocols and file formats still ask for it, not because it is a good choice for anything new.  The algorithms that nothing should ask for any more -- MD5, 3DES, RC4, CBC mode -- are left out.++ML-KEM and ML-DSA are left out too, though 2.1.8 brought both and they are+fast -- `mlkem-native` and `mldsa-native`, from the PQ Code Package.  No+comparison is reported here: the implementations to measure against are+still developing.
benchs/Bench.hs view
@@ -195,7 +195,9 @@   where     cp k (ini, plain) =         let iniState =-                throwCryptoError $ CP.initialize k (throwCryptoError $ CP.nonce12 nonce12)+                CP.initialize+                    (throwCryptoError $ CP.key k)+                    (throwCryptoError $ CP.nonce12 nonce12)             afterAAD = CP.finalizeAAD (CP.appendAAD ini iniState)             (out, afterEncrypt) = CP.encrypt plain afterAAD             outtag = CP.finalize afterEncrypt
+ cbits/LICENSE.go view
@@ -0,0 +1,38 @@+The arrangement of the AArch64 multiply-accumulate loop in+cbits/crypton_bignum.h -- four limbs to an iteration, the low halves of the+products and the high halves accumulated in two chains -- follows+addMulVVWx in Go's crypto/internal/fips140/bigmod/nat_arm64.s, written out+in the assembler that file's compiler speaks.  Go is at+https://github.com/golang/go and carries the licence below.  That file's+own header reads "Copyright 2013 The Go Authors. All rights reserved.  Use+of this source code is governed by a BSD-style license that can be found in+the LICENSE file", and the LICENSE file it means is this one.+++Copyright 2009 The Go Authors.++Redistribution and use in source and binary forms, with or without+modification, are permitted provided that the following conditions are+met:++   * Redistributions of source code must retain the above copyright+notice, this list of conditions and the following disclaimer.+   * Redistributions in binary form must reproduce the above+copyright notice, this list of conditions and the following disclaimer+in the documentation and/or other materials provided with the+distribution.+   * Neither the name of Google LLC nor the names of its+contributors may be used to endorse or promote products derived from+this software without specific prior written permission.++THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ cbits/aes/LICENSE.fusion view
@@ -0,0 +1,29 @@+Parts of cbits/aes/gcm_fused_x86.c follow the AES-GCM implementation in+picotls, lib/fusion.c, which is under the MIT license reproduced below.+The design is described by its author at++    http://blog.kazuhooku.com/2020/06/quicaes-gcm-12.html+    http://blog.kazuhooku.com/2020/06/quicaes-gcm-22.html++and the source is at https://github.com/h2o/picotls.+++Copyright (c) 2020-2022 Fastly, Kazuho Oku++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to+deal in the Software without restriction, including without limitation the+rights to use, copy, modify, merge, publish, distribute, sublicense, and/or+sell copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in+all copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS+IN THE SOFTWARE.
cbits/aes/armv8.c view
@@ -20,12 +20,9 @@ #include <stdint.h> #include <string.h> #include <arm_neon.h>-#if defined(__linux__)-#include <sys/auxv.h>-#include <asm/hwcap.h>-#endif #include "crypton_aes.h" #include "crypton_bitfn.h"+#include "crypton_cpu.h"  /*  * The AES and PMULL instructions are extensions, so a translation unit@@ -39,11 +36,7 @@  *  * "+crypto" rather than "crypto": GCC rejects the latter.  */-#ifdef WITH_TARGET_ATTRIBUTES-#define TARGET_ARMV8_CRYPTO __attribute__((target("+crypto")))-#else-#define TARGET_ARMV8_CRYPTO-#endif+#include "crypton_armv8_target.h"  /* forward round keys: nbr + 1 of them, written by the generic key expansion */ #define FWD(key)  ((const uint8_t *) (key)->data)@@ -70,14 +63,14 @@  * instructions exist to remove -- but a key schedule is the one thing an  * attacker most wants and it costs little to keep it out of the cache.  */-TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO static uint32x4_t sub_word(uint32x4_t w) { 	return vreinterpretq_u32_u8( 	    vaeseq_u8(vreinterpretq_u8_u32(w), vdupq_n_u8(0))); } -TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO static uint32x4_t sub_rot_word(uint32x4_t w) { 	const uint8x16_t s = vreinterpretq_u8_u32(sub_word(w));@@ -85,7 +78,7 @@ 	return vreinterpretq_u32_u8(vextq_u8(s, s, 1)); } -TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO void crypton_aes_armv8_init(aes_key *key, uint8_t *origkey, uint8_t size) { 	/* 2^0 .. 2^9 in GF(2^8), which is as far as any key size reaches */@@ -127,191 +120,194 @@ }  /*- * Whether the extensions are actually present.- *- * They are mandatory on Apple silicon, and on other AArch64 systems the- * kernel reports them through the auxiliary vector.  A system without them- * keeps the generic implementation.+ * Whether the extensions are actually present.  crypton_cpu.c asks the+ * system once, in the one place that knows how each system answers; a+ * processor without them keeps the generic implementation.  */ int crypton_aes_armv8_available(void) {-#if defined(__APPLE__)-	return 1;-#elif defined(__linux__)-	return (getauxval(AT_HWCAP) & HWCAP_AES) != 0;-#else-	return 0;-#endif+	return (crypton_arm_features() & CRYPTON_ARM_AES) != 0; }  /*  * GHASH using PMULL, the AArch64 counterpart to PCLMULQDQ.  *- * This is a transliteration of gfmul_pclmuldq in x86ni.c rather than a fresh- * formulation: that code is already pinned by the GCM known-answer tests, and- * every operation it uses has a direct NEON equivalent, so translating it is- * easier to check than reasoning about a new reduction from scratch.+ * Not the transliteration of gfmul_pclmuldq in x86ni.c this used to be.  The+ * x86 formulation keeps H in the order GCM writes it and pays, at the end of+ * every batch, a reduction that first has to undo GCM's bit reflection: some+ * twenty-five shifts and XORs, and a batch of eight costs it once.  *- *   _mm_shuffle_epi8 with a reversing mask  ->  vrev64q_u8 then vextq_u8- *   _mm_clmulepi64_si128                    ->  vmull_p64 / vmull_high_p64- *   _mm_slli_si128 / _mm_srli_si128         ->  vextq_u8 against zero- *   _mm_slli_epi32 / _mm_srli_epi32         ->  vshlq_n_u32 / vshrq_n_u32+ * Instead H is twisted once, at key setup, so that the reflection is already+ * undone and a reversed-polynomial multiply lands in the right place.  Two+ * things follow.  The reduction becomes two PMULL against 0xC2000..0 and six+ * EOR, a third of what it was.  And because nothing has to be byte-reversed+ * back and forth, Karatsuba pays: three PMULL a block rather than four, with+ * the middle terms accumulated in a third register and tidied up once per+ * batch.+ *+ * crypton tried Karatsuba in the old representation and measured it 1.6 per+ * cent slower -- the saved PMULL did not cover the extra EOR when the+ * reduction stayed as expensive as it was.  It is the pair that pays.+ * Measured over 16 KiB messages, this against the old code:+ *+ *                      Apple M4        Neoverse N2+ *     AES-128-GCM        1.30             1.25+ *     AES-192-GCM        1.22             1.25+ *     AES-256-GCM        1.19             1.24+ *+ * The scheme is ARM's, from the 'big' AES-GCM kernel of+ * https://github.com/ARM-software/AArch64cryptolib, which is BSD-3-Clause,+ * (c) 2018-2019 ARM Limited.  Their kernels under AArch64cryptolib_opt_bigger+ * are faster again and are NOT under that licence, whatever the repository's+ * LICENSE.md says; nothing here comes from those files.+ *+ * The table holds the twisted powers H^1 .. H^8 at htable[0 .. 7] and the+ * Karatsuba half of each -- its high 64 bits XOR its low -- in the first+ * eight bytes of htable[8 .. 15].  The running tag is kept the way GCM+ * writes it at every boundary, and swapped into the internal form on the way+ * in and out, which is two instructions.  */ -/* reverse all 16 bytes */-TARGET_ARMV8_CRYPTO-static inline uint8x16_t bswap128(uint8x16_t v)-{-	return vextq_u8(vrev64q_u8(v), vrev64q_u8(v), 8);-}--/* shift the whole register left by n bytes, as _mm_slli_si128 does */-#define SHIFT_LEFT_BYTES(v, n)  vextq_u8(vdupq_n_u8(0), (v), 16 - (n))-/* and right, as _mm_srli_si128 does */-#define SHIFT_RIGHT_BYTES(v, n) vextq_u8((v), vdupq_n_u8(0), (n))--#define SHL32(v, n) vreinterpretq_u8_u32(vshlq_n_u32(vreinterpretq_u32_u8(v), (n)))-#define SHR32(v, n) vreinterpretq_u8_u32(vshrq_n_u32(vreinterpretq_u32_u8(v), (n)))--TARGET_ARMV8_CRYPTO-static inline uint8x16_t clmul_ll(uint8x16_t a, uint8x16_t b)-{-	return vreinterpretq_u8_p128(vmull_p64(-	    (poly64_t) vgetq_lane_u64(vreinterpretq_u64_u8(a), 0),-	    (poly64_t) vgetq_lane_u64(vreinterpretq_u64_u8(b), 0)));-}+#define GHASH_MODC ((poly64_t) 0xC200000000000000ul) -TARGET_ARMV8_CRYPTO-static inline uint8x16_t clmul_lh(uint8x16_t a, uint8x16_t b)+/* the internal accumulator form, and back again: its own inverse */+CRYPTON_TARGET_ARMV8_CRYPTO+static inline uint8x16_t ghash_swap(uint8x16_t t) {-	return vreinterpretq_u8_p128(vmull_p64(-	    (poly64_t) vgetq_lane_u64(vreinterpretq_u64_u8(a), 0),-	    (poly64_t) vgetq_lane_u64(vreinterpretq_u64_u8(b), 1)));+	t = vrev64q_u8(t);+	return vextq_u8(t, t, 8); } -TARGET_ARMV8_CRYPTO-static inline uint8x16_t clmul_hl(uint8x16_t a, uint8x16_t b)+/* the high and low halves XORed together, which is what Karatsuba wants */+CRYPTON_TARGET_ARMV8_CRYPTO+static inline poly64_t ghash_karat(poly64x2_t v) {-	return vreinterpretq_u8_p128(vmull_p64(-	    (poly64_t) vgetq_lane_u64(vreinterpretq_u64_u8(a), 1),-	    (poly64_t) vgetq_lane_u64(vreinterpretq_u64_u8(b), 0)));+	return (poly64_t) veor_u64(vget_high_u64(vreinterpretq_u64_p64(v)),+	                           vget_low_u64(vreinterpretq_u64_p64(v))); } -TARGET_ARMV8_CRYPTO-static inline uint8x16_t clmul_hh(uint8x16_t a, uint8x16_t b)-{-	return vreinterpretq_u8_p128(vmull_high_p64(-	    vreinterpretq_p64_u8(a), vreinterpretq_p64_u8(b)));-}+/* the twisted power H^(i+1) */+#define GHASH_POW(ht, i)                                                      \+	vreinterpretq_p64_u8(vld1q_u8((const uint8_t *) &(ht)[i]))+/* and its Karatsuba half */+#define GHASH_KARAT(ht, i)                                                    \+	((poly64_t) vgetq_lane_u64(                                           \+	    vreinterpretq_u64_u8(vld1q_u8((const uint8_t *) &(ht)[8 + (i)])), 0))  /*- * The 256-bit carry-less product of a (normal byte order) and b (already- * reversed, as it sits in the table), before the reflection fixup and the- * reduction.  Split out from the reduction because both of those are linear- * over XOR: several products can be added together and fixed up just once,- * which is what gf_mul4 below does.+ * One block's three partial products, XORed into the accumulators.  b is+ * already in the internal form; hp and hk are the power it is to meet.  */-TARGET_ARMV8_CRYPTO-static inline void clmul_pmull(uint8x16_t a, uint8x16_t b,-                               uint8x16_t *lo, uint8x16_t *hi)-{-	uint8x16_t t3, t4, t5, t6;--	a = bswap128(a);--	t3 = clmul_ll(a, b);-	t4 = clmul_lh(a, b);-	t5 = clmul_hl(a, b);-	t6 = clmul_hh(a, b);--	t4 = veorq_u8(t4, t5);-	t5 = SHIFT_LEFT_BYTES(t4, 8);-	t4 = SHIFT_RIGHT_BYTES(t4, 8);--	*lo = veorq_u8(t3, t5);-	*hi = veorq_u8(t6, t4);-}+#define GHASH_MUL(b, hp, hk, H, M, L)                                         \+	do {                                                                  \+		poly64x2_t b__ = (b);                                         \+		poly64x2_t hp__ = (hp);                                       \+		(H) = veorq_u64((H), vreinterpretq_u64_p128(                  \+		    vmull_high_p64(b__, hp__)));                              \+		(L) = veorq_u64((L), vreinterpretq_u64_p128(vmull_p64(        \+		    (poly64_t) vgetq_lane_u64(vreinterpretq_u64_p64(b__), 0), \+		    (poly64_t) vgetq_lane_u64(vreinterpretq_u64_p64(hp__), 0)))); \+		(M) = veorq_u64((M), vreinterpretq_u64_p128(                  \+		    vmull_p64(ghash_karat(b__), (hk))));                      \+	} while (0) -/* Shift the 256-bit product left by one to undo GCM's bit reflection, then- * reduce modulo the GCM polynomial.  This is the expensive half. */-TARGET_ARMV8_CRYPTO-static inline uint8x16_t gfred_pmull(uint8x16_t t3, uint8x16_t t6)+/*+ * Finish the Karatsuba -- the middle accumulator still holds only the+ * (ah^al)(bh^bl) terms and wants the other two taken out of it -- and reduce+ * the 256 bits modulo the GCM polynomial.  The result is in internal form.+ */+CRYPTON_TARGET_ARMV8_CRYPTO+static inline uint64x2_t ghash_reduce(uint64x2_t H, uint64x2_t M, uint64x2_t L) {-	uint8x16_t t2, t4, t5, t7, t8, t9;--	t7 = SHR32(t3, 31);-	t8 = SHR32(t6, 31);-	t3 = SHL32(t3, 1);-	t6 = SHL32(t6, 1);--	t9 = SHIFT_RIGHT_BYTES(t7, 12);-	t8 = SHIFT_LEFT_BYTES(t8, 4);-	t7 = SHIFT_LEFT_BYTES(t7, 4);-	t3 = vorrq_u8(t3, t7);-	t6 = vorrq_u8(t6, t8);-	t6 = vorrq_u8(t6, t9);--	t7 = SHL32(t3, 31);-	t8 = SHL32(t3, 30);-	t9 = SHL32(t3, 25);+	uint64x2_t t; -	t7 = veorq_u8(t7, t8);-	t7 = veorq_u8(t7, t9);-	t8 = SHIFT_RIGHT_BYTES(t7, 4);-	t7 = SHIFT_LEFT_BYTES(t7, 12);-	t3 = veorq_u8(t3, t7);+	M = veorq_u64(M, H);+	M = veorq_u64(M, L); -	t2 = SHR32(t3, 1);-	t4 = SHR32(t3, 2);-	t5 = SHR32(t3, 7);-	t2 = veorq_u8(t2, t4);-	t2 = veorq_u8(t2, t5);-	t2 = veorq_u8(t2, t8);-	t3 = veorq_u8(t3, t2);-	t6 = veorq_u8(t6, t3);+	t = vreinterpretq_u64_p128(vmull_p64(+	    (poly64_t) vgetq_lane_u64(H, 0), GHASH_MODC));+	H = vreinterpretq_u64_u8(vextq_u8(vreinterpretq_u8_u64(H),+	                                  vreinterpretq_u8_u64(H), 8));+	M = veorq_u64(M, t);+	M = veorq_u64(M, H); -	return bswap128(t6);+	t = vreinterpretq_u64_p128(vmull_p64(+	    (poly64_t) vgetq_lane_u64(M, 0), GHASH_MODC));+	M = vreinterpretq_u64_u8(vextq_u8(vreinterpretq_u8_u64(M),+	                                  vreinterpretq_u8_u64(M), 8));+	L = veorq_u64(L, t);+	return veorq_u64(L, M); } -TARGET_ARMV8_CRYPTO-static uint8x16_t gfmul_pmull(uint8x16_t a, const uint8_t *htable)+/* a single block against H^1, accumulator in internal form */+CRYPTON_TARGET_ARMV8_CRYPTO+static inline uint64x2_t ghash_one(uint64x2_t acc, uint8x16_t blk,+                                   const block128 *ht) {-	uint8x16_t lo, hi;+	uint64x2_t H = vdupq_n_u64(0), M = H, L = H;+	poly64x2_t b; -	clmul_pmull(a, vld1q_u8(htable), &lo, &hi);-	return gfred_pmull(lo, hi);+	acc = vreinterpretq_u64_u8(vextq_u8(vreinterpretq_u8_u64(acc),+	                                    vreinterpretq_u8_u64(acc), 8));+	b = vreinterpretq_p64_u64(veorq_u64(+	    vreinterpretq_u64_u8(vrev64q_u8(blk)), acc));+	GHASH_MUL(b, GHASH_POW(ht, 0), GHASH_KARAT(ht, 0), H, M, L);+	return ghash_reduce(H, M, L); }  /*- * With PMULL there is no 4-bit table to fill: H goes in at index 0, byte- * reversed, so that gfmul_pmull does not have to swap it every time.  This- * mirrors crypton_aesni_hinit_pclmul.+ * Twist H and raise it to the powers a batch needs.  *- * Indices 1..7 get H^2 .. H^8, which is what lets a group of blocks fold- * into one reduction: gf_mul4 uses the first four, the GCM loop all eight.- * The table has sixteen slots, so they are free.+ * The twist is a shift left by one with 0xC2000..01 folded back in when a+ * bit falls off the top -- the same correction the old reduction applied to+ * every product, done once here instead.  Each further power is one multiply+ * in the twisted domain; the result comes out of ghash_reduce with its+ * halves swapped, which a batch undoes on the way in, so a stored power has+ * to be swapped back.  */-TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO void crypton_aes_armv8_hinit_pmull(block128 *htable, const block128 *h) {-	uint8x16_t p;+	uint8x16_t hk = vrev64q_u8(vld1q_u8((const uint8_t *) h));+	uint64x2_t shl = vshlq_n_u64(vreinterpretq_u64_u8(hk), 1);+	uint64x2_t shr = vreinterpretq_u64_s64(+	    vshrq_n_s64(vreinterpretq_s64_u8(hk), 63));+	uint8x16_t mask = vextq_u8(vreinterpretq_u8_u64(shr),+	                           vreinterpretq_u8_u64(shr), 12);+	uint64x2_t tc = vdupq_n_u64(0);+	poly64x2_t base, p; 	int i; -	htable[0].q[0] = bitfn_swap64(h->q[1]);-	htable[0].q[1] = bitfn_swap64(h->q[0]);+	tc = vsetq_lane_u64(0xC200000000000001ul, tc, 0);+	tc = vsetq_lane_u64(1, tc, 1);+	tc = vandq_u64(vreinterpretq_u64_u8(mask), tc);+	base = vreinterpretq_p64_u64(veorq_u64(tc, shl)); -	p = vld1q_u8((const uint8_t *) h);-	for (i = 1; i < 8; i++) {-		p = gfmul_pmull(p, (const uint8_t *) &htable[0]);-		vst1q_u8((uint8_t *) &htable[i], bswap128(p));+	p = base;+	for (i = 0; i < 8; i++) {+		uint64x2_t H = vdupq_n_u64(0), M = H, L = H, r;++		vst1q_u8((uint8_t *) &htable[i],+		         vreinterpretq_u8_p64(p));+		vst1q_u8((uint8_t *) &htable[8 + i],+		         vreinterpretq_u8_u64(+		             vdupq_n_u64((uint64_t) ghash_karat(p))));++		GHASH_MUL(p, base, ghash_karat(base), H, M, L);+		r = ghash_reduce(H, M, L);+		p = vreinterpretq_p64_u8(vextq_u8(vreinterpretq_u8_u64(r),+		                                  vreinterpretq_u8_u64(r), 8)); 	} } -TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO void crypton_aes_armv8_gf_mul_pmull(block128 *a, const block128 *htable) {-	vst1q_u8((uint8_t *) a,-	         gfmul_pmull(vld1q_u8((const uint8_t *) a), (const uint8_t *) htable));+	uint64x2_t acc = vreinterpretq_u64_u8(+	    ghash_swap(vld1q_u8((const uint8_t *) a)));++	acc = ghash_one(acc, vdupq_n_u8(0), htable);+	vst1q_u8((uint8_t *) a, ghash_swap(vreinterpretq_u8_u64(acc))); }  /*@@ -320,36 +316,38 @@  * four products can be summed first and reduced once, which is where the  * time goes.  Aggregated reduction, from the Intel GCM paper.  */-TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO void crypton_aes_armv8_gf_mul4_pmull(block128 *a, const block128 *blocks,                                      const block128 *htable) {-	uint8x16_t lo, hi, l, h;+	uint64x2_t acc = vreinterpretq_u64_u8(+	    ghash_swap(vld1q_u8((const uint8_t *) a)));+	uint64x2_t H = vdupq_n_u64(0), M = H, L = H;+	poly64x2_t b; 	int i; -	clmul_pmull(veorq_u8(vld1q_u8((const uint8_t *) a),-	                     vld1q_u8((const uint8_t *) &blocks[0])),-	            vld1q_u8((const uint8_t *) &htable[3]), &lo, &hi);+	acc = vreinterpretq_u64_u8(vextq_u8(vreinterpretq_u8_u64(acc),+	                                    vreinterpretq_u8_u64(acc), 8));+	b = vreinterpretq_p64_u64(veorq_u64(+	    vreinterpretq_u64_u8(vrev64q_u8(+	        vld1q_u8((const uint8_t *) &blocks[0]))), acc));+	GHASH_MUL(b, GHASH_POW(htable, 3), GHASH_KARAT(htable, 3), H, M, L);  	for (i = 1; i < 4; i++) {-		clmul_pmull(vld1q_u8((const uint8_t *) &blocks[i]),-		            vld1q_u8((const uint8_t *) &htable[3 - i]), &l, &h);-		lo = veorq_u8(lo, l);-		hi = veorq_u8(hi, h);+		b = vreinterpretq_p64_u8(vrev64q_u8(+		    vld1q_u8((const uint8_t *) &blocks[i])));+		GHASH_MUL(b, GHASH_POW(htable, 3 - i),+		          GHASH_KARAT(htable, 3 - i), H, M, L); 	} -	vst1q_u8((uint8_t *) a, gfred_pmull(lo, hi));+	acc = ghash_reduce(H, M, L);+	vst1q_u8((uint8_t *) a, ghash_swap(vreinterpretq_u8_u64(acc))); } + int crypton_aes_armv8_pmull_available(void) {-#if defined(__APPLE__)-	return 1;-#elif defined(__linux__)-	return (getauxval(AT_HWCAP) & HWCAP_PMULL) != 0;-#else-	return 0;-#endif+	return (crypton_arm_features() & CRYPTON_ARM_PMULL) != 0; }  /*@@ -359,7 +357,7 @@  * the high half, and fold the bit that leaves the top back in as 0x87.  The  * block is little-endian, so lane 0 is the low half.  */-TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO static inline uint8x16_t gfmulx_neon(uint8x16_t v) { 	const uint64x2_t x = vreinterpretq_u64_u8(v);@@ -396,3 +394,59 @@ #include <aes/armv8_impl.c> #undef SIZED #undef NBR++/*+ * The fused entry point, over the three key sizes.  Each was generated with+ * its round count fixed, which is what lets the eight chains stay in+ * registers; the choice between them is made once per message here.+ */+CRYPTON_TARGET_ARMV8_CRYPTO+void crypton_aes_armv8_gcm_fused(uint8_t *out, const block128 *ht,+                                 aes_key *key, const uint8_t *nonce,+                                 const uint8_t *aad, uint32_t aadlen,+                                 const uint8_t *in, uint32_t inlen,+                                 uint32_t taglen, aes_key *hpkey,+                                 uint32_t sampleoff, uint8_t *mask)+{+	switch (key->strength) {+	case 0:+		crypton_aes_armv8_gcm_fused128(out, ht, key, nonce, aad, aadlen,+		                               in, inlen, taglen, hpkey,+		                               sampleoff, mask);+		break;+	case 1:+		crypton_aes_armv8_gcm_fused192(out, ht, key, nonce, aad, aadlen,+		                               in, inlen, taglen, hpkey,+		                               sampleoff, mask);+		break;+	default:+		crypton_aes_armv8_gcm_fused256(out, ht, key, nonce, aad, aadlen,+		                               in, inlen, taglen, hpkey,+		                               sampleoff, mask);+		break;+	}+}++CRYPTON_TARGET_ARMV8_CRYPTO+int crypton_aes_armv8_gcm_fused_dec(uint8_t *out, const block128 *ht,+                                    aes_key *key, const uint8_t *nonce,+                                    const uint8_t *aad, uint32_t aadlen,+                                    const uint8_t *in, uint32_t inlen,+                                    const uint8_t *tag, uint32_t taglen,+                                    uint8_t *outtag)+{+	switch (key->strength) {+	case 0:+		return crypton_aes_armv8_gcm_fused_dec128(out, ht, key, nonce,+		                                          aad, aadlen, in, inlen,+		                                          tag, taglen, outtag);+	case 1:+		return crypton_aes_armv8_gcm_fused_dec192(out, ht, key, nonce,+		                                          aad, aadlen, in, inlen,+		                                          tag, taglen, outtag);+	default:+		return crypton_aes_armv8_gcm_fused_dec256(out, ht, key, nonce,+		                                          aad, aadlen, in, inlen,+		                                          tag, taglen, outtag);+	}+}
cbits/aes/armv8_impl.c view
@@ -25,8 +25,6 @@  #define EACH1(m) m(0) #define EACH8(m) m(0) m(1) m(2) m(3) m(4) m(5) m(6) m(7)-/* the blocks after the first; GHASH folds block 0 in with the tag */-#define EACH7(m) m(1) m(2) m(3) m(4) m(5) m(6) m(7)  #define LOAD_IN(i)   s[i] = vld1q_u8((const uint8_t *) (input + (i))); #define STORE_OUT(i) vst1q_u8((uint8_t *) (output + (i)), s[i]);@@ -74,7 +72,7 @@ 		}                                                            \ 	} while (0) -TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_encrypt_block)(aes_block *output, aes_key *key, aes_block *input) { 	const uint8_t *rk = FWD(key);@@ -85,7 +83,7 @@ 	EACH1(STORE_OUT); } -TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_decrypt_block)(aes_block *output, aes_key *key, aes_block *input) { 	const uint8_t *fwd = FWD(key);@@ -97,7 +95,7 @@ 	EACH1(STORE_OUT); } -TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_encrypt_ecb)(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks) { 	const uint8_t *rk = FWD(key);@@ -115,7 +113,7 @@ 	} } -TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_decrypt_ecb)(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks) { 	const uint8_t *fwd = FWD(key);@@ -136,7 +134,7 @@  /* CBC encryption chains, so there is nothing to interleave.  It still gains  * the round keys staying put. */-TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_encrypt_cbc)(aes_block *output, aes_key *key, aes_block *_iv, aes_block *input, uint32_t nb_blocks) { 	const uint8_t *rk = FWD(key);@@ -158,7 +156,7 @@ #define CBC_KEEP(i)  c[(i) + 1] = s[i]; #define CBC_XOR(i)   vst1q_u8((uint8_t *) (output + (i)), veorq_u8(s[i], c[i])); -TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_decrypt_cbc)(aes_block *output, aes_key *key, aes_block *_iv, aes_block *input, uint32_t nb_blocks) { 	const uint8_t *fwd = FWD(key);@@ -192,7 +190,7 @@ #define CTR_XOR(i)  vst1q_u8(output + 16 * (i), \                              veorq_u8(s[i], vld1q_u8(input + 16 * (i)))); -TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_encrypt_ctr)(uint8_t *output, aes_key *key, aes_block *iv, uint8_t *input, uint32_t len) { 	const uint8_t *rk = FWD(key);@@ -262,6 +260,47 @@  *  * GCM's counter is the low 32 bits only and wraps there, so unlike CTR  * there is no carry to chase: the top twelve bytes never move.+ *+ * What this loop is short of is not overlap but instructions.  A group of+ * eight blocks compiles to 383 of them on an Apple M4, and only 152 are the+ * cipher -- eighty AESE and seventy-two AESMC.  The rest is the GHASH and+ * the counters.  Measured against OpenSSL on the same machine, 16 KiB+ * messages under AES-128:+ *+ *     crypton, this loop with the GHASH taken out     19883 MB/s+ *     OpenSSL, AES-128-CTR                            17176+ *     OpenSSL, AES-128-GCM                             9869+ *     crypton, AES-128-GCM                             8641+ *+ * The cipher here is ahead of OpenSSL's; all of the 0.87 is the GHASH.+ * Four ways of closing it were tried and every one measured worse, so they+ * are written down here rather than tried again.  Each was checked to give+ * the same ciphertext and tag as this code for three key sizes and+ * thirty-five lengths, encrypt and decrypt, before being timed:+ *+ *     the GHASH held back a group and spread through the next    -1.3%+ *       group's AES rounds, so that the two do not queue         to -3.6%+ *     Karatsuba: three multiplications for the two halves        -1.6%+ *       instead of four+ *     the same with every product on a lane the instruction       0.0%+ *       reaches, which does remove sixteen fmov a group+ *     the same again with the H powers' halves added together    -3%+ *       already, in the spare half of htable+ *+ * The first fails because there was nothing to gain: a group's AES depends+ * on nothing in the group before it, so a wide out-of-order core already+ * runs the two together, and holding a group back only adds copies.  The+ * rest fail for one reason -- none of them makes the loop shorter.+ * Karatsuba buys a PMULL for two EOR and an EXT, and the folded table turns+ * sixteen `dup` into sixteen `ld1r` and sixteen more address adds.+ *+ * That last sentence used to end by saying a count which does come down+ * wants the data laid out differently.  It does, and since the GHASH was+ * rewritten against a twisted H -- see cbits/aes/armv8.c -- it is laid out+ * differently, so the figures above are what the *previous* GHASH gave.+ * Karatsuba pays now that the reduction it has to carry is a third of what+ * it was; the other three are untried in the new representation and the+ * first of them has no more reason to work than it had.  */ #define GCM_CTR(i)   s[i] = vreinterpretq_u8_u32(vsetq_lane_u32(cpu_to_be32(c + 1 + (i)), base, 3)); #define GCM_ENC(i)   { const uint8x16_t m_ = vld1q_u8(input + 16 * (i)); \@@ -270,27 +309,33 @@ #define GCM_DEC(i)   { const uint8x16_t m_ = vld1q_u8(input + 16 * (i)); \                        vst1q_u8(output + 16 * (i), veorq_u8(s[i], m_)); \                        s[i] = m_; }-#define GCM_GHASH(i) { uint8x16_t l_, h_; \-                       clmul_pmull(s[i], vld1q_u8((const uint8_t *) &ht[WAY - 1 - (i)]), \-                                   &l_, &h_); \-                       glo = veorq_u8(glo, l_); ghi = veorq_u8(ghi, h_); }+#define GCM_GHASH(i)                                                          \+	{                                                                     \+		poly64x2_t b_ = vreinterpretq_p64_u8(vrev64q_u8(s[i]));       \+		if ((i) == 0)                                                 \+			b_ = vreinterpretq_p64_u64(veorq_u64(                 \+			    vreinterpretq_u64_p64(b_), acc));                 \+		GHASH_MUL(b_, GHASH_POW(ht, WAY - 1 - (i)),                   \+		          GHASH_KARAT(ht, WAY - 1 - (i)), gH, gM, gL);        \+	}  /* the eight blocks now in s[] are the ciphertext; fold them into the tag */ #define GCM_FOLD()                                                            \ 	do {                                                                  \-		uint8x16_t glo, ghi;                                          \-		clmul_pmull(veorq_u8(tag, s[0]),                              \-		            vld1q_u8((const uint8_t *) &ht[WAY - 1]),         \-		            &glo, &ghi);                                      \-		EACH7(GCM_GHASH)                                              \-		tag = gfred_pmull(glo, ghi);                                  \+		uint64x2_t gH = vdupq_n_u64(0), gM = gH, gL = gH;             \+		acc = vreinterpretq_u64_u8(                                   \+		    vextq_u8(vreinterpretq_u8_u64(acc),                       \+		             vreinterpretq_u8_u64(acc), 8));                  \+		EACH8(GCM_GHASH)                                              \+		acc = ghash_reduce(gH, gM, gL);                               \ 	} while (0)  #define GCM_PROLOGUE                                                          \ 	const uint8_t *rk = FWD(key);                                         \ 	const block128 *ht = gcm->htable;                                     \ 	uint8x16_t s[WAY];                                                    \-	uint8x16_t tag = vld1q_u8((const uint8_t *) &gcm->tag);               \+	uint64x2_t acc = vreinterpretq_u64_u8(                                \+	    ghash_swap(vld1q_u8((const uint8_t *) &gcm->tag)));               \ 	uint32_t c = be32_to_cpu(gcm->civ.d[3]);                              \ 	uint32x4_t base = vreinterpretq_u32_u8(vld1q_u8((const uint8_t *) &gcm->civ)) @@ -303,16 +348,17 @@ 		ENC_ROUNDS(EACH1);                                            \ 		s[0] = veorq_u8(s[0], m_);                                    \ 		(store_c);                                                    \-		tag = gfmul_pmull(veorq_u8(tag, (ghash_of)), (const uint8_t *) ht); \+		acc = ghash_one(acc, (ghash_of), ht);                         \ 	} while (0)  #define GCM_EPILOGUE                                                          \ 	do {                                                                  \ 		gcm->civ.d[3] = cpu_to_be32(c);                               \-		vst1q_u8((uint8_t *) &gcm->tag, tag);                         \+		vst1q_u8((uint8_t *) &gcm->tag,                               \+		         ghash_swap(vreinterpretq_u8_u64(acc)));              \ 	} while (0) -TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_gcm_encrypt)(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length) { 	GCM_PROLOGUE;@@ -343,13 +389,12 @@ 		block128_zero(&m); 		for (i = 0; i < length; i++) 			output[i] = m.b[i] = o.b[i];-		tag = gfmul_pmull(veorq_u8(tag, vld1q_u8((const uint8_t *) &m)),-		                  (const uint8_t *) ht);+		acc = ghash_one(acc, vld1q_u8((const uint8_t *) &m), ht); 	} 	GCM_EPILOGUE; } -TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_gcm_decrypt)(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length) { 	GCM_PROLOGUE;@@ -381,8 +426,7 @@ 		vst1q_u8((uint8_t *) &o, s[0]); 		for (i = 0; i < length; i++) 			output[i] = o.b[i];-		tag = gfmul_pmull(veorq_u8(tag, vld1q_u8((const uint8_t *) &m)),-		                  (const uint8_t *) ht);+		acc = ghash_one(acc, vld1q_u8((const uint8_t *) &m), ht); 	} 	GCM_EPILOGUE; }@@ -433,7 +477,7 @@ } while (0); #define XTS_TWEAK_ROLL(i) do { t[i] = tn[i]; } while (0); -TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_encrypt_xts)(aes_block *output, aes_key *key, aes_key *key2, aes_block *dataunit, uint32_t spoint, aes_block *input, uint32_t nb_blocks) { 	const uint8_t *rk = FWD(key);@@ -477,7 +521,7 @@ 	} } -TARGET_ARMV8_CRYPTO+CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_decrypt_xts)(aes_block *output, aes_key *key, aes_key *key2, aes_block *dataunit, uint32_t spoint, aes_block *input, uint32_t nb_blocks) { 	const uint8_t *fwd = FWD(key);@@ -523,9 +567,237 @@ 	} } +/*+ * One message, one call: the additional data, the counter-mode encryption,+ * the tag and the QUIC header protection mask, with the running tag and the+ * counter kept in registers from end to end.+ *+ * What this saves over composing crypton_aes_gcm_aad, _encrypt and _finish is+ * not the arithmetic but the boundaries.  Each of those reaches its+ * primitives through a branch table, so the 128-bit state goes back to memory+ * at every step and a header of one block pays a reduction of its own.  On an+ * Apple M4 that framing was 0.07 of the 0.112 microseconds a 100-byte packet+ * cost -- more than the encryption of the packet itself.+ *+ * The GHASH is taken in batches of WAY against the powers of H the key+ * already holds, so a batch costs one reduction rather than one per block,+ * and the additional data and the length block ride in the same batches as+ * the ciphertext instead of being multiplied on their own.+ */++/* start a batch, or continue one; blen is how many blocks this batch holds */+#define FG_ABSORB(blk)                                                        \+	do {                                                                  \+		poly64x2_t b_ = vreinterpretq_p64_u8(vrev64q_u8(blk));        \+		if (bn == 0) {                                                \+			uint32_t left_ = gtotal - gidx;                       \+			blen = left_ < WAY ? left_ : WAY;                     \+			acc = vreinterpretq_u64_u8(                           \+			    vextq_u8(vreinterpretq_u8_u64(acc),               \+			             vreinterpretq_u8_u64(acc), 8));          \+			b_ = vreinterpretq_p64_u64(veorq_u64(                 \+			    vreinterpretq_u64_p64(b_), acc));                 \+			gH = vdupq_n_u64(0);                                  \+			gM = gH;                                              \+			gL = gH;                                              \+		}                                                             \+		GHASH_MUL(b_, GHASH_POW(ht, blen - bn - 1),                   \+		          GHASH_KARAT(ht, blen - bn - 1), gH, gM, gL);        \+		gidx++; bn++;                                                 \+		if (bn == blen) { acc = ghash_reduce(gH, gM, gL); bn = 0; }   \+	} while (0)++/* a block that is short, zero padded, as GHASH wants it */+#define FG_PARTIAL(p, n)                                                      \+	({ uint8_t buf_[16]; memset(buf_, 0, 16); memcpy(buf_, (p), (n));     \+	   vld1q_u8(buf_); })++CRYPTON_TARGET_ARMV8_CRYPTO+void SIZED(crypton_aes_armv8_gcm_fused)(uint8_t *out, const block128 *ht,+                                        aes_key *key, const uint8_t *nonce,+                                        const uint8_t *aad, uint32_t aadlen,+                                        const uint8_t *in, uint32_t inlen,+                                        uint32_t taglen, aes_key *hpkey,+                                        uint32_t sampleoff, uint8_t *mask)+{+	const uint8_t *rk = FWD(key);+	uint8x16_t s[WAY];+	uint8x16_t ek0;+	uint64x2_t acc = vdupq_n_u64(0), gH = acc, gM = acc, gL = acc;+	uint32x4_t base;+	uint32_t c = 1, bn = 0, blen = 0, gidx = 0;+	uint32_t gtotal = (aadlen + 15) / 16 + (inlen + 15) / 16 + 1;+	uint32_t i, done;+	uint8_t y0[16], lenb[16];+	uint64_t la, lc;++	memcpy(y0, nonce, 12);+	y0[12] = 0; y0[13] = 0; y0[14] = 0; y0[15] = 1;+	base = vreinterpretq_u32_u8(vld1q_u8(y0));++	s[0] = vld1q_u8(y0);+	ENC_ROUNDS(EACH1);+	ek0 = s[0];++	for (i = 0; i + 16 <= aadlen; i += 16)+		FG_ABSORB(vld1q_u8(aad + i));+	if (i < aadlen)+		FG_ABSORB(FG_PARTIAL(aad + i, aadlen - i));++	for (done = 0; done + 16 * WAY <= inlen; done += 16 * WAY) {+		const uint8_t *p = in + done;+		uint8_t *q = out + done;++		EACH8(GCM_CTR);+		c += WAY;+		ENC_ROUNDS(EACH8);+		{+			const uint8_t *input = p;+			uint8_t *output = q;+			EACH8(GCM_ENC);+		}+		FG_ABSORB(s[0]); FG_ABSORB(s[1]); FG_ABSORB(s[2]); FG_ABSORB(s[3]);+		FG_ABSORB(s[4]); FG_ABSORB(s[5]); FG_ABSORB(s[6]); FG_ABSORB(s[7]);+	}++	for (; done < inlen; done += 16) {+		uint32_t n = inlen - done < 16 ? inlen - done : 16;+		uint8x16_t m_ = n == 16 ? vld1q_u8(in + done)+		                        : FG_PARTIAL(in + done, n);+		c++;+		s[0] = vreinterpretq_u8_u32(vsetq_lane_u32(cpu_to_be32(c), base, 3));+		ENC_ROUNDS(EACH1);+		s[0] = veorq_u8(s[0], m_);+		if (n == 16) {+			vst1q_u8(out + done, s[0]);+		} else {+			uint8_t buf_[16];+			vst1q_u8(buf_, s[0]);+			memcpy(out + done, buf_, n);+			memset(buf_ + n, 0, 16 - n);+			s[0] = vld1q_u8(buf_);+		}+		FG_ABSORB(s[0]);+	}++	la = (uint64_t) aadlen << 3;+	lc = (uint64_t) inlen << 3;+	for (i = 0; i < 8; i++) lenb[i] = (uint8_t) (la >> (56 - 8 * i));+	for (i = 0; i < 8; i++) lenb[8 + i] = (uint8_t) (lc >> (56 - 8 * i));+	FG_ABSORB(vld1q_u8(lenb));++	{+		uint8_t tbuf[16];+		vst1q_u8(tbuf, veorq_u8(ghash_swap(vreinterpretq_u8_u64(acc)), ek0));+		memcpy(out + inlen, tbuf, taglen);+	}++	if (hpkey != 0 && mask != 0) {+		block128 sample, m;+		memcpy(&sample, out + sampleoff, 16);+		crypton_aes_encrypt_ecb(&m, hpkey, &sample, 1);+		memcpy(mask, &m, 16);+	}+}+++/*+ * The same for decryption.  GCM_DEC leaves the ciphertext in s[] once it has+ * written the plaintext out, which is what GHASH wants, so the only other+ * difference is the end: the tag is compared here rather than written, every+ * byte of it whichever way the answer goes.+ */+CRYPTON_TARGET_ARMV8_CRYPTO+int SIZED(crypton_aes_armv8_gcm_fused_dec)(uint8_t *out, const block128 *ht,+                                           aes_key *key, const uint8_t *nonce,+                                           const uint8_t *aad, uint32_t aadlen,+                                           const uint8_t *in, uint32_t inlen,+                                           const uint8_t *tagp, uint32_t taglen,+                                           uint8_t *outtag)+{+	const uint8_t *rk = FWD(key);+	uint8x16_t s[WAY];+	uint8x16_t ek0;+	uint64x2_t acc = vdupq_n_u64(0), gH = acc, gM = acc, gL = acc;+	uint32x4_t base;+	uint32_t c = 1, bn = 0, blen = 0, gidx = 0;+	uint32_t gtotal = (aadlen + 15) / 16 + (inlen + 15) / 16 + 1;+	uint32_t i, done;+	uint8_t y0[16], lenb[16], want[16];+	uint64_t la, lc;+	uint8_t diff = 0;++	memcpy(y0, nonce, 12);+	y0[12] = 0; y0[13] = 0; y0[14] = 0; y0[15] = 1;+	base = vreinterpretq_u32_u8(vld1q_u8(y0));++	s[0] = vld1q_u8(y0);+	ENC_ROUNDS(EACH1);+	ek0 = s[0];++	for (i = 0; i + 16 <= aadlen; i += 16)+		FG_ABSORB(vld1q_u8(aad + i));+	if (i < aadlen)+		FG_ABSORB(FG_PARTIAL(aad + i, aadlen - i));++	for (done = 0; done + 16 * WAY <= inlen; done += 16 * WAY) {+		const uint8_t *p = in + done;+		uint8_t *q = out + done;++		EACH8(GCM_CTR);+		c += WAY;+		ENC_ROUNDS(EACH8);+		{+			const uint8_t *input = p;+			uint8_t *output = q;+			EACH8(GCM_DEC);+		}+		FG_ABSORB(s[0]); FG_ABSORB(s[1]); FG_ABSORB(s[2]); FG_ABSORB(s[3]);+		FG_ABSORB(s[4]); FG_ABSORB(s[5]); FG_ABSORB(s[6]); FG_ABSORB(s[7]);+	}++	for (; done < inlen; done += 16) {+		uint32_t n = inlen - done < 16 ? inlen - done : 16;+		uint8x16_t m_ = n == 16 ? vld1q_u8(in + done)+		                        : FG_PARTIAL(in + done, n);+		c++;+		s[0] = vreinterpretq_u8_u32(vsetq_lane_u32(cpu_to_be32(c), base, 3));+		ENC_ROUNDS(EACH1);+		{+			uint8x16_t pl = veorq_u8(s[0], m_);+			if (n == 16) {+				vst1q_u8(out + done, pl);+			} else {+				uint8_t buf_[16];+				vst1q_u8(buf_, pl);+				memcpy(out + done, buf_, n);+			}+		}+		FG_ABSORB(m_);+	}++	la = (uint64_t) aadlen << 3;+	lc = (uint64_t) inlen << 3;+	for (i = 0; i < 8; i++) lenb[i] = (uint8_t) (la >> (56 - 8 * i));+	for (i = 0; i < 8; i++) lenb[8 + i] = (uint8_t) (lc >> (56 - 8 * i));+	FG_ABSORB(vld1q_u8(lenb));++	vst1q_u8(want, veorq_u8(ghash_swap(vreinterpretq_u8_u64(acc)), ek0));+	if (outtag) {+		/* The caller holds the expected tag and will compare it itself. */+		memcpy(outtag, want, taglen);+		return 1;+	}+	for (i = 0; i < taglen; i++)+		diff |= (uint8_t) (want[i] ^ tagp[i]);+	return diff == 0;+}++#undef FG_ABSORB+#undef FG_PARTIAL+ #undef WAY #undef EACH1-#undef EACH7 #undef EACH8 #undef LOAD_IN #undef STORE_OUT
cbits/aes/block128.h view
@@ -34,7 +34,21 @@ #include <crypton_bitfn.h> #include <crypton_align.h> -typedef union {+/* Packed, so that the union asks nothing of the address it is at.+ *+ * Several callers here make one of these out of a pointer of their own -- a+ * ciphertext, a tag, a nonce -- and without this that cast produces a pointer+ * the standard says may not exist, and reading q[] out of it is a member+ * access at an address the type is not aligned for.  crypton_align.h used to+ * answer that with need_alignment, which is zero on i386 and x86-64: the two+ * places the access is architecturally fine and the standard still says+ * nothing about it.  UndefinedBehaviorSanitizer reported it.+ *+ * With the alignment declared to be one, the compiler is the one that knows+ * what the target can do: on i386, x86-64 and AArch64 it emits the same load+ * and store it emitted before, and where a target cannot read a word off an+ * odd address it emits what that target needs. */+typedef union __attribute__((packed)) {        uint64_t q[2];        uint32_t d[4];        uint16_t w[8];
+ cbits/aes/gcm_fused_x86.c view
@@ -0,0 +1,1013 @@+/*+ * A fused AES-GCM for x86-64, following the design Kazuho Oku sets out in+ * "QUICむけにAES-GCM実装を最適化した話" and implements in picotls's+ * lib/fusion.c: keep AES-NI issuing every clock and fit everything else --+ * the additional data, the tag, the QUIC header protection mask -- into the+ * gaps it leaves.  Written in C with intrinsics rather than assembly, for+ * the same reason he gives: the scheduling is what is complicated here, and+ * it has to stay readable to stay correct.+ *+ * Parts of this file follow fusion closely enough to say so: `loadn` and the+ * two tables it reads, `loadn_page_end` and `storen` are its `loadn128`,+ * `loadn_end_of_page` and `storen128` in another spelling, and the+ * reduction of a 256-bit product is the sequence fusion takes from Gueron's+ * "AES-GCM for Efficient Authenticated Encryption".  fusion is under the MIT+ * license, which is in cbits/aes/LICENSE.fusion beside this file.+ *+ * The powers of H are built once per key, so the additional data, the+ * ciphertext and the length block are absorbed against them in batches that+ * share one reduction, rather than each block paying for a reduction of its+ * own.  How many powers, and so how large a batch, is+ * CRYPTON_GCM_FUSED_POWERS in crypton_aes.h.+ *+ * Only messages shorter than CRYPTON_GCM_FUSED_MAX_MESSAGE come here.  Above+ * that the stitched assembly in cbits/asm is faster, and crypton_aes.c sends+ * them there instead; below it, that assembly will not start at all.+ */++#include <crypton_cpu.h>++#ifdef WITH_GCM_FUSED++#include <stdint.h>+#include <string.h>+#include <wmmintrin.h>+#include <smmintrin.h>+#include <tmmintrin.h>++#include <crypton_aes.h>+#include <aes/gcm_fused_x86.h>++/*+ * aes_key is a struct of bytes, so its round keys sit wherever the members+ * before them leave them -- eight bytes in, as it happens.  A __m128i *+ * pointed at that gets an aligned load and a fault, so each round key is+ * fetched with an unaligned load instead.  Copying them somewhere aligned+ * would cost a copy per call, which at these message lengths is a tenth of+ * the whole; the load is free from L1 and the round key is fetched once for+ * all six lanes.+ */+#define RK(p, i) \+    _mm_loadu_si128((const __m128i *) ((const uint8_t *) (p) + 16 * (size_t) (i)))++/* a full sixteen-byte reversal: mask bytes 15,14,...,0 */+static const __m128i BSWAP = {0x08090a0b0c0d0e0fLL, 0x0001020304050607LL};+++#define TGT __attribute__((target("aes,pclmul,sse4.1")))++/* the two halves of a value added together: the term Karatsuba needs, and it+ * does not depend on what the value is multiplied by */+TGT static inline __m128i fold(__m128i a)+{+    return _mm_xor_si128(a, _mm_unpackhi_epi64(a, a));+}++/* H multiplied by x in the field, which is what puts H and its powers one+ * bit up: GCM numbers the bits of a field element the other way round from+ * the way the carry-less multiply does, and pre-shifting is what saves the+ * correction after every multiply.+ *+ * Written from the definition.  The field is GF(2)[x] modulo x^128 + x^127 ++ * x^126 + x^121 + 1, so multiplying by x is a shift of one place, and the+ * term that leaves the top comes back as the other four. */+TGT static __m128i twist(__m128i h)+{+    /* x^127 + x^126 + x^121 + 1, the terms x^128 is congruent to */+    const __m128i poly = _mm_set_epi64x(0xc200000000000000ULL, 1);+    /* the top bit of each half */+    __m128i tops = _mm_srli_epi64(h, 63);+    /* doubling a polynomial is a shift by one: each half doubles, and the+     * low half's top bit becomes the high half's bottom bit */+    __m128i doubled = _mm_or_si128(_mm_add_epi64(h, h),+                                   _mm_slli_si128(tops, 8));+    /* bit 127 is the one that leaves the field; spread it to a mask by+     * subtracting it from zero, and it brings the four terms back */+    __m128i mask = _mm_sub_epi64(_mm_setzero_si128(),+                                 _mm_unpackhi_epi64(tops, tops));++    return _mm_xor_si128(doubled, _mm_and_si128(mask, poly));+}++/* one reduction of a 256-bit product back into the field */+TGT static __m128i reduce256(__m128i lo, __m128i hi)+{+    const __m128i poly = _mm_set_epi64x(0xc200000000000000ULL, 1);+    __m128i t;++    t = _mm_clmulepi64_si128(lo, poly, 0x10);+    lo = _mm_xor_si128(_mm_shuffle_epi32(lo, 0x4e), t);+    t = _mm_clmulepi64_si128(lo, poly, 0x10);+    lo = _mm_xor_si128(_mm_shuffle_epi32(lo, 0x4e), t);+    return _mm_xor_si128(hi, lo);+}++/*+ * One multiply in exactly the form the hot loop uses it: the left operand+ * plain, the right one already twisted.  Building the table with the same+ * multiply that consumes it is the only way the two conventions cannot drift+ * apart.+ */+TGT static __m128i mul_twisted(__m128i a, __m128i ht)+{+    __m128i lo = _mm_clmulepi64_si128(a, ht, 0x00);+    __m128i hi = _mm_clmulepi64_si128(a, ht, 0x11);+    __m128i mid = _mm_clmulepi64_si128(fold(a), fold(ht), 0x00);++    mid = _mm_xor_si128(mid, _mm_xor_si128(lo, hi));+    lo = _mm_xor_si128(lo, _mm_slli_si128(mid, 8));+    hi = _mm_xor_si128(hi, _mm_srli_si128(mid, 8));+    return reduce256(lo, hi);+}++TGT void crypton_gcm_fused_key_init(aes_gcm_fused *fk, const aes_key *key)+{+    const uint8_t *rk = key->data;+    const int rounds = key->nbr;+    __m128i h, p;+    int i;++    /* H = E_K(0) */+    h = RK(rk, 0);+    for (i = 1; i < rounds; i++) h = _mm_aesenc_si128(h, RK(rk, i));+    h = _mm_aesenclast_si128(h, RK(rk, rounds));+    h = _mm_shuffle_epi8(h, BSWAP);++    {+        __m128i ht = twist(h);+        p = h;+        for (i = 0; i < CRYPTON_GCM_FUSED_POWERS; i++) {+            __m128i t = twist(p);+            _mm_storeu_si128((__m128i *) &fk->p[i].h, t);+            _mm_storeu_si128((__m128i *) &fk->p[i].r, fold(t));+            p = mul_twisted(p, ht);+        }+    }+}++/*+ * The running product.  Three plain locals and a macro, not a struct behind+ * a pointer: taking the address of the accumulators is enough to keep them+ * out of registers, and then every multiply reloads and restores them.  That+ * is the same mistake as reaching a table through an index the compiler+ * cannot fold, and it costs more here because it is on the inner path.+ */+#define GHASH_DECL __m128i glo = _mm_setzero_si128(),                        \+                            ghi = _mm_setzero_si128(),                       \+                            gmid = _mm_setzero_si128(),                      \+                            gtag = _mm_setzero_si128();                      \+                   int gidx = 0, gblen = 0, gbpos = 0++/*+ * Absorb one block.  Blocks are taken in batches of at most CRYPTON_GCM_FUSED_POWERS: the+ * first of a batch carries in the value the batch before it reduced to, the+ * rest go in against descending powers, and the batch ends with the one+ * reduction they share.  With the batch as long as the message this is+ * picotls's single reduction; with it fixed, the state stays a fixed size.+ */+#define GHASH_ONE(blk, unused_power)                                         \+    do {                                                                     \+        __m128i _b = (blk);                                                  \+        __m128i _h, _r;                                                      \+        if (gbpos == 0) {                                                    \+            int _left = gtotal - gidx;                                       \+            gblen = _left < CRYPTON_GCM_FUSED_POWERS ? _left : CRYPTON_GCM_FUSED_POWERS;             \+            _b = _mm_xor_si128(_b, gtag);                                    \+            glo = ghi = gmid = _mm_setzero_si128();                          \+        }                                                                    \+        _h = _mm_loadu_si128((const __m128i *) &fk->p[gblen-gbpos-1].h);     \+        _r = _mm_loadu_si128((const __m128i *) &fk->p[gblen-gbpos-1].r);     \+        glo = _mm_xor_si128(glo, _mm_clmulepi64_si128(_b, _h, 0x00));        \+        ghi = _mm_xor_si128(ghi, _mm_clmulepi64_si128(_b, _h, 0x11));        \+        gmid = _mm_xor_si128(gmid,                                           \+                   _mm_clmulepi64_si128(fold(_b), _r, 0x00));                \+        gidx++; gbpos++;                                                     \+        if (gbpos == gblen) {                                                \+            gtag = ghash_reduce(glo, ghi, gmid);                             \+            gbpos = 0;                                                       \+        }                                                                    \+    } while (0)++TGT static __m128i ghash_reduce(__m128i glo, __m128i ghi, __m128i gmid)+{+    __m128i mid = _mm_xor_si128(gmid, _mm_xor_si128(glo, ghi));+    __m128i lo = _mm_xor_si128(glo, _mm_slli_si128(mid, 8));+    __m128i hi = _mm_xor_si128(ghi, _mm_srli_si128(mid, 8));++    return reduce256(lo, hi);+}++/* zero every byte from n onwards, so a partial block can be fed to GHASH+ * without being written out and read back */+TGT static __m128i clampn(__m128i v, size_t n)+{+    const __m128i idx = {0x0706050403020100LL, 0x0f0e0d0c0b0a0908LL};+    return _mm_and_si128(v, _mm_cmpgt_epi8(_mm_set1_epi8((char) n), idx));+}++/*+ * A short block, zero padded, without going through the stack.+ *+ * The obvious way -- zero sixteen bytes, copy n in, load them back -- is+ * three trips to memory with a store the load has to wait for, and at these+ * lengths that is a tenth of the whole call.  Reading the sixteen bytes and+ * masking off what is above n is two instructions, but it reads past the end+ * of what the caller gave, so it has to be sure those bytes exist.+ *+ * They do unless the block ends a page.  A load of sixteen bytes that starts+ * at least sixteen from the end of a page stays inside it; and if the n bytes+ * asked for themselves cross the boundary then the next page is there to be+ * read as well.  What is left is a block near the end of a page whose own+ * bytes stop short of it, and that is read aligned -- which cannot leave the+ * page -- and shuffled down.  This is how picotls's fusion does it.+ */++/* thirty-two bytes of ones, then thirty-one of zeros: sixteen loaded from+ * 32 - n give n bytes of ones and the rest zeros */+static const uint8_t loadn_mask[63] = {+    0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,+    0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,+    0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,+    0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff};++/* the first sixteen map to byte offsets, the rest to zero */+static const uint8_t loadn_shuffle[31] = {+    0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07,+    0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,+    0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80,+    0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80};++#if defined(__has_feature)+#if __has_feature(address_sanitizer)+#define NO_ASAN __attribute__((no_sanitize_address))+#endif+#elif defined(__SANITIZE_ADDRESS__)+#define NO_ASAN __attribute__((no_sanitize_address))+#endif+#ifndef NO_ASAN+#define NO_ASAN+#endif++TGT NO_ASAN static __m128i loadn_page_end(const uint8_t *p, size_t n)+{+    uintptr_t shift = (uintptr_t) p & 15;+    __m128i pattern = _mm_loadu_si128((const __m128i *) (loadn_shuffle + shift));++    (void) n;+    return _mm_shuffle_epi8(+        _mm_load_si128((const __m128i *) ((uintptr_t) p - shift)), pattern);+}++TGT NO_ASAN static __m128i loadn(const uint8_t *p, size_t n)+{+    __m128i mask = _mm_loadu_si128((const __m128i *) (loadn_mask + 32 - n));+    uintptr_t mod4k = (uintptr_t) p % 4096;+    __m128i v;++    if (mod4k <= 4096 - 16 || mod4k + n > 4096)+        v = _mm_loadu_si128((const __m128i *) p);+    else+        v = loadn_page_end(p, n);+    return _mm_and_si128(v, mask);+}++TGT static void storen(uint8_t *p, __m128i v, size_t n)+{+    uint8_t buf[16];+    _mm_storeu_si128((__m128i *) buf, v);+    memcpy(p, buf, n);+}++/* One block.  The ten rounds of the common case are written out for the+ * same reason the six-wide group is: a loop over a round count that lives in+ * the key leaves every round key fetched through an index the compiler+ * cannot fold, and adds a branch to a chain that is already latency-bound. */+TGT static __m128i aes_one_block(const uint8_t *rk, int rounds, __m128i v)+{+    const uint8_t *k = rk;+    int i;++    if (rounds == 10) {+        v = _mm_xor_si128(v, RK(k, 0));+        v = _mm_aesenc_si128(v, RK(k, 1));+        v = _mm_aesenc_si128(v, RK(k, 2));+        v = _mm_aesenc_si128(v, RK(k, 3));+        v = _mm_aesenc_si128(v, RK(k, 4));+        v = _mm_aesenc_si128(v, RK(k, 5));+        v = _mm_aesenc_si128(v, RK(k, 6));+        v = _mm_aesenc_si128(v, RK(k, 7));+        v = _mm_aesenc_si128(v, RK(k, 8));+        v = _mm_aesenc_si128(v, RK(k, 9));+        return _mm_aesenclast_si128(v, RK(k, 10));+    }+    v = _mm_xor_si128(v, RK(k, 0));+    for (i = 1; i < rounds; i++) v = _mm_aesenc_si128(v, RK(k, i));+    return _mm_aesenclast_si128(v, RK(k, rounds));+}++/*+ * Six blocks at once, with lane 5 free to run a different key schedule from+ * the rest.  Which schedule that lane uses is chosen once, into a pointer,+ * rather than tested inside the rounds, and the six live in named variables+ * rather than an array -- an array indexed by a running variable goes to+ * memory, and then every round is a load and a store instead of a register+ * to register operation, which is the whole of what this is trying to avoid.+ *+ * Lanes beyond what the caller needs still run.  Six are in flight whatever+ * the message length, so the spare ones cost nothing, and that is exactly+ * why the header protection mask and E(K,Y0) are worth putting in them+ * instead of giving each a dependent chain of its own.+ */+#define WIDE6(alt)                                                           \+    do {                                                                     \+        const uint8_t *ak = (alt);                                           \+        int r;                                                               \+        t0 = _mm_xor_si128(t0, RK(rk, 0));                                   \+        t1 = _mm_xor_si128(t1, RK(rk, 0));                                   \+        t2 = _mm_xor_si128(t2, RK(rk, 0));                                   \+        t3 = _mm_xor_si128(t3, RK(rk, 0));                                   \+        t4 = _mm_xor_si128(t4, RK(rk, 0));                                   \+        t5 = _mm_xor_si128(t5, RK(ak, 0));                                       \+        for (r = 1; r < rounds; r++) {                                   \+            __m128i k = RK(rk, r);                                           \+            t0 = _mm_aesenc_si128(t0, k);                                    \+            t1 = _mm_aesenc_si128(t1, k);                                    \+            t2 = _mm_aesenc_si128(t2, k);                                    \+            t3 = _mm_aesenc_si128(t3, k);                                    \+            t4 = _mm_aesenc_si128(t4, k);                                    \+            t5 = _mm_aesenc_si128(t5, RK(ak, r));                                \+            GSTEP();                                                         \+        }                                                                    \+        {                                                                    \+            __m128i k = RK(rk, rounds);                                  \+            t0 = _mm_aesenclast_si128(t0, k);                                \+            t1 = _mm_aesenclast_si128(t1, k);                                \+            t2 = _mm_aesenclast_si128(t2, k);                                \+            t3 = _mm_aesenclast_si128(t3, k);                                \+            t4 = _mm_aesenclast_si128(t4, k);                                \+            t5 = _mm_aesenclast_si128(t5, RK(ak, rounds));                   \+        }                                                                    \+    } while (0)++/* the same with nothing queued to absorb: the decryption side takes its+ * GHASH straight from the input and has no queue to drain */+#define WIDE6_NOQ(alt)                                                           \+    do {                                                                     \+        const uint8_t *ak = (alt);                                           \+        int r;                                                               \+        t0 = _mm_xor_si128(t0, RK(rk, 0));                                   \+        t1 = _mm_xor_si128(t1, RK(rk, 0));                                   \+        t2 = _mm_xor_si128(t2, RK(rk, 0));                                   \+        t3 = _mm_xor_si128(t3, RK(rk, 0));                                   \+        t4 = _mm_xor_si128(t4, RK(rk, 0));                                   \+        t5 = _mm_xor_si128(t5, RK(ak, 0));                                       \+        for (r = 1; r < rounds; r++) {                                   \+            __m128i k = RK(rk, r);                                           \+            t0 = _mm_aesenc_si128(t0, k);                                    \+            t1 = _mm_aesenc_si128(t1, k);                                    \+            t2 = _mm_aesenc_si128(t2, k);                                    \+            t3 = _mm_aesenc_si128(t3, k);                                    \+            t4 = _mm_aesenc_si128(t4, k);                                    \+            t5 = _mm_aesenc_si128(t5, RK(ak, r));                                \+        }                                                                    \+        {                                                                    \+            __m128i k = RK(rk, rounds);                                  \+            t0 = _mm_aesenclast_si128(t0, k);                                \+            t1 = _mm_aesenclast_si128(t1, k);                                \+            t2 = _mm_aesenclast_si128(t2, k);                                \+            t3 = _mm_aesenclast_si128(t3, k);                                \+            t4 = _mm_aesenclast_si128(t4, k);                                \+            t5 = _mm_aesenclast_si128(t5, RK(ak, rounds));                   \+        }                                                                    \+    } while (0)++/*+ * The same written out for the ten rounds of AES-128, with a slot for one+ * queued multiply between each of the first six.  The loop above cannot take+ * them: the round count is a value in the key, so there is no place the+ * compiler knows is a round apart from the next, and a test before each+ * multiply would end the basic block the scheduler works inside.  This pass+ * runs with a group's multiplies still waiting, and on a short message that+ * is most of what it has to do.+ */+#define WROUND6(r, ak)                                                       \+    do {                                                                     \+        __m128i k = RK(rk, r);                                               \+        t0 = _mm_aesenc_si128(t0, k);                                        \+        t1 = _mm_aesenc_si128(t1, k);                                        \+        t2 = _mm_aesenc_si128(t2, k);                                        \+        t3 = _mm_aesenc_si128(t3, k);                                        \+        t4 = _mm_aesenc_si128(t4, k);                                        \+        t5 = _mm_aesenc_si128(t5, RK(ak, r));                                \+    } while (0)++#define WIDE6_10(alt)                                                        \+    do {                                                                     \+        const uint8_t *ak = (alt);                                           \+        t0 = _mm_xor_si128(t0, RK(rk, 0));                                   \+        t1 = _mm_xor_si128(t1, RK(rk, 0));                                   \+        t2 = _mm_xor_si128(t2, RK(rk, 0));                                   \+        t3 = _mm_xor_si128(t3, RK(rk, 0));                                   \+        t4 = _mm_xor_si128(t4, RK(rk, 0));                                   \+        t5 = _mm_xor_si128(t5, RK(ak, 0));                                   \+        WROUND6(1, ak); GAT(0);                                              \+        WROUND6(2, ak); GAT(1);                                              \+        WROUND6(3, ak); GAT(2);                                              \+        WROUND6(4, ak); GAT(3);                                              \+        WROUND6(5, ak); GAT(4);                                              \+        WROUND6(6, ak); GAT(5);                                              \+        WROUND6(7, ak);                                                      \+        WROUND6(8, ak);                                                      \+        WROUND6(9, ak);                                                      \+        {                                                                    \+            __m128i k = RK(rk, 10);                                          \+            t0 = _mm_aesenclast_si128(t0, k);                                \+            t1 = _mm_aesenclast_si128(t1, k);                                \+            t2 = _mm_aesenclast_si128(t2, k);                                \+            t3 = _mm_aesenclast_si128(t3, k);                                \+            t4 = _mm_aesenclast_si128(t4, k);                                \+            t5 = _mm_aesenclast_si128(t5, RK(ak, 10));                       \+        }                                                                    \+    } while (0)++/*+ * v2: six blocks of AES in flight at once, so the ten rounds of one block no+ * longer wait on each other -- AES-NI is pipelined and will take one+ * instruction a clock as long as the instructions in flight are independent.+ * The GHASH multiplies of the group just finished are issued between the+ * rounds of the group now running, which is the stitching: they do not want+ * the same execution port, so held against each other they cost about what+ * the rounds alone cost.+ */++/*+ * The counter block, built without leaving the vector registers.+ *+ * GCM counts in the low 32 bits of the block, big endian, and wraps there.+ * ctr holds the block with its bytes reversed, so those four bytes are the+ * low lane and _mm_add_epi32 steps them without carrying into the nonce+ * above -- which is the wrap GCM asks for.  A shuffle puts the bytes back.+ *+ * The obvious way -- increment a uint32_t, byte swap it, pinsrd it in --+ * costs a move from a general register to a vector one for every lane, six+ * to a group, and those do not come free.+ */+#define CTR6(j)                                                              \+    do {                                                                     \+        ctr = _mm_add_epi32(ctr, one32);                                     \+        b##j = _mm_xor_si128(_mm_shuffle_epi8(ctr, BSWAP), RK(rk, 0));       \+    } while (0)++#define ROUND6(r)                                                            \+    do {                                                                     \+        __m128i k = RK(rk, r);                                               \+        b0 = _mm_aesenc_si128(b0, k);                                        \+        b1 = _mm_aesenc_si128(b1, k);                                        \+        b2 = _mm_aesenc_si128(b2, k);                                        \+        b3 = _mm_aesenc_si128(b3, k);                                        \+        b4 = _mm_aesenc_si128(b4, k);                                        \+        b5 = _mm_aesenc_si128(b5, k);                                        \+    } while (0)++#define LAST6(r)                                                             \+    do {                                                                     \+        __m128i k = RK(rk, r);                                               \+        b0 = _mm_aesenclast_si128(b0, k);                                    \+        b1 = _mm_aesenclast_si128(b1, k);                                    \+        b2 = _mm_aesenclast_si128(b2, k);                                    \+        b3 = _mm_aesenclast_si128(b3, k);                                    \+        b4 = _mm_aesenclast_si128(b4, k);                                    \+        b5 = _mm_aesenclast_si128(b5, k);                                    \+    } while (0)++/* one GHASH multiply, taken from a queue of blocks waiting to be absorbed,+ * to be issued in the gaps between AES rounds */+/* A ring, so that a block queued while others are still waiting costs an+ * index and not a move: the queue is walked from both ends and never+ * compacted. */+#define GQ_MASK 15++#define GPUSH(v)                                                             \+    do { gq[gw] = (v); gw++; gn++; } while (0)++#define GSTEP()                                                              \+    do {                                                                     \+        if (gn > 0) {                                                        \+            GHASH_ONE(gq[gi], gp);                                   \+            gi++; gp--; gn--;                                                \+        }                                                                    \+    } while (0)++/* the same at a slot the compiler can see, for the unrolled group below */+/*+ * One queued block, at a slot the compiler can see and with nothing to test+ * before it.  A test here would end the basic block, and the scheduler works+ * inside one: six tests turn the group into twelve blocks and the multiplies+ * can no longer be moved up among the rounds, which is the whole point of+ * writing them there.  The group below is entered only when the queue is+ * full, so there is nothing to test.+ */+/*+ * The block a slot names, read from the output where the group before it+ * left the ciphertext rather than from a copy kept beside it.  The copy+ * cost six stores a group for bytes already in memory; picotls's fusion+ * points its GHASH at the output it has just written for the same reason.+ */+#define GAT(j) GHASH_ONE(_mm_shuffle_epi8(                                   \+        _mm_loadu_si128((const __m128i *) (prev + 16 * (j))), BSWAP), gp - (j))++TGT void crypton_gcm_fused_encrypt(uint8_t *out, const aes_gcm_fused *fk,+                                   const aes_key *key, const uint8_t *nonce,+                                   const uint8_t *aad, size_t aadlen,+                                   const uint8_t *in, size_t inlen, size_t taglen,+                                   const aes_key *hpkey, size_t sampleoff,+                                   uint8_t *mask)+{+    const uint8_t *rk = key->data;+    const uint8_t *hprk = hpkey != 0 ? hpkey->data : rk;+    const int rounds = key->nbr;+    const int hprounds = hpkey != 0 ? hpkey->nbr : rounds;+    GHASH_DECL;+    __m128i ctrbase, ctr, one32, ek0, tag, b0, b1, b2, b3, b4, b5;+    const int ntail_pre = (int) ((inlen % 96 + 15) / 16);+    int lane_ek0;+    __m128i gq[6];+    unsigned gi = 0, gw = 0;+    int gn = 0;+    size_t nblk = (aadlen + 15) / 16 + (inlen + 15) / 16 + 1;+    const int gtotal = (int) nblk;+    int gp = (int) nblk;+    size_t i;+    size_t done;+    int lane_mask = 0;++    /*+     * Y0: the twelve bytes of nonce and a counter of one.  loadn reads the+     * nonce where it lies and masks what is above it, so this is one load+     * rather than three of four bytes each and a set built from them.+     */+    ctrbase = _mm_insert_epi32(loadn(nonce, 12), (int) __builtin_bswap32(1), 3);+    ctr = _mm_shuffle_epi8(ctrbase, BSWAP);+    one32 = _mm_set_epi32(0, 0, 0, 1);++    lane_ek0 = ntail_pre > 0 && ntail_pre <= 4;+    if (!lane_ek0)+        ek0 = aes_one_block(rk, rounds, ctrbase);++    /* The additional data goes in first and takes the highest powers, but it+     * is only queued here: absorbing it takes multiplies, and the multiplies+     * belong in the gaps between the AES rounds below rather than in front+     * of them where nothing else is running. */+    /*+     * The additional data goes in first and takes the highest powers.  It is+     * absorbed here rather than queued: what the queue is for is giving the+     * rounds below something to interleave with, and a queue that sometimes+     * holds the header and sometimes does not forces a test before every+     * multiply -- which is what stopped the interleaving from happening at+     * all.  See the peeled first group below.+     */+    {+        size_t nfull = aadlen / 16;+        size_t rest = aadlen % 16;++        for (i = 0; i < nfull; i++)+            GHASH_ONE(_mm_shuffle_epi8(+                _mm_loadu_si128((const __m128i *) (aad + i * 16)), BSWAP), 0);+        if (rest)+            GHASH_ONE(_mm_shuffle_epi8(loadn(aad + nfull * 16, rest), BSWAP), 0);+    }++    /* Whole groups of six.  The rounds are written out rather than looped:+     * the number of them is a value in the key, so a loop over it leaves the+     * compiler fetching each round key through an index it cannot fold, and+     * the six lanes go to memory with them.  Written out, the whole group+     * stays in registers, and the six multiplies of the group before can be+     * placed between the rounds by hand -- which is the stitching: AES-NI+     * and PCLMULQDQ do not contend for the same port, so the multiplies are+     * very nearly free.+     */+    done = 0;+    if (rounds == 10) {+        if (done + 96 <= inlen) {+            const uint8_t *p = in + done;+            uint8_t *q = out + done;++            CTR6(0); CTR6(1); CTR6(2); CTR6(3); CTR6(4); CTR6(5);+            ROUND6(1);+            ROUND6(2);+            ROUND6(3);+            ROUND6(4);+            ROUND6(5);+            ROUND6(6);+            ROUND6(7);+            ROUND6(8);+            ROUND6(9);+            LAST6(10);+            gn = 0; gi = 0; gw = 0;++            b0 = _mm_xor_si128(b0, _mm_loadu_si128((const __m128i *) p));+            b1 = _mm_xor_si128(b1, _mm_loadu_si128((const __m128i *) (p + 16)));+            b2 = _mm_xor_si128(b2, _mm_loadu_si128((const __m128i *) (p + 32)));+            b3 = _mm_xor_si128(b3, _mm_loadu_si128((const __m128i *) (p + 48)));+            b4 = _mm_xor_si128(b4, _mm_loadu_si128((const __m128i *) (p + 64)));+            b5 = _mm_xor_si128(b5, _mm_loadu_si128((const __m128i *) (p + 80)));+            _mm_storeu_si128((__m128i *) q, b0);+            _mm_storeu_si128((__m128i *) (q + 16), b1);+            _mm_storeu_si128((__m128i *) (q + 32), b2);+            _mm_storeu_si128((__m128i *) (q + 48), b3);+            _mm_storeu_si128((__m128i *) (q + 64), b4);+            _mm_storeu_si128((__m128i *) (q + 80), b5);++            done += 96;+        }+        for (; done + 96 <= inlen; done += 96) {+            const uint8_t *p = in + done;+            uint8_t *q = out + done;+            const uint8_t *prev = out + done - 96;++            CTR6(0); CTR6(1); CTR6(2); CTR6(3); CTR6(4); CTR6(5);+            ROUND6(1); GAT(0);+            ROUND6(2); GAT(1);+            ROUND6(3); GAT(2);+            ROUND6(4); GAT(3);+            ROUND6(5); GAT(4);+            ROUND6(6); GAT(5);+            ROUND6(7);+            ROUND6(8);+            ROUND6(9);+            LAST6(10);+            gp -= 6;++            b0 = _mm_xor_si128(b0, _mm_loadu_si128((const __m128i *) p));+            b1 = _mm_xor_si128(b1, _mm_loadu_si128((const __m128i *) (p + 16)));+            b2 = _mm_xor_si128(b2, _mm_loadu_si128((const __m128i *) (p + 32)));+            b3 = _mm_xor_si128(b3, _mm_loadu_si128((const __m128i *) (p + 48)));+            b4 = _mm_xor_si128(b4, _mm_loadu_si128((const __m128i *) (p + 64)));+            b5 = _mm_xor_si128(b5, _mm_loadu_si128((const __m128i *) (p + 80)));+            _mm_storeu_si128((__m128i *) q, b0);+            _mm_storeu_si128((__m128i *) (q + 16), b1);+            _mm_storeu_si128((__m128i *) (q + 32), b2);+            _mm_storeu_si128((__m128i *) (q + 48), b3);+            _mm_storeu_si128((__m128i *) (q + 64), b4);+            _mm_storeu_si128((__m128i *) (q + 80), b5);++            /* straight from the registers the last round left them in: the+             * queue existed only to hold them until the next group's rounds+             * could hide the multiplies, and that is 192 bytes of store and+             * load per 96 bytes of payload */+        }++    } else {+        for (done = 0; done + 96 <= inlen; done += 96) {+                const uint8_t *p = in + done;+                uint8_t *q = out + done;+                int r;++                CTR6(0); CTR6(1); CTR6(2); CTR6(3); CTR6(4); CTR6(5);+                for (r = 1; r < rounds; r++) {+                    ROUND6(r);+                    GSTEP();+                }+                LAST6(rounds);++                b0 = _mm_xor_si128(b0, _mm_loadu_si128((const __m128i *) p));+                b1 = _mm_xor_si128(b1, _mm_loadu_si128((const __m128i *) (p + 16)));+                b2 = _mm_xor_si128(b2, _mm_loadu_si128((const __m128i *) (p + 32)));+                b3 = _mm_xor_si128(b3, _mm_loadu_si128((const __m128i *) (p + 48)));+                b4 = _mm_xor_si128(b4, _mm_loadu_si128((const __m128i *) (p + 64)));+                b5 = _mm_xor_si128(b5, _mm_loadu_si128((const __m128i *) (p + 80)));+                _mm_storeu_si128((__m128i *) q, b0);+                _mm_storeu_si128((__m128i *) (q + 16), b1);+                _mm_storeu_si128((__m128i *) (q + 32), b2);+                _mm_storeu_si128((__m128i *) (q + 48), b3);+                _mm_storeu_si128((__m128i *) (q + 64), b4);+                _mm_storeu_si128((__m128i *) (q + 80), b5);++                while (gn > 0) GSTEP();+                gi = 0; gw = 0;+                gq[0] = _mm_shuffle_epi8(b0, BSWAP);+                gq[1] = _mm_shuffle_epi8(b1, BSWAP);+                gq[2] = _mm_shuffle_epi8(b2, BSWAP);+                gq[3] = _mm_shuffle_epi8(b3, BSWAP);+                gq[4] = _mm_shuffle_epi8(b4, BSWAP);+                gq[5] = _mm_shuffle_epi8(b5, BSWAP);+                gn = 6;+        }+    }++    /* The tail.  The wide pass below runs only when there are blocks for it:+     * sixty AES instructions to fill one lane is not worth it, so a message+     * that ends on a group boundary leaves E(K,Y0) the chain it was given+     * above and takes the mask on one of its own.+     *+     * The spare lane carries the mask only when two things hold.  The sample+     * has to lie entirely in output the groups above have already written:+     * this pass reads it while it runs, and the blocks it is itself+     * computing are stored after it, so a sample reaching into them would be+     * read before it exists.  And the two key schedules have to have the+     * same number of rounds, because the lanes share the loop that counts+     * them and a shorter schedule would be read past its end.  TLS and QUIC+     * satisfy both; anything else gets the mask on a chain of its own, which+     * is what it would have had anyway. */+    {+        __m128i t0, t1, t2, t3, t4, t5;+        __m128i tv[6];+        size_t toff[6];+        int ntail = 0, j;++        if (done >= inlen) goto no_tail;++        for (i = done; i < inlen; i += 16) {+            toff[ntail] = i;+            ctr = _mm_add_epi32(ctr, one32);+            tv[ntail] = _mm_shuffle_epi8(ctr, BSWAP);+            ntail++;+        }++        lane_mask = ntail > 0 && ntail <= 5 && hpkey != 0 && mask != 0+                 && hprounds == rounds+                 && sampleoff + 16 <= done;++        if (ntail > 0) {+            t0 = tv[0];+            t1 = ntail > 1 ? tv[1] : ctrbase;+            t2 = ntail > 2 ? tv[2] : ctrbase;+            t3 = ntail > 3 ? tv[3] : ctrbase;+            t4 = lane_ek0 ? ctrbase : (ntail > 4 ? tv[4] : ctrbase);+            t5 = lane_mask+               ? _mm_loadu_si128((const __m128i *) (out + sampleoff))+               : (ntail > 5 ? tv[5] : ctrbase);+            /* A group leaves exactly six queued, which is what lets the+             * slots below be named at compile time.  Where no group ran+             * there is nothing to place and the plain pass will do. */+            if (rounds == 10 && done >= 96) {+                const uint8_t *prev = out + done - 96;+                WIDE6_10(lane_mask ? hprk : rk);+            } else {+                WIDE6(lane_mask ? hprk : rk);+            }+            if (lane_ek0)+                ek0 = t4;+            else+                tv[4] = t4;+            if (lane_mask)+                _mm_storeu_si128((__m128i *) mask, t5);+            else if (ntail > 5)+                tv[5] = t5;+        }+no_tail:+        while (gn > 0) GSTEP();++        /* The last group's ciphertext is absorbed by the pass above where+         * there is one to absorb it.  A message that ends on a group+         * boundary has no such pass, so it is taken here. */+        if (rounds == 10 && done >= 96 && ntail == 0) {+            const uint8_t *prev = out + done - 96;+            GAT(0); GAT(1); GAT(2); GAT(3); GAT(4); GAT(5);+        }++        /*+         * The tail blocks, from the registers the pass left them in.  They+         * were going through an array indexed by the loop variable, which+         * the compiler cannot see through and so keeps in memory: every+         * block then reloaded its own keystream.  Named one per block and+         * reached by a test on a count instead, they stay where they are.+         */+#define TAILBLK(j, reg)                                                      \+        do {                                                                 \+            size_t off = done + 16 * (j);                                    \+            size_t n = inlen - off < 16 ? inlen - off : 16;                  \+            __m128i c = _mm_xor_si128(reg, n == 16                           \+                ? _mm_loadu_si128((const __m128i *) (in + off))              \+                : loadn(in + off, n));                                       \+            if (n == 16) {                                                   \+                _mm_storeu_si128((__m128i *) (out + off), c);                \+            } else {                                                         \+                /* the tag goes in directly above, so those bytes are        \+                 * written over anyway where there are sixteen to spare */   \+                if (n + taglen >= 16)                                        \+                    _mm_storeu_si128((__m128i *) (out + off), c);            \+                else                                                         \+                    storen(out + off, c, n);                                 \+                c = clampn(c, n);                                            \+            }                                                                \+            GHASH_ONE(_mm_shuffle_epi8(c, BSWAP), gp);                       \+            gp--;                                                            \+        } while (0)++        if (ntail > 0) TAILBLK(0, t0);+        if (ntail > 1) TAILBLK(1, t1);+        if (ntail > 2) TAILBLK(2, t2);+        if (ntail > 3) TAILBLK(3, t3);+        if (ntail > 4) TAILBLK(4, tv[4]);+        if (ntail > 5) TAILBLK(5, tv[5]);+#undef TAILBLK+    }++    {+        /*+         * The length block: the additional data's bit count and the+         * message's, each big endian in a half, and then reversed like+         * every other block on its way to GHASH.+         *+         * Reversed, that block is the two counts as ordinary little endian+         * words with the message's in the low half -- which is one set, and+         * no shuffle.  Sixteen byte stores to the stack and a load back is+         * what it cost before.+         */+        GHASH_ONE(_mm_set_epi64x((long long) ((uint64_t) aadlen << 3),+                                 (long long) ((uint64_t) inlen << 3)), gp);+    }++    tag = _mm_shuffle_epi8(gtag, BSWAP);+    tag = _mm_xor_si128(tag, ek0);+    if (taglen == 16)+        _mm_storeu_si128((__m128i *) (out + inlen), tag);+    else+        storen(out + inlen, tag, taglen);++    /* A sample the pass above could not reach -- because it covered blocks+     * that pass was still computing, or the tag, which is written just now+     * -- is taken here instead, where everything it can cover exists. */+    if (!lane_mask && hpkey != 0 && mask != 0)+        _mm_storeu_si128((__m128i *) mask,+                         aes_one_block(hprk, hprounds, _mm_loadu_si128(+                             (const __m128i *) (out + sampleoff))));+}+++/*+ * The same for decryption, which is the simpler of the two.+ *+ * What GHASH absorbs here is the ciphertext, and the ciphertext is the+ * input: it is there before any of the AES has run.  So there is no queue --+ * the multiplies of a group go between the rounds of that same group rather+ * than waiting for the one after, and nothing is stored and loaded back to+ * carry them across.+ *+ * The tag is compared here, every byte of it whichever way the answer goes,+ * and the answer is 1 for a message whose tag matched.+ */++/* one ciphertext block straight from the input, at a slot named here */+#define DAT(j) GHASH_ONE(_mm_shuffle_epi8(                                   \+        _mm_loadu_si128((const __m128i *) (p + 16 * (j))), BSWAP), 0)++/* the next counter block, into a named register */+#define CTRT(t)                                                              \+    do { ctr = _mm_add_epi32(ctr, one32);                                    \+         t = _mm_shuffle_epi8(ctr, BSWAP); } while (0)++TGT int crypton_gcm_fused_decrypt(uint8_t *out, const aes_gcm_fused *fk,+                                  const aes_key *key, const uint8_t *nonce,+                                  const uint8_t *aad, size_t aadlen,+                                  const uint8_t *in, size_t inlen,+                                  const uint8_t *tag, size_t taglen,+                                  uint8_t *outtag)+{+    const uint8_t *rk = key->data;+    const int rounds = key->nbr;+    GHASH_DECL;+    __m128i ctrbase, ctr, one32, ek0, want, b0, b1, b2, b3, b4, b5;+    const int ntail_pre = (int) ((inlen % 96 + 15) / 16);+    int lane_ek0, gp = 0;+    const int gtotal = (int) ((aadlen + 15) / 16 + (inlen + 15) / 16 + 1);+    size_t i;+    size_t done;+    uint8_t diff = 0;++    ctrbase = _mm_insert_epi32(loadn(nonce, 12), (int) __builtin_bswap32(1), 3);+    ctr = _mm_shuffle_epi8(ctrbase, BSWAP);+    one32 = _mm_set_epi32(0, 0, 0, 1);++    lane_ek0 = ntail_pre > 0 && ntail_pre <= 5;+    if (!lane_ek0)+        ek0 = aes_one_block(rk, rounds, ctrbase);++    {+        size_t nfull = aadlen / 16;+        size_t rest = aadlen % 16;++        for (i = 0; i < nfull; i++)+            GHASH_ONE(_mm_shuffle_epi8(+                _mm_loadu_si128((const __m128i *) (aad + i * 16)), BSWAP), 0);+        if (rest)+            GHASH_ONE(_mm_shuffle_epi8(loadn(aad + nfull * 16, rest), BSWAP), 0);+    }++    done = 0;+    if (rounds == 10) {+        for (; done + 96 <= inlen; done += 96) {+            const uint8_t *p = in + done;+            uint8_t *q = out + done;++            CTR6(0); CTR6(1); CTR6(2); CTR6(3); CTR6(4); CTR6(5);+            ROUND6(1); DAT(0);+            ROUND6(2); DAT(1);+            ROUND6(3); DAT(2);+            ROUND6(4); DAT(3);+            ROUND6(5); DAT(4);+            ROUND6(6); DAT(5);+            ROUND6(7);+            ROUND6(8);+            ROUND6(9);+            LAST6(10);++            _mm_storeu_si128((__m128i *) q,+                _mm_xor_si128(b0, _mm_loadu_si128((const __m128i *) p)));+            _mm_storeu_si128((__m128i *) (q + 16),+                _mm_xor_si128(b1, _mm_loadu_si128((const __m128i *) (p + 16))));+            _mm_storeu_si128((__m128i *) (q + 32),+                _mm_xor_si128(b2, _mm_loadu_si128((const __m128i *) (p + 32))));+            _mm_storeu_si128((__m128i *) (q + 48),+                _mm_xor_si128(b3, _mm_loadu_si128((const __m128i *) (p + 48))));+            _mm_storeu_si128((__m128i *) (q + 64),+                _mm_xor_si128(b4, _mm_loadu_si128((const __m128i *) (p + 64))));+            _mm_storeu_si128((__m128i *) (q + 80),+                _mm_xor_si128(b5, _mm_loadu_si128((const __m128i *) (p + 80))));+        }+    } else {+        for (; done + 96 <= inlen; done += 96) {+            const uint8_t *p = in + done;+            uint8_t *q = out + done;+            int r;++            CTR6(0); CTR6(1); CTR6(2); CTR6(3); CTR6(4); CTR6(5);+            for (r = 1; r < rounds; r++) {+                ROUND6(r);+                if (r <= 6) DAT(r - 1);+            }+            LAST6(rounds);++            _mm_storeu_si128((__m128i *) q,+                _mm_xor_si128(b0, _mm_loadu_si128((const __m128i *) p)));+            _mm_storeu_si128((__m128i *) (q + 16),+                _mm_xor_si128(b1, _mm_loadu_si128((const __m128i *) (p + 16))));+            _mm_storeu_si128((__m128i *) (q + 32),+                _mm_xor_si128(b2, _mm_loadu_si128((const __m128i *) (p + 32))));+            _mm_storeu_si128((__m128i *) (q + 48),+                _mm_xor_si128(b3, _mm_loadu_si128((const __m128i *) (p + 48))));+            _mm_storeu_si128((__m128i *) (q + 64),+                _mm_xor_si128(b4, _mm_loadu_si128((const __m128i *) (p + 64))));+            _mm_storeu_si128((__m128i *) (q + 80),+                _mm_xor_si128(b5, _mm_loadu_si128((const __m128i *) (p + 80))));+        }+    }++    /* the tail, with E(K,Y0) in a lane the length leaves idle */+    {+        __m128i t0, t1, t2, t3, t4, t5;+        int ntail = (int) ((inlen - done + 15) / 16), j;++        /* the counter is where the groups left it */+        t0 = t1 = t2 = t3 = t4 = t5 = ctrbase;+        if (ntail > 0) CTRT(t0);+        if (ntail > 1) CTRT(t1);+        if (ntail > 2) CTRT(t2);+        if (ntail > 3) CTRT(t3);+        if (ntail > 4) CTRT(t4);+        if (ntail > 5) CTRT(t5);++        if (ntail > 0) {+            WIDE6_NOQ(rk);+            if (lane_ek0) ek0 = t5;+        }++        for (j = 0; j < ntail; j++) {+            size_t off = done + 16 * (size_t) j;+            size_t n = inlen - off < 16 ? inlen - off : 16;+            __m128i c = n == 16 ? _mm_loadu_si128((const __m128i *) (in + off))+                                : loadn(in + off, n);+            __m128i ks = j == 0 ? t0 : j == 1 ? t1 : j == 2 ? t2+                       : j == 3 ? t3 : j == 4 ? t4 : t5;++            GHASH_ONE(_mm_shuffle_epi8(c, BSWAP), 0);+            {+                __m128i pl = _mm_xor_si128(ks, c);+                if (n == 16)+                    _mm_storeu_si128((__m128i *) (out + off), pl);+                else+                    storen(out + off, pl, n);+            }+        }+    }++    GHASH_ONE(_mm_set_epi64x((long long) ((uint64_t) aadlen << 3),+                             (long long) ((uint64_t) inlen << 3)), gp);++    want = _mm_xor_si128(_mm_shuffle_epi8(gtag, BSWAP), ek0);+    {+        uint8_t got[16];+        _mm_storeu_si128((__m128i *) got, want);+        if (outtag) {+            /* The caller holds the expected tag and will compare it itself. */+            memcpy(outtag, got, taglen);+            return 1;+        }+        for (i = 0; i < taglen; i++)+            diff |= (uint8_t) (got[i] ^ tag[i]);+    }+    return diff == 0;+}++#endif /* WITH_GCM_FUSED */
+ cbits/aes/gcm_fused_x86.h view
@@ -0,0 +1,55 @@+/*+ * Copyright (c) 2026 Kazu Yamamoto <kazu@iij.ad.jp>+ *+ * All rights reserved.+ *+ * Redistribution and use in source and binary forms, with or without+ * modification, are permitted provided that the following conditions+ * are met:+ * 1. Redistributions of source code must retain the above copyright+ *    notice, this list of conditions and the following disclaimer.+ * 2. Redistributions in binary form must reproduce the above copyright+ *    notice, this list of conditions and the following disclaimer in the+ *    documentation and/or other materials provided with the distribution.+ * 3. Neither the name of the author nor the names of his contributors+ *    may be used to endorse or promote products derived from this software+ *    without specific prior written permission.+ *+ * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE+ * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF+ * SUCH DAMAGE.+ */++#ifndef CRYPTON_GCM_FUSED_X86_H+#define CRYPTON_GCM_FUSED_X86_H++#include <stdint.h>+#include <stddef.h>+#include <crypton_aes.h>++void crypton_gcm_fused_key_init(aes_gcm_fused *fk, const aes_key *key);++void crypton_gcm_fused_encrypt(uint8_t *out, const aes_gcm_fused *fk,+                               const aes_key *key,+                               const uint8_t *nonce,+                               const uint8_t *aad, size_t aadlen,+                               const uint8_t *in, size_t inlen, size_t taglen,+                               const aes_key *hpkey, size_t sampleoff,+                               uint8_t *mask);++int crypton_gcm_fused_decrypt(uint8_t *out, const aes_gcm_fused *fk,+                              const aes_key *key, const uint8_t *nonce,+                              const uint8_t *aad, size_t aadlen,+                              const uint8_t *in, size_t inlen,+                              const uint8_t *tag, size_t taglen,+                              uint8_t *outtag);++#endif
+ cbits/aes/gcm_vaes512_x86.c view
@@ -0,0 +1,364 @@+/*+ * AES-GCM through VAES and VPCLMULQDQ in their 512-bit form, which takes+ * four blocks where the 256-bit form in cbits/aes/gcm_vaes_x86.c takes two+ * and the 128-bit one takes one.  The instruction rate is the same, so the+ * work per group halves again.+ *+ * This is that file widened and nothing else: the same group of sixteen+ * blocks, the same descending powers of H sharing one reduction, the same+ * round keys read from memory rather than held in registers.  Four blocks to+ * a register means the group fills four of them rather than eight, which is+ * what leaves room for the group's own ciphertext to be kept for the GHASH+ * when encrypting.+ *+ * Nothing here is borrowed.  OpenSSL's and BoringSSL's AVX-512 AES-GCM is+ * Apache-2.0 and s2n-bignum has no GCM at all.+ *+ * The reduction at the end is a copy of the one in gcm_vaes_x86.c rather+ * than a call to it: the two files are compiled for different instruction+ * sets, and a function compiled for one cannot be inlined into the other.+ */+#include "aes/gcm_vaes512_x86.h"++#ifdef WITH_GCM_VAES512++#include <string.h>+#include <immintrin.h>++#include <aes/gf.h>+#include <aes/block128.h>++#if defined(__clang__) || defined(__GNUC__)+#define V512_TARGET \+	__attribute__((target("avx512f,avx512bw,avx512vl,aes,pclmul,vaes,vpclmulqdq")))+#else+#define V512_TARGET+#endif++/*+ * Thirty-two blocks to a group, four to a register, so eight registers are+ * in flight.  The number of registers is what matters as much as the blocks+ * per instruction: AES-NI has a latency of four cycles against a throughput+ * of one, so it takes eight independent chains to keep two ports busy.  Four+ * registers of four blocks was written first and measured *slower* than the+ * 256-bit path -- the blocks per instruction had doubled and the chains had+ * halved.+ *+ * The table holds sixteen powers of H, so the GHASH of a group is two passes+ * of sixteen blocks, the second picking up the tag the first leaves.+ */+#define V512WIDE 8+#define V512HALF 4+#define V512BYTES 512++/*+ * The 128-bit multiply of cbits/aes/x86ni.c, done in all four lanes at once.+ * Every shuffle and shift here works inside its own 128-bit lane, so the+ * four products never mix: what comes out is four independent carry-less+ * products, accumulated by the caller and reduced together at the end.+ */+V512_TARGET+static inline void clmul512(__m512i a, __m512i b, __m512i *lo, __m512i *hi)+{+	const __m512i bswap = _mm512_set4_epi32(+		0x00010203, 0x04050607, 0x08090a0b, 0x0c0d0e0f);+	__m512i t3, t4, t5, t6;++	a = _mm512_shuffle_epi8(a, bswap);++	/* Karatsuba, as in the 128-bit one: three multiplies, not four */+	t3 = _mm512_clmulepi64_epi128(a, b, 0x00);+	t6 = _mm512_clmulepi64_epi128(a, b, 0x11);+	t4 = _mm512_clmulepi64_epi128(+		_mm512_xor_si512(a, _mm512_shuffle_epi32(a, _MM_PERM_BADC)),+		_mm512_xor_si512(b, _mm512_shuffle_epi32(b, _MM_PERM_BADC)),+		0x00);+	t4 = _mm512_xor_si512(t4, _mm512_xor_si512(t3, t6));++	t5 = _mm512_bslli_epi128(t4, 8);+	t4 = _mm512_bsrli_epi128(t4, 8);++	*lo = _mm512_xor_si512(t3, t5);+	*hi = _mm512_xor_si512(t6, t4);+}++/*+ * The reduction of cbits/aes/x86ni.c.  By the time it runs the four lanes+ * have been folded into one, so there is one 256-bit product to reduce.+ */+V512_TARGET+static inline __m128i gfred512(__m128i t3, __m128i t6)+{+	const __m128i bswap = _mm_set_epi8(0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15);+	__m128i t2, t4, t5, t7, t8, t9;++	t7 = _mm_srli_epi32(t3, 31);+	t8 = _mm_srli_epi32(t6, 31);+	t3 = _mm_slli_epi32(t3, 1);+	t6 = _mm_slli_epi32(t6, 1);++	t9 = _mm_srli_si128(t7, 12);+	t8 = _mm_slli_si128(t8, 4);+	t7 = _mm_slli_si128(t7, 4);+	t3 = _mm_or_si128(t3, t7);+	t6 = _mm_or_si128(t6, t8);+	t6 = _mm_or_si128(t6, t9);++	t7 = _mm_slli_epi32(t3, 31);+	t8 = _mm_slli_epi32(t3, 30);+	t9 = _mm_slli_epi32(t3, 25);++	t7 = _mm_xor_si128(t7, t8);+	t7 = _mm_xor_si128(t7, t9);+	t8 = _mm_srli_si128(t7, 4);+	t7 = _mm_slli_si128(t7, 12);+	t3 = _mm_xor_si128(t3, t7);++	t2 = _mm_srli_epi32(t3, 1);+	t4 = _mm_srli_epi32(t3, 2);+	t5 = _mm_srli_epi32(t3, 7);+	t2 = _mm_xor_si128(t2, t4);+	t2 = _mm_xor_si128(t2, t5);+	t2 = _mm_xor_si128(t2, t8);+	t3 = _mm_xor_si128(t3, t2);+	t6 = _mm_xor_si128(t6, t3);++	return _mm_shuffle_epi8(t6, bswap);+}++/* the four 128-bit lanes of a register added together */+V512_TARGET+static inline __m128i fold512(__m512i v)+{+	__m256i h = _mm256_xor_si256(_mm512_castsi512_si256(v),+	                             _mm512_extracti64x4_epi64(v, 1));++	return _mm_xor_si128(_mm256_castsi256_si128(h),+	                     _mm256_extracti128_si256(h, 1));+}++/*+ * Sixteen blocks against H^16 .. H^1, one reduction.  v[j] holds blocks 4j+ * to 4j+3 in its four lanes, so the powers for it are H^(16-4j) down to+ * H^(13-4j) -- the table's own order the other way round, hence the four+ * 128-bit loads rather than one 512-bit one.+ */+V512_TARGET+static inline __m128i ghash16(__m128i tag, const table_4bit htable,+                              const __m512i *v, int fromwire)+{+	__m512i lo = _mm512_setzero_si512(), hi = _mm512_setzero_si512();+	__m512i l, h, b;+	int j;++	for (j = 0; j < V512HALF; j++) {+		const __m128i p0 =+			_mm_loadu_si128((const __m128i *) &htable[15 - 4 * j]);+		const __m128i p1 =+			_mm_loadu_si128((const __m128i *) &htable[14 - 4 * j]);+		const __m128i p2 =+			_mm_loadu_si128((const __m128i *) &htable[13 - 4 * j]);+		const __m128i p3 =+			_mm_loadu_si128((const __m128i *) &htable[12 - 4 * j]);+		__m512i hp = _mm512_castsi128_si512(p0);++		hp = _mm512_inserti32x4(hp, p1, 1);+		hp = _mm512_inserti32x4(hp, p2, 2);+		hp = _mm512_inserti32x4(hp, p3, 3);++		b = fromwire ? _mm512_loadu_si512(v + j) : v[j];+		if (j == 0) /* the running tag joins the first block */+			b = _mm512_xor_si512(+				b, _mm512_inserti32x4(+					_mm512_setzero_si512(), tag, 0));+		clmul512(b, hp, &l, &h);+		lo = _mm512_xor_si512(lo, l);+		hi = _mm512_xor_si512(hi, h);+	}++	/* the four lanes are independent products of the same sum: fold them */+	return gfred512(fold512(lo), fold512(hi));+}++#define KK512(r) _mm512_broadcast_i32x4(_mm_loadu_si128(k_ + (r)))++#define AESENC32(K)                                                         \+	do {                                                                \+		const __m512i rk = (K);                                     \+		v[0] = _mm512_aesenc_epi128(v[0], rk);                      \+		v[1] = _mm512_aesenc_epi128(v[1], rk);                      \+		v[2] = _mm512_aesenc_epi128(v[2], rk);                      \+		v[3] = _mm512_aesenc_epi128(v[3], rk);                      \+		v[4] = _mm512_aesenc_epi128(v[4], rk);                      \+		v[5] = _mm512_aesenc_epi128(v[5], rk);                      \+		v[6] = _mm512_aesenc_epi128(v[6], rk);                      \+		v[7] = _mm512_aesenc_epi128(v[7], rk);                      \+	} while (0)++#define AESLAST32(K)                                                        \+	do {                                                                \+		const __m512i rk = (K);                                     \+		v[0] = _mm512_aesenclast_epi128(v[0], rk);                  \+		v[1] = _mm512_aesenclast_epi128(v[1], rk);                  \+		v[2] = _mm512_aesenclast_epi128(v[2], rk);                  \+		v[3] = _mm512_aesenclast_epi128(v[3], rk);                  \+		v[4] = _mm512_aesenclast_epi128(v[4], rk);                  \+		v[5] = _mm512_aesenclast_epi128(v[5], rk);                  \+		v[6] = _mm512_aesenclast_epi128(v[6], rk);                  \+		v[7] = _mm512_aesenclast_epi128(v[7], rk);                  \+	} while (0)++#define XOR32(K)                                                            \+	do {                                                                \+		const __m512i rk = (K);                                     \+		v[0] = _mm512_xor_si512(v[0], rk);                          \+		v[1] = _mm512_xor_si512(v[1], rk);                          \+		v[2] = _mm512_xor_si512(v[2], rk);                          \+		v[3] = _mm512_xor_si512(v[3], rk);                          \+		v[4] = _mm512_xor_si512(v[4], rk);                          \+		v[5] = _mm512_xor_si512(v[5], rk);                          \+		v[6] = _mm512_xor_si512(v[6], rk);                          \+		v[7] = _mm512_xor_si512(v[7], rk);                          \+	} while (0)++/*+ * The rounds are written out rather than looped for the reason the 128-bit+ * loop gives: the count is a value in the key, and a loop over it leaves the+ * round key reached through an index the compiler cannot fold.+ */+V512_TARGET+static inline __attribute__((always_inline)) void+rounds32(__m512i *v, const uint8_t *k, const int nbr)+{+	const __m128i *k_ = (const __m128i *) k;++	XOR32(KK512(0));+	AESENC32(KK512(1)); AESENC32(KK512(2)); AESENC32(KK512(3));+	AESENC32(KK512(4)); AESENC32(KK512(5)); AESENC32(KK512(6));+	AESENC32(KK512(7)); AESENC32(KK512(8)); AESENC32(KK512(9));+	if (nbr > 10) {+		AESENC32(KK512(10)); AESENC32(KK512(11));+		if (nbr > 12) {+			AESENC32(KK512(12)); AESENC32(KK512(13));+		}+	}+	AESLAST32(_mm512_broadcast_i32x4(_mm_loadu_si128(k_ + nbr)));+}++/* sixteen consecutive counters, four to a register.  GCM counts in the low+ * thirty-two bits and wraps there, which is what _mm_add_epi32 does. */+V512_TARGET+static inline __m128i counters32(__m512i *v, __m128i iv, __m128i one,+                                 __m128i bswap)+{+	int j;++	for (j = 0; j < V512WIDE; j++) {+		__m128i c0, c1, c2, c3;+		__m512i c;++		iv = _mm_add_epi32(iv, one);+		c0 = _mm_shuffle_epi8(iv, bswap);+		iv = _mm_add_epi32(iv, one);+		c1 = _mm_shuffle_epi8(iv, bswap);+		iv = _mm_add_epi32(iv, one);+		c2 = _mm_shuffle_epi8(iv, bswap);+		iv = _mm_add_epi32(iv, one);+		c3 = _mm_shuffle_epi8(iv, bswap);++		c = _mm512_castsi128_si512(c0);+		c = _mm512_inserti32x4(c, c1, 1);+		c = _mm512_inserti32x4(c, c2, 2);+		c = _mm512_inserti32x4(c, c3, 3);+		v[j] = c;+	}+	return iv;+}++/*+ * Inlined into three callers with the round count a constant in each, which+ * folds away the tests inside the group loop -- the same reason the 256-bit+ * file gives, where without it AES-256 lost what AES-128 gained.+ */+V512_TARGET+static inline __attribute__((always_inline)) uint32_t+bulk_n(uint8_t *output, aes_gcm *gcm, const aes_key *key,+       const uint8_t *input, uint32_t length, int decrypt, const int nbr)+{+	const __m128i bswap = _mm_setr_epi8(7,6,5,4,3,2,1,0,15,14,13,12,11,10,9,8);+	const __m128i one = _mm_set_epi32(0, 1, 0, 0);+	__m512i v[V512WIDE];+	__m128i iv, tag;+	uint32_t groups = length / V512BYTES;+	uint32_t done = 0;+	uint32_t g;+	int j;++	if (groups == 0)+		return 0;++	iv = _mm_shuffle_epi8(_mm_loadu_si128((const __m128i *) &gcm->civ), bswap);+	tag = _mm_loadu_si128((const __m128i *) &gcm->tag);++	for (g = 0; g < groups; g++, input += V512BYTES, output += V512BYTES,+	     done += V512BYTES) {+		iv = counters32(v, iv, one, bswap);+		rounds32(v, key->data, nbr);++		for (j = 0; j < V512WIDE; j++) {+			const __m512i in =+				_mm512_loadu_si512((const __m512i *) (input + 64 * j));++			v[j] = _mm512_xor_si512(v[j], in);+			_mm512_storeu_si512((__m512i *) (output + 64 * j), v[j]);+		}+		/* sixteen blocks to a pass, since that is how many powers of+		 * H the table holds; the second picks up the tag the first+		 * leaves */+		tag = ghash16(tag, gcm->htable,+		              decrypt ? (const __m512i *) input : v,+		              decrypt);+		tag = ghash16(tag, gcm->htable,+		              decrypt ? (const __m512i *) (input + 256)+		                      : v + V512HALF,+		              decrypt);+	}++	_mm_storeu_si128((__m128i *) &gcm->civ, _mm_shuffle_epi8(iv, bswap));+	_mm_storeu_si128((__m128i *) &gcm->tag, tag);+	return done;+}++V512_TARGET+static uint32_t bulk(uint8_t *output, aes_gcm *gcm, const aes_key *key,+                     const uint8_t *input, uint32_t length, int decrypt)+{+	switch (key->nbr) {+	case 10:+		return bulk_n(output, gcm, key, input, length, decrypt, 10);+	case 12:+		return bulk_n(output, gcm, key, input, length, decrypt, 12);+	case 14:+		return bulk_n(output, gcm, key, input, length, decrypt, 14);+	default:+		return 0; /* not a key length AES has */+	}+}++uint32_t crypton_gcm_vaes512_bulk_encrypt(uint8_t *output, aes_gcm *gcm,+                                          const aes_key *key,+                                          const uint8_t *input,+                                          uint32_t length)+{+	return bulk(output, gcm, key, input, length, 0);+}++uint32_t crypton_gcm_vaes512_bulk_decrypt(uint8_t *output, aes_gcm *gcm,+                                          const aes_key *key,+                                          const uint8_t *input,+                                          uint32_t length)+{+	return bulk(output, gcm, key, input, length, 1);+}++#endif
+ cbits/aes/gcm_vaes512_x86.h view
@@ -0,0 +1,37 @@+/*+ * AES-GCM in the 512-bit form of the AES and carry-less multiply+ * instructions, which do four blocks where the 128-bit ones do one and the+ * 256-bit ones in cbits/aes/gcm_vaes_x86.c do two.+ */+#ifndef CRYPTON_GCM_VAES512_X86_H+#define CRYPTON_GCM_VAES512_X86_H++#include <crypton_cpu.h>++#if defined(ARCH_X86) && defined(__x86_64__) && defined(WITH_AESNI) \+    && defined(WITH_PCLMUL)+#define WITH_GCM_VAES512+#endif++#ifdef WITH_GCM_VAES512++#include <stdint.h>+#include <crypton_aes.h>++/* The same contract as the 256-bit pair: whole groups off the front, the+ * counter left in gcm->civ and the running tag in gcm->tag, and the number+ * of bytes taken returned, a multiple of 512 and possibly zero. */+uint32_t crypton_gcm_vaes512_bulk_encrypt(uint8_t *output, aes_gcm *gcm,+                                          const aes_key *key,+                                          const uint8_t *input,+                                          uint32_t length);+uint32_t crypton_gcm_vaes512_bulk_decrypt(uint8_t *output, aes_gcm *gcm,+                                          const aes_key *key,+                                          const uint8_t *input,+                                          uint32_t length);++/* Thirty-two blocks is the least it will start on. */+#define GCM_VAES512_MIN_BLOCKS 32++#endif+#endif
+ cbits/aes/gcm_vaes_x86.c view
@@ -0,0 +1,325 @@+/*+ * AES-GCM through VAES and VPCLMULQDQ: the same AES and carry-less multiply+ * instructions the rest of this directory uses, in their 256-bit form, which+ * takes two blocks where the 128-bit form takes one.  The instruction rate is+ * the same, so the throughput is twice -- measured at 2.00 on an EPYC 9V74,+ * for both halves, with nothing else in the loop.+ *+ * Nothing here is borrowed.  OpenSSL's and BoringSSL's wide AES-GCM is+ * Apache-2.0, s2n-bignum has no GCM at all, and the CRYPTOGAMS assembly in+ * cbits/asm is 128-bit throughout -- its `vaesenc` is the VEX encoding of+ * AESENC on XMM, not the VAES extension.  So this is the 128-bit loop in+ * cbits/aes/x86ni_impl.c widened, and it keeps that loop's shape: a group of+ * counters through the rounds together, the round keys read from memory+ * rather than held in registers, and the group's GHASH folded against+ * descending powers of H so that sixteen blocks share one reduction.+ *+ * The powers come from the table crypton_aesni_hinit_pclmul fills.  It has+ * sixteen slots and the 128-bit loop uses eight of them; this uses all+ * sixteen, which is why that function now fills them.+ */+#include "aes/gcm_vaes_x86.h"++#ifdef WITH_GCM_VAES++#include <string.h>+#include <immintrin.h>++#include <aes/gf.h>+#include <aes/block128.h>++#if defined(__clang__) || defined(__GNUC__)+#define VAES_TARGET __attribute__((target("avx2,aes,pclmul,vaes,vpclmulqdq")))+#else+#define VAES_TARGET+#endif++/* sixteen blocks to a group, two to a register */+#define VWIDE 8++/*+ * The 128-bit multiply of cbits/aes/x86ni.c, done in both lanes at once.+ * Every shuffle and shift here works inside its own 128-bit half, so the two+ * products never mix: what comes out is two independent carry-less products,+ * accumulated by the caller and reduced together at the end.+ */+VAES_TARGET+static inline void clmul256(__m256i a, __m256i b, __m256i *lo, __m256i *hi)+{+	const __m256i bswap = _mm256_setr_epi8(+		15,14,13,12,11,10,9,8,7,6,5,4,3,2,1,0,+		15,14,13,12,11,10,9,8,7,6,5,4,3,2,1,0);+	__m256i t3, t4, t5, t6;++	a = _mm256_shuffle_epi8(a, bswap);++	/* Karatsuba, as in the 128-bit one: three multiplies, not four */+	t3 = _mm256_clmulepi64_epi128(a, b, 0x00);+	t6 = _mm256_clmulepi64_epi128(a, b, 0x11);+	t4 = _mm256_clmulepi64_epi128(+		_mm256_xor_si256(a, _mm256_shuffle_epi32(a, 0x4e)),+		_mm256_xor_si256(b, _mm256_shuffle_epi32(b, 0x4e)), 0x00);+	t4 = _mm256_xor_si256(t4, _mm256_xor_si256(t3, t6));++	t5 = _mm256_slli_si256(t4, 8);+	t4 = _mm256_srli_si256(t4, 8);++	*lo = _mm256_xor_si256(t3, t5);+	*hi = _mm256_xor_si256(t6, t4);+}++/*+ * The reduction of cbits/aes/x86ni.c, unchanged: by the time it runs the two+ * lanes have been folded into one, so there is one 256-bit product to reduce+ * and no reason to do it twice.+ */+VAES_TARGET+static inline __m128i gfred(__m128i t3, __m128i t6)+{+	const __m128i bswap = _mm_set_epi8(0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15);+	__m128i t2, t4, t5, t7, t8, t9;++	t7 = _mm_srli_epi32(t3, 31);+	t8 = _mm_srli_epi32(t6, 31);+	t3 = _mm_slli_epi32(t3, 1);+	t6 = _mm_slli_epi32(t6, 1);++	t9 = _mm_srli_si128(t7, 12);+	t8 = _mm_slli_si128(t8, 4);+	t7 = _mm_slli_si128(t7, 4);+	t3 = _mm_or_si128(t3, t7);+	t6 = _mm_or_si128(t6, t8);+	t6 = _mm_or_si128(t6, t9);++	t7 = _mm_slli_epi32(t3, 31);+	t8 = _mm_slli_epi32(t3, 30);+	t9 = _mm_slli_epi32(t3, 25);++	t7 = _mm_xor_si128(t7, t8);+	t7 = _mm_xor_si128(t7, t9);+	t8 = _mm_srli_si128(t7, 4);+	t7 = _mm_slli_si128(t7, 12);+	t3 = _mm_xor_si128(t3, t7);++	t2 = _mm_srli_epi32(t3, 1);+	t4 = _mm_srli_epi32(t3, 2);+	t5 = _mm_srli_epi32(t3, 7);+	t2 = _mm_xor_si128(t2, t4);+	t2 = _mm_xor_si128(t2, t5);+	t2 = _mm_xor_si128(t2, t8);+	t3 = _mm_xor_si128(t3, t2);+	t6 = _mm_xor_si128(t6, t3);++	return _mm_shuffle_epi8(t6, bswap);+}++/*+ * Sixteen blocks against H^16 .. H^1, one reduction.  v[j] holds blocks 2j+ * and 2j+1 in its low and high halves, so the powers for it are H^(16-2j)+ * low and H^(15-2j) high -- the table's own order the other way round, hence+ * the pair of 128-bit loads rather than one 256-bit one.+ */+VAES_TARGET+static inline __m128i ghash16(__m128i tag, const table_4bit htable,+                              const __m256i *v, int fromwire)+{+	__m256i lo = _mm256_setzero_si256(), hi = _mm256_setzero_si256();+	__m256i l, h, b;+	int j;++	for (j = 0; j < VWIDE; j++) {+		const __m256i hp = _mm256_set_m128i(+			_mm_loadu_si128((const __m128i *) &htable[14 - 2 * j]),+			_mm_loadu_si128((const __m128i *) &htable[15 - 2 * j]));++		b = fromwire ? _mm256_loadu_si256(v + j) : v[j];+		if (j == 0) /* the running tag joins the first block */+			b = _mm256_xor_si256(+				b, _mm256_inserti128_si256(+					_mm256_setzero_si256(), tag, 0));+		clmul256(b, hp, &l, &h);+		lo = _mm256_xor_si256(lo, l);+		hi = _mm256_xor_si256(hi, h);+	}++	/* the two lanes are independent products of the same sum: fold them */+	return gfred(_mm_xor_si128(_mm256_castsi256_si128(lo),+	                           _mm256_extracti128_si256(lo, 1)),+	             _mm_xor_si128(_mm256_castsi256_si128(hi),+	                           _mm256_extracti128_si256(hi, 1)));+}++#define KK(r) _mm256_broadcastsi128_si256(_mm_loadu_si128(k_ + (r)))++#define AESENC16(K)                                                          \+	do {                                                                 \+		const __m256i rk = (K);                                      \+		v[0] = _mm256_aesenc_epi128(v[0], rk);                       \+		v[1] = _mm256_aesenc_epi128(v[1], rk);                       \+		v[2] = _mm256_aesenc_epi128(v[2], rk);                       \+		v[3] = _mm256_aesenc_epi128(v[3], rk);                       \+		v[4] = _mm256_aesenc_epi128(v[4], rk);                       \+		v[5] = _mm256_aesenc_epi128(v[5], rk);                       \+		v[6] = _mm256_aesenc_epi128(v[6], rk);                       \+		v[7] = _mm256_aesenc_epi128(v[7], rk);                       \+	} while (0)++#define AESLAST16(K)                                                         \+	do {                                                                 \+		const __m256i rk = (K);                                      \+		v[0] = _mm256_aesenclast_epi128(v[0], rk);                   \+		v[1] = _mm256_aesenclast_epi128(v[1], rk);                   \+		v[2] = _mm256_aesenclast_epi128(v[2], rk);                   \+		v[3] = _mm256_aesenclast_epi128(v[3], rk);                   \+		v[4] = _mm256_aesenclast_epi128(v[4], rk);                   \+		v[5] = _mm256_aesenclast_epi128(v[5], rk);                   \+		v[6] = _mm256_aesenclast_epi128(v[6], rk);                   \+		v[7] = _mm256_aesenclast_epi128(v[7], rk);                   \+	} while (0)++#define XOR16(K)                                                             \+	do {                                                                 \+		const __m256i rk = (K);                                      \+		v[0] = _mm256_xor_si256(v[0], rk);                           \+		v[1] = _mm256_xor_si256(v[1], rk);                           \+		v[2] = _mm256_xor_si256(v[2], rk);                           \+		v[3] = _mm256_xor_si256(v[3], rk);                           \+		v[4] = _mm256_xor_si256(v[4], rk);                           \+		v[5] = _mm256_xor_si256(v[5], rk);                           \+		v[6] = _mm256_xor_si256(v[6], rk);                           \+		v[7] = _mm256_xor_si256(v[7], rk);                           \+	} while (0)++/*+ * The rounds are written out rather than looped for the reason the 128-bit+ * loop gives: the count is a value in the key, and a loop over it leaves the+ * round key reached through an index the compiler cannot fold.+ */+VAES_TARGET+static inline __attribute__((always_inline)) void+rounds16(__m256i *v, const uint8_t *k, const int nbr)+{+	const __m128i *k_ = (const __m128i *) k;++	XOR16(KK(0));+	AESENC16(KK(1)); AESENC16(KK(2)); AESENC16(KK(3));+	AESENC16(KK(4)); AESENC16(KK(5)); AESENC16(KK(6));+	AESENC16(KK(7)); AESENC16(KK(8)); AESENC16(KK(9));+	if (nbr > 10) {+		AESENC16(KK(10)); AESENC16(KK(11));+		if (nbr > 12) {+			AESENC16(KK(12)); AESENC16(KK(13));+		}+	}+	AESLAST16(_mm256_broadcastsi128_si256(_mm_loadu_si128(k_ + nbr)));+}++/* sixteen consecutive counters, two to a register.  GCM counts in the low+ * thirty-two bits and wraps there, which is what _mm_add_epi32 does. */+VAES_TARGET+static inline __m128i counters16(__m256i *v, __m128i iv, __m128i one,+                                 __m128i bswap)+{+	int j;++	for (j = 0; j < VWIDE; j++) {+		__m128i c0, c1;++		iv = _mm_add_epi32(iv, one);+		c0 = _mm_shuffle_epi8(iv, bswap);+		iv = _mm_add_epi32(iv, one);+		c1 = _mm_shuffle_epi8(iv, bswap);+		v[j] = _mm256_set_m128i(c1, c0);+	}+	return iv;+}++/*+ * The round count is a value in the key, and a test on it inside the group+ * loop is a branch the 128-bit path does not have: that one compiles a+ * separate function for each key length through the SIZED macro.  This does+ * the same thing by being inlined into three callers with the count a+ * constant in each, which folds the tests away.  Without it AES-256 lost+ * what AES-128 gained.+ */+VAES_TARGET+static inline __attribute__((always_inline)) uint32_t+bulk_n(uint8_t *output, aes_gcm *gcm, const aes_key *key,+       const uint8_t *input, uint32_t length, int decrypt, const int nbr)+{+	const __m128i bswap = _mm_setr_epi8(7,6,5,4,3,2,1,0,15,14,13,12,11,10,9,8);+	const __m128i one = _mm_set_epi32(0, 1, 0, 0);+	__m256i v[VWIDE];+	__m128i iv, tag;+	uint32_t groups = length / 256;+	uint32_t done = 0;+	uint32_t g;+	int j;++	if (groups == 0)+		return 0;++	iv = _mm_shuffle_epi8(_mm_loadu_si128((const __m128i *) &gcm->civ), bswap);+	tag = _mm_loadu_si128((const __m128i *) &gcm->tag);++	for (g = 0; g < groups; g++, input += 256, output += 256, done += 256) {+		iv = counters16(v, iv, one, bswap);+		rounds16(v, key->data, nbr);++		/*+		 * The ciphertext is what the tag is taken over, and after+		 * this exclusive or it is in v itself when encrypting.  When+		 * decrypting it is the input, which the GHASH below reads+		 * again rather than keeping: there are sixteen vector+		 * registers, the group fills eight of them, and a second+		 * eight held aside is what makes the compiler spill.  The+		 * input is in L1 from the load a moment ago.+		 */+		for (j = 0; j < VWIDE; j++) {+			const __m256i in =+				_mm256_loadu_si256((const __m256i *) (input + 32 * j));++			v[j] = _mm256_xor_si256(v[j], in);+			_mm256_storeu_si256((__m256i *) (output + 32 * j), v[j]);+		}+		tag = ghash16(tag, gcm->htable,+		              decrypt ? (const __m256i *) input : v,+		              decrypt);+	}++	_mm_storeu_si128((__m128i *) &gcm->civ, _mm_shuffle_epi8(iv, bswap));+	_mm_storeu_si128((__m128i *) &gcm->tag, tag);+	return done;+}++VAES_TARGET+static uint32_t bulk(uint8_t *output, aes_gcm *gcm, const aes_key *key,+                     const uint8_t *input, uint32_t length, int decrypt)+{+	switch (key->nbr) {+	case 10:+		return bulk_n(output, gcm, key, input, length, decrypt, 10);+	case 12:+		return bulk_n(output, gcm, key, input, length, decrypt, 12);+	case 14:+		return bulk_n(output, gcm, key, input, length, decrypt, 14);+	default:+		return 0; /* not a key length AES has */+	}+}++uint32_t crypton_gcm_vaes_bulk_encrypt(uint8_t *output, aes_gcm *gcm,+                                       const aes_key *key,+                                       const uint8_t *input, uint32_t length)+{+	return bulk(output, gcm, key, input, length, 0);+}++uint32_t crypton_gcm_vaes_bulk_decrypt(uint8_t *output, aes_gcm *gcm,+                                       const aes_key *key,+                                       const uint8_t *input, uint32_t length)+{+	return bulk(output, gcm, key, input, length, 1);+}++#endif
+ cbits/aes/gcm_vaes_x86.h view
@@ -0,0 +1,35 @@+/*+ * AES-GCM in the 256-bit form of the AES and carry-less multiply+ * instructions, which do two blocks where the 128-bit ones do one.+ */+#ifndef CRYPTON_GCM_VAES_X86_H+#define CRYPTON_GCM_VAES_X86_H++#include <crypton_cpu.h>++#if defined(ARCH_X86) && defined(__x86_64__) && defined(WITH_AESNI) \+    && defined(WITH_PCLMUL)+#define WITH_GCM_VAES+#endif++#ifdef WITH_GCM_VAES++#include <stdint.h>+#include <crypton_aes.h>++/* The bulk of a message in whole groups of sixteen blocks, leaving the+ * counter in gcm->civ and the running tag in gcm->tag where the caller's own+ * loop expects to find them.  Returns the number of bytes taken, which is a+ * multiple of 256 and may be zero. */+uint32_t crypton_gcm_vaes_bulk_encrypt(uint8_t *output, aes_gcm *gcm,+                                       const aes_key *key,+                                       const uint8_t *input, uint32_t length);+uint32_t crypton_gcm_vaes_bulk_decrypt(uint8_t *output, aes_gcm *gcm,+                                       const aes_key *key,+                                       const uint8_t *input, uint32_t length);++/* Sixteen blocks is the least it will start on. */+#define GCM_VAES_MIN_BLOCKS 16++#endif+#endif
cbits/aes/gcm_x86_asm.c view
@@ -92,19 +92,29 @@  * H, and every power of it, is kept shifted up by one bit: GCM numbers the  * bits of a field element the other way round from the way the carry-less  * multiply does, and pre-shifting the operand is what saves the correction- * after each multiply.  The bit that falls off the top is the one that the- * polynomial reduces.+ * after each multiply.+ *+ * Written from the definition.  The field is GF(2)[x] modulo x^128 + x^127 ++ * x^126 + x^121 + 1, so multiplying by x is a shift of one place, and the+ * term that leaves the top comes back as the other four.  */ TARGET_PCLMUL static __m128i twist(__m128i h) {+	/* x^127 + x^126 + x^121 + 1, the terms x^128 is congruent to */ 	const __m128i poly = _mm_set_epi64x(0xc200000000000000ULL, 1);-	__m128i carried = _mm_slli_si128(_mm_srli_epi64(h, 63), 8);-	__m128i top = _mm_shuffle_epi32(h, 0xff);-	__m128i reduce = _mm_cmpgt_epi32(_mm_setzero_si128(), top);+	/* the top bit of each half */+	__m128i tops = _mm_srli_epi64(h, 63);+	/* doubling a polynomial is a shift by one: each half doubles, and the+	 * low half's top bit becomes the high half's bottom bit */+	__m128i doubled = _mm_or_si128(_mm_add_epi64(h, h),+	                               _mm_slli_si128(tops, 8));+	/* bit 127 is the one that leaves the field; spread it to a mask by+	 * subtracting it from zero, and it brings the four terms back */+	__m128i mask = _mm_sub_epi64(_mm_setzero_si128(),+	                             _mm_unpackhi_epi64(tops, tops)); -	h = _mm_or_si128(_mm_slli_epi64(h, 1), carried);-	return _mm_xor_si128(h, _mm_and_si128(reduce, poly));+	return _mm_xor_si128(doubled, _mm_and_si128(mask, poly)); }  /* the two halves of a value added together, which is the term Karatsuba
cbits/aes/generic.c view
@@ -27,420 +27,191 @@  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF  * SUCH DAMAGE.  *- * AES implementation+ * AES, portable, for machines whose processor has no AES instructions or+ * whose build was not given them.+ *+ * This was a table-driven implementation: an S-box indexed by a byte of the+ * state, in every round and in the key expansion, which is what made it fast+ * and what made it variable-time.  It is now BearSSL's bitsliced aes_ct64,+ * which holds four blocks interleaved across eight 64-bit words and computes+ * the S-box as boolean algebra -- no table, and so no address derived from a+ * secret.  See cbits/bearssl/README.md.+ *+ * What is here is only the glue.  Nothing in this file branches or indexes on+ * the key or the data.  */  #include <stdint.h>-#include <stdlib.h>+#include <string.h> #include <crypton_aes.h>-#include <crypton_bitfn.h>--static uint8_t sbox[256] = {-	0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5, 0x30, 0x01, 0x67, 0x2b, 0xfe,-	0xd7, 0xab, 0x76, 0xca, 0x82, 0xc9, 0x7d, 0xfa, 0x59, 0x47, 0xf0, 0xad, 0xd4,-	0xa2, 0xaf, 0x9c, 0xa4, 0x72, 0xc0, 0xb7, 0xfd, 0x93, 0x26, 0x36, 0x3f, 0xf7,-	0xcc, 0x34, 0xa5, 0xe5, 0xf1, 0x71, 0xd8, 0x31, 0x15, 0x04, 0xc7, 0x23, 0xc3,-	0x18, 0x96, 0x05, 0x9a, 0x07, 0x12, 0x80, 0xe2, 0xeb, 0x27, 0xb2, 0x75, 0x09,-	0x83, 0x2c, 0x1a, 0x1b, 0x6e, 0x5a, 0xa0, 0x52, 0x3b, 0xd6, 0xb3, 0x29, 0xe3,-	0x2f, 0x84, 0x53, 0xd1, 0x00, 0xed, 0x20, 0xfc, 0xb1, 0x5b, 0x6a, 0xcb, 0xbe,-	0x39, 0x4a, 0x4c, 0x58, 0xcf, 0xd0, 0xef, 0xaa, 0xfb, 0x43, 0x4d, 0x33, 0x85,-	0x45, 0xf9, 0x02, 0x7f, 0x50, 0x3c, 0x9f, 0xa8, 0x51, 0xa3, 0x40, 0x8f, 0x92,-	0x9d, 0x38, 0xf5, 0xbc, 0xb6, 0xda, 0x21, 0x10, 0xff, 0xf3, 0xd2, 0xcd, 0x0c,-	0x13, 0xec, 0x5f, 0x97, 0x44, 0x17, 0xc4, 0xa7, 0x7e, 0x3d, 0x64, 0x5d, 0x19,-	0x73, 0x60, 0x81, 0x4f, 0xdc, 0x22, 0x2a, 0x90, 0x88, 0x46, 0xee, 0xb8, 0x14,-	0xde, 0x5e, 0x0b, 0xdb, 0xe0, 0x32, 0x3a, 0x0a, 0x49, 0x06, 0x24, 0x5c, 0xc2,-	0xd3, 0xac, 0x62, 0x91, 0x95, 0xe4, 0x79, 0xe7, 0xc8, 0x37, 0x6d, 0x8d, 0xd5,-	0x4e, 0xa9, 0x6c, 0x56, 0xf4, 0xea, 0x65, 0x7a, 0xae, 0x08, 0xba, 0x78, 0x25,-	0x2e, 0x1c, 0xa6, 0xb4, 0xc6, 0xe8, 0xdd, 0x74, 0x1f, 0x4b, 0xbd, 0x8b, 0x8a,-	0x70, 0x3e, 0xb5, 0x66, 0x48, 0x03, 0xf6, 0x0e, 0x61, 0x35, 0x57, 0xb9, 0x86,-	0xc1, 0x1d, 0x9e, 0xe1, 0xf8, 0x98, 0x11, 0x69, 0xd9, 0x8e, 0x94, 0x9b, 0x1e,-	0x87, 0xe9, 0xce, 0x55, 0x28, 0xdf, 0x8c, 0xa1, 0x89, 0x0d, 0xbf, 0xe6, 0x42,-	0x68, 0x41, 0x99, 0x2d, 0x0f, 0xb0, 0x54, 0xbb, 0x16-};--static uint8_t rsbox[256] = {-	0x52, 0x09, 0x6a, 0xd5, 0x30, 0x36, 0xa5, 0x38, 0xbf, 0x40, 0xa3, 0x9e, 0x81,-	0xf3, 0xd7, 0xfb, 0x7c, 0xe3, 0x39, 0x82, 0x9b, 0x2f, 0xff, 0x87, 0x34, 0x8e,-	0x43, 0x44, 0xc4, 0xde, 0xe9, 0xcb, 0x54, 0x7b, 0x94, 0x32, 0xa6, 0xc2, 0x23,-	0x3d, 0xee, 0x4c, 0x95, 0x0b, 0x42, 0xfa, 0xc3, 0x4e, 0x08, 0x2e, 0xa1, 0x66,-	0x28, 0xd9, 0x24, 0xb2, 0x76, 0x5b, 0xa2, 0x49, 0x6d, 0x8b, 0xd1, 0x25, 0x72,-	0xf8, 0xf6, 0x64, 0x86, 0x68, 0x98, 0x16, 0xd4, 0xa4, 0x5c, 0xcc, 0x5d, 0x65,-	0xb6, 0x92, 0x6c, 0x70, 0x48, 0x50, 0xfd, 0xed, 0xb9, 0xda, 0x5e, 0x15, 0x46,-	0x57, 0xa7, 0x8d, 0x9d, 0x84, 0x90, 0xd8, 0xab, 0x00, 0x8c, 0xbc, 0xd3, 0x0a,-	0xf7, 0xe4, 0x58, 0x05, 0xb8, 0xb3, 0x45, 0x06, 0xd0, 0x2c, 0x1e, 0x8f, 0xca,-	0x3f, 0x0f, 0x02, 0xc1, 0xaf, 0xbd, 0x03, 0x01, 0x13, 0x8a, 0x6b, 0x3a, 0x91,-	0x11, 0x41, 0x4f, 0x67, 0xdc, 0xea, 0x97, 0xf2, 0xcf, 0xce, 0xf0, 0xb4, 0xe6,-	0x73, 0x96, 0xac, 0x74, 0x22, 0xe7, 0xad, 0x35, 0x85, 0xe2, 0xf9, 0x37, 0xe8,-	0x1c, 0x75, 0xdf, 0x6e, 0x47, 0xf1, 0x1a, 0x71, 0x1d, 0x29, 0xc5, 0x89, 0x6f,-	0xb7, 0x62, 0x0e, 0xaa, 0x18, 0xbe, 0x1b, 0xfc, 0x56, 0x3e, 0x4b, 0xc6, 0xd2,-	0x79, 0x20, 0x9a, 0xdb, 0xc0, 0xfe, 0x78, 0xcd, 0x5a, 0xf4, 0x1f, 0xdd, 0xa8,-	0x33, 0x88, 0x07, 0xc7, 0x31, 0xb1, 0x12, 0x10, 0x59, 0x27, 0x80, 0xec, 0x5f,-	0x60, 0x51, 0x7f, 0xa9, 0x19, 0xb5, 0x4a, 0x0d, 0x2d, 0xe5, 0x7a, 0x9f, 0x93,-	0xc9, 0x9c, 0xef, 0xa0, 0xe0, 0x3b, 0x4d, 0xae, 0x2a, 0xf5, 0xb0, 0xc8, 0xeb,-	0xbb, 0x3c, 0x83, 0x53, 0x99, 0x61, 0x17, 0x2b, 0x04, 0x7e, 0xba, 0x77, 0xd6,-	0x26, 0xe1, 0x69, 0x14, 0x63, 0x55, 0x21, 0x0c, 0x7d-};--static uint8_t Rcon[] = {-	0x8d, 0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1b, 0x36, 0x6c, 0xd8,-	0xab, 0x4d, 0x9a, 0x2f, 0x5e, 0xbc, 0x63, 0xc6, 0x97, 0x35, 0x6a, 0xd4, 0xb3,-	0x7d, 0xfa, 0xef, 0xc5, 0x91, 0x39, 0x72, 0xe4, 0xd3, 0xbd, 0x61, 0xc2, 0x9f,-	0x25, 0x4a, 0x94, 0x33, 0x66, 0xcc, 0x83, 0x1d, 0x3a, 0x74, 0xe8, 0xcb,-};+#include "bearssl/inner.h"+#include "aes/block128.h"+#include "aes/generic.h" -#define G(a,b,c,d,e,f) { a,b,c,d,e,f }-static uint8_t gmtab[256][6] =-{-	G(0x00, 0x00, 0x00, 0x00, 0x00, 0x00), G(0x02, 0x03, 0x09, 0x0b, 0x0d, 0x0e),-	G(0x04, 0x06, 0x12, 0x16, 0x1a, 0x1c), G(0x06, 0x05, 0x1b, 0x1d, 0x17, 0x12),-	G(0x08, 0x0c, 0x24, 0x2c, 0x34, 0x38), G(0x0a, 0x0f, 0x2d, 0x27, 0x39, 0x36),-	G(0x0c, 0x0a, 0x36, 0x3a, 0x2e, 0x24), G(0x0e, 0x09, 0x3f, 0x31, 0x23, 0x2a),-	G(0x10, 0x18, 0x48, 0x58, 0x68, 0x70), G(0x12, 0x1b, 0x41, 0x53, 0x65, 0x7e),-	G(0x14, 0x1e, 0x5a, 0x4e, 0x72, 0x6c), G(0x16, 0x1d, 0x53, 0x45, 0x7f, 0x62),-	G(0x18, 0x14, 0x6c, 0x74, 0x5c, 0x48), G(0x1a, 0x17, 0x65, 0x7f, 0x51, 0x46),-	G(0x1c, 0x12, 0x7e, 0x62, 0x46, 0x54), G(0x1e, 0x11, 0x77, 0x69, 0x4b, 0x5a),-	G(0x20, 0x30, 0x90, 0xb0, 0xd0, 0xe0), G(0x22, 0x33, 0x99, 0xbb, 0xdd, 0xee),-	G(0x24, 0x36, 0x82, 0xa6, 0xca, 0xfc), G(0x26, 0x35, 0x8b, 0xad, 0xc7, 0xf2),-	G(0x28, 0x3c, 0xb4, 0x9c, 0xe4, 0xd8), G(0x2a, 0x3f, 0xbd, 0x97, 0xe9, 0xd6),-	G(0x2c, 0x3a, 0xa6, 0x8a, 0xfe, 0xc4), G(0x2e, 0x39, 0xaf, 0x81, 0xf3, 0xca),-	G(0x30, 0x28, 0xd8, 0xe8, 0xb8, 0x90), G(0x32, 0x2b, 0xd1, 0xe3, 0xb5, 0x9e),-	G(0x34, 0x2e, 0xca, 0xfe, 0xa2, 0x8c), G(0x36, 0x2d, 0xc3, 0xf5, 0xaf, 0x82),-	G(0x38, 0x24, 0xfc, 0xc4, 0x8c, 0xa8), G(0x3a, 0x27, 0xf5, 0xcf, 0x81, 0xa6),-	G(0x3c, 0x22, 0xee, 0xd2, 0x96, 0xb4), G(0x3e, 0x21, 0xe7, 0xd9, 0x9b, 0xba),-	G(0x40, 0x60, 0x3b, 0x7b, 0xbb, 0xdb), G(0x42, 0x63, 0x32, 0x70, 0xb6, 0xd5),-	G(0x44, 0x66, 0x29, 0x6d, 0xa1, 0xc7), G(0x46, 0x65, 0x20, 0x66, 0xac, 0xc9),-	G(0x48, 0x6c, 0x1f, 0x57, 0x8f, 0xe3), G(0x4a, 0x6f, 0x16, 0x5c, 0x82, 0xed),-	G(0x4c, 0x6a, 0x0d, 0x41, 0x95, 0xff), G(0x4e, 0x69, 0x04, 0x4a, 0x98, 0xf1),-	G(0x50, 0x78, 0x73, 0x23, 0xd3, 0xab), G(0x52, 0x7b, 0x7a, 0x28, 0xde, 0xa5),-	G(0x54, 0x7e, 0x61, 0x35, 0xc9, 0xb7), G(0x56, 0x7d, 0x68, 0x3e, 0xc4, 0xb9),-	G(0x58, 0x74, 0x57, 0x0f, 0xe7, 0x93), G(0x5a, 0x77, 0x5e, 0x04, 0xea, 0x9d),-	G(0x5c, 0x72, 0x45, 0x19, 0xfd, 0x8f), G(0x5e, 0x71, 0x4c, 0x12, 0xf0, 0x81),-	G(0x60, 0x50, 0xab, 0xcb, 0x6b, 0x3b), G(0x62, 0x53, 0xa2, 0xc0, 0x66, 0x35),-	G(0x64, 0x56, 0xb9, 0xdd, 0x71, 0x27), G(0x66, 0x55, 0xb0, 0xd6, 0x7c, 0x29),-	G(0x68, 0x5c, 0x8f, 0xe7, 0x5f, 0x03), G(0x6a, 0x5f, 0x86, 0xec, 0x52, 0x0d),-	G(0x6c, 0x5a, 0x9d, 0xf1, 0x45, 0x1f), G(0x6e, 0x59, 0x94, 0xfa, 0x48, 0x11),-	G(0x70, 0x48, 0xe3, 0x93, 0x03, 0x4b), G(0x72, 0x4b, 0xea, 0x98, 0x0e, 0x45),-	G(0x74, 0x4e, 0xf1, 0x85, 0x19, 0x57), G(0x76, 0x4d, 0xf8, 0x8e, 0x14, 0x59),-	G(0x78, 0x44, 0xc7, 0xbf, 0x37, 0x73), G(0x7a, 0x47, 0xce, 0xb4, 0x3a, 0x7d),-	G(0x7c, 0x42, 0xd5, 0xa9, 0x2d, 0x6f), G(0x7e, 0x41, 0xdc, 0xa2, 0x20, 0x61),-	G(0x80, 0xc0, 0x76, 0xf6, 0x6d, 0xad), G(0x82, 0xc3, 0x7f, 0xfd, 0x60, 0xa3),-	G(0x84, 0xc6, 0x64, 0xe0, 0x77, 0xb1), G(0x86, 0xc5, 0x6d, 0xeb, 0x7a, 0xbf),-	G(0x88, 0xcc, 0x52, 0xda, 0x59, 0x95), G(0x8a, 0xcf, 0x5b, 0xd1, 0x54, 0x9b),-	G(0x8c, 0xca, 0x40, 0xcc, 0x43, 0x89), G(0x8e, 0xc9, 0x49, 0xc7, 0x4e, 0x87),-	G(0x90, 0xd8, 0x3e, 0xae, 0x05, 0xdd), G(0x92, 0xdb, 0x37, 0xa5, 0x08, 0xd3),-	G(0x94, 0xde, 0x2c, 0xb8, 0x1f, 0xc1), G(0x96, 0xdd, 0x25, 0xb3, 0x12, 0xcf),-	G(0x98, 0xd4, 0x1a, 0x82, 0x31, 0xe5), G(0x9a, 0xd7, 0x13, 0x89, 0x3c, 0xeb),-	G(0x9c, 0xd2, 0x08, 0x94, 0x2b, 0xf9), G(0x9e, 0xd1, 0x01, 0x9f, 0x26, 0xf7),-	G(0xa0, 0xf0, 0xe6, 0x46, 0xbd, 0x4d), G(0xa2, 0xf3, 0xef, 0x4d, 0xb0, 0x43),-	G(0xa4, 0xf6, 0xf4, 0x50, 0xa7, 0x51), G(0xa6, 0xf5, 0xfd, 0x5b, 0xaa, 0x5f),-	G(0xa8, 0xfc, 0xc2, 0x6a, 0x89, 0x75), G(0xaa, 0xff, 0xcb, 0x61, 0x84, 0x7b),-	G(0xac, 0xfa, 0xd0, 0x7c, 0x93, 0x69), G(0xae, 0xf9, 0xd9, 0x77, 0x9e, 0x67),-	G(0xb0, 0xe8, 0xae, 0x1e, 0xd5, 0x3d), G(0xb2, 0xeb, 0xa7, 0x15, 0xd8, 0x33),-	G(0xb4, 0xee, 0xbc, 0x08, 0xcf, 0x21), G(0xb6, 0xed, 0xb5, 0x03, 0xc2, 0x2f),-	G(0xb8, 0xe4, 0x8a, 0x32, 0xe1, 0x05), G(0xba, 0xe7, 0x83, 0x39, 0xec, 0x0b),-	G(0xbc, 0xe2, 0x98, 0x24, 0xfb, 0x19), G(0xbe, 0xe1, 0x91, 0x2f, 0xf6, 0x17),-	G(0xc0, 0xa0, 0x4d, 0x8d, 0xd6, 0x76), G(0xc2, 0xa3, 0x44, 0x86, 0xdb, 0x78),-	G(0xc4, 0xa6, 0x5f, 0x9b, 0xcc, 0x6a), G(0xc6, 0xa5, 0x56, 0x90, 0xc1, 0x64),-	G(0xc8, 0xac, 0x69, 0xa1, 0xe2, 0x4e), G(0xca, 0xaf, 0x60, 0xaa, 0xef, 0x40),-	G(0xcc, 0xaa, 0x7b, 0xb7, 0xf8, 0x52), G(0xce, 0xa9, 0x72, 0xbc, 0xf5, 0x5c),-	G(0xd0, 0xb8, 0x05, 0xd5, 0xbe, 0x06), G(0xd2, 0xbb, 0x0c, 0xde, 0xb3, 0x08),-	G(0xd4, 0xbe, 0x17, 0xc3, 0xa4, 0x1a), G(0xd6, 0xbd, 0x1e, 0xc8, 0xa9, 0x14),-	G(0xd8, 0xb4, 0x21, 0xf9, 0x8a, 0x3e), G(0xda, 0xb7, 0x28, 0xf2, 0x87, 0x30),-	G(0xdc, 0xb2, 0x33, 0xef, 0x90, 0x22), G(0xde, 0xb1, 0x3a, 0xe4, 0x9d, 0x2c),-	G(0xe0, 0x90, 0xdd, 0x3d, 0x06, 0x96), G(0xe2, 0x93, 0xd4, 0x36, 0x0b, 0x98),-	G(0xe4, 0x96, 0xcf, 0x2b, 0x1c, 0x8a), G(0xe6, 0x95, 0xc6, 0x20, 0x11, 0x84),-	G(0xe8, 0x9c, 0xf9, 0x11, 0x32, 0xae), G(0xea, 0x9f, 0xf0, 0x1a, 0x3f, 0xa0),-	G(0xec, 0x9a, 0xeb, 0x07, 0x28, 0xb2), G(0xee, 0x99, 0xe2, 0x0c, 0x25, 0xbc),-	G(0xf0, 0x88, 0x95, 0x65, 0x6e, 0xe6), G(0xf2, 0x8b, 0x9c, 0x6e, 0x63, 0xe8),-	G(0xf4, 0x8e, 0x87, 0x73, 0x74, 0xfa), G(0xf6, 0x8d, 0x8e, 0x78, 0x79, 0xf4),-	G(0xf8, 0x84, 0xb1, 0x49, 0x5a, 0xde), G(0xfa, 0x87, 0xb8, 0x42, 0x57, 0xd0),-	G(0xfc, 0x82, 0xa3, 0x5f, 0x40, 0xc2), G(0xfe, 0x81, 0xaa, 0x54, 0x4d, 0xcc),-	G(0x1b, 0x9b, 0xec, 0xf7, 0xda, 0x41), G(0x19, 0x98, 0xe5, 0xfc, 0xd7, 0x4f),-	G(0x1f, 0x9d, 0xfe, 0xe1, 0xc0, 0x5d), G(0x1d, 0x9e, 0xf7, 0xea, 0xcd, 0x53),-	G(0x13, 0x97, 0xc8, 0xdb, 0xee, 0x79), G(0x11, 0x94, 0xc1, 0xd0, 0xe3, 0x77),-	G(0x17, 0x91, 0xda, 0xcd, 0xf4, 0x65), G(0x15, 0x92, 0xd3, 0xc6, 0xf9, 0x6b),-	G(0x0b, 0x83, 0xa4, 0xaf, 0xb2, 0x31), G(0x09, 0x80, 0xad, 0xa4, 0xbf, 0x3f),-	G(0x0f, 0x85, 0xb6, 0xb9, 0xa8, 0x2d), G(0x0d, 0x86, 0xbf, 0xb2, 0xa5, 0x23),-	G(0x03, 0x8f, 0x80, 0x83, 0x86, 0x09), G(0x01, 0x8c, 0x89, 0x88, 0x8b, 0x07),-	G(0x07, 0x89, 0x92, 0x95, 0x9c, 0x15), G(0x05, 0x8a, 0x9b, 0x9e, 0x91, 0x1b),-	G(0x3b, 0xab, 0x7c, 0x47, 0x0a, 0xa1), G(0x39, 0xa8, 0x75, 0x4c, 0x07, 0xaf),-	G(0x3f, 0xad, 0x6e, 0x51, 0x10, 0xbd), G(0x3d, 0xae, 0x67, 0x5a, 0x1d, 0xb3),-	G(0x33, 0xa7, 0x58, 0x6b, 0x3e, 0x99), G(0x31, 0xa4, 0x51, 0x60, 0x33, 0x97),-	G(0x37, 0xa1, 0x4a, 0x7d, 0x24, 0x85), G(0x35, 0xa2, 0x43, 0x76, 0x29, 0x8b),-	G(0x2b, 0xb3, 0x34, 0x1f, 0x62, 0xd1), G(0x29, 0xb0, 0x3d, 0x14, 0x6f, 0xdf),-	G(0x2f, 0xb5, 0x26, 0x09, 0x78, 0xcd), G(0x2d, 0xb6, 0x2f, 0x02, 0x75, 0xc3),-	G(0x23, 0xbf, 0x10, 0x33, 0x56, 0xe9), G(0x21, 0xbc, 0x19, 0x38, 0x5b, 0xe7),-	G(0x27, 0xb9, 0x02, 0x25, 0x4c, 0xf5), G(0x25, 0xba, 0x0b, 0x2e, 0x41, 0xfb),-	G(0x5b, 0xfb, 0xd7, 0x8c, 0x61, 0x9a), G(0x59, 0xf8, 0xde, 0x87, 0x6c, 0x94),-	G(0x5f, 0xfd, 0xc5, 0x9a, 0x7b, 0x86), G(0x5d, 0xfe, 0xcc, 0x91, 0x76, 0x88),-	G(0x53, 0xf7, 0xf3, 0xa0, 0x55, 0xa2), G(0x51, 0xf4, 0xfa, 0xab, 0x58, 0xac),-	G(0x57, 0xf1, 0xe1, 0xb6, 0x4f, 0xbe), G(0x55, 0xf2, 0xe8, 0xbd, 0x42, 0xb0),-	G(0x4b, 0xe3, 0x9f, 0xd4, 0x09, 0xea), G(0x49, 0xe0, 0x96, 0xdf, 0x04, 0xe4),-	G(0x4f, 0xe5, 0x8d, 0xc2, 0x13, 0xf6), G(0x4d, 0xe6, 0x84, 0xc9, 0x1e, 0xf8),-	G(0x43, 0xef, 0xbb, 0xf8, 0x3d, 0xd2), G(0x41, 0xec, 0xb2, 0xf3, 0x30, 0xdc),-	G(0x47, 0xe9, 0xa9, 0xee, 0x27, 0xce), G(0x45, 0xea, 0xa0, 0xe5, 0x2a, 0xc0),-	G(0x7b, 0xcb, 0x47, 0x3c, 0xb1, 0x7a), G(0x79, 0xc8, 0x4e, 0x37, 0xbc, 0x74),-	G(0x7f, 0xcd, 0x55, 0x2a, 0xab, 0x66), G(0x7d, 0xce, 0x5c, 0x21, 0xa6, 0x68),-	G(0x73, 0xc7, 0x63, 0x10, 0x85, 0x42), G(0x71, 0xc4, 0x6a, 0x1b, 0x88, 0x4c),-	G(0x77, 0xc1, 0x71, 0x06, 0x9f, 0x5e), G(0x75, 0xc2, 0x78, 0x0d, 0x92, 0x50),-	G(0x6b, 0xd3, 0x0f, 0x64, 0xd9, 0x0a), G(0x69, 0xd0, 0x06, 0x6f, 0xd4, 0x04),-	G(0x6f, 0xd5, 0x1d, 0x72, 0xc3, 0x16), G(0x6d, 0xd6, 0x14, 0x79, 0xce, 0x18),-	G(0x63, 0xdf, 0x2b, 0x48, 0xed, 0x32), G(0x61, 0xdc, 0x22, 0x43, 0xe0, 0x3c),-	G(0x67, 0xd9, 0x39, 0x5e, 0xf7, 0x2e), G(0x65, 0xda, 0x30, 0x55, 0xfa, 0x20),-	G(0x9b, 0x5b, 0x9a, 0x01, 0xb7, 0xec), G(0x99, 0x58, 0x93, 0x0a, 0xba, 0xe2),-	G(0x9f, 0x5d, 0x88, 0x17, 0xad, 0xf0), G(0x9d, 0x5e, 0x81, 0x1c, 0xa0, 0xfe),-	G(0x93, 0x57, 0xbe, 0x2d, 0x83, 0xd4), G(0x91, 0x54, 0xb7, 0x26, 0x8e, 0xda),-	G(0x97, 0x51, 0xac, 0x3b, 0x99, 0xc8), G(0x95, 0x52, 0xa5, 0x30, 0x94, 0xc6),-	G(0x8b, 0x43, 0xd2, 0x59, 0xdf, 0x9c), G(0x89, 0x40, 0xdb, 0x52, 0xd2, 0x92),-	G(0x8f, 0x45, 0xc0, 0x4f, 0xc5, 0x80), G(0x8d, 0x46, 0xc9, 0x44, 0xc8, 0x8e),-	G(0x83, 0x4f, 0xf6, 0x75, 0xeb, 0xa4), G(0x81, 0x4c, 0xff, 0x7e, 0xe6, 0xaa),-	G(0x87, 0x49, 0xe4, 0x63, 0xf1, 0xb8), G(0x85, 0x4a, 0xed, 0x68, 0xfc, 0xb6),-	G(0xbb, 0x6b, 0x0a, 0xb1, 0x67, 0x0c), G(0xb9, 0x68, 0x03, 0xba, 0x6a, 0x02),-	G(0xbf, 0x6d, 0x18, 0xa7, 0x7d, 0x10), G(0xbd, 0x6e, 0x11, 0xac, 0x70, 0x1e),-	G(0xb3, 0x67, 0x2e, 0x9d, 0x53, 0x34), G(0xb1, 0x64, 0x27, 0x96, 0x5e, 0x3a),-	G(0xb7, 0x61, 0x3c, 0x8b, 0x49, 0x28), G(0xb5, 0x62, 0x35, 0x80, 0x44, 0x26),-	G(0xab, 0x73, 0x42, 0xe9, 0x0f, 0x7c), G(0xa9, 0x70, 0x4b, 0xe2, 0x02, 0x72),-	G(0xaf, 0x75, 0x50, 0xff, 0x15, 0x60), G(0xad, 0x76, 0x59, 0xf4, 0x18, 0x6e),-	G(0xa3, 0x7f, 0x66, 0xc5, 0x3b, 0x44), G(0xa1, 0x7c, 0x6f, 0xce, 0x36, 0x4a),-	G(0xa7, 0x79, 0x74, 0xd3, 0x21, 0x58), G(0xa5, 0x7a, 0x7d, 0xd8, 0x2c, 0x56),-	G(0xdb, 0x3b, 0xa1, 0x7a, 0x0c, 0x37), G(0xd9, 0x38, 0xa8, 0x71, 0x01, 0x39),-	G(0xdf, 0x3d, 0xb3, 0x6c, 0x16, 0x2b), G(0xdd, 0x3e, 0xba, 0x67, 0x1b, 0x25),-	G(0xd3, 0x37, 0x85, 0x56, 0x38, 0x0f), G(0xd1, 0x34, 0x8c, 0x5d, 0x35, 0x01),-	G(0xd7, 0x31, 0x97, 0x40, 0x22, 0x13), G(0xd5, 0x32, 0x9e, 0x4b, 0x2f, 0x1d),-	G(0xcb, 0x23, 0xe9, 0x22, 0x64, 0x47), G(0xc9, 0x20, 0xe0, 0x29, 0x69, 0x49),-	G(0xcf, 0x25, 0xfb, 0x34, 0x7e, 0x5b), G(0xcd, 0x26, 0xf2, 0x3f, 0x73, 0x55),-	G(0xc3, 0x2f, 0xcd, 0x0e, 0x50, 0x7f), G(0xc1, 0x2c, 0xc4, 0x05, 0x5d, 0x71),-	G(0xc7, 0x29, 0xdf, 0x18, 0x4a, 0x63), G(0xc5, 0x2a, 0xd6, 0x13, 0x47, 0x6d),-	G(0xfb, 0x0b, 0x31, 0xca, 0xdc, 0xd7), G(0xf9, 0x08, 0x38, 0xc1, 0xd1, 0xd9),-	G(0xff, 0x0d, 0x23, 0xdc, 0xc6, 0xcb), G(0xfd, 0x0e, 0x2a, 0xd7, 0xcb, 0xc5),-	G(0xf3, 0x07, 0x15, 0xe6, 0xe8, 0xef), G(0xf1, 0x04, 0x1c, 0xed, 0xe5, 0xe1),-	G(0xf7, 0x01, 0x07, 0xf0, 0xf2, 0xf3), G(0xf5, 0x02, 0x0e, 0xfb, 0xff, 0xfd),-	G(0xeb, 0x13, 0x79, 0x92, 0xb4, 0xa7), G(0xe9, 0x10, 0x70, 0x99, 0xb9, 0xa9),-	G(0xef, 0x15, 0x6b, 0x84, 0xae, 0xbb), G(0xed, 0x16, 0x62, 0x8f, 0xa3, 0xb5),-	G(0xe3, 0x1f, 0x5d, 0xbe, 0x80, 0x9f), G(0xe1, 0x1c, 0x54, 0xb5, 0x8d, 0x91),-	G(0xe7, 0x19, 0x4f, 0xa8, 0x9a, 0x83), G(0xe5, 0x1a, 0x46, 0xa3, 0x97, 0x8d),-};-#undef G+/*+ * The schedule is kept in its compressed form, 30 words of it, inside the+ * aes_key the caller already has -- comfortably inside the 448 bytes that+ * held the round keys before.  br_aes_ct64_skey_expand blows it up to 120+ * words on the stack once per call, which is how BearSSL's own CTR and CBC+ * use it.+ *+ * It travels through memcpy rather than a cast because aes_key is all+ * uint8_t and so carries no alignment of its own, whatever the allocator+ * happens to give it.+ */+#define COMP_SKEY_WORDS 30 -static void expand_key(uint8_t *expandedKey, uint8_t *key, int size, size_t expandedKeySize)+void crypton_aes_generic_schedule(aes_sched *sched, const aes_key *key) {-	int csz;-	int i;-	uint8_t t[4] = { 0 };--	for (i = 0; i < size; i++)-		expandedKey[i] = key[i];-	csz = size;+	uint64_t comp_skey[COMP_SKEY_WORDS]; -	i = 1;-	while (csz < expandedKeySize) {-		t[0] = expandedKey[(csz - 4) + 0];-		t[1] = expandedKey[(csz - 4) + 1];-		t[2] = expandedKey[(csz - 4) + 2];-		t[3] = expandedKey[(csz - 4) + 3];+	memcpy(comp_skey, key->data, sizeof comp_skey);+	sched->nbr = key->nbr;+	br_aes_ct64_skey_expand(sched->sk_exp, sched->nbr, comp_skey);+} -		if (csz % size == 0) {-			uint8_t tmp;+/*+ * Up to four blocks through the bitsliced core at once.  Fewer than four is+ * the same work as four -- the lanes are there whether anything is in them+ * -- so a caller with four to offer gets them for what one used to cost.+ */+static void pass(uint8_t *output, const uint8_t *input, unsigned n,+                 const aes_sched *sched, int decrypt)+{+	uint32_t w[16];+	uint64_t q[8];+	unsigned i; -			tmp = t[0];-			t[0] = sbox[t[1]] ^ Rcon[i++ % sizeof(Rcon)];-			t[1] = sbox[t[2]];-			t[2] = sbox[t[3]];-			t[3] = sbox[tmp];-		}+	memset(w, 0, sizeof w);+	for (i = 0; i < n; i++) {+		w[4 * i]     = br_dec32le(input + 16 * i);+		w[4 * i + 1] = br_dec32le(input + 16 * i + 4);+		w[4 * i + 2] = br_dec32le(input + 16 * i + 8);+		w[4 * i + 3] = br_dec32le(input + 16 * i + 12);+	} -		if (size == 32 && ((csz % size) == 16)) {-			t[0] = sbox[t[0]];-			t[1] = sbox[t[1]];-			t[2] = sbox[t[2]];-			t[3] = sbox[t[3]];-		}+	for (i = 0; i < 4; i++)+		br_aes_ct64_interleave_in(&q[i], &q[i + 4], w + 4 * i);+	br_aes_ct64_ortho(q);+	if (decrypt)+		br_aes_ct64_bitslice_decrypt(sched->nbr, sched->sk_exp, q);+	else+		br_aes_ct64_bitslice_encrypt(sched->nbr, sched->sk_exp, q);+	br_aes_ct64_ortho(q);+	for (i = 0; i < 4; i++)+		br_aes_ct64_interleave_out(w + 4 * i, q[i], q[i + 4]); -		expandedKey[csz] = expandedKey[csz - size] ^ t[0]; csz++;-		expandedKey[csz] = expandedKey[csz - size] ^ t[1]; csz++;-		expandedKey[csz] = expandedKey[csz - size] ^ t[2]; csz++;-		expandedKey[csz] = expandedKey[csz - size] ^ t[3]; csz++;+	for (i = 0; i < n; i++) {+		br_enc32le(output + 16 * i,      w[4 * i]);+		br_enc32le(output + 16 * i + 4,  w[4 * i + 1]);+		br_enc32le(output + 16 * i + 8,  w[4 * i + 2]);+		br_enc32le(output + 16 * i + 12, w[4 * i + 3]); 	} } -static void shift_rows(uint8_t *state)+void crypton_aes_generic_blocks(uint8_t *output, const uint8_t *input,+                                uint32_t nb_blocks, const aes_sched *sched,+                                int decrypt) {-	uint32_t *s32;-	int i;--	for (i = 0; i < 16; i++)-		state[i] = sbox[state[i]];-	s32 = (uint32_t *) state;-	s32[1] = rol32_be(s32[1], 8);-	s32[2] = rol32_be(s32[2], 16);-	s32[3] = rol32_be(s32[3], 24);+	while (nb_blocks >= 4) {+		pass(output, input, 4, sched, decrypt);+		output += 64;+		input += 64;+		nb_blocks -= 4;+	}+	if (nb_blocks > 0)+		pass(output, input, (unsigned) nb_blocks, sched, decrypt); } -static void add_round_key(uint8_t *state, uint8_t *rk)+static void one(aes_block *output, aes_key *key, aes_block *input, int decrypt) {-	uint32_t *s32, *r32;+	aes_sched sched; -	s32 = (uint32_t *) state;-	r32 = (uint32_t *) rk;-	s32[0] ^= r32[0];-	s32[1] ^= r32[1];-	s32[2] ^= r32[2];-	s32[3] ^= r32[3];+	crypton_aes_generic_schedule(&sched, key);+	crypton_aes_generic_blocks((uint8_t *) output, (const uint8_t *) input,+	                           1, &sched, decrypt); } -#define gm1(a) (a)-#define gm2(a) gmtab[a][0]-#define gm3(a) gmtab[a][1]-#define gm9(a) gmtab[a][2]-#define gm11(a) gmtab[a][3]-#define gm13(a) gmtab[a][4]-#define gm14(a) gmtab[a][5]--static void mix_columns(uint8_t *state)+void crypton_aes_generic_encrypt_block(aes_block *output, aes_key *key, aes_block *input) {-	int i;-	uint8_t cpy[4];--	for (i = 0; i < 4; i++) {-		cpy[0] = state[0 * 4 + i];-		cpy[1] = state[1 * 4 + i];-		cpy[2] = state[2 * 4 + i];-		cpy[3] = state[3 * 4 + i];-		state[i] = gm2(cpy[0]) ^ gm1(cpy[3]) ^ gm1(cpy[2]) ^ gm3(cpy[1]);-		state[4+i] = gm2(cpy[1]) ^ gm1(cpy[0]) ^ gm1(cpy[3]) ^ gm3(cpy[2]);-		state[8+i] = gm2(cpy[2]) ^ gm1(cpy[1]) ^ gm1(cpy[0]) ^ gm3(cpy[3]);-		state[12+i] = gm2(cpy[3]) ^ gm1(cpy[2]) ^ gm1(cpy[1]) ^ gm3(cpy[0]);-	}+	one(output, key, input, 0); } -static void create_round_key(uint8_t *expandedKey, uint8_t *rk)+void crypton_aes_generic_decrypt_block(aes_block *output, aes_key *key, aes_block *input) {-	int i,j;-	for (i = 0; i < 4; i++)-		for (j = 0; j < 4; j++)-			rk[i + j * 4] = expandedKey[i * 4 + j];+	one(output, key, input, 1); } -static void aes_main(aes_key *key, uint8_t *state)+void crypton_aes_generic_init(aes_key *key, uint8_t *origkey, uint8_t size) {-	int i = 0;-	uint32_t rk[4];-	uint8_t *rkptr = (uint8_t *) rk;--	create_round_key(key->data, rkptr);-	add_round_key(state, rkptr);--	for (i = 1; i < key->nbr; i++) {-		create_round_key(key->data + 16 * i, rkptr);-		shift_rows(state);-		mix_columns(state);-		add_round_key(state, rkptr);-	}--	create_round_key(key->data + 16 * key->nbr, rkptr);-	shift_rows(state);-	add_round_key(state, rkptr);-}+	uint64_t comp_skey[COMP_SKEY_WORDS];+	unsigned nbr; -static void shift_rows_inv(uint8_t *state)-{-	uint32_t *s32;-	int i;+	/* 0 for a key length that is not 16, 24 or 32; the old code returned+	 * without touching the key in that case and so does this */+	nbr = br_aes_ct64_keysched(comp_skey, origkey, size);+	if (nbr == 0)+		return; -	s32 = (uint32_t *) state;-	s32[1] = ror32_be(s32[1], 8);-	s32[2] = ror32_be(s32[2], 16);-	s32[3] = ror32_be(s32[3], 24);-	for (i = 0; i < 16; i++)-		state[i] = rsbox[state[i]];+	key->nbr = (uint8_t) nbr;+	memcpy(key->data, comp_skey, sizeof comp_skey); } -static void mix_columns_inv(uint8_t *state)+/*+ * CTR, four counter blocks at a time.  The counter itself is serial, but+ * nothing about it depends on the keystream, so the four blocks it will+ * reach next can be written down before any of them is encrypted.+ */+static void ctr(uint8_t *output, aes_key *key, aes_block *iv,+                uint8_t *input, uint32_t len, int c32) {-	int i;-	uint8_t cpy[4];+	aes_sched sched;+	aes_block counter;+	uint8_t ks[64];+	uint32_t nb_blocks = len / 16;+	uint32_t tail = len % 16;+	uint32_t i; -	for (i = 0; i < 4; i++) {-		cpy[0] = state[0 * 4 + i];-		cpy[1] = state[1 * 4 + i];-		cpy[2] = state[2 * 4 + i];-		cpy[3] = state[3 * 4 + i];-		state[i] = gm14(cpy[0]) ^ gm9(cpy[3]) ^ gm13(cpy[2]) ^ gm11(cpy[1]);-		state[4+i] = gm14(cpy[1]) ^ gm9(cpy[0]) ^ gm13(cpy[3]) ^ gm11(cpy[2]);-		state[8+i] = gm14(cpy[2]) ^ gm9(cpy[1]) ^ gm13(cpy[0]) ^ gm11(cpy[3]);-		state[12+i] = gm14(cpy[3]) ^ gm9(cpy[2]) ^ gm13(cpy[1]) ^ gm11(cpy[0]);-	}-}+	crypton_aes_generic_schedule(&sched, key);+	block128_copy(&counter, iv); -static void aes_main_inv(aes_key *key, uint8_t *state)-{-	int i = 0;-	uint32_t rk[4];-	uint8_t *rkptr = (uint8_t *) rk;+	while (nb_blocks > 0) {+		uint32_t n = nb_blocks < 4 ? nb_blocks : 4; -	create_round_key(key->data + 16 * key->nbr, rkptr);-	add_round_key(state, rkptr);+		for (i = 0; i < n; i++) {+			block128_copy((block128 *) (ks + 16 * i), &counter);+			if (c32)+				block128_inc32_le(&counter);+			else+				block128_inc_be(&counter);+		}+		crypton_aes_generic_blocks(ks, ks, n, &sched, 0);+		for (i = 0; i < n * 16; i++)+			output[i] = ks[i] ^ input[i]; -	for (i = key->nbr - 1; i > 0; i--) {-		create_round_key(key->data + 16 * i, rkptr);-		shift_rows_inv(state);-		add_round_key(state, rkptr);-		mix_columns_inv(state);+		output += n * 16;+		input += n * 16;+		nb_blocks -= n; 	} -	create_round_key(key->data, rkptr);-	shift_rows_inv(state);-	add_round_key(state, rkptr);-}--/* Set the block values, for the block:- * a0,0 a0,1 a0,2 a0,3- * a1,0 a1,1 a1,2 a1,3 -> a0,0 a1,0 a2,0 a3,0 a0,1 a1,1 ... a2,3 a3,3- * a2,0 a2,1 a2,2 a2,3- * a3,0 a3,1 a3,2 a3,3- */-#define swap_block(t, f) \-	t[0] = f[0]; t[4] = f[1]; t[8] = f[2]; t[12] = f[3]; \-	t[1] = f[4]; t[5] = f[5]; t[9] = f[6]; t[13] = f[7]; \-	t[2] = f[8]; t[6] = f[9]; t[10] = f[10]; t[14] = f[11]; \-	t[3] = f[12]; t[7] = f[13]; t[11] = f[14]; t[15] = f[15]--void crypton_aes_generic_encrypt_block(aes_block *output, aes_key *key, aes_block *input)-{-	uint32_t block[4];-	uint8_t *iptr, *optr, *bptr;--	iptr = (uint8_t *) input;-	optr = (uint8_t *) output;-	bptr = (uint8_t *) block;-	swap_block(bptr, iptr);-	aes_main(key, bptr);-	swap_block(optr, bptr);+	if (tail != 0) {+		block128_copy((block128 *) ks, &counter);+		crypton_aes_generic_blocks(ks, ks, 1, &sched, 0);+		for (i = 0; i < tail; i++)+			output[i] = ks[i] ^ input[i];+	} } -void crypton_aes_generic_decrypt_block(aes_block *output, aes_key *key, aes_block *input)+void crypton_aes_bitsliced_encrypt_ctr(uint8_t *output, aes_key *key,+                                       aes_block *iv, uint8_t *input,+                                       uint32_t len) {-	uint32_t block[4];-	uint8_t *iptr, *optr, *bptr;--	iptr = (uint8_t *) input;-	optr = (uint8_t *) output;-	bptr = (uint8_t *) block;-	swap_block(bptr, iptr);-	aes_main_inv(key, bptr);-	swap_block(optr, bptr);+	ctr(output, key, iv, input, len, 0); } -void crypton_aes_generic_init(aes_key *key, uint8_t *origkey, uint8_t size)+void crypton_aes_bitsliced_encrypt_c32(uint8_t *output, aes_key *key,+                                       aes_block *iv, uint8_t *input,+                                       uint32_t len) {-	int esz;--	switch (size) {-	case 16: key->nbr = 10; esz = 176; break;-	case 24: key->nbr = 12; esz = 208; break;-	case 32: key->nbr = 14; esz = 240; break;-	default: return;-	}-	expand_key(key->data, origkey, size, esz);-	return;+	ctr(output, key, iv, input, len, 1); }
cbits/aes/generic.h view
@@ -32,3 +32,35 @@ void crypton_aes_generic_encrypt_block(aes_block *output, aes_key *key, aes_block *input); void crypton_aes_generic_decrypt_block(aes_block *output, aes_key *key, aes_block *input); void crypton_aes_generic_init(aes_key *key, uint8_t *origkey, uint8_t size);++/*+ * The bitsliced core takes four blocks at a time, and the schedule it reads+ * is the expanded one rather than the compressed form the aes_key holds.+ * Expanding costs about what a block costs, so a mode with more than one+ * block to do expands once, here, and hands the result to every group.+ */+typedef struct {+	uint64_t sk_exp[120];+	unsigned nbr;+} aes_sched;++void crypton_aes_generic_schedule(aes_sched *sched, const aes_key *key);++/* nb_blocks of them, four at a pass; decrypt selects the direction */+void crypton_aes_generic_blocks(uint8_t *output, const uint8_t *input,+                                uint32_t nb_blocks, const aes_sched *sched,+                                int decrypt);++/*+ * CTR with the two counters crypton uses.  These are not the generic+ * entries: those go through the branch table for the block itself and so+ * run on accelerated machines too, where the key holds a different+ * schedule.  crypton_aes.c installs these only when nothing was+ * accelerated.+ */+void crypton_aes_bitsliced_encrypt_ctr(uint8_t *output, aes_key *key,+                                       aes_block *iv, uint8_t *input,+                                       uint32_t len);+void crypton_aes_bitsliced_encrypt_c32(uint8_t *output, aes_key *key,+                                       aes_block *iv, uint8_t *input,+                                       uint32_t len);
cbits/aes/gf.c view
@@ -33,6 +33,7 @@ #include <crypton_cpu.h> #include <aes/gf.h> #include <aes/x86ni.h>+#include "bearssl/inner.h"  /* inplace GFMUL for xts mode */ void crypton_aes_generic_gf_mulx(block128 *a)@@ -45,118 +46,43 @@   /*- * GF multiplication with Shoup's method and 4-bit table.+ * GHASH, without a table.  *- * We precompute the products of H with all 4-bit polynomials and store them in- * a 'table_4bit' array.  To avoid unnecessary byte swapping, the 16 blocks are- * written to the table with qwords already converted to CPU order.  Table- * indices use the reflected bit ordering, i.e. polynomials X^0, X^1, X^2, X^3- * map to bit positions 3, 2, 1, 0 respectively.+ * This was Shoup's method: the products of H with all sixteen 4-bit+ * polynomials, precomputed, and thirty-two lookups a block at indices taken+ * from the accumulator -- which is to say, thirty-two addresses derived from+ * a secret.  It is now BearSSL's ghash_ctmul64, which builds the GF(2^128)+ * multiply out of shifts, masks and integer multiplies and looks nothing up.+ * See cbits/bearssl/README.md.  *- * To multiply an arbitrary block with H, the input block is decomposed in 4-bit- * segments.  We get the final result after 32 table lookups and additions, one- * for each segment, interleaving multiplication by P(X)=X^4.+ * The table_4bit the interface names is sixteen blocks wide because the+ * table needed it.  This keeps H in the first and leaves the rest alone; the+ * PMULL and PCLMUL implementations keep their own powers of H in that same+ * space, so the width stays as it is.  */ -/* convert block128 qwords between BE and CPU order */-static inline void block128_cpu_swap_be(block128 *a, const block128 *b)-{-	a->q[1] = cpu_to_be64(b->q[1]);-	a->q[0] = cpu_to_be64(b->q[0]);-}--/* multiplication by P(X)=X, assuming qwords already in CPU order */-static inline void cpu_gf_mulx(block128 *a, const block128 *b)-{-	uint64_t v0 = b->q[0];-	uint64_t v1 = b->q[1];-	a->q[1] = v1 >> 1 | v0 << 63;-	a->q[0] = v0 >> 1 ^ ((0-(v1 & 1)) & 0xe100000000000000ULL);-}--static const uint64_t r4_0[] =-	{ 0x0000000000000000ULL, 0x1c20000000000000ULL-	, 0x3840000000000000ULL, 0x2460000000000000ULL-	, 0x7080000000000000ULL, 0x6ca0000000000000ULL-	, 0x48c0000000000000ULL, 0x54e0000000000000ULL-	, 0xe100000000000000ULL, 0xfd20000000000000ULL-	, 0xd940000000000000ULL, 0xc560000000000000ULL-	, 0x9180000000000000ULL, 0x8da0000000000000ULL-	, 0xa9c0000000000000ULL, 0xb5e0000000000000ULL-	};--/* multiplication by P(X)=X^4, assuming qwords already in CPU order */-static inline void cpu_gf_mulx4(block128 *a, const block128 *b)-{-	uint64_t v0 = b->q[0];-	uint64_t v1 = b->q[1];-	a->q[1] = v1 >> 4 | v0 << 60;-	a->q[0] = v0 >> 4 ^ r4_0[v1 & 0xf];-}--/* initialize the 4-bit table given H */+/* remember H */ void crypton_aes_generic_hinit(table_4bit htable, const block128 *h) {-	block128 v, *p;-	int i, j;--	/* multiplication by 0 is 0 */-	block128_zero(&htable[0]);--	/* at index 8=2^3 we have H.X^0 = H */-	i = 8;-	block128_cpu_swap_be(&htable[i], h); /* in CPU order */-	p = &htable[i];--	/* for other powers of 2, repeat multiplication by P(X)=X */-	for (i = 4; i > 0; i >>= 1)-	{-		cpu_gf_mulx(&htable[i], p);-		p = &htable[i];-	}--	/* remaining elements are linear combinations */-	for (i = 2; i < 16; i <<= 1) {-		p = &htable[i];-		v = *p;-		for (j = 1; j < i; j++) {-			p[j] = v;-			block128_xor_aligned(&p[j], &htable[j]);-		}-	}+	block128_copy(&htable[0], h); } -/* multiply a block with H */+/*+ * br_ghash_ctmul64 computes y = (y ^ x) * H for each block x it is given, so+ * a block of zeros is the bare multiply this entry is asked for.+ */ void crypton_aes_generic_gf_mul(block128 *a, const table_4bit htable) {-	block128 b;-	int i;-	block128_zero(&b);-	for (i = 15; i >= 0; i--)-	{-		uint8_t v = a->b[i];-		block128_xor_aligned(&b, &htable[v & 0xf]); /* high bits (reflected) */-		cpu_gf_mulx4(&b, &b);-		block128_xor_aligned(&b, &htable[v >> 4]);  /* low bits (reflected) */-		if (i > 0)-			cpu_gf_mulx4(&b, &b);-		else-			block128_cpu_swap_be(a, &b); /* restore BE order when done */-	}+	static const uint8_t zero[16] = { 0 };++	br_ghash_ctmul64(a, &htable[0], zero, sizeof zero); }  /*- * Four GHASH steps at once.  The generic table-driven multiply has no cheaper- * way to do this than one block at a time; the point of the entry is that the- * PMULL and PCLMUL versions can fold the four products into one reduction, so- * the GCM loops hand over four blocks whenever they have them.+ * Four GHASH steps at once, which here is one call rather than four: the+ * loop inside br_ghash_ctmul64 takes the blocks as they come.  */ void crypton_aes_generic_gf_mul4(block128 *a, const block128 *blocks, const table_4bit htable) {-	int i;--	for (i = 0; i < 4; i++) {-		block128_xor(a, &blocks[i]);-		crypton_aes_generic_gf_mul(a, htable);-	}+	br_ghash_ctmul64(a, &htable[0], blocks, 4 * sizeof(block128)); }
+ cbits/aes/ppc8.c view
@@ -0,0 +1,287 @@+/*+ * AES and GHASH using the PowerISA 2.07 vector instructions, first+ * implemented by POWER8.+ *+ * The instructions themselves come from CRYPTOGAMS, assembled from+ * cbits/asm/aesp8-ppc-*.S and cbits/asm/ghashp8-ppc-*.S; what is here is the+ * glue that puts them behind crypton's branch table.  Nothing in this file+ * branches or indexes on a key or on data.+ *+ * == Where the key schedule lives+ *+ * The assembly takes OpenSSL's AES_KEY -- sixty round-key words and a round+ * count, 244 bytes -- and its set_decrypt_key writes a second, complete one+ * rather than sharing ends with the first the way cbits/aes/armv8.c does.+ * Two of those are 488 bytes and aes_key.data is 448, so both do not fit.+ *+ * So the forward schedule is kept there, with the key the caller gave after+ * it, and the inverse is built on the stack by the operations that need it.+ * Those are all bulk -- ECB, CBC and XTS decryption -- so it is one key+ * schedule per call rather than per block.  Single-block decryption pays for+ * one too, and is reached by nothing that runs in a loop: OCB and CCM drive+ * the block function in the encrypting direction.+ */++#include <stdint.h>+#include <string.h>+#include <crypton_aes.h>+#include <crypton_bitfn.h>+#include "aes/block128.h"+#include "aes/gf.h"+#include "aes/ppc8.h"++/* OpenSSL's AES_KEY, which is what the assembly was written against */+typedef struct {+	unsigned int rd_key[60];+	int rounds;+} p8_key;++int  crypton_aes_p8_set_encrypt_key(const unsigned char *, int, p8_key *);+int  crypton_aes_p8_set_decrypt_key(const unsigned char *, int, p8_key *);+void crypton_aes_p8_encrypt(const unsigned char *, unsigned char *, const p8_key *);+void crypton_aes_p8_decrypt(const unsigned char *, unsigned char *, const p8_key *);+void crypton_aes_p8_cbc_encrypt(const unsigned char *, unsigned char *, size_t,+                                const p8_key *, unsigned char *, int);+void crypton_aes_p8_ctr32_encrypt_blocks(const unsigned char *, unsigned char *,+                                         size_t, const p8_key *,+                                         const unsigned char *);+void crypton_aes_p8_xts_encrypt(const unsigned char *, unsigned char *, size_t,+                                const p8_key *, const p8_key *,+                                const unsigned char *);+void crypton_aes_p8_xts_decrypt(const unsigned char *, unsigned char *, size_t,+                                const p8_key *, const p8_key *,+                                const unsigned char *);+/* void * rather than uint64_t *: the assembly loads these with lvx_u and so+ * does not want the alignment a uint64_t pointer would promise, and+ * block128 is packed. */+void crypton_gcm_init_p8(void *Htable, const void *H);+void crypton_gcm_gmult_p8(void *Xi, const void *Htable);+void crypton_gcm_ghash_p8(void *Xi, const void *Htable,+                          const unsigned char *inp, size_t len);++#define FORWARD(k)  ((p8_key *) (void *) (k)->data)+#define USERKEY(k)  ((uint8_t *) (k)->data + sizeof(p8_key))+#define USERLEN(k)  (*((uint8_t *) (k)->data + sizeof(p8_key) + 32))++static void inverse(p8_key *dk, aes_key *key)+{+	crypton_aes_p8_set_decrypt_key(USERKEY(key), USERLEN(key) * 8, dk);+}++void crypton_aes_ppc8_init(aes_key *key, uint8_t *origkey, uint8_t size)+{+	if (size != 16 && size != 24 && size != 32)+		return;+	crypton_aes_p8_set_encrypt_key(origkey, size * 8, FORWARD(key));+	memcpy(USERKEY(key), origkey, size);+	USERLEN(key) = size;+}++void crypton_aes_ppc8_encrypt_block(aes_block *output, aes_key *key, aes_block *input)+{+	crypton_aes_p8_encrypt((const unsigned char *) input,+	                       (unsigned char *) output, FORWARD(key));+}++void crypton_aes_ppc8_decrypt_block(aes_block *output, aes_key *key, aes_block *input)+{+	p8_key dk;++	inverse(&dk, key);+	crypton_aes_p8_decrypt((const unsigned char *) input,+	                       (unsigned char *) output, &dk);+	memset(&dk, 0, sizeof dk);+}++/* The assembly has no ECB entry: it is the block function in a loop, which+ * is what the generic implementation does too. */+void crypton_aes_ppc8_encrypt_ecb(aes_block *output, aes_key *key, aes_block *input,+                                  uint32_t nb_blocks)+{+	for (; nb_blocks-- > 0; input++, output++)+		crypton_aes_p8_encrypt((const unsigned char *) input,+		                       (unsigned char *) output, FORWARD(key));+}++void crypton_aes_ppc8_decrypt_ecb(aes_block *output, aes_key *key, aes_block *input,+                                  uint32_t nb_blocks)+{+	p8_key dk;++	inverse(&dk, key);+	for (; nb_blocks-- > 0; input++, output++)+		crypton_aes_p8_decrypt((const unsigned char *) input,+		                       (unsigned char *) output, &dk);+	memset(&dk, 0, sizeof dk);+}++void crypton_aes_ppc8_encrypt_cbc(aes_block *output, aes_key *key, aes_block *iv,+                                  aes_block *input, uint32_t nb_blocks)+{+	uint8_t ivbuf[16];++	/* a copy, because the assembly writes the last block back through this+	 * pointer and the callers of this entry do not expect their IV touched */+	memcpy(ivbuf, iv, 16);+	crypton_aes_p8_cbc_encrypt((const unsigned char *) input,+	                           (unsigned char *) output,+	                           (size_t) nb_blocks * 16, FORWARD(key), ivbuf, 1);+}++void crypton_aes_ppc8_decrypt_cbc(aes_block *output, aes_key *key, aes_block *iv,+                                  aes_block *input, uint32_t nb_blocks)+{+	p8_key dk;+	uint8_t ivbuf[16];++	inverse(&dk, key);+	memcpy(ivbuf, iv, 16);+	crypton_aes_p8_cbc_encrypt((const unsigned char *) input,+	                           (unsigned char *) output,+	                           (size_t) nb_blocks * 16, &dk, ivbuf, 0);+	memset(&dk, 0, sizeof dk);+}++/*+ * CTR, which is where the two counters have to be reconciled.+ *+ * crypton counts with block128_inc_be, over the whole 128 bits; the assembly+ * counts over the low 32 only.  They agree until that word wraps, so the+ * work is handed over in runs that stop there, and the carry into the upper+ * bits is done here.  This is the same arrangement OpenSSL makes for the+ * same reason.+ */+void crypton_aes_ppc8_encrypt_ctr(uint8_t *output, aes_key *key, aes_block *iv,+                                  uint8_t *input, uint32_t len)+{+	block128 ctr;+	uint32_t nb_blocks = len / 16;+	uint32_t tail = len % 16;+	uint32_t i;++	block128_copy(&ctr, iv);++	while (nb_blocks > 0) {+		uint64_t low = (uint64_t) be32_to_cpu(ctr.d[3]);+		uint64_t room = 0x100000000ULL - low;   /* blocks before it wraps */+		uint32_t n;++		/* room is 2^32 when the word is at zero, which does not fit the+		 * type the comparison would narrow it to -- and a zero n here+		 * would not advance */+		if (room > (uint64_t) nb_blocks)+			room = (uint64_t) nb_blocks;+		n = (uint32_t) room;++		crypton_aes_p8_ctr32_encrypt_blocks((const unsigned char *) input,+		                                    (unsigned char *) output,+		                                    n, FORWARD(key),+		                                    (const unsigned char *) &ctr);+		/* the assembly does not write the counter back */+		for (i = 0; i < n; i++)+			block128_inc_be(&ctr);++		output += (size_t) n * 16;+		input += (size_t) n * 16;+		nb_blocks -= n;+	}++	if (tail != 0) {+		block128 ks;++		crypton_aes_p8_encrypt((const unsigned char *) &ctr,+		                       (unsigned char *) &ks, FORWARD(key));+		for (i = 0; i < tail; i++)+			output[i] = ks.b[i] ^ input[i];+	}+}++/*+ * XTS.  The assembly encrypts the tweak with its second key, unless that key+ * is NULL, in which case it takes the tweak already encrypted -- which is+ * what this needs, because crypton's entry also carries a starting point and+ * the tweak has to be advanced that many doublings before any block is+ * enciphered.+ */+static void xts_tweak(block128 *tweak, aes_key *k2, aes_block *dataunit,+                      uint32_t spoint)+{+	block128_copy(tweak, dataunit);+	crypton_aes_p8_encrypt((const unsigned char *) tweak,+	                       (unsigned char *) tweak, FORWARD(k2));+	while (spoint-- > 0)+		crypton_aes_generic_gf_mulx(tweak);+}++void crypton_aes_ppc8_encrypt_xts(aes_block *output, aes_key *k1, aes_key *k2,+                                  aes_block *dataunit, uint32_t spoint,+                                  aes_block *input, uint32_t nb_blocks)+{+	block128 tweak;++	xts_tweak(&tweak, k2, dataunit, spoint);+	crypton_aes_p8_xts_encrypt((const unsigned char *) input,+	                           (unsigned char *) output,+	                           (size_t) nb_blocks * 16, FORWARD(k1), NULL,+	                           (const unsigned char *) &tweak);+}++void crypton_aes_ppc8_decrypt_xts(aes_block *output, aes_key *k1, aes_key *k2,+                                  aes_block *dataunit, uint32_t spoint,+                                  aes_block *input, uint32_t nb_blocks)+{+	block128 tweak;+	p8_key dk;++	/* the tweak is enciphered with k2 forwards either way */+	xts_tweak(&tweak, k2, dataunit, spoint);+	inverse(&dk, k1);+	crypton_aes_p8_xts_decrypt((const unsigned char *) input,+	                           (unsigned char *) output,+	                           (size_t) nb_blocks * 16, &dk, NULL,+	                           (const unsigned char *) &tweak);+	memset(&dk, 0, sizeof dk);+}++/*+ * GHASH.  The table the assembly builds is 192 bytes, which fits the sixteen+ * blocks the interface names.+ *+ * The two arguments do not take the same convention, which is worth saying+ * because it does not look like an accident until it is checked.  H arrives+ * here as the sixteen bytes GCM defines, and gcm_init_p8 wants those as a+ * pair of host-order words -- so they are swapped on a little-endian machine+ * and left alone on a big-endian one, which be64_to_cpu does.  The+ * accumulator is the byte string throughout, in and out, and is passed as it+ * stands.  OpenSSL makes the same pair of choices.+ *+ * Checked rather than assumed: against crypton's own GHASH, over two+ * different H and with a non-zero accumulator, so that neither a symmetric+ * value nor a zero start could hide a wrong one.+ */+void crypton_aes_ppc8_hinit(table_4bit htable, const block128 *h)+{+	uint64_t H[2];++	memcpy(H, h, sizeof H);+	H[0] = be64_to_cpu(H[0]);+	H[1] = be64_to_cpu(H[1]);+	crypton_gcm_init_p8(htable, H);+}++void crypton_aes_ppc8_gf_mul(block128 *a, const table_4bit htable)+{+	crypton_gcm_gmult_p8(a, htable);+}++void crypton_aes_ppc8_gf_mul4(block128 *a, const block128 *blocks,+                              const table_4bit htable)+{+	crypton_gcm_ghash_p8(a, htable, (const unsigned char *) blocks,+	                     4 * sizeof(block128));+}++int crypton_aes_ppc8_available(void)+{+	return (crypton_ppc_features() & CRYPTON_PPC_VCRYPTO) != 0;+}
+ cbits/aes/ppc8.h view
@@ -0,0 +1,29 @@+/*+ * AES and GHASH on PowerISA 2.07, as implemented by cbits/aes/ppc8.c over+ * the CRYPTOGAMS assembly.  crypton_aes.c installs these when+ * crypton_aes_ppc8_available says the processor has the instructions.+ */+#ifndef CRYPTON_AES_PPC8_H+#define CRYPTON_AES_PPC8_H++#include "crypton_aes.h"+#include "aes/gf.h"+#include "crypton_cpu.h"++int  crypton_aes_ppc8_available(void);++void crypton_aes_ppc8_init(aes_key *key, uint8_t *origkey, uint8_t size);+void crypton_aes_ppc8_encrypt_block(aes_block *output, aes_key *key, aes_block *input);+void crypton_aes_ppc8_decrypt_block(aes_block *output, aes_key *key, aes_block *input);+void crypton_aes_ppc8_encrypt_ecb(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks);+void crypton_aes_ppc8_decrypt_ecb(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks);+void crypton_aes_ppc8_encrypt_cbc(aes_block *output, aes_key *key, aes_block *iv, aes_block *input, uint32_t nb_blocks);+void crypton_aes_ppc8_decrypt_cbc(aes_block *output, aes_key *key, aes_block *iv, aes_block *input, uint32_t nb_blocks);+void crypton_aes_ppc8_encrypt_ctr(uint8_t *output, aes_key *key, aes_block *iv, uint8_t *input, uint32_t len);+void crypton_aes_ppc8_encrypt_xts(aes_block *output, aes_key *k1, aes_key *k2, aes_block *dataunit, uint32_t spoint, aes_block *input, uint32_t nb_blocks);+void crypton_aes_ppc8_decrypt_xts(aes_block *output, aes_key *k1, aes_key *k2, aes_block *dataunit, uint32_t spoint, aes_block *input, uint32_t nb_blocks);+void crypton_aes_ppc8_hinit(table_4bit htable, const block128 *h);+void crypton_aes_ppc8_gf_mul(block128 *a, const table_4bit htable);+void crypton_aes_ppc8_gf_mul4(block128 *a, const block128 *blocks, const table_4bit htable);++#endif
cbits/aes/x86ni.c view
@@ -37,6 +37,8 @@ #include <crypton_cpu.h> #include <aes/gf.h> #include <aes/x86ni.h>+#include <aes/gcm_vaes_x86.h>+#include <aes/gcm_vaes512_x86.h> #include <aes/block128.h> #include <aes/gcm_x86_asm.h> @@ -207,13 +209,23 @@ 	return v; } +/* memcpy rather than a cast, as everything else that moves bytes between a+ * crypton structure and a word does since block128 was packed.  The cast+ * this replaces was written in 2014, when block128 was a plain union and+ * taking a __m128i * to one promised nothing the type did not already+ * offer.  Packing it dropped its alignment to one, and the promise with it:+ * gcc has reported the cast ever since, and it is right to -- the attribute+ * on the local below is what makes the promise true, and nothing obliges+ * the next edit to keep it.  Sixteen bytes of memcpy between a __m128i and+ * a sixteen-byte object is one movdqu, or nothing at all when both stay in+ * registers. */ TARGET_AESNI static __m128i gfmul_generic(__m128i tag, const table_4bit htable) {-	aes_block _t ALIGNMENT(16);-	_mm_store_si128((__m128i *) &_t, tag);+	aes_block _t;+	memcpy(&_t, &tag, sizeof _t); 	crypton_aes_generic_gf_mul(&_t, htable);-	tag = _mm_load_si128((__m128i *) &_t);+	memcpy(&tag, &_t, sizeof tag); 	return tag; } @@ -357,11 +369,14 @@ 	htable[0].q[0] = bitfn_swap64(h->q[1]); 	htable[0].q[1] = bitfn_swap64(h->q[0]); -	/* Indices 1..7 get H^2 .. H^8, which is what lets a group of blocks-	 * fold into one reduction: gf_mul4 uses the first four, the GCM loop-	 * all eight.  The table has sixteen slots. */+	/* Indices 1..15 get H^2 .. H^16, which is what lets a group of blocks+	 * fold into one reduction: gf_mul4 uses the first four, the 128-bit+	 * GCM loop eight, and the 256-bit one all sixteen.  The table has+	 * sixteen slots and now they are all used.  Filling the upper half+	 * costs eight multiplies once per key, which is nothing beside a+	 * message. */ 	p = _mm_loadu_si128((const __m128i *) h);-	for (i = 1; i < 8; i++) {+	for (i = 1; i < 16; i++) { 		p = gfmul_pclmuldq(p, htable); 		_mm_storeu_si128((__m128i *) &htable[i], 		                 _mm_shuffle_epi8(p, bswap_mask));
cbits/aes/x86ni_impl.c view
@@ -335,9 +335,43 @@  	gcm->length_input += length; +#ifdef WITH_GCM_VAES512+	/*+	 * The widest form the processor has, first: four blocks to an+	 * instruction where the 256-bit one below takes two and the assembly+	 * after that takes one.  Same contract throughout -- whole groups off+	 * the front, the counter and the tag left behind.+	 */+	if (nb_blocks >= GCM_VAES512_MIN_BLOCKS+	    && (crypton_x86_simd_features() & CRYPTON_X86_VAES512)) {+		uint32_t done = crypton_gcm_vaes512_bulk_encrypt(+			output, gcm, key, input, nb_blocks * 16);++		output += done;+		input += done;+		nb_blocks -= done / 16;+	}+#endif+#ifdef WITH_GCM_VAES+	/*+	 * The 256-bit instructions next: they take two blocks where the ones+	 * below take one, and the assembly that follows is 128-bit+	 * throughout.+	 */+	if (nb_blocks >= GCM_VAES_MIN_BLOCKS+	    && (crypton_x86_simd_features() & CRYPTON_X86_VAES)) {+		uint32_t done = crypton_gcm_vaes_bulk_encrypt(output, gcm, key,+		                                              input,+		                                              nb_blocks * 16);++		output += done;+		input += done;+		nb_blocks -= done / 16;+	}+#endif #if defined(WITH_X86_GCM_ASM) && defined(WITH_PCLMUL) 	/*-	 * The stitched assembly first, which takes whole groups of six+	 * The stitched assembly next, which takes whole groups of six 	 * blocks off the front of the message and leaves the counter and the 	 * running tag where the loop below expects to find them.  It wants 	 * eighteen blocks before it will start, and answers with what it did.@@ -483,6 +517,31 @@  	gcm->length_input += length; +#ifdef WITH_GCM_VAES512+	/* as in encryption; the tag is taken over the input here */+	if (nb_blocks >= GCM_VAES512_MIN_BLOCKS+	    && (crypton_x86_simd_features() & CRYPTON_X86_VAES512)) {+		uint32_t done = crypton_gcm_vaes512_bulk_decrypt(+			output, gcm, key, input, nb_blocks * 16);++		output += done;+		input += done;+		nb_blocks -= done / 16;+	}+#endif+#ifdef WITH_GCM_VAES+	/* as in encryption; the tag is taken over the input here */+	if (nb_blocks >= GCM_VAES_MIN_BLOCKS+	    && (crypton_x86_simd_features() & CRYPTON_X86_VAES)) {+		uint32_t done = crypton_gcm_vaes_bulk_decrypt(output, gcm, key,+		                                              input,+		                                              nb_blocks * 16);++		output += done;+		input += done;+		nb_blocks -= done / 16;+	}+#endif #if defined(WITH_X86_GCM_ASM) && defined(WITH_PCLMUL) 	/* the same as encryption, except that decryption has nothing to 	 * hold back and so will start on six blocks */
cbits/asm/README.md view
@@ -2,7 +2,7 @@  ## What is here -Two modules from [CRYPTOGAMS](https://github.com/dot-asm/cryptogams), by Andy+Modules from [CRYPTOGAMS](https://github.com/dot-asm/cryptogams), by Andy Polyakov, checked in unmodified together with the translators they need:  | generator | what it is |@@ -17,6 +17,8 @@ | `sha1-armv8.pl` | SHA-1 for AArch64 | | `sha512-armv8.pl` | SHA-256 for AArch64 (the generator emits SHA-512 or SHA-256 according to the name it is given, and only the latter is wanted) | | `keccak1600-armv8.pl` | Keccak for AArch64 |+| `aesp8-ppc.pl` | AES for PowerISA 2.07, which POWER8 was the first to implement |+| `ghashp8-ppc.pl` | GHASH for the same, over `vpmsumd` |  `x86_64-xlate.pl`, `arm-xlate.pl` and `arm_arch.h` are the machinery those modules use.  `generate.sh` runs the generators to produce the `.S` files, which are
+ cbits/asm/aesp8-ppc-linux64le.S view
@@ -0,0 +1,3658 @@+.machine	"any"++.abiversion	2+.text++.align	7+rcon:+.byte	0x00,0x00,0x00,0x01,0x00,0x00,0x00,0x01,0x00,0x00,0x00,0x01,0x00,0x00,0x00,0x01+.byte	0x00,0x00,0x00,0x1b,0x00,0x00,0x00,0x1b,0x00,0x00,0x00,0x1b,0x00,0x00,0x00,0x1b+.byte	0x0c,0x0f,0x0e,0x0d,0x0c,0x0f,0x0e,0x0d,0x0c,0x0f,0x0e,0x0d,0x0c,0x0f,0x0e,0x0d+.byte	0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00+.Lconsts:+	mflr	0+	bcl	20,31,$+4+	mflr	6+	addi	6,6,-0x48+	mtlr	0+	blr	+.long	0+.byte	0,12,0x14,0,0,0,0,0+.byte	65,69,83,32,102,111,114,32,80,111,119,101,114,73,83,65,32,50,46,48,55,44,67,82,89,80,84,79,71,65,77,83,32,98,121,32,60,97,112,112,114,111,64,111,112,101,110,115,115,108,46,111,114,103,62,0+.align	2++.globl	crypton_aes_p8_set_encrypt_key+.type	crypton_aes_p8_set_encrypt_key,@function+.align	5+crypton_aes_p8_set_encrypt_key:+.localentry	crypton_aes_p8_set_encrypt_key,0++.Lset_encrypt_key:+	mflr	11+	std	11,16(1)++	li	6,-1+	cmpldi	3,0+	beq-	.Lenc_key_abort+	cmpldi	5,0+	beq-	.Lenc_key_abort+	li	6,-2+	cmpwi	4,128+	blt-	.Lenc_key_abort+	cmpwi	4,256+	bgt-	.Lenc_key_abort+	andi.	0,4,0x3f+	bne-	.Lenc_key_abort++	lis	0,0xfff0+	li	12,-1+	or	0,0,0++	bl	.Lconsts+	mtlr	11++	neg	9,3+	lvx	1,0,3+	addi	3,3,15+	lvsr	3,0,9+	li	8,0x20+	cmpwi	4,192+	lvx	2,0,3+	vspltisb	5,0x0f+	lvx	4,0,6+	vxor	3,3,5+	lvx	5,8,6+	addi	6,6,0x10+	vperm	1,1,2,3+	li	7,8+	vxor	0,0,0+	mtctr	7++	lvsl	8,0,5+	vspltisb	9,-1+	lvx	10,0,5+	vperm	9,9,0,8++	blt	.Loop128+	addi	3,3,8+	beq	.L192+	addi	3,3,8+	b	.L256++.align	4+.Loop128:+	vperm	3,1,1,5+	vsldoi	6,0,1,12+	vperm	11,1,1,8+	vsel	7,10,11,9+	vor	10,11,11+	.long	0x10632509+	stvx	7,0,5+	addi	5,5,16++	vxor	1,1,6+	vsldoi	6,0,6,12+	vxor	1,1,6+	vsldoi	6,0,6,12+	vxor	1,1,6+	vadduwm	4,4,4+	vxor	1,1,3+	bdnz	.Loop128++	lvx	4,0,6++	vperm	3,1,1,5+	vsldoi	6,0,1,12+	vperm	11,1,1,8+	vsel	7,10,11,9+	vor	10,11,11+	.long	0x10632509+	stvx	7,0,5+	addi	5,5,16++	vxor	1,1,6+	vsldoi	6,0,6,12+	vxor	1,1,6+	vsldoi	6,0,6,12+	vxor	1,1,6+	vadduwm	4,4,4+	vxor	1,1,3++	vperm	3,1,1,5+	vsldoi	6,0,1,12+	vperm	11,1,1,8+	vsel	7,10,11,9+	vor	10,11,11+	.long	0x10632509+	stvx	7,0,5+	addi	5,5,16++	vxor	1,1,6+	vsldoi	6,0,6,12+	vxor	1,1,6+	vsldoi	6,0,6,12+	vxor	1,1,6+	vxor	1,1,3+	vperm	11,1,1,8+	vsel	7,10,11,9+	vor	10,11,11+	stvx	7,0,5++	addi	3,5,15+	addi	5,5,0x50++	li	8,10+	b	.Ldone++.align	4+.L192:+	lvx	6,0,3+	li	7,4+	vperm	11,1,1,8+	vsel	7,10,11,9+	vor	10,11,11+	stvx	7,0,5+	addi	5,5,16+	vperm	2,2,6,3+	vspltisb	3,8+	mtctr	7+	vsububm	5,5,3++.Loop192:+	vperm	3,2,2,5+	vsldoi	6,0,1,12+	.long	0x10632509++	vxor	1,1,6+	vsldoi	6,0,6,12+	vxor	1,1,6+	vsldoi	6,0,6,12+	vxor	1,1,6++	vsldoi	7,0,2,8+	vspltw	6,1,3+	vxor	6,6,2+	vsldoi	2,0,2,12+	vadduwm	4,4,4+	vxor	2,2,6+	vxor	1,1,3+	vxor	2,2,3+	vsldoi	7,7,1,8++	vperm	3,2,2,5+	vsldoi	6,0,1,12+	vperm	11,7,7,8+	vsel	7,10,11,9+	vor	10,11,11+	.long	0x10632509+	stvx	7,0,5+	addi	5,5,16++	vsldoi	7,1,2,8+	vxor	1,1,6+	vsldoi	6,0,6,12+	vperm	11,7,7,8+	vsel	7,10,11,9+	vor	10,11,11+	vxor	1,1,6+	vsldoi	6,0,6,12+	vxor	1,1,6+	stvx	7,0,5+	addi	5,5,16++	vspltw	6,1,3+	vxor	6,6,2+	vsldoi	2,0,2,12+	vadduwm	4,4,4+	vxor	2,2,6+	vxor	1,1,3+	vxor	2,2,3+	vperm	11,1,1,8+	vsel	7,10,11,9+	vor	10,11,11+	stvx	7,0,5+	addi	3,5,15+	addi	5,5,16+	bdnz	.Loop192++	li	8,12+	addi	5,5,0x20+	b	.Ldone++.align	4+.L256:+	lvx	6,0,3+	li	7,7+	li	8,14+	vperm	11,1,1,8+	vsel	7,10,11,9+	vor	10,11,11+	stvx	7,0,5+	addi	5,5,16+	vperm	2,2,6,3+	mtctr	7++.Loop256:+	vperm	3,2,2,5+	vsldoi	6,0,1,12+	vperm	11,2,2,8+	vsel	7,10,11,9+	vor	10,11,11+	.long	0x10632509+	stvx	7,0,5+	addi	5,5,16++	vxor	1,1,6+	vsldoi	6,0,6,12+	vxor	1,1,6+	vsldoi	6,0,6,12+	vxor	1,1,6+	vadduwm	4,4,4+	vxor	1,1,3+	vperm	11,1,1,8+	vsel	7,10,11,9+	vor	10,11,11+	stvx	7,0,5+	addi	3,5,15+	addi	5,5,16+	bdz	.Ldone++	vspltw	3,1,3+	vsldoi	6,0,2,12+	.long	0x106305C8++	vxor	2,2,6+	vsldoi	6,0,6,12+	vxor	2,2,6+	vsldoi	6,0,6,12+	vxor	2,2,6++	vxor	2,2,3+	b	.Loop256++.align	4+.Ldone:+	lvx	2,0,3+	vsel	2,10,2,9+	stvx	2,0,3+	li	6,0+	or	12,12,12+	stw	8,0(5)++.Lenc_key_abort:+	mr	3,6+	blr	+.long	0+.byte	0,12,0x14,1,0,0,3,0+.long	0+.size	crypton_aes_p8_set_encrypt_key,.-crypton_aes_p8_set_encrypt_key++.globl	crypton_aes_p8_set_decrypt_key+.type	crypton_aes_p8_set_decrypt_key,@function+.align	5+crypton_aes_p8_set_decrypt_key:+.localentry	crypton_aes_p8_set_decrypt_key,0++	stdu	1,-64(1)+	mflr	10+	std	10,64+16(1)+	bl	.Lset_encrypt_key+	mtlr	10++	cmpwi	3,0+	bne-	.Ldec_key_abort++	slwi	7,8,4+	subi	3,5,240+	srwi	8,8,1+	add	5,3,7+	mtctr	8++.Ldeckey:+	lwz	0, 0(3)+	lwz	6, 4(3)+	lwz	7, 8(3)+	lwz	8, 12(3)+	addi	3,3,16+	lwz	9, 0(5)+	lwz	10,4(5)+	lwz	11,8(5)+	lwz	12,12(5)+	stw	0, 0(5)+	stw	6, 4(5)+	stw	7, 8(5)+	stw	8, 12(5)+	subi	5,5,16+	stw	9, -16(3)+	stw	10,-12(3)+	stw	11,-8(3)+	stw	12,-4(3)+	bdnz	.Ldeckey++	xor	3,3,3+.Ldec_key_abort:+	addi	1,1,64+	blr	+.long	0+.byte	0,12,4,1,0x80,0,3,0+.long	0+.size	crypton_aes_p8_set_decrypt_key,.-crypton_aes_p8_set_decrypt_key+.globl	crypton_aes_p8_encrypt+.type	crypton_aes_p8_encrypt,@function+.align	5+crypton_aes_p8_encrypt:+.localentry	crypton_aes_p8_encrypt,0++	lwz	6,240(5)+	lis	0,0xfc00+	li	12,-1+	li	7,15+	or	0,0,0++	lvx	0,0,3+	neg	11,4+	lvx	1,7,3+	lvsl	2,0,3+	vspltisb	4,0x0f+	lvsr	3,0,11+	vxor	2,2,4+	li	7,16+	vperm	0,0,1,2+	lvx	1,0,5+	lvsr	5,0,5+	srwi	6,6,1+	lvx	2,7,5+	addi	7,7,16+	subi	6,6,1+	vperm	1,2,1,5++	vxor	0,0,1+	lvx	1,7,5+	addi	7,7,16+	mtctr	6++.Loop_enc:+	vperm	2,1,2,5+	.long	0x10001508+	lvx	2,7,5+	addi	7,7,16+	vperm	1,2,1,5+	.long	0x10000D08+	lvx	1,7,5+	addi	7,7,16+	bdnz	.Loop_enc++	vperm	2,1,2,5+	.long	0x10001508+	lvx	2,7,5+	vperm	1,2,1,5+	.long	0x10000D09++	vspltisb	2,-1+	vxor	1,1,1+	li	7,15+	vperm	2,2,1,3+	vxor	3,3,4+	lvx	1,0,4+	vperm	0,0,0,3+	vsel	1,1,0,2+	lvx	4,7,4+	stvx	1,0,4+	vsel	0,0,4,2+	stvx	0,7,4++	or	12,12,12+	blr	+.long	0+.byte	0,12,0x14,0,0,0,3,0+.long	0+.size	crypton_aes_p8_encrypt,.-crypton_aes_p8_encrypt+.globl	crypton_aes_p8_decrypt+.type	crypton_aes_p8_decrypt,@function+.align	5+crypton_aes_p8_decrypt:+.localentry	crypton_aes_p8_decrypt,0++	lwz	6,240(5)+	lis	0,0xfc00+	li	12,-1+	li	7,15+	or	0,0,0++	lvx	0,0,3+	neg	11,4+	lvx	1,7,3+	lvsl	2,0,3+	vspltisb	4,0x0f+	lvsr	3,0,11+	vxor	2,2,4+	li	7,16+	vperm	0,0,1,2+	lvx	1,0,5+	lvsr	5,0,5+	srwi	6,6,1+	lvx	2,7,5+	addi	7,7,16+	subi	6,6,1+	vperm	1,2,1,5++	vxor	0,0,1+	lvx	1,7,5+	addi	7,7,16+	mtctr	6++.Loop_dec:+	vperm	2,1,2,5+	.long	0x10001548+	lvx	2,7,5+	addi	7,7,16+	vperm	1,2,1,5+	.long	0x10000D48+	lvx	1,7,5+	addi	7,7,16+	bdnz	.Loop_dec++	vperm	2,1,2,5+	.long	0x10001548+	lvx	2,7,5+	vperm	1,2,1,5+	.long	0x10000D49++	vspltisb	2,-1+	vxor	1,1,1+	li	7,15+	vperm	2,2,1,3+	vxor	3,3,4+	lvx	1,0,4+	vperm	0,0,0,3+	vsel	1,1,0,2+	lvx	4,7,4+	stvx	1,0,4+	vsel	0,0,4,2+	stvx	0,7,4++	or	12,12,12+	blr	+.long	0+.byte	0,12,0x14,0,0,0,3,0+.long	0+.size	crypton_aes_p8_decrypt,.-crypton_aes_p8_decrypt+.globl	crypton_aes_p8_cbc_encrypt+.type	crypton_aes_p8_cbc_encrypt,@function+.align	5+crypton_aes_p8_cbc_encrypt:+.localentry	crypton_aes_p8_cbc_encrypt,0++	cmpldi	5,16+	.long	0x4dc00020++	cmpwi	8,0+	lis	0,0xffe0+	li	12,-1+	or	0,0,0++	li	10,15+	vxor	0,0,0+	vspltisb	3,0x0f++	lvx	4,0,7+	lvsl	6,0,7+	lvx	5,10,7+	vxor	6,6,3+	vperm	4,4,5,6++	neg	11,3+	lvsr	10,0,6+	lwz	9,240(6)++	lvsr	6,0,11+	lvx	5,0,3+	addi	3,3,15+	vxor	6,6,3++	lvsl	8,0,4+	vspltisb	9,-1+	lvx	7,0,4+	vperm	9,9,0,8+	vxor	8,8,3++	srwi	9,9,1+	li	10,16+	subi	9,9,1+	beq	.Lcbc_dec++.Lcbc_enc:+	vor	2,5,5+	lvx	5,0,3+	addi	3,3,16+	mtctr	9+	subi	5,5,16++	lvx	0,0,6+	vperm	2,2,5,6+	lvx	1,10,6+	addi	10,10,16+	vperm	0,1,0,10+	vxor	2,2,0+	lvx	0,10,6+	addi	10,10,16+	vxor	2,2,4++.Loop_cbc_enc:+	vperm	1,0,1,10+	.long	0x10420D08+	lvx	1,10,6+	addi	10,10,16+	vperm	0,1,0,10+	.long	0x10420508+	lvx	0,10,6+	addi	10,10,16+	bdnz	.Loop_cbc_enc++	vperm	1,0,1,10+	.long	0x10420D08+	lvx	1,10,6+	li	10,16+	vperm	0,1,0,10+	.long	0x10820509+	cmpldi	5,16++	vperm	3,4,4,8+	vsel	2,7,3,9+	vor	7,3,3+	stvx	2,0,4+	addi	4,4,16+	bge	.Lcbc_enc++	b	.Lcbc_done++.align	4+.Lcbc_dec:+	cmpldi	5,128+	bge	_aesp8_cbc_decrypt8x+	vor	3,5,5+	lvx	5,0,3+	addi	3,3,16+	mtctr	9+	subi	5,5,16++	lvx	0,0,6+	vperm	3,3,5,6+	lvx	1,10,6+	addi	10,10,16+	vperm	0,1,0,10+	vxor	2,3,0+	lvx	0,10,6+	addi	10,10,16++.Loop_cbc_dec:+	vperm	1,0,1,10+	.long	0x10420D48+	lvx	1,10,6+	addi	10,10,16+	vperm	0,1,0,10+	.long	0x10420548+	lvx	0,10,6+	addi	10,10,16+	bdnz	.Loop_cbc_dec++	vperm	1,0,1,10+	.long	0x10420D48+	lvx	1,10,6+	li	10,16+	vperm	0,1,0,10+	.long	0x10420549+	cmpldi	5,16++	vxor	2,2,4+	vor	4,3,3+	vperm	3,2,2,8+	vsel	2,7,3,9+	vor	7,3,3+	stvx	2,0,4+	addi	4,4,16+	bge	.Lcbc_dec++.Lcbc_done:+	addi	4,4,-1+	lvx	2,0,4+	vsel	2,7,2,9+	stvx	2,0,4++	neg	8,7+	li	10,15+	vxor	0,0,0+	vspltisb	9,-1+	vspltisb	3,0x0f+	lvsr	8,0,8+	vperm	9,9,0,8+	vxor	8,8,3+	lvx	7,0,7+	vperm	4,4,4,8+	vsel	2,7,4,9+	lvx	5,10,7+	stvx	2,0,7+	vsel	2,4,5,9+	stvx	2,10,7++	or	12,12,12+	blr	+.long	0+.byte	0,12,0x14,0,0,0,6,0+.long	0+.align	5+_aesp8_cbc_decrypt8x:+	stdu	1,-448(1)+	li	10,207+	li	11,223+	stvx	20,10,1+	addi	10,10,32+	stvx	21,11,1+	addi	11,11,32+	stvx	22,10,1+	addi	10,10,32+	stvx	23,11,1+	addi	11,11,32+	stvx	24,10,1+	addi	10,10,32+	stvx	25,11,1+	addi	11,11,32+	stvx	26,10,1+	addi	10,10,32+	stvx	27,11,1+	addi	11,11,32+	stvx	28,10,1+	addi	10,10,32+	stvx	29,11,1+	addi	11,11,32+	stvx	30,10,1+	stvx	31,11,1+	li	0,-1+	stw	12,396(1)+	li	8,0x10+	std	26,400(1)+	li	26,0x20+	std	27,408(1)+	li	27,0x30+	std	28,416(1)+	li	28,0x40+	std	29,424(1)+	li	29,0x50+	std	30,432(1)+	li	30,0x60+	std	31,440(1)+	li	31,0x70+	or	0,0,0++	subi	9,9,3+	subi	5,5,128++	lvx	23,0,6+	lvx	30,8,6+	addi	6,6,0x20+	lvx	31,0,6+	vperm	23,30,23,10+	addi	11,1,64+15+	mtctr	9++.Load_cbc_dec_key:+	vperm	24,31,30,10+	lvx	30,8,6+	addi	6,6,0x20+	stvx	24,0,11+	vperm	25,30,31,10+	lvx	31,0,6+	stvx	25,8,11+	addi	11,11,0x20+	bdnz	.Load_cbc_dec_key++	lvx	26,8,6+	vperm	24,31,30,10+	lvx	27,26,6+	stvx	24,0,11+	vperm	25,26,31,10+	lvx	28,27,6+	stvx	25,8,11+	addi	11,1,64+15+	vperm	26,27,26,10+	lvx	29,28,6+	vperm	27,28,27,10+	lvx	30,29,6+	vperm	28,29,28,10+	lvx	31,30,6+	vperm	29,30,29,10+	lvx	14,31,6+	vperm	30,31,30,10+	lvx	24,0,11+	vperm	31,14,31,10+	lvx	25,8,11++++	subi	3,3,15++	li	10,8+	.long	0x7C001E99+	lvsl	6,0,10+	vspltisb	3,0x0f+	.long	0x7C281E99+	vxor	6,6,3+	.long	0x7C5A1E99+	vperm	0,0,0,6+	.long	0x7C7B1E99+	vperm	1,1,1,6+	.long	0x7D5C1E99+	vperm	2,2,2,6+	vxor	14,0,23+	.long	0x7D7D1E99+	vperm	3,3,3,6+	vxor	15,1,23+	.long	0x7D9E1E99+	vperm	10,10,10,6+	vxor	16,2,23+	.long	0x7DBF1E99+	addi	3,3,0x80+	vperm	11,11,11,6+	vxor	17,3,23+	vperm	12,12,12,6+	vxor	18,10,23+	vperm	13,13,13,6+	vxor	19,11,23+	vxor	20,12,23+	vxor	21,13,23++	mtctr	9+	b	.Loop_cbc_dec8x+.align	5+.Loop_cbc_dec8x:+	.long	0x11CEC548+	.long	0x11EFC548+	.long	0x1210C548+	.long	0x1231C548+	.long	0x1252C548+	.long	0x1273C548+	.long	0x1294C548+	.long	0x12B5C548+	lvx	24,26,11+	addi	11,11,0x20++	.long	0x11CECD48+	.long	0x11EFCD48+	.long	0x1210CD48+	.long	0x1231CD48+	.long	0x1252CD48+	.long	0x1273CD48+	.long	0x1294CD48+	.long	0x12B5CD48+	lvx	25,8,11+	bdnz	.Loop_cbc_dec8x++	subic	5,5,128+	.long	0x11CEC548+	.long	0x11EFC548+	.long	0x1210C548+	.long	0x1231C548+	.long	0x1252C548+	.long	0x1273C548+	.long	0x1294C548+	.long	0x12B5C548++	subfe.	0,0,0+	.long	0x11CECD48+	.long	0x11EFCD48+	.long	0x1210CD48+	.long	0x1231CD48+	.long	0x1252CD48+	.long	0x1273CD48+	.long	0x1294CD48+	.long	0x12B5CD48++	and	0,0,5+	.long	0x11CED548+	.long	0x11EFD548+	.long	0x1210D548+	.long	0x1231D548+	.long	0x1252D548+	.long	0x1273D548+	.long	0x1294D548+	.long	0x12B5D548++	add	3,3,0++++	.long	0x11CEDD48+	.long	0x11EFDD48+	.long	0x1210DD48+	.long	0x1231DD48+	.long	0x1252DD48+	.long	0x1273DD48+	.long	0x1294DD48+	.long	0x12B5DD48++	addi	11,1,64+15+	.long	0x11CEE548+	.long	0x11EFE548+	.long	0x1210E548+	.long	0x1231E548+	.long	0x1252E548+	.long	0x1273E548+	.long	0x1294E548+	.long	0x12B5E548+	lvx	24,0,11++	.long	0x11CEED48+	.long	0x11EFED48+	.long	0x1210ED48+	.long	0x1231ED48+	.long	0x1252ED48+	.long	0x1273ED48+	.long	0x1294ED48+	.long	0x12B5ED48+	lvx	25,8,11++	.long	0x11CEF548+	vxor	4,4,31+	.long	0x11EFF548+	vxor	0,0,31+	.long	0x1210F548+	vxor	1,1,31+	.long	0x1231F548+	vxor	2,2,31+	.long	0x1252F548+	vxor	3,3,31+	.long	0x1273F548+	vxor	10,10,31+	.long	0x1294F548+	vxor	11,11,31+	.long	0x12B5F548+	vxor	12,12,31++	.long	0x11CE2549+	.long	0x11EF0549+	.long	0x7C001E99+	.long	0x12100D49+	.long	0x7C281E99+	.long	0x12311549+	vperm	0,0,0,6+	.long	0x7C5A1E99+	.long	0x12521D49+	vperm	1,1,1,6+	.long	0x7C7B1E99+	.long	0x12735549+	vperm	2,2,2,6+	.long	0x7D5C1E99+	.long	0x12945D49+	vperm	3,3,3,6+	.long	0x7D7D1E99+	.long	0x12B56549+	vperm	10,10,10,6+	.long	0x7D9E1E99+	vor	4,13,13+	vperm	11,11,11,6+	.long	0x7DBF1E99+	addi	3,3,0x80++	vperm	14,14,14,6+	vperm	15,15,15,6+	.long	0x7DC02799+	vperm	12,12,12,6+	vxor	14,0,23+	vperm	16,16,16,6+	.long	0x7DE82799+	vperm	13,13,13,6+	vxor	15,1,23+	vperm	17,17,17,6+	.long	0x7E1A2799+	vxor	16,2,23+	vperm	18,18,18,6+	.long	0x7E3B2799+	vxor	17,3,23+	vperm	19,19,19,6+	.long	0x7E5C2799+	vxor	18,10,23+	vperm	20,20,20,6+	.long	0x7E7D2799+	vxor	19,11,23+	vperm	21,21,21,6+	.long	0x7E9E2799+	vxor	20,12,23+	.long	0x7EBF2799+	addi	4,4,0x80+	vxor	21,13,23++	mtctr	9+	beq	.Loop_cbc_dec8x++	addic.	5,5,128+	beq	.Lcbc_dec8x_done+	nop	+	nop	++.Loop_cbc_dec8x_tail:+	.long	0x11EFC548+	.long	0x1210C548+	.long	0x1231C548+	.long	0x1252C548+	.long	0x1273C548+	.long	0x1294C548+	.long	0x12B5C548+	lvx	24,26,11+	addi	11,11,0x20++	.long	0x11EFCD48+	.long	0x1210CD48+	.long	0x1231CD48+	.long	0x1252CD48+	.long	0x1273CD48+	.long	0x1294CD48+	.long	0x12B5CD48+	lvx	25,8,11+	bdnz	.Loop_cbc_dec8x_tail++	.long	0x11EFC548+	.long	0x1210C548+	.long	0x1231C548+	.long	0x1252C548+	.long	0x1273C548+	.long	0x1294C548+	.long	0x12B5C548++	.long	0x11EFCD48+	.long	0x1210CD48+	.long	0x1231CD48+	.long	0x1252CD48+	.long	0x1273CD48+	.long	0x1294CD48+	.long	0x12B5CD48++	.long	0x11EFD548+	.long	0x1210D548+	.long	0x1231D548+	.long	0x1252D548+	.long	0x1273D548+	.long	0x1294D548+	.long	0x12B5D548++	.long	0x11EFDD48+	.long	0x1210DD48+	.long	0x1231DD48+	.long	0x1252DD48+	.long	0x1273DD48+	.long	0x1294DD48+	.long	0x12B5DD48++	.long	0x11EFE548+	.long	0x1210E548+	.long	0x1231E548+	.long	0x1252E548+	.long	0x1273E548+	.long	0x1294E548+	.long	0x12B5E548++	.long	0x11EFED48+	.long	0x1210ED48+	.long	0x1231ED48+	.long	0x1252ED48+	.long	0x1273ED48+	.long	0x1294ED48+	.long	0x12B5ED48++	.long	0x11EFF548+	vxor	4,4,31+	.long	0x1210F548+	vxor	1,1,31+	.long	0x1231F548+	vxor	2,2,31+	.long	0x1252F548+	vxor	3,3,31+	.long	0x1273F548+	vxor	10,10,31+	.long	0x1294F548+	vxor	11,11,31+	.long	0x12B5F548+	vxor	12,12,31++	cmplwi	5,32+	blt	.Lcbc_dec8x_one+	nop	+	beq	.Lcbc_dec8x_two+	cmplwi	5,64+	blt	.Lcbc_dec8x_three+	nop	+	beq	.Lcbc_dec8x_four+	cmplwi	5,96+	blt	.Lcbc_dec8x_five+	nop	+	beq	.Lcbc_dec8x_six++.Lcbc_dec8x_seven:+	.long	0x11EF2549+	.long	0x12100D49+	.long	0x12311549+	.long	0x12521D49+	.long	0x12735549+	.long	0x12945D49+	.long	0x12B56549+	vor	4,13,13++	vperm	15,15,15,6+	vperm	16,16,16,6+	.long	0x7DE02799+	vperm	17,17,17,6+	.long	0x7E082799+	vperm	18,18,18,6+	.long	0x7E3A2799+	vperm	19,19,19,6+	.long	0x7E5B2799+	vperm	20,20,20,6+	.long	0x7E7C2799+	vperm	21,21,21,6+	.long	0x7E9D2799+	.long	0x7EBE2799+	addi	4,4,0x70+	b	.Lcbc_dec8x_done++.align	5+.Lcbc_dec8x_six:+	.long	0x12102549+	.long	0x12311549+	.long	0x12521D49+	.long	0x12735549+	.long	0x12945D49+	.long	0x12B56549+	vor	4,13,13++	vperm	16,16,16,6+	vperm	17,17,17,6+	.long	0x7E002799+	vperm	18,18,18,6+	.long	0x7E282799+	vperm	19,19,19,6+	.long	0x7E5A2799+	vperm	20,20,20,6+	.long	0x7E7B2799+	vperm	21,21,21,6+	.long	0x7E9C2799+	.long	0x7EBD2799+	addi	4,4,0x60+	b	.Lcbc_dec8x_done++.align	5+.Lcbc_dec8x_five:+	.long	0x12312549+	.long	0x12521D49+	.long	0x12735549+	.long	0x12945D49+	.long	0x12B56549+	vor	4,13,13++	vperm	17,17,17,6+	vperm	18,18,18,6+	.long	0x7E202799+	vperm	19,19,19,6+	.long	0x7E482799+	vperm	20,20,20,6+	.long	0x7E7A2799+	vperm	21,21,21,6+	.long	0x7E9B2799+	.long	0x7EBC2799+	addi	4,4,0x50+	b	.Lcbc_dec8x_done++.align	5+.Lcbc_dec8x_four:+	.long	0x12522549+	.long	0x12735549+	.long	0x12945D49+	.long	0x12B56549+	vor	4,13,13++	vperm	18,18,18,6+	vperm	19,19,19,6+	.long	0x7E402799+	vperm	20,20,20,6+	.long	0x7E682799+	vperm	21,21,21,6+	.long	0x7E9A2799+	.long	0x7EBB2799+	addi	4,4,0x40+	b	.Lcbc_dec8x_done++.align	5+.Lcbc_dec8x_three:+	.long	0x12732549+	.long	0x12945D49+	.long	0x12B56549+	vor	4,13,13++	vperm	19,19,19,6+	vperm	20,20,20,6+	.long	0x7E602799+	vperm	21,21,21,6+	.long	0x7E882799+	.long	0x7EBA2799+	addi	4,4,0x30+	b	.Lcbc_dec8x_done++.align	5+.Lcbc_dec8x_two:+	.long	0x12942549+	.long	0x12B56549+	vor	4,13,13++	vperm	20,20,20,6+	vperm	21,21,21,6+	.long	0x7E802799+	.long	0x7EA82799+	addi	4,4,0x20+	b	.Lcbc_dec8x_done++.align	5+.Lcbc_dec8x_one:+	.long	0x12B52549+	vor	4,13,13++	vperm	21,21,21,6+	.long	0x7EA02799+	addi	4,4,0x10++.Lcbc_dec8x_done:+	vperm	4,4,4,6+	.long	0x7C803F99++	li	10,79+	li	11,95+	stvx	6,10,1+	addi	10,10,32+	stvx	6,11,1+	addi	11,11,32+	stvx	6,10,1+	addi	10,10,32+	stvx	6,11,1+	addi	11,11,32+	stvx	6,10,1+	addi	10,10,32+	stvx	6,11,1+	addi	11,11,32+	stvx	6,10,1+	addi	10,10,32+	stvx	6,11,1+	addi	11,11,32++	or	12,12,12+	lvx	20,10,1+	addi	10,10,32+	lvx	21,11,1+	addi	11,11,32+	lvx	22,10,1+	addi	10,10,32+	lvx	23,11,1+	addi	11,11,32+	lvx	24,10,1+	addi	10,10,32+	lvx	25,11,1+	addi	11,11,32+	lvx	26,10,1+	addi	10,10,32+	lvx	27,11,1+	addi	11,11,32+	lvx	28,10,1+	addi	10,10,32+	lvx	29,11,1+	addi	11,11,32+	lvx	30,10,1+	lvx	31,11,1+	ld	26,400(1)+	ld	27,408(1)+	ld	28,416(1)+	ld	29,424(1)+	ld	30,432(1)+	ld	31,440(1)+	addi	1,1,448+	blr	+.long	0+.byte	0,12,0x04,0,0x80,6,6,0+.long	0+.size	crypton_aes_p8_cbc_encrypt,.-crypton_aes_p8_cbc_encrypt+.globl	crypton_aes_p8_ctr32_encrypt_blocks+.type	crypton_aes_p8_ctr32_encrypt_blocks,@function+.align	5+crypton_aes_p8_ctr32_encrypt_blocks:+.localentry	crypton_aes_p8_ctr32_encrypt_blocks,0++	cmpldi	5,1+	.long	0x4dc00020++	lis	0,0xfff0+	li	12,-1+	or	0,0,0++	li	10,15+	vxor	0,0,0+	vspltisb	3,0x0f++	lvx	4,0,7+	lvsl	6,0,7+	lvx	5,10,7+	vspltisb	11,1+	vxor	6,6,3+	vperm	4,4,5,6+	vsldoi	11,0,11,1++	neg	11,3+	lvsr	10,0,6+	lwz	9,240(6)++	lvsr	6,0,11+	lvx	5,0,3+	addi	3,3,15+	vxor	6,6,3++	srwi	9,9,1+	li	10,16+	subi	9,9,1++	cmpldi	5,8+	bge	_aesp8_ctr32_encrypt8x++	lvsl	8,0,4+	vspltisb	9,-1+	lvx	7,0,4+	vperm	9,9,0,8+	vxor	8,8,3++	lvx	0,0,6+	mtctr	9+	lvx	1,10,6+	addi	10,10,16+	vperm	0,1,0,10+	vxor	2,4,0+	lvx	0,10,6+	addi	10,10,16+	b	.Loop_ctr32_enc++.align	5+.Loop_ctr32_enc:+	vperm	1,0,1,10+	.long	0x10420D08+	lvx	1,10,6+	addi	10,10,16+	vperm	0,1,0,10+	.long	0x10420508+	lvx	0,10,6+	addi	10,10,16+	bdnz	.Loop_ctr32_enc++	vadduwm	4,4,11+	vor	3,5,5+	lvx	5,0,3+	addi	3,3,16+	subic.	5,5,1++	vperm	1,0,1,10+	.long	0x10420D08+	lvx	1,10,6+	vperm	3,3,5,6+	li	10,16+	vperm	1,1,0,10+	lvx	0,0,6+	vxor	3,3,1+	.long	0x10421D09++	lvx	1,10,6+	addi	10,10,16+	vperm	2,2,2,8+	vsel	3,7,2,9+	mtctr	9+	vperm	0,1,0,10+	vor	7,2,2+	vxor	2,4,0+	lvx	0,10,6+	addi	10,10,16+	stvx	3,0,4+	addi	4,4,16+	bne	.Loop_ctr32_enc++	addi	4,4,-1+	lvx	2,0,4+	vsel	2,7,2,9+	stvx	2,0,4++	or	12,12,12+	blr	+.long	0+.byte	0,12,0x14,0,0,0,6,0+.long	0+.align	5+_aesp8_ctr32_encrypt8x:+	stdu	1,-448(1)+	li	10,207+	li	11,223+	stvx	20,10,1+	addi	10,10,32+	stvx	21,11,1+	addi	11,11,32+	stvx	22,10,1+	addi	10,10,32+	stvx	23,11,1+	addi	11,11,32+	stvx	24,10,1+	addi	10,10,32+	stvx	25,11,1+	addi	11,11,32+	stvx	26,10,1+	addi	10,10,32+	stvx	27,11,1+	addi	11,11,32+	stvx	28,10,1+	addi	10,10,32+	stvx	29,11,1+	addi	11,11,32+	stvx	30,10,1+	stvx	31,11,1+	li	0,-1+	stw	12,396(1)+	li	8,0x10+	std	26,400(1)+	li	26,0x20+	std	27,408(1)+	li	27,0x30+	std	28,416(1)+	li	28,0x40+	std	29,424(1)+	li	29,0x50+	std	30,432(1)+	li	30,0x60+	std	31,440(1)+	li	31,0x70+	or	0,0,0++	subi	9,9,3++	lvx	23,0,6+	lvx	30,8,6+	addi	6,6,0x20+	lvx	31,0,6+	vperm	23,30,23,10+	addi	11,1,64+15+	mtctr	9++.Load_ctr32_enc_key:+	vperm	24,31,30,10+	lvx	30,8,6+	addi	6,6,0x20+	stvx	24,0,11+	vperm	25,30,31,10+	lvx	31,0,6+	stvx	25,8,11+	addi	11,11,0x20+	bdnz	.Load_ctr32_enc_key++	lvx	26,8,6+	vperm	24,31,30,10+	lvx	27,26,6+	stvx	24,0,11+	vperm	25,26,31,10+	lvx	28,27,6+	stvx	25,8,11+	addi	11,1,64+15+	vperm	26,27,26,10+	lvx	29,28,6+	vperm	27,28,27,10+	lvx	30,29,6+	vperm	28,29,28,10+	lvx	31,30,6+	vperm	29,30,29,10+	lvx	15,31,6+	vperm	30,31,30,10+	lvx	24,0,11+	vperm	31,15,31,10+	lvx	25,8,11++	vadduwm	7,11,11+	subi	3,3,15+	sldi	5,5,4++	vadduwm	16,4,11+	vadduwm	17,4,7+	vxor	15,4,23+	li	10,8+	vadduwm	18,16,7+	vxor	16,16,23+	lvsl	6,0,10+	vadduwm	19,17,7+	vxor	17,17,23+	vspltisb	3,0x0f+	vadduwm	20,18,7+	vxor	18,18,23+	vxor	6,6,3+	vadduwm	21,19,7+	vxor	19,19,23+	vadduwm	22,20,7+	vxor	20,20,23+	vadduwm	4,21,7+	vxor	21,21,23+	vxor	22,22,23++	mtctr	9+	b	.Loop_ctr32_enc8x+.align	5+.Loop_ctr32_enc8x:+	.long	0x11EFC508+	.long	0x1210C508+	.long	0x1231C508+	.long	0x1252C508+	.long	0x1273C508+	.long	0x1294C508+	.long	0x12B5C508+	.long	0x12D6C508+.Loop_ctr32_enc8x_middle:+	lvx	24,26,11+	addi	11,11,0x20++	.long	0x11EFCD08+	.long	0x1210CD08+	.long	0x1231CD08+	.long	0x1252CD08+	.long	0x1273CD08+	.long	0x1294CD08+	.long	0x12B5CD08+	.long	0x12D6CD08+	lvx	25,8,11+	bdnz	.Loop_ctr32_enc8x++	subic	11,5,256+	.long	0x11EFC508+	.long	0x1210C508+	.long	0x1231C508+	.long	0x1252C508+	.long	0x1273C508+	.long	0x1294C508+	.long	0x12B5C508+	.long	0x12D6C508++	subfe	0,0,0+	.long	0x11EFCD08+	.long	0x1210CD08+	.long	0x1231CD08+	.long	0x1252CD08+	.long	0x1273CD08+	.long	0x1294CD08+	.long	0x12B5CD08+	.long	0x12D6CD08++	and	0,0,11+	addi	11,1,64+15+	.long	0x11EFD508+	.long	0x1210D508+	.long	0x1231D508+	.long	0x1252D508+	.long	0x1273D508+	.long	0x1294D508+	.long	0x12B5D508+	.long	0x12D6D508+	lvx	24,0,11++	subic	5,5,129+	.long	0x11EFDD08+	addi	5,5,1+	.long	0x1210DD08+	.long	0x1231DD08+	.long	0x1252DD08+	.long	0x1273DD08+	.long	0x1294DD08+	.long	0x12B5DD08+	.long	0x12D6DD08+	lvx	25,8,11++	.long	0x11EFE508+	.long	0x7C001E99+	.long	0x1210E508+	.long	0x7C281E99+	.long	0x1231E508+	.long	0x7C5A1E99+	.long	0x1252E508+	.long	0x7C7B1E99+	.long	0x1273E508+	.long	0x7D5C1E99+	.long	0x1294E508+	.long	0x7D9D1E99+	.long	0x12B5E508+	.long	0x7DBE1E99+	.long	0x12D6E508+	.long	0x7DDF1E99+	addi	3,3,0x80++	.long	0x11EFED08+	vperm	0,0,0,6+	.long	0x1210ED08+	vperm	1,1,1,6+	.long	0x1231ED08+	vperm	2,2,2,6+	.long	0x1252ED08+	vperm	3,3,3,6+	.long	0x1273ED08+	vperm	10,10,10,6+	.long	0x1294ED08+	vperm	12,12,12,6+	.long	0x12B5ED08+	vperm	13,13,13,6+	.long	0x12D6ED08+	vperm	14,14,14,6++	add	3,3,0++++	subfe.	0,0,0+	.long	0x11EFF508+	vxor	0,0,31+	.long	0x1210F508+	vxor	1,1,31+	.long	0x1231F508+	vxor	2,2,31+	.long	0x1252F508+	vxor	3,3,31+	.long	0x1273F508+	vxor	10,10,31+	.long	0x1294F508+	vxor	12,12,31+	.long	0x12B5F508+	vxor	13,13,31+	.long	0x12D6F508+	vxor	14,14,31++	bne	.Lctr32_enc8x_break++	.long	0x100F0509+	.long	0x10300D09+	vadduwm	16,4,11+	.long	0x10511509+	vadduwm	17,4,7+	vxor	15,4,23+	.long	0x10721D09+	vadduwm	18,16,7+	vxor	16,16,23+	.long	0x11535509+	vadduwm	19,17,7+	vxor	17,17,23+	.long	0x11946509+	vadduwm	20,18,7+	vxor	18,18,23+	.long	0x11B56D09+	vadduwm	21,19,7+	vxor	19,19,23+	.long	0x11D67509+	vadduwm	22,20,7+	vxor	20,20,23+	vperm	0,0,0,6+	vadduwm	4,21,7+	vxor	21,21,23+	vperm	1,1,1,6+	vxor	22,22,23+	mtctr	9++	.long	0x11EFC508+	.long	0x7C002799+	vperm	2,2,2,6+	.long	0x1210C508+	.long	0x7C282799+	vperm	3,3,3,6+	.long	0x1231C508+	.long	0x7C5A2799+	vperm	10,10,10,6+	.long	0x1252C508+	.long	0x7C7B2799+	vperm	12,12,12,6+	.long	0x1273C508+	.long	0x7D5C2799+	vperm	13,13,13,6+	.long	0x1294C508+	.long	0x7D9D2799+	vperm	14,14,14,6+	.long	0x12B5C508+	.long	0x7DBE2799+	.long	0x12D6C508+	.long	0x7DDF2799+	addi	4,4,0x80++	b	.Loop_ctr32_enc8x_middle++.align	5+.Lctr32_enc8x_break:+	cmpwi	5,-0x60+	blt	.Lctr32_enc8x_one+	nop	+	beq	.Lctr32_enc8x_two+	cmpwi	5,-0x40+	blt	.Lctr32_enc8x_three+	nop	+	beq	.Lctr32_enc8x_four+	cmpwi	5,-0x20+	blt	.Lctr32_enc8x_five+	nop	+	beq	.Lctr32_enc8x_six+	cmpwi	5,0x00+	blt	.Lctr32_enc8x_seven++.Lctr32_enc8x_eight:+	.long	0x11EF0509+	.long	0x12100D09+	.long	0x12311509+	.long	0x12521D09+	.long	0x12735509+	.long	0x12946509+	.long	0x12B56D09+	.long	0x12D67509++	vperm	15,15,15,6+	vperm	16,16,16,6+	.long	0x7DE02799+	vperm	17,17,17,6+	.long	0x7E082799+	vperm	18,18,18,6+	.long	0x7E3A2799+	vperm	19,19,19,6+	.long	0x7E5B2799+	vperm	20,20,20,6+	.long	0x7E7C2799+	vperm	21,21,21,6+	.long	0x7E9D2799+	vperm	22,22,22,6+	.long	0x7EBE2799+	.long	0x7EDF2799+	addi	4,4,0x80+	b	.Lctr32_enc8x_done++.align	5+.Lctr32_enc8x_seven:+	.long	0x11EF0D09+	.long	0x12101509+	.long	0x12311D09+	.long	0x12525509+	.long	0x12736509+	.long	0x12946D09+	.long	0x12B57509++	vperm	15,15,15,6+	vperm	16,16,16,6+	.long	0x7DE02799+	vperm	17,17,17,6+	.long	0x7E082799+	vperm	18,18,18,6+	.long	0x7E3A2799+	vperm	19,19,19,6+	.long	0x7E5B2799+	vperm	20,20,20,6+	.long	0x7E7C2799+	vperm	21,21,21,6+	.long	0x7E9D2799+	.long	0x7EBE2799+	addi	4,4,0x70+	b	.Lctr32_enc8x_done++.align	5+.Lctr32_enc8x_six:+	.long	0x11EF1509+	.long	0x12101D09+	.long	0x12315509+	.long	0x12526509+	.long	0x12736D09+	.long	0x12947509++	vperm	15,15,15,6+	vperm	16,16,16,6+	.long	0x7DE02799+	vperm	17,17,17,6+	.long	0x7E082799+	vperm	18,18,18,6+	.long	0x7E3A2799+	vperm	19,19,19,6+	.long	0x7E5B2799+	vperm	20,20,20,6+	.long	0x7E7C2799+	.long	0x7E9D2799+	addi	4,4,0x60+	b	.Lctr32_enc8x_done++.align	5+.Lctr32_enc8x_five:+	.long	0x11EF1D09+	.long	0x12105509+	.long	0x12316509+	.long	0x12526D09+	.long	0x12737509++	vperm	15,15,15,6+	vperm	16,16,16,6+	.long	0x7DE02799+	vperm	17,17,17,6+	.long	0x7E082799+	vperm	18,18,18,6+	.long	0x7E3A2799+	vperm	19,19,19,6+	.long	0x7E5B2799+	.long	0x7E7C2799+	addi	4,4,0x50+	b	.Lctr32_enc8x_done++.align	5+.Lctr32_enc8x_four:+	.long	0x11EF5509+	.long	0x12106509+	.long	0x12316D09+	.long	0x12527509++	vperm	15,15,15,6+	vperm	16,16,16,6+	.long	0x7DE02799+	vperm	17,17,17,6+	.long	0x7E082799+	vperm	18,18,18,6+	.long	0x7E3A2799+	.long	0x7E5B2799+	addi	4,4,0x40+	b	.Lctr32_enc8x_done++.align	5+.Lctr32_enc8x_three:+	.long	0x11EF6509+	.long	0x12106D09+	.long	0x12317509++	vperm	15,15,15,6+	vperm	16,16,16,6+	.long	0x7DE02799+	vperm	17,17,17,6+	.long	0x7E082799+	.long	0x7E3A2799+	addi	4,4,0x30+	b	.Lctr32_enc8x_done++.align	5+.Lctr32_enc8x_two:+	.long	0x11EF6D09+	.long	0x12107509++	vperm	15,15,15,6+	vperm	16,16,16,6+	.long	0x7DE02799+	.long	0x7E082799+	addi	4,4,0x20+	b	.Lctr32_enc8x_done++.align	5+.Lctr32_enc8x_one:+	.long	0x11EF7509++	vperm	15,15,15,6+	.long	0x7DE02799+	addi	4,4,0x10++.Lctr32_enc8x_done:+	li	10,79+	li	11,95+	stvx	6,10,1+	addi	10,10,32+	stvx	6,11,1+	addi	11,11,32+	stvx	6,10,1+	addi	10,10,32+	stvx	6,11,1+	addi	11,11,32+	stvx	6,10,1+	addi	10,10,32+	stvx	6,11,1+	addi	11,11,32+	stvx	6,10,1+	addi	10,10,32+	stvx	6,11,1+	addi	11,11,32++	or	12,12,12+	lvx	20,10,1+	addi	10,10,32+	lvx	21,11,1+	addi	11,11,32+	lvx	22,10,1+	addi	10,10,32+	lvx	23,11,1+	addi	11,11,32+	lvx	24,10,1+	addi	10,10,32+	lvx	25,11,1+	addi	11,11,32+	lvx	26,10,1+	addi	10,10,32+	lvx	27,11,1+	addi	11,11,32+	lvx	28,10,1+	addi	10,10,32+	lvx	29,11,1+	addi	11,11,32+	lvx	30,10,1+	lvx	31,11,1+	ld	26,400(1)+	ld	27,408(1)+	ld	28,416(1)+	ld	29,424(1)+	ld	30,432(1)+	ld	31,440(1)+	addi	1,1,448+	blr	+.long	0+.byte	0,12,0x04,0,0x80,6,6,0+.long	0+.size	crypton_aes_p8_ctr32_encrypt_blocks,.-crypton_aes_p8_ctr32_encrypt_blocks+.globl	crypton_aes_p8_xts_encrypt+.type	crypton_aes_p8_xts_encrypt,@function+.align	5+crypton_aes_p8_xts_encrypt:+.localentry	crypton_aes_p8_xts_encrypt,0++	mr	10,3+	li	3,-1+	cmpldi	5,16+	.long	0x4dc00020++	lis	0,0xfff0+	li	12,-1+	li	11,0+	or	0,0,0++	vspltisb	9,0x07+	lvsl	6,11,11+	vspltisb	11,0x0f+	vxor	6,6,9++	li	3,15+	lvx	8,0,8+	lvsl	5,0,8+	lvx	4,3,8+	vxor	5,5,11+	vperm	8,8,4,5++	neg	11,10+	lvsr	5,0,11+	lvx	2,0,10+	addi	10,10,15+	vxor	5,5,11++	cmpldi	7,0+	beq	.Lxts_enc_no_key2++	lvsr	7,0,7+	lwz	9,240(7)+	srwi	9,9,1+	subi	9,9,1+	li	3,16++	lvx	0,0,7+	lvx	1,3,7+	addi	3,3,16+	vperm	0,1,0,7+	vxor	8,8,0+	lvx	0,3,7+	addi	3,3,16+	mtctr	9++.Ltweak_xts_enc:+	vperm	1,0,1,7+	.long	0x11080D08+	lvx	1,3,7+	addi	3,3,16+	vperm	0,1,0,7+	.long	0x11080508+	lvx	0,3,7+	addi	3,3,16+	bdnz	.Ltweak_xts_enc++	vperm	1,0,1,7+	.long	0x11080D08+	lvx	1,3,7+	vperm	0,1,0,7+	.long	0x11080509++	li	8,0+	b	.Lxts_enc++.Lxts_enc_no_key2:+	li	3,-16+	and	5,5,3+++.Lxts_enc:+	lvx	4,0,10+	addi	10,10,16++	lvsr	7,0,6+	lwz	9,240(6)+	srwi	9,9,1+	subi	9,9,1+	li	3,16++	vslb	10,9,9+	vor	10,10,9+	vspltisb	11,1+	vsldoi	10,10,11,15++	cmpldi	5,96+	bge	_aesp8_xts_encrypt6x++	andi.	7,5,15+	subic	0,5,32+	subi	7,7,16+	subfe	0,0,0+	and	0,0,7+	add	10,10,0++	lvx	0,0,6+	lvx	1,3,6+	addi	3,3,16+	vperm	2,2,4,5+	vperm	0,1,0,7+	vxor	2,2,8+	vxor	2,2,0+	lvx	0,3,6+	addi	3,3,16+	mtctr	9+	b	.Loop_xts_enc++.align	5+.Loop_xts_enc:+	vperm	1,0,1,7+	.long	0x10420D08+	lvx	1,3,6+	addi	3,3,16+	vperm	0,1,0,7+	.long	0x10420508+	lvx	0,3,6+	addi	3,3,16+	bdnz	.Loop_xts_enc++	vperm	1,0,1,7+	.long	0x10420D08+	lvx	1,3,6+	li	3,16+	vperm	0,1,0,7+	vxor	0,0,8+	.long	0x10620509++	vperm	11,3,3,6++	.long	0x7D602799++	addi	4,4,16++	subic.	5,5,16+	beq	.Lxts_enc_done++	vor	2,4,4+	lvx	4,0,10+	addi	10,10,16+	lvx	0,0,6+	lvx	1,3,6+	addi	3,3,16++	subic	0,5,32+	subfe	0,0,0+	and	0,0,7+	add	10,10,0++	vsrab	11,8,9+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	vand	11,11,10+	vxor	8,8,11++	vperm	2,2,4,5+	vperm	0,1,0,7+	vxor	2,2,8+	vxor	3,3,0+	vxor	2,2,0+	lvx	0,3,6+	addi	3,3,16++	mtctr	9+	cmpldi	5,16+	bge	.Loop_xts_enc++	vxor	3,3,8+	lvsr	5,0,5+	vxor	4,4,4+	vspltisb	11,-1+	vperm	4,4,11,5+	vsel	2,2,3,4++	subi	11,4,17+	subi	4,4,16+	mtctr	5+	li	5,16+.Loop_xts_enc_steal:+	lbzu	0,1(11)+	stb	0,16(11)+	bdnz	.Loop_xts_enc_steal++	mtctr	9+	b	.Loop_xts_enc++.Lxts_enc_done:+	cmpldi	8,0+	beq	.Lxts_enc_ret++	vsrab	11,8,9+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	vand	11,11,10+	vxor	8,8,11++	vperm	8,8,8,6+	.long	0x7D004799++.Lxts_enc_ret:+	or	12,12,12+	li	3,0+	blr	+.long	0+.byte	0,12,0x04,0,0x80,6,6,0+.long	0+.size	crypton_aes_p8_xts_encrypt,.-crypton_aes_p8_xts_encrypt++.globl	crypton_aes_p8_xts_decrypt+.type	crypton_aes_p8_xts_decrypt,@function+.align	5+crypton_aes_p8_xts_decrypt:+.localentry	crypton_aes_p8_xts_decrypt,0++	mr	10,3+	li	3,-1+	cmpldi	5,16+	.long	0x4dc00020++	lis	0,0xfff8+	li	12,-1+	li	11,0+	or	0,0,0++	andi.	0,5,15+	neg	0,0+	andi.	0,0,16+	sub	5,5,0++	vspltisb	9,0x07+	lvsl	6,11,11+	vspltisb	11,0x0f+	vxor	6,6,9++	li	3,15+	lvx	8,0,8+	lvsl	5,0,8+	lvx	4,3,8+	vxor	5,5,11+	vperm	8,8,4,5++	neg	11,10+	lvsr	5,0,11+	lvx	2,0,10+	addi	10,10,15+	vxor	5,5,11++	cmpldi	7,0+	beq	.Lxts_dec_no_key2++	lvsr	7,0,7+	lwz	9,240(7)+	srwi	9,9,1+	subi	9,9,1+	li	3,16++	lvx	0,0,7+	lvx	1,3,7+	addi	3,3,16+	vperm	0,1,0,7+	vxor	8,8,0+	lvx	0,3,7+	addi	3,3,16+	mtctr	9++.Ltweak_xts_dec:+	vperm	1,0,1,7+	.long	0x11080D08+	lvx	1,3,7+	addi	3,3,16+	vperm	0,1,0,7+	.long	0x11080508+	lvx	0,3,7+	addi	3,3,16+	bdnz	.Ltweak_xts_dec++	vperm	1,0,1,7+	.long	0x11080D08+	lvx	1,3,7+	vperm	0,1,0,7+	.long	0x11080509++	li	8,0+	b	.Lxts_dec++.Lxts_dec_no_key2:+	neg	3,5+	andi.	3,3,15+	add	5,5,3+++.Lxts_dec:+	lvx	4,0,10+	addi	10,10,16++	lvsr	7,0,6+	lwz	9,240(6)+	srwi	9,9,1+	subi	9,9,1+	li	3,16++	vslb	10,9,9+	vor	10,10,9+	vspltisb	11,1+	vsldoi	10,10,11,15++	cmpldi	5,96+	bge	_aesp8_xts_decrypt6x++	lvx	0,0,6+	lvx	1,3,6+	addi	3,3,16+	vperm	2,2,4,5+	vperm	0,1,0,7+	vxor	2,2,8+	vxor	2,2,0+	lvx	0,3,6+	addi	3,3,16+	mtctr	9++	cmpldi	5,16+	blt	.Ltail_xts_dec+++.align	5+.Loop_xts_dec:+	vperm	1,0,1,7+	.long	0x10420D48+	lvx	1,3,6+	addi	3,3,16+	vperm	0,1,0,7+	.long	0x10420548+	lvx	0,3,6+	addi	3,3,16+	bdnz	.Loop_xts_dec++	vperm	1,0,1,7+	.long	0x10420D48+	lvx	1,3,6+	li	3,16+	vperm	0,1,0,7+	vxor	0,0,8+	.long	0x10620549++	vperm	11,3,3,6++	.long	0x7D602799++	addi	4,4,16++	subic.	5,5,16+	beq	.Lxts_dec_done++	vor	2,4,4+	lvx	4,0,10+	addi	10,10,16+	lvx	0,0,6+	lvx	1,3,6+	addi	3,3,16++	vsrab	11,8,9+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	vand	11,11,10+	vxor	8,8,11++	vperm	2,2,4,5+	vperm	0,1,0,7+	vxor	2,2,8+	vxor	2,2,0+	lvx	0,3,6+	addi	3,3,16++	mtctr	9+	cmpldi	5,16+	bge	.Loop_xts_dec++.Ltail_xts_dec:+	vsrab	11,8,9+	vaddubm	12,8,8+	vsldoi	11,11,11,15+	vand	11,11,10+	vxor	12,12,11++	subi	10,10,16+	add	10,10,5++	vxor	2,2,8+	vxor	2,2,12++.Loop_xts_dec_short:+	vperm	1,0,1,7+	.long	0x10420D48+	lvx	1,3,6+	addi	3,3,16+	vperm	0,1,0,7+	.long	0x10420548+	lvx	0,3,6+	addi	3,3,16+	bdnz	.Loop_xts_dec_short++	vperm	1,0,1,7+	.long	0x10420D48+	lvx	1,3,6+	li	3,16+	vperm	0,1,0,7+	vxor	0,0,12+	.long	0x10620549++	vperm	11,3,3,6++	.long	0x7D602799+++	vor	2,4,4+	lvx	4,0,10++	lvx	0,0,6+	lvx	1,3,6+	addi	3,3,16+	vperm	2,2,4,5+	vperm	0,1,0,7++	lvsr	5,0,5+	vxor	4,4,4+	vspltisb	11,-1+	vperm	4,4,11,5+	vsel	2,2,3,4++	vxor	0,0,8+	vxor	2,2,0+	lvx	0,3,6+	addi	3,3,16++	subi	11,4,1+	mtctr	5+	li	5,16+.Loop_xts_dec_steal:+	lbzu	0,1(11)+	stb	0,16(11)+	bdnz	.Loop_xts_dec_steal++	mtctr	9+	b	.Loop_xts_dec++.Lxts_dec_done:+	cmpldi	8,0+	beq	.Lxts_dec_ret++	vsrab	11,8,9+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	vand	11,11,10+	vxor	8,8,11++	vperm	8,8,8,6+	.long	0x7D004799++.Lxts_dec_ret:+	or	12,12,12+	li	3,0+	blr	+.long	0+.byte	0,12,0x04,0,0x80,6,6,0+.long	0+.size	crypton_aes_p8_xts_decrypt,.-crypton_aes_p8_xts_decrypt+.align	5+_aesp8_xts_encrypt6x:+	stdu	1,-448(1)+	mflr	11+	li	7,207+	li	3,223+	std	11,464(1)+	stvx	20,7,1+	addi	7,7,32+	stvx	21,3,1+	addi	3,3,32+	stvx	22,7,1+	addi	7,7,32+	stvx	23,3,1+	addi	3,3,32+	stvx	24,7,1+	addi	7,7,32+	stvx	25,3,1+	addi	3,3,32+	stvx	26,7,1+	addi	7,7,32+	stvx	27,3,1+	addi	3,3,32+	stvx	28,7,1+	addi	7,7,32+	stvx	29,3,1+	addi	3,3,32+	stvx	30,7,1+	stvx	31,3,1+	li	0,-1+	stw	12,396(1)+	li	3,0x10+	std	26,400(1)+	li	26,0x20+	std	27,408(1)+	li	27,0x30+	std	28,416(1)+	li	28,0x40+	std	29,424(1)+	li	29,0x50+	std	30,432(1)+	li	30,0x60+	std	31,440(1)+	li	31,0x70+	or	0,0,0++	subi	9,9,3++	lvx	23,0,6+	lvx	30,3,6+	addi	6,6,0x20+	lvx	31,0,6+	vperm	23,30,23,7+	addi	7,1,64+15+	mtctr	9++.Load_xts_enc_key:+	vperm	24,31,30,7+	lvx	30,3,6+	addi	6,6,0x20+	stvx	24,0,7+	vperm	25,30,31,7+	lvx	31,0,6+	stvx	25,3,7+	addi	7,7,0x20+	bdnz	.Load_xts_enc_key++	lvx	26,3,6+	vperm	24,31,30,7+	lvx	27,26,6+	stvx	24,0,7+	vperm	25,26,31,7+	lvx	28,27,6+	stvx	25,3,7+	addi	7,1,64+15+	vperm	26,27,26,7+	lvx	29,28,6+	vperm	27,28,27,7+	lvx	30,29,6+	vperm	28,29,28,7+	lvx	31,30,6+	vperm	29,30,29,7+	lvx	22,31,6+	vperm	30,31,30,7+	lvx	24,0,7+	vperm	31,22,31,7+	lvx	25,3,7++	vperm	0,2,4,5+	subi	10,10,31+	vxor	17,8,23+	vsrab	11,8,9+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	vand	11,11,10+	vxor	7,0,17+	vxor	8,8,11++	.long	0x7C235699+	vxor	18,8,23+	vsrab	11,8,9+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	vperm	1,1,1,6+	vand	11,11,10+	vxor	12,1,18+	vxor	8,8,11++	.long	0x7C5A5699+	andi.	31,5,15+	vxor	19,8,23+	vsrab	11,8,9+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	vperm	2,2,2,6+	vand	11,11,10+	vxor	13,2,19+	vxor	8,8,11++	.long	0x7C7B5699+	sub	5,5,31+	vxor	20,8,23+	vsrab	11,8,9+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	vperm	3,3,3,6+	vand	11,11,10+	vxor	14,3,20+	vxor	8,8,11++	.long	0x7C9C5699+	subi	5,5,0x60+	vxor	21,8,23+	vsrab	11,8,9+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	vperm	4,4,4,6+	vand	11,11,10+	vxor	15,4,21+	vxor	8,8,11++	.long	0x7CBD5699+	addi	10,10,0x60+	vxor	22,8,23+	vsrab	11,8,9+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	vperm	5,5,5,6+	vand	11,11,10+	vxor	16,5,22+	vxor	8,8,11++	vxor	31,31,23+	mtctr	9+	b	.Loop_xts_enc6x++.align	5+.Loop_xts_enc6x:+	.long	0x10E7C508+	.long	0x118CC508+	.long	0x11ADC508+	.long	0x11CEC508+	.long	0x11EFC508+	.long	0x1210C508+	lvx	24,26,7+	addi	7,7,0x20++	.long	0x10E7CD08+	.long	0x118CCD08+	.long	0x11ADCD08+	.long	0x11CECD08+	.long	0x11EFCD08+	.long	0x1210CD08+	lvx	25,3,7+	bdnz	.Loop_xts_enc6x++	subic	5,5,96+	vxor	0,17,31+	.long	0x10E7C508+	.long	0x118CC508+	vsrab	11,8,9+	vxor	17,8,23+	vaddubm	8,8,8+	.long	0x11ADC508+	.long	0x11CEC508+	vsldoi	11,11,11,15+	.long	0x11EFC508+	.long	0x1210C508++	subfe.	0,0,0+	vand	11,11,10+	.long	0x10E7CD08+	.long	0x118CCD08+	vxor	8,8,11+	.long	0x11ADCD08+	.long	0x11CECD08+	vxor	1,18,31+	vsrab	11,8,9+	vxor	18,8,23+	.long	0x11EFCD08+	.long	0x1210CD08++	and	0,0,5+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	.long	0x10E7D508+	.long	0x118CD508+	vand	11,11,10+	.long	0x11ADD508+	.long	0x11CED508+	vxor	8,8,11+	.long	0x11EFD508+	.long	0x1210D508++	add	10,10,0++++	vxor	2,19,31+	vsrab	11,8,9+	vxor	19,8,23+	vaddubm	8,8,8+	.long	0x10E7DD08+	.long	0x118CDD08+	vsldoi	11,11,11,15+	.long	0x11ADDD08+	.long	0x11CEDD08+	vand	11,11,10+	.long	0x11EFDD08+	.long	0x1210DD08++	addi	7,1,64+15+	vxor	8,8,11+	.long	0x10E7E508+	.long	0x118CE508+	vxor	3,20,31+	vsrab	11,8,9+	vxor	20,8,23+	.long	0x11ADE508+	.long	0x11CEE508+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	.long	0x11EFE508+	.long	0x1210E508+	lvx	24,0,7+	vand	11,11,10++	.long	0x10E7ED08+	.long	0x118CED08+	vxor	8,8,11+	.long	0x11ADED08+	.long	0x11CEED08+	vxor	4,21,31+	vsrab	11,8,9+	vxor	21,8,23+	.long	0x11EFED08+	.long	0x1210ED08+	lvx	25,3,7+	vaddubm	8,8,8+	vsldoi	11,11,11,15++	.long	0x10E7F508+	.long	0x118CF508+	vand	11,11,10+	.long	0x11ADF508+	.long	0x11CEF508+	vxor	8,8,11+	.long	0x11EFF508+	.long	0x1210F508+	vxor	5,22,31+	vsrab	11,8,9+	vxor	22,8,23++	.long	0x10E70509+	.long	0x7C005699+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	.long	0x118C0D09+	.long	0x7C235699+	.long	0x11AD1509+	vperm	0,0,0,6+	.long	0x7C5A5699+	vand	11,11,10+	.long	0x11CE1D09+	vperm	1,1,1,6+	.long	0x7C7B5699+	.long	0x11EF2509+	vperm	2,2,2,6+	.long	0x7C9C5699+	vxor	8,8,11+	.long	0x11702D09++	vperm	3,3,3,6+	.long	0x7CBD5699+	addi	10,10,0x60+	vperm	4,4,4,6+	vperm	5,5,5,6++	vperm	7,7,7,6+	vperm	12,12,12,6+	.long	0x7CE02799+	vxor	7,0,17+	vperm	13,13,13,6+	.long	0x7D832799+	vxor	12,1,18+	vperm	14,14,14,6+	.long	0x7DBA2799+	vxor	13,2,19+	vperm	15,15,15,6+	.long	0x7DDB2799+	vxor	14,3,20+	vperm	16,11,11,6+	.long	0x7DFC2799+	vxor	15,4,21+	.long	0x7E1D2799++	vxor	16,5,22+	addi	4,4,0x60++	mtctr	9+	beq	.Loop_xts_enc6x++	addic.	5,5,0x60+	beq	.Lxts_enc6x_zero+	cmpwi	5,0x20+	blt	.Lxts_enc6x_one+	nop	+	beq	.Lxts_enc6x_two+	cmpwi	5,0x40+	blt	.Lxts_enc6x_three+	nop	+	beq	.Lxts_enc6x_four++.Lxts_enc6x_five:+	vxor	7,1,17+	vxor	12,2,18+	vxor	13,3,19+	vxor	14,4,20+	vxor	15,5,21++	bl	_aesp8_xts_enc5x++	vperm	7,7,7,6+	vor	17,22,22+	vperm	12,12,12,6+	.long	0x7CE02799+	vperm	13,13,13,6+	.long	0x7D832799+	vperm	14,14,14,6+	.long	0x7DBA2799+	vxor	11,15,22+	vperm	15,15,15,6+	.long	0x7DDB2799+	.long	0x7DFC2799+	addi	4,4,0x50+	bne	.Lxts_enc6x_steal+	b	.Lxts_enc6x_done++.align	4+.Lxts_enc6x_four:+	vxor	7,2,17+	vxor	12,3,18+	vxor	13,4,19+	vxor	14,5,20+	vxor	15,15,15++	bl	_aesp8_xts_enc5x++	vperm	7,7,7,6+	vor	17,21,21+	vperm	12,12,12,6+	.long	0x7CE02799+	vperm	13,13,13,6+	.long	0x7D832799+	vxor	11,14,21+	vperm	14,14,14,6+	.long	0x7DBA2799+	.long	0x7DDB2799+	addi	4,4,0x40+	bne	.Lxts_enc6x_steal+	b	.Lxts_enc6x_done++.align	4+.Lxts_enc6x_three:+	vxor	7,3,17+	vxor	12,4,18+	vxor	13,5,19+	vxor	14,14,14+	vxor	15,15,15++	bl	_aesp8_xts_enc5x++	vperm	7,7,7,6+	vor	17,20,20+	vperm	12,12,12,6+	.long	0x7CE02799+	vxor	11,13,20+	vperm	13,13,13,6+	.long	0x7D832799+	.long	0x7DBA2799+	addi	4,4,0x30+	bne	.Lxts_enc6x_steal+	b	.Lxts_enc6x_done++.align	4+.Lxts_enc6x_two:+	vxor	7,4,17+	vxor	12,5,18+	vxor	13,13,13+	vxor	14,14,14+	vxor	15,15,15++	bl	_aesp8_xts_enc5x++	vperm	7,7,7,6+	vor	17,19,19+	vxor	11,12,19+	vperm	12,12,12,6+	.long	0x7CE02799+	.long	0x7D832799+	addi	4,4,0x20+	bne	.Lxts_enc6x_steal+	b	.Lxts_enc6x_done++.align	4+.Lxts_enc6x_one:+	vxor	7,5,17+	nop	+.Loop_xts_enc1x:+	.long	0x10E7C508+	lvx	24,26,7+	addi	7,7,0x20++	.long	0x10E7CD08+	lvx	25,3,7+	bdnz	.Loop_xts_enc1x++	add	10,10,31+	cmpwi	31,0+	.long	0x10E7C508++	subi	10,10,16+	.long	0x10E7CD08++	lvsr	5,0,31+	.long	0x10E7D508++	.long	0x7C005699+	.long	0x10E7DD08++	addi	7,1,64+15+	.long	0x10E7E508+	lvx	24,0,7++	.long	0x10E7ED08+	lvx	25,3,7+	vxor	17,17,31++	vperm	0,0,0,6+	.long	0x10E7F508++	vperm	0,0,0,5+	.long	0x10E78D09++	vor	17,18,18+	vxor	11,7,18+	vperm	7,7,7,6+	.long	0x7CE02799+	addi	4,4,0x10+	bne	.Lxts_enc6x_steal+	b	.Lxts_enc6x_done++.align	4+.Lxts_enc6x_zero:+	cmpwi	31,0+	beq	.Lxts_enc6x_done++	add	10,10,31+	subi	10,10,16+	.long	0x7C005699+	lvsr	5,0,31+	vperm	0,0,0,6+	vperm	0,0,0,5+	vxor	11,11,17+.Lxts_enc6x_steal:+	vxor	0,0,17+	vxor	7,7,7+	vspltisb	12,-1+	vperm	7,7,12,5+	vsel	7,0,11,7++	subi	30,4,17+	subi	4,4,16+	mtctr	31+.Loop_xts_enc6x_steal:+	lbzu	0,1(30)+	stb	0,16(30)+	bdnz	.Loop_xts_enc6x_steal++	li	31,0+	mtctr	9+	b	.Loop_xts_enc1x++.align	4+.Lxts_enc6x_done:+	cmpldi	8,0+	beq	.Lxts_enc6x_ret++	vxor	8,17,23+	vperm	8,8,8,6+	.long	0x7D004799++.Lxts_enc6x_ret:+	mtlr	11+	li	10,79+	li	11,95+	stvx	9,10,1+	addi	10,10,32+	stvx	9,11,1+	addi	11,11,32+	stvx	9,10,1+	addi	10,10,32+	stvx	9,11,1+	addi	11,11,32+	stvx	9,10,1+	addi	10,10,32+	stvx	9,11,1+	addi	11,11,32+	stvx	9,10,1+	addi	10,10,32+	stvx	9,11,1+	addi	11,11,32++	or	12,12,12+	lvx	20,10,1+	addi	10,10,32+	lvx	21,11,1+	addi	11,11,32+	lvx	22,10,1+	addi	10,10,32+	lvx	23,11,1+	addi	11,11,32+	lvx	24,10,1+	addi	10,10,32+	lvx	25,11,1+	addi	11,11,32+	lvx	26,10,1+	addi	10,10,32+	lvx	27,11,1+	addi	11,11,32+	lvx	28,10,1+	addi	10,10,32+	lvx	29,11,1+	addi	11,11,32+	lvx	30,10,1+	lvx	31,11,1+	ld	26,400(1)+	ld	27,408(1)+	ld	28,416(1)+	ld	29,424(1)+	ld	30,432(1)+	ld	31,440(1)+	addi	1,1,448+	blr	+.long	0+.byte	0,12,0x04,1,0x80,6,6,0+.long	0++.align	5+_aesp8_xts_enc5x:+	.long	0x10E7C508+	.long	0x118CC508+	.long	0x11ADC508+	.long	0x11CEC508+	.long	0x11EFC508+	lvx	24,26,7+	addi	7,7,0x20++	.long	0x10E7CD08+	.long	0x118CCD08+	.long	0x11ADCD08+	.long	0x11CECD08+	.long	0x11EFCD08+	lvx	25,3,7+	bdnz	_aesp8_xts_enc5x++	add	10,10,31+	cmpwi	31,0+	.long	0x10E7C508+	.long	0x118CC508+	.long	0x11ADC508+	.long	0x11CEC508+	.long	0x11EFC508++	subi	10,10,16+	.long	0x10E7CD08+	.long	0x118CCD08+	.long	0x11ADCD08+	.long	0x11CECD08+	.long	0x11EFCD08+	vxor	17,17,31++	.long	0x10E7D508+	lvsr	5,0,31+	.long	0x118CD508+	.long	0x11ADD508+	.long	0x11CED508+	.long	0x11EFD508+	vxor	1,18,31++	.long	0x10E7DD08+	.long	0x7C005699+	.long	0x118CDD08+	.long	0x11ADDD08+	.long	0x11CEDD08+	.long	0x11EFDD08+	vxor	2,19,31++	addi	7,1,64+15+	.long	0x10E7E508+	.long	0x118CE508+	.long	0x11ADE508+	.long	0x11CEE508+	.long	0x11EFE508+	lvx	24,0,7+	vxor	3,20,31++	.long	0x10E7ED08+	vperm	0,0,0,6+	.long	0x118CED08+	.long	0x11ADED08+	.long	0x11CEED08+	.long	0x11EFED08+	lvx	25,3,7+	vxor	4,21,31++	.long	0x10E7F508+	vperm	0,0,0,5+	.long	0x118CF508+	.long	0x11ADF508+	.long	0x11CEF508+	.long	0x11EFF508++	.long	0x10E78D09+	.long	0x118C0D09+	.long	0x11AD1509+	.long	0x11CE1D09+	.long	0x11EF2509+	blr	+.long	0+.byte	0,12,0x14,0,0,0,0,0++.align	5+_aesp8_xts_decrypt6x:+	stdu	1,-448(1)+	mflr	11+	li	7,207+	li	3,223+	std	11,464(1)+	stvx	20,7,1+	addi	7,7,32+	stvx	21,3,1+	addi	3,3,32+	stvx	22,7,1+	addi	7,7,32+	stvx	23,3,1+	addi	3,3,32+	stvx	24,7,1+	addi	7,7,32+	stvx	25,3,1+	addi	3,3,32+	stvx	26,7,1+	addi	7,7,32+	stvx	27,3,1+	addi	3,3,32+	stvx	28,7,1+	addi	7,7,32+	stvx	29,3,1+	addi	3,3,32+	stvx	30,7,1+	stvx	31,3,1+	li	0,-1+	stw	12,396(1)+	li	3,0x10+	std	26,400(1)+	li	26,0x20+	std	27,408(1)+	li	27,0x30+	std	28,416(1)+	li	28,0x40+	std	29,424(1)+	li	29,0x50+	std	30,432(1)+	li	30,0x60+	std	31,440(1)+	li	31,0x70+	or	0,0,0++	subi	9,9,3++	lvx	23,0,6+	lvx	30,3,6+	addi	6,6,0x20+	lvx	31,0,6+	vperm	23,30,23,7+	addi	7,1,64+15+	mtctr	9++.Load_xts_dec_key:+	vperm	24,31,30,7+	lvx	30,3,6+	addi	6,6,0x20+	stvx	24,0,7+	vperm	25,30,31,7+	lvx	31,0,6+	stvx	25,3,7+	addi	7,7,0x20+	bdnz	.Load_xts_dec_key++	lvx	26,3,6+	vperm	24,31,30,7+	lvx	27,26,6+	stvx	24,0,7+	vperm	25,26,31,7+	lvx	28,27,6+	stvx	25,3,7+	addi	7,1,64+15+	vperm	26,27,26,7+	lvx	29,28,6+	vperm	27,28,27,7+	lvx	30,29,6+	vperm	28,29,28,7+	lvx	31,30,6+	vperm	29,30,29,7+	lvx	22,31,6+	vperm	30,31,30,7+	lvx	24,0,7+	vperm	31,22,31,7+	lvx	25,3,7++	vperm	0,2,4,5+	subi	10,10,31+	vxor	17,8,23+	vsrab	11,8,9+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	vand	11,11,10+	vxor	7,0,17+	vxor	8,8,11++	.long	0x7C235699+	vxor	18,8,23+	vsrab	11,8,9+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	vperm	1,1,1,6+	vand	11,11,10+	vxor	12,1,18+	vxor	8,8,11++	.long	0x7C5A5699+	andi.	31,5,15+	vxor	19,8,23+	vsrab	11,8,9+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	vperm	2,2,2,6+	vand	11,11,10+	vxor	13,2,19+	vxor	8,8,11++	.long	0x7C7B5699+	sub	5,5,31+	vxor	20,8,23+	vsrab	11,8,9+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	vperm	3,3,3,6+	vand	11,11,10+	vxor	14,3,20+	vxor	8,8,11++	.long	0x7C9C5699+	subi	5,5,0x60+	vxor	21,8,23+	vsrab	11,8,9+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	vperm	4,4,4,6+	vand	11,11,10+	vxor	15,4,21+	vxor	8,8,11++	.long	0x7CBD5699+	addi	10,10,0x60+	vxor	22,8,23+	vsrab	11,8,9+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	vperm	5,5,5,6+	vand	11,11,10+	vxor	16,5,22+	vxor	8,8,11++	vxor	31,31,23+	mtctr	9+	b	.Loop_xts_dec6x++.align	5+.Loop_xts_dec6x:+	.long	0x10E7C548+	.long	0x118CC548+	.long	0x11ADC548+	.long	0x11CEC548+	.long	0x11EFC548+	.long	0x1210C548+	lvx	24,26,7+	addi	7,7,0x20++	.long	0x10E7CD48+	.long	0x118CCD48+	.long	0x11ADCD48+	.long	0x11CECD48+	.long	0x11EFCD48+	.long	0x1210CD48+	lvx	25,3,7+	bdnz	.Loop_xts_dec6x++	subic	5,5,96+	vxor	0,17,31+	.long	0x10E7C548+	.long	0x118CC548+	vsrab	11,8,9+	vxor	17,8,23+	vaddubm	8,8,8+	.long	0x11ADC548+	.long	0x11CEC548+	vsldoi	11,11,11,15+	.long	0x11EFC548+	.long	0x1210C548++	subfe.	0,0,0+	vand	11,11,10+	.long	0x10E7CD48+	.long	0x118CCD48+	vxor	8,8,11+	.long	0x11ADCD48+	.long	0x11CECD48+	vxor	1,18,31+	vsrab	11,8,9+	vxor	18,8,23+	.long	0x11EFCD48+	.long	0x1210CD48++	and	0,0,5+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	.long	0x10E7D548+	.long	0x118CD548+	vand	11,11,10+	.long	0x11ADD548+	.long	0x11CED548+	vxor	8,8,11+	.long	0x11EFD548+	.long	0x1210D548++	add	10,10,0++++	vxor	2,19,31+	vsrab	11,8,9+	vxor	19,8,23+	vaddubm	8,8,8+	.long	0x10E7DD48+	.long	0x118CDD48+	vsldoi	11,11,11,15+	.long	0x11ADDD48+	.long	0x11CEDD48+	vand	11,11,10+	.long	0x11EFDD48+	.long	0x1210DD48++	addi	7,1,64+15+	vxor	8,8,11+	.long	0x10E7E548+	.long	0x118CE548+	vxor	3,20,31+	vsrab	11,8,9+	vxor	20,8,23+	.long	0x11ADE548+	.long	0x11CEE548+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	.long	0x11EFE548+	.long	0x1210E548+	lvx	24,0,7+	vand	11,11,10++	.long	0x10E7ED48+	.long	0x118CED48+	vxor	8,8,11+	.long	0x11ADED48+	.long	0x11CEED48+	vxor	4,21,31+	vsrab	11,8,9+	vxor	21,8,23+	.long	0x11EFED48+	.long	0x1210ED48+	lvx	25,3,7+	vaddubm	8,8,8+	vsldoi	11,11,11,15++	.long	0x10E7F548+	.long	0x118CF548+	vand	11,11,10+	.long	0x11ADF548+	.long	0x11CEF548+	vxor	8,8,11+	.long	0x11EFF548+	.long	0x1210F548+	vxor	5,22,31+	vsrab	11,8,9+	vxor	22,8,23++	.long	0x10E70549+	.long	0x7C005699+	vaddubm	8,8,8+	vsldoi	11,11,11,15+	.long	0x118C0D49+	.long	0x7C235699+	.long	0x11AD1549+	vperm	0,0,0,6+	.long	0x7C5A5699+	vand	11,11,10+	.long	0x11CE1D49+	vperm	1,1,1,6+	.long	0x7C7B5699+	.long	0x11EF2549+	vperm	2,2,2,6+	.long	0x7C9C5699+	vxor	8,8,11+	.long	0x12102D49+	vperm	3,3,3,6+	.long	0x7CBD5699+	addi	10,10,0x60+	vperm	4,4,4,6+	vperm	5,5,5,6++	vperm	7,7,7,6+	vperm	12,12,12,6+	.long	0x7CE02799+	vxor	7,0,17+	vperm	13,13,13,6+	.long	0x7D832799+	vxor	12,1,18+	vperm	14,14,14,6+	.long	0x7DBA2799+	vxor	13,2,19+	vperm	15,15,15,6+	.long	0x7DDB2799+	vxor	14,3,20+	vperm	16,16,16,6+	.long	0x7DFC2799+	vxor	15,4,21+	.long	0x7E1D2799+	vxor	16,5,22+	addi	4,4,0x60++	mtctr	9+	beq	.Loop_xts_dec6x++	addic.	5,5,0x60+	beq	.Lxts_dec6x_zero+	cmpwi	5,0x20+	blt	.Lxts_dec6x_one+	nop	+	beq	.Lxts_dec6x_two+	cmpwi	5,0x40+	blt	.Lxts_dec6x_three+	nop	+	beq	.Lxts_dec6x_four++.Lxts_dec6x_five:+	vxor	7,1,17+	vxor	12,2,18+	vxor	13,3,19+	vxor	14,4,20+	vxor	15,5,21++	bl	_aesp8_xts_dec5x++	vperm	7,7,7,6+	vor	17,22,22+	vxor	18,8,23+	vperm	12,12,12,6+	.long	0x7CE02799+	vxor	7,0,18+	vperm	13,13,13,6+	.long	0x7D832799+	vperm	14,14,14,6+	.long	0x7DBA2799+	vperm	15,15,15,6+	.long	0x7DDB2799+	.long	0x7DFC2799+	addi	4,4,0x50+	bne	.Lxts_dec6x_steal+	b	.Lxts_dec6x_done++.align	4+.Lxts_dec6x_four:+	vxor	7,2,17+	vxor	12,3,18+	vxor	13,4,19+	vxor	14,5,20+	vxor	15,15,15++	bl	_aesp8_xts_dec5x++	vperm	7,7,7,6+	vor	17,21,21+	vor	18,22,22+	vperm	12,12,12,6+	.long	0x7CE02799+	vxor	7,0,22+	vperm	13,13,13,6+	.long	0x7D832799+	vperm	14,14,14,6+	.long	0x7DBA2799+	.long	0x7DDB2799+	addi	4,4,0x40+	bne	.Lxts_dec6x_steal+	b	.Lxts_dec6x_done++.align	4+.Lxts_dec6x_three:+	vxor	7,3,17+	vxor	12,4,18+	vxor	13,5,19+	vxor	14,14,14+	vxor	15,15,15++	bl	_aesp8_xts_dec5x++	vperm	7,7,7,6+	vor	17,20,20+	vor	18,21,21+	vperm	12,12,12,6+	.long	0x7CE02799+	vxor	7,0,21+	vperm	13,13,13,6+	.long	0x7D832799+	.long	0x7DBA2799+	addi	4,4,0x30+	bne	.Lxts_dec6x_steal+	b	.Lxts_dec6x_done++.align	4+.Lxts_dec6x_two:+	vxor	7,4,17+	vxor	12,5,18+	vxor	13,13,13+	vxor	14,14,14+	vxor	15,15,15++	bl	_aesp8_xts_dec5x++	vperm	7,7,7,6+	vor	17,19,19+	vor	18,20,20+	vperm	12,12,12,6+	.long	0x7CE02799+	vxor	7,0,20+	.long	0x7D832799+	addi	4,4,0x20+	bne	.Lxts_dec6x_steal+	b	.Lxts_dec6x_done++.align	4+.Lxts_dec6x_one:+	vxor	7,5,17+	nop	+.Loop_xts_dec1x:+	.long	0x10E7C548+	lvx	24,26,7+	addi	7,7,0x20++	.long	0x10E7CD48+	lvx	25,3,7+	bdnz	.Loop_xts_dec1x++	subi	0,31,1+	.long	0x10E7C548++	andi.	0,0,16+	cmpwi	31,0+	.long	0x10E7CD48++	sub	10,10,0+	.long	0x10E7D548++	.long	0x7C005699+	.long	0x10E7DD48++	addi	7,1,64+15+	.long	0x10E7E548+	lvx	24,0,7++	.long	0x10E7ED48+	lvx	25,3,7+	vxor	17,17,31++	vperm	0,0,0,6+	.long	0x10E7F548++	mtctr	9+	.long	0x10E78D49++	vor	17,18,18+	vor	18,19,19+	vperm	7,7,7,6+	.long	0x7CE02799+	addi	4,4,0x10+	vxor	7,0,19+	bne	.Lxts_dec6x_steal+	b	.Lxts_dec6x_done++.align	4+.Lxts_dec6x_zero:+	cmpwi	31,0+	beq	.Lxts_dec6x_done++	.long	0x7C005699+	vperm	0,0,0,6+	vxor	7,0,18+.Lxts_dec6x_steal:+	.long	0x10E7C548+	lvx	24,26,7+	addi	7,7,0x20++	.long	0x10E7CD48+	lvx	25,3,7+	bdnz	.Lxts_dec6x_steal++	add	10,10,31+	.long	0x10E7C548++	cmpwi	31,0+	.long	0x10E7CD48++	.long	0x7C005699+	.long	0x10E7D548++	lvsr	5,0,31+	.long	0x10E7DD48++	addi	7,1,64+15+	.long	0x10E7E548+	lvx	24,0,7++	.long	0x10E7ED48+	lvx	25,3,7+	vxor	18,18,31++	vperm	0,0,0,6+	.long	0x10E7F548++	vperm	0,0,0,5+	.long	0x11679549++	vperm	7,11,11,6+	.long	0x7CE02799+++	vxor	7,7,7+	vspltisb	12,-1+	vperm	7,7,12,5+	vsel	7,0,11,7+	vxor	7,7,17++	subi	30,4,1+	mtctr	31+.Loop_xts_dec6x_steal:+	lbzu	0,1(30)+	stb	0,16(30)+	bdnz	.Loop_xts_dec6x_steal++	li	31,0+	mtctr	9+	b	.Loop_xts_dec1x++.align	4+.Lxts_dec6x_done:+	cmpldi	8,0+	beq	.Lxts_dec6x_ret++	vxor	8,17,23+	vperm	8,8,8,6+	.long	0x7D004799++.Lxts_dec6x_ret:+	mtlr	11+	li	10,79+	li	11,95+	stvx	9,10,1+	addi	10,10,32+	stvx	9,11,1+	addi	11,11,32+	stvx	9,10,1+	addi	10,10,32+	stvx	9,11,1+	addi	11,11,32+	stvx	9,10,1+	addi	10,10,32+	stvx	9,11,1+	addi	11,11,32+	stvx	9,10,1+	addi	10,10,32+	stvx	9,11,1+	addi	11,11,32++	or	12,12,12+	lvx	20,10,1+	addi	10,10,32+	lvx	21,11,1+	addi	11,11,32+	lvx	22,10,1+	addi	10,10,32+	lvx	23,11,1+	addi	11,11,32+	lvx	24,10,1+	addi	10,10,32+	lvx	25,11,1+	addi	11,11,32+	lvx	26,10,1+	addi	10,10,32+	lvx	27,11,1+	addi	11,11,32+	lvx	28,10,1+	addi	10,10,32+	lvx	29,11,1+	addi	11,11,32+	lvx	30,10,1+	lvx	31,11,1+	ld	26,400(1)+	ld	27,408(1)+	ld	28,416(1)+	ld	29,424(1)+	ld	30,432(1)+	ld	31,440(1)+	addi	1,1,448+	blr	+.long	0+.byte	0,12,0x04,1,0x80,6,6,0+.long	0++.align	5+_aesp8_xts_dec5x:+	.long	0x10E7C548+	.long	0x118CC548+	.long	0x11ADC548+	.long	0x11CEC548+	.long	0x11EFC548+	lvx	24,26,7+	addi	7,7,0x20++	.long	0x10E7CD48+	.long	0x118CCD48+	.long	0x11ADCD48+	.long	0x11CECD48+	.long	0x11EFCD48+	lvx	25,3,7+	bdnz	_aesp8_xts_dec5x++	subi	0,31,1+	.long	0x10E7C548+	.long	0x118CC548+	.long	0x11ADC548+	.long	0x11CEC548+	.long	0x11EFC548++	andi.	0,0,16+	cmpwi	31,0+	.long	0x10E7CD48+	.long	0x118CCD48+	.long	0x11ADCD48+	.long	0x11CECD48+	.long	0x11EFCD48+	vxor	17,17,31++	sub	10,10,0+	.long	0x10E7D548+	.long	0x118CD548+	.long	0x11ADD548+	.long	0x11CED548+	.long	0x11EFD548+	vxor	1,18,31++	.long	0x10E7DD48+	.long	0x7C005699+	.long	0x118CDD48+	.long	0x11ADDD48+	.long	0x11CEDD48+	.long	0x11EFDD48+	vxor	2,19,31++	addi	7,1,64+15+	.long	0x10E7E548+	.long	0x118CE548+	.long	0x11ADE548+	.long	0x11CEE548+	.long	0x11EFE548+	lvx	24,0,7+	vxor	3,20,31++	.long	0x10E7ED48+	vperm	0,0,0,6+	.long	0x118CED48+	.long	0x11ADED48+	.long	0x11CEED48+	.long	0x11EFED48+	lvx	25,3,7+	vxor	4,21,31++	.long	0x10E7F548+	.long	0x118CF548+	.long	0x11ADF548+	.long	0x11CEF548+	.long	0x11EFF548++	.long	0x10E78D49+	.long	0x118C0D49+	.long	0x11AD1549+	.long	0x11CE1D49+	.long	0x11EF2549+	mtctr	9+	blr	+.long	0+.byte	0,12,0x14,0,0,0,0,0
+ cbits/asm/aesp8-ppc.pl view
@@ -0,0 +1,3801 @@+#!/usr/bin/env perl+#+# ====================================================================+# Written by Andy Polyakov <appro@openssl.org> for the OpenSSL+# project. The module is, however, dual licensed under OpenSSL and+# CRYPTOGAMS licenses depending on where you obtain it. For further+# details see http://www.openssl.org/~appro/cryptogams/.+# ====================================================================+#+# This module implements support for AES instructions as per PowerISA+# specification version 2.07, first implemented by POWER8 processor.+# The module is endian-agnostic in sense that it supports both big-+# and little-endian cases. Data alignment in parallelizable modes is+# handled with VSX loads and stores, which implies MSR.VSX flag being+# set. It should also be noted that ISA specification doesn't prohibit+# alignment exceptions for these instructions on page boundaries.+# Initially alignment was handled in pure AltiVec/VMX way [when data+# is aligned programmatically, which in turn guarantees exception-+# free execution], but it turned to hamper performance when vcipher+# instructions are interleaved. It's reckoned that eventual+# misalignment penalties at page boundaries are in average lower+# than additional overhead in pure AltiVec approach.+#+# May 2016+#+# Add XTS subroutine, 9x on little- and 12x improvement on big-endian+# systems were measured.+#+######################################################################+# Current large-block performance in cycles per byte processed with+# 128-bit key (less is better).+#+#		CBC en-/decrypt	CTR	XTS+# POWER8[le]	3.96/0.72	0.74	1.1+# POWER8[be]	3.75/0.65	0.66	1.0+# POWER9[le]	4.02/0.86	0.84	1.05+# POWER9[be]	3.99/0.78	0.79	0.97+++$flavour = shift;++if ($flavour =~ /64/) {+	$SIZE_T	=8;+	$LRSAVE	=2*$SIZE_T;+	$STU	="stdu";+	$POP	="ld";+	$PUSH	="std";+	$UCMP	="cmpld";+	$SHL	="sldi";+} elsif ($flavour =~ /32/) {+	$SIZE_T	=4;+	$LRSAVE	=$SIZE_T;+	$STU	="stwu";+	$POP	="lwz";+	$PUSH	="stw";+	$UCMP	="cmplw";+	$SHL	="slwi";+} else { die "nonsense $flavour"; }++$LITTLE_ENDIAN = ($flavour=~/le$/) ? $SIZE_T : 0;++$0 =~ m/(.*[\/\\])[^\/\\]+$/; $dir=$1;+( $xlate="${dir}ppc-xlate.pl" and -f $xlate ) or+( $xlate="${dir}../../perlasm/ppc-xlate.pl" and -f $xlate) or+die "can't locate ppc-xlate.pl";++open STDOUT,"| $^X $xlate $flavour ".shift || die "can't call $xlate: $!";++$FRAME=8*$SIZE_T;+$prefix="aes_p8";++$sp="r1";+$vrsave="r12";++#########################################################################+{{{	# Key setup procedures						#+my ($inp,$bits,$out,$ptr,$cnt,$rounds)=map("r$_",(3..8));+my ($zero,$in0,$in1,$key,$rcon,$mask,$tmp)=map("v$_",(0..6));+my ($stage,$outperm,$outmask,$outhead,$outtail)=map("v$_",(7..11));++$code.=<<___;+.machine	"any"++.text++.align	7+rcon:+.long	0x01000000, 0x01000000, 0x01000000, 0x01000000	?rev+.long	0x1b000000, 0x1b000000, 0x1b000000, 0x1b000000	?rev+.long	0x0d0e0f0c, 0x0d0e0f0c, 0x0d0e0f0c, 0x0d0e0f0c	?rev+.long	0,0,0,0						?asis+Lconsts:+	mflr	r0+	bcl	20,31,\$+4+	mflr	$ptr	 #vvvvv "distance between . and rcon+	addi	$ptr,$ptr,-0x48+	mtlr	r0+	blr+	.long	0+	.byte	0,12,0x14,0,0,0,0,0+.asciz	"AES for PowerISA 2.07, CRYPTOGAMS by <appro\@openssl.org>"++.globl	.${prefix}_set_encrypt_key+.align	5+.${prefix}_set_encrypt_key:+Lset_encrypt_key:+	mflr		r11+	$PUSH		r11,$LRSAVE($sp)++	li		$ptr,-1+	${UCMP}i	$inp,0+	beq-		Lenc_key_abort		# if ($inp==0) return -1;+	${UCMP}i	$out,0+	beq-		Lenc_key_abort		# if ($out==0) return -1;+	li		$ptr,-2+	cmpwi		$bits,128+	blt-		Lenc_key_abort+	cmpwi		$bits,256+	bgt-		Lenc_key_abort+	andi.		r0,$bits,0x3f+	bne-		Lenc_key_abort++	lis		r0,0xfff0+	mfspr		$vrsave,256+	mtspr		256,r0++	bl		Lconsts+	mtlr		r11++	neg		r9,$inp+	lvx		$in0,0,$inp+	addi		$inp,$inp,15		# 15 is not typo+	lvsr		$key,0,r9		# borrow $key+	li		r8,0x20+	cmpwi		$bits,192+	lvx		$in1,0,$inp+	le?vspltisb	$mask,0x0f		# borrow $mask+	lvx		$rcon,0,$ptr+	le?vxor		$key,$key,$mask		# adjust for byte swap+	lvx		$mask,r8,$ptr+	addi		$ptr,$ptr,0x10+	vperm		$in0,$in0,$in1,$key	# align [and byte swap in LE]+	li		$cnt,8+	vxor		$zero,$zero,$zero+	mtctr		$cnt++	?lvsr		$outperm,0,$out+	vspltisb	$outmask,-1+	lvx		$outhead,0,$out+	?vperm		$outmask,$zero,$outmask,$outperm++	blt		Loop128+	addi		$inp,$inp,8+	beq		L192+	addi		$inp,$inp,8+	b		L256++.align	4+Loop128:+	vperm		$key,$in0,$in0,$mask	# rotate-n-splat+	vsldoi		$tmp,$zero,$in0,12	# >>32+	 vperm		$outtail,$in0,$in0,$outperm	# rotate+	 vsel		$stage,$outhead,$outtail,$outmask+	 vmr		$outhead,$outtail+	vcipherlast	$key,$key,$rcon+	 stvx		$stage,0,$out+	 addi		$out,$out,16++	vxor		$in0,$in0,$tmp+	vsldoi		$tmp,$zero,$tmp,12	# >>32+	vxor		$in0,$in0,$tmp+	vsldoi		$tmp,$zero,$tmp,12	# >>32+	vxor		$in0,$in0,$tmp+	 vadduwm	$rcon,$rcon,$rcon+	vxor		$in0,$in0,$key+	bdnz		Loop128++	lvx		$rcon,0,$ptr		# last two round keys++	vperm		$key,$in0,$in0,$mask	# rotate-n-splat+	vsldoi		$tmp,$zero,$in0,12	# >>32+	 vperm		$outtail,$in0,$in0,$outperm	# rotate+	 vsel		$stage,$outhead,$outtail,$outmask+	 vmr		$outhead,$outtail+	vcipherlast	$key,$key,$rcon+	 stvx		$stage,0,$out+	 addi		$out,$out,16++	vxor		$in0,$in0,$tmp+	vsldoi		$tmp,$zero,$tmp,12	# >>32+	vxor		$in0,$in0,$tmp+	vsldoi		$tmp,$zero,$tmp,12	# >>32+	vxor		$in0,$in0,$tmp+	 vadduwm	$rcon,$rcon,$rcon+	vxor		$in0,$in0,$key++	vperm		$key,$in0,$in0,$mask	# rotate-n-splat+	vsldoi		$tmp,$zero,$in0,12	# >>32+	 vperm		$outtail,$in0,$in0,$outperm	# rotate+	 vsel		$stage,$outhead,$outtail,$outmask+	 vmr		$outhead,$outtail+	vcipherlast	$key,$key,$rcon+	 stvx		$stage,0,$out+	 addi		$out,$out,16++	vxor		$in0,$in0,$tmp+	vsldoi		$tmp,$zero,$tmp,12	# >>32+	vxor		$in0,$in0,$tmp+	vsldoi		$tmp,$zero,$tmp,12	# >>32+	vxor		$in0,$in0,$tmp+	vxor		$in0,$in0,$key+	 vperm		$outtail,$in0,$in0,$outperm	# rotate+	 vsel		$stage,$outhead,$outtail,$outmask+	 vmr		$outhead,$outtail+	 stvx		$stage,0,$out++	addi		$inp,$out,15		# 15 is not typo+	addi		$out,$out,0x50++	li		$rounds,10+	b		Ldone++.align	4+L192:+	lvx		$tmp,0,$inp+	li		$cnt,4+	 vperm		$outtail,$in0,$in0,$outperm	# rotate+	 vsel		$stage,$outhead,$outtail,$outmask+	 vmr		$outhead,$outtail+	 stvx		$stage,0,$out+	 addi		$out,$out,16+	vperm		$in1,$in1,$tmp,$key	# align [and byte swap in LE]+	vspltisb	$key,8			# borrow $key+	mtctr		$cnt+	vsububm		$mask,$mask,$key	# adjust the mask++Loop192:+	vperm		$key,$in1,$in1,$mask	# roate-n-splat+	vsldoi		$tmp,$zero,$in0,12	# >>32+	vcipherlast	$key,$key,$rcon++	vxor		$in0,$in0,$tmp+	vsldoi		$tmp,$zero,$tmp,12	# >>32+	vxor		$in0,$in0,$tmp+	vsldoi		$tmp,$zero,$tmp,12	# >>32+	vxor		$in0,$in0,$tmp++	 vsldoi		$stage,$zero,$in1,8+	vspltw		$tmp,$in0,3+	vxor		$tmp,$tmp,$in1+	vsldoi		$in1,$zero,$in1,12	# >>32+	 vadduwm	$rcon,$rcon,$rcon+	vxor		$in1,$in1,$tmp+	vxor		$in0,$in0,$key+	vxor		$in1,$in1,$key+	 vsldoi		$stage,$stage,$in0,8++	vperm		$key,$in1,$in1,$mask	# rotate-n-splat+	vsldoi		$tmp,$zero,$in0,12	# >>32+	 vperm		$outtail,$stage,$stage,$outperm	# rotate+	 vsel		$stage,$outhead,$outtail,$outmask+	 vmr		$outhead,$outtail+	vcipherlast	$key,$key,$rcon+	 stvx		$stage,0,$out+	 addi		$out,$out,16++	 vsldoi		$stage,$in0,$in1,8+	vxor		$in0,$in0,$tmp+	vsldoi		$tmp,$zero,$tmp,12	# >>32+	 vperm		$outtail,$stage,$stage,$outperm	# rotate+	 vsel		$stage,$outhead,$outtail,$outmask+	 vmr		$outhead,$outtail+	vxor		$in0,$in0,$tmp+	vsldoi		$tmp,$zero,$tmp,12	# >>32+	vxor		$in0,$in0,$tmp+	 stvx		$stage,0,$out+	 addi		$out,$out,16++	vspltw		$tmp,$in0,3+	vxor		$tmp,$tmp,$in1+	vsldoi		$in1,$zero,$in1,12	# >>32+	 vadduwm	$rcon,$rcon,$rcon+	vxor		$in1,$in1,$tmp+	vxor		$in0,$in0,$key+	vxor		$in1,$in1,$key+	 vperm		$outtail,$in0,$in0,$outperm	# rotate+	 vsel		$stage,$outhead,$outtail,$outmask+	 vmr		$outhead,$outtail+	 stvx		$stage,0,$out+	 addi		$inp,$out,15		# 15 is not typo+	 addi		$out,$out,16+	bdnz		Loop192++	li		$rounds,12+	addi		$out,$out,0x20+	b		Ldone++.align	4+L256:+	lvx		$tmp,0,$inp+	li		$cnt,7+	li		$rounds,14+	 vperm		$outtail,$in0,$in0,$outperm	# rotate+	 vsel		$stage,$outhead,$outtail,$outmask+	 vmr		$outhead,$outtail+	 stvx		$stage,0,$out+	 addi		$out,$out,16+	vperm		$in1,$in1,$tmp,$key	# align [and byte swap in LE]+	mtctr		$cnt++Loop256:+	vperm		$key,$in1,$in1,$mask	# rotate-n-splat+	vsldoi		$tmp,$zero,$in0,12	# >>32+	 vperm		$outtail,$in1,$in1,$outperm	# rotate+	 vsel		$stage,$outhead,$outtail,$outmask+	 vmr		$outhead,$outtail+	vcipherlast	$key,$key,$rcon+	 stvx		$stage,0,$out+	 addi		$out,$out,16++	vxor		$in0,$in0,$tmp+	vsldoi		$tmp,$zero,$tmp,12	# >>32+	vxor		$in0,$in0,$tmp+	vsldoi		$tmp,$zero,$tmp,12	# >>32+	vxor		$in0,$in0,$tmp+	 vadduwm	$rcon,$rcon,$rcon+	vxor		$in0,$in0,$key+	 vperm		$outtail,$in0,$in0,$outperm	# rotate+	 vsel		$stage,$outhead,$outtail,$outmask+	 vmr		$outhead,$outtail+	 stvx		$stage,0,$out+	 addi		$inp,$out,15		# 15 is not typo+	 addi		$out,$out,16+	bdz		Ldone++	vspltw		$key,$in0,3		# just splat+	vsldoi		$tmp,$zero,$in1,12	# >>32+	vsbox		$key,$key++	vxor		$in1,$in1,$tmp+	vsldoi		$tmp,$zero,$tmp,12	# >>32+	vxor		$in1,$in1,$tmp+	vsldoi		$tmp,$zero,$tmp,12	# >>32+	vxor		$in1,$in1,$tmp++	vxor		$in1,$in1,$key+	b		Loop256++.align	4+Ldone:+	lvx		$in1,0,$inp		# redundant in aligned case+	vsel		$in1,$outhead,$in1,$outmask+	stvx		$in1,0,$inp+	li		$ptr,0+	mtspr		256,$vrsave+	stw		$rounds,0($out)++Lenc_key_abort:+	mr		r3,$ptr+	blr+	.long		0+	.byte		0,12,0x14,1,0,0,3,0+	.long		0+.size	.${prefix}_set_encrypt_key,.-.${prefix}_set_encrypt_key++.globl	.${prefix}_set_decrypt_key+.align	5+.${prefix}_set_decrypt_key:+	$STU		$sp,-$FRAME($sp)+	mflr		r10+	$PUSH		r10,$FRAME+$LRSAVE($sp)+	bl		Lset_encrypt_key+	mtlr		r10++	cmpwi		r3,0+	bne-		Ldec_key_abort++	slwi		$cnt,$rounds,4+	subi		$inp,$out,240		# first round key+	srwi		$rounds,$rounds,1+	add		$out,$inp,$cnt		# last round key+	mtctr		$rounds++Ldeckey:+	lwz		r0, 0($inp)+	lwz		r6, 4($inp)+	lwz		r7, 8($inp)+	lwz		r8, 12($inp)+	addi		$inp,$inp,16+	lwz		r9, 0($out)+	lwz		r10,4($out)+	lwz		r11,8($out)+	lwz		r12,12($out)+	stw		r0, 0($out)+	stw		r6, 4($out)+	stw		r7, 8($out)+	stw		r8, 12($out)+	subi		$out,$out,16+	stw		r9, -16($inp)+	stw		r10,-12($inp)+	stw		r11,-8($inp)+	stw		r12,-4($inp)+	bdnz		Ldeckey++	xor		r3,r3,r3		# return value+Ldec_key_abort:+	addi		$sp,$sp,$FRAME+	blr+	.long		0+	.byte		0,12,4,1,0x80,0,3,0+	.long		0+.size	.${prefix}_set_decrypt_key,.-.${prefix}_set_decrypt_key+___+}}}+#########################################################################+{{{	# Single block en- and decrypt procedures			#+sub gen_block () {+my $dir = shift;+my $n   = $dir eq "de" ? "n" : "";+my ($inp,$out,$key,$rounds,$idx)=map("r$_",(3..7));++$code.=<<___;+.globl	.${prefix}_${dir}crypt+.align	5+.${prefix}_${dir}crypt:+	lwz		$rounds,240($key)+	lis		r0,0xfc00+	mfspr		$vrsave,256+	li		$idx,15			# 15 is not typo+	mtspr		256,r0++	lvx		v0,0,$inp+	neg		r11,$out+	lvx		v1,$idx,$inp+	lvsl		v2,0,$inp		# inpperm+	le?vspltisb	v4,0x0f+	?lvsl		v3,0,r11		# outperm+	le?vxor		v2,v2,v4+	li		$idx,16+	vperm		v0,v0,v1,v2		# align [and byte swap in LE]+	lvx		v1,0,$key+	?lvsl		v5,0,$key		# keyperm+	srwi		$rounds,$rounds,1+	lvx		v2,$idx,$key+	addi		$idx,$idx,16+	subi		$rounds,$rounds,1+	?vperm		v1,v1,v2,v5		# align round key++	vxor		v0,v0,v1+	lvx		v1,$idx,$key+	addi		$idx,$idx,16+	mtctr		$rounds++Loop_${dir}c:+	?vperm		v2,v2,v1,v5+	v${n}cipher	v0,v0,v2+	lvx		v2,$idx,$key+	addi		$idx,$idx,16+	?vperm		v1,v1,v2,v5+	v${n}cipher	v0,v0,v1+	lvx		v1,$idx,$key+	addi		$idx,$idx,16+	bdnz		Loop_${dir}c++	?vperm		v2,v2,v1,v5+	v${n}cipher	v0,v0,v2+	lvx		v2,$idx,$key+	?vperm		v1,v1,v2,v5+	v${n}cipherlast	v0,v0,v1++	vspltisb	v2,-1+	vxor		v1,v1,v1+	li		$idx,15			# 15 is not typo+	?vperm		v2,v1,v2,v3		# outmask+	le?vxor		v3,v3,v4+	lvx		v1,0,$out		# outhead+	vperm		v0,v0,v0,v3		# rotate [and byte swap in LE]+	vsel		v1,v1,v0,v2+	lvx		v4,$idx,$out+	stvx		v1,0,$out+	vsel		v0,v0,v4,v2+	stvx		v0,$idx,$out++	mtspr		256,$vrsave+	blr+	.long		0+	.byte		0,12,0x14,0,0,0,3,0+	.long		0+.size	.${prefix}_${dir}crypt,.-.${prefix}_${dir}crypt+___+}+&gen_block("en");+&gen_block("de");+}}}+#########################################################################+{{{	# CBC en- and decrypt procedures				#+my ($inp,$out,$len,$key,$ivp,$enc,$rounds,$idx)=map("r$_",(3..10));+my ($rndkey0,$rndkey1,$inout,$tmp)=		map("v$_",(0..3));+my ($ivec,$inptail,$inpperm,$outhead,$outperm,$outmask,$keyperm)=+						map("v$_",(4..10));+$code.=<<___;+.globl	.${prefix}_cbc_encrypt+.align	5+.${prefix}_cbc_encrypt:+	${UCMP}i	$len,16+	bltlr-++	cmpwi		$enc,0			# test direction+	lis		r0,0xffe0+	mfspr		$vrsave,256+	mtspr		256,r0++	li		$idx,15+	vxor		$rndkey0,$rndkey0,$rndkey0+	le?vspltisb	$tmp,0x0f++	lvx		$ivec,0,$ivp		# load [unaligned] iv+	lvsl		$inpperm,0,$ivp+	lvx		$inptail,$idx,$ivp+	le?vxor		$inpperm,$inpperm,$tmp+	vperm		$ivec,$ivec,$inptail,$inpperm++	neg		r11,$inp+	?lvsl		$keyperm,0,$key		# prepare for unaligned key+	lwz		$rounds,240($key)++	lvsr		$inpperm,0,r11		# prepare for unaligned load+	lvx		$inptail,0,$inp+	addi		$inp,$inp,15		# 15 is not typo+	le?vxor		$inpperm,$inpperm,$tmp++	?lvsr		$outperm,0,$out		# prepare for unaligned store+	vspltisb	$outmask,-1+	lvx		$outhead,0,$out+	?vperm		$outmask,$rndkey0,$outmask,$outperm+	le?vxor		$outperm,$outperm,$tmp++	srwi		$rounds,$rounds,1+	li		$idx,16+	subi		$rounds,$rounds,1+	beq		Lcbc_dec++Lcbc_enc:+	vmr		$inout,$inptail+	lvx		$inptail,0,$inp+	addi		$inp,$inp,16+	mtctr		$rounds+	subi		$len,$len,16		# len-=16++	lvx		$rndkey0,0,$key+	 vperm		$inout,$inout,$inptail,$inpperm+	lvx		$rndkey1,$idx,$key+	addi		$idx,$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vxor		$inout,$inout,$rndkey0+	lvx		$rndkey0,$idx,$key+	addi		$idx,$idx,16+	vxor		$inout,$inout,$ivec++Loop_cbc_enc:+	?vperm		$rndkey1,$rndkey1,$rndkey0,$keyperm+	vcipher		$inout,$inout,$rndkey1+	lvx		$rndkey1,$idx,$key+	addi		$idx,$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vcipher		$inout,$inout,$rndkey0+	lvx		$rndkey0,$idx,$key+	addi		$idx,$idx,16+	bdnz		Loop_cbc_enc++	?vperm		$rndkey1,$rndkey1,$rndkey0,$keyperm+	vcipher		$inout,$inout,$rndkey1+	lvx		$rndkey1,$idx,$key+	li		$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vcipherlast	$ivec,$inout,$rndkey0+	${UCMP}i	$len,16++	vperm		$tmp,$ivec,$ivec,$outperm+	vsel		$inout,$outhead,$tmp,$outmask+	vmr		$outhead,$tmp+	stvx		$inout,0,$out+	addi		$out,$out,16+	bge		Lcbc_enc++	b		Lcbc_done++.align	4+Lcbc_dec:+	${UCMP}i	$len,128+	bge		_aesp8_cbc_decrypt8x+	vmr		$tmp,$inptail+	lvx		$inptail,0,$inp+	addi		$inp,$inp,16+	mtctr		$rounds+	subi		$len,$len,16		# len-=16++	lvx		$rndkey0,0,$key+	 vperm		$tmp,$tmp,$inptail,$inpperm+	lvx		$rndkey1,$idx,$key+	addi		$idx,$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vxor		$inout,$tmp,$rndkey0+	lvx		$rndkey0,$idx,$key+	addi		$idx,$idx,16++Loop_cbc_dec:+	?vperm		$rndkey1,$rndkey1,$rndkey0,$keyperm+	vncipher	$inout,$inout,$rndkey1+	lvx		$rndkey1,$idx,$key+	addi		$idx,$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vncipher	$inout,$inout,$rndkey0+	lvx		$rndkey0,$idx,$key+	addi		$idx,$idx,16+	bdnz		Loop_cbc_dec++	?vperm		$rndkey1,$rndkey1,$rndkey0,$keyperm+	vncipher	$inout,$inout,$rndkey1+	lvx		$rndkey1,$idx,$key+	li		$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vncipherlast	$inout,$inout,$rndkey0+	${UCMP}i	$len,16++	vxor		$inout,$inout,$ivec+	vmr		$ivec,$tmp+	vperm		$tmp,$inout,$inout,$outperm+	vsel		$inout,$outhead,$tmp,$outmask+	vmr		$outhead,$tmp+	stvx		$inout,0,$out+	addi		$out,$out,16+	bge		Lcbc_dec++Lcbc_done:+	addi		$out,$out,-1+	lvx		$inout,0,$out		# redundant in aligned case+	vsel		$inout,$outhead,$inout,$outmask+	stvx		$inout,0,$out++	neg		$enc,$ivp		# write [unaligned] iv+	li		$idx,15			# 15 is not typo+	vxor		$rndkey0,$rndkey0,$rndkey0+	vspltisb	$outmask,-1+	le?vspltisb	$tmp,0x0f+	?lvsl		$outperm,0,$enc+	?vperm		$outmask,$rndkey0,$outmask,$outperm+	le?vxor		$outperm,$outperm,$tmp+	lvx		$outhead,0,$ivp+	vperm		$ivec,$ivec,$ivec,$outperm+	vsel		$inout,$outhead,$ivec,$outmask+	lvx		$inptail,$idx,$ivp+	stvx		$inout,0,$ivp+	vsel		$inout,$ivec,$inptail,$outmask+	stvx		$inout,$idx,$ivp++	mtspr		256,$vrsave+	blr+	.long		0+	.byte		0,12,0x14,0,0,0,6,0+	.long		0+___+#########################################################################+{{	# Optimized CBC decrypt procedure				#+my $key_="r11";+my ($x00,$x10,$x20,$x30,$x40,$x50,$x60,$x70)=map("r$_",(0,8,26..31));+    $x00=0 if ($flavour =~ /osx/);+my ($in0, $in1, $in2, $in3, $in4, $in5, $in6, $in7 )=map("v$_",(0..3,10..13));+my ($out0,$out1,$out2,$out3,$out4,$out5,$out6,$out7)=map("v$_",(14..21));+my $rndkey0="v23";	# v24-v25 rotating buffer for first found keys+			# v26-v31 last 6 round keys+my ($tmp,$keyperm)=($in3,$in4);	# aliases with "caller", redundant assignment++$code.=<<___;+.align	5+_aesp8_cbc_decrypt8x:+	$STU		$sp,-`($FRAME+21*16+6*$SIZE_T)`($sp)+	li		r10,`$FRAME+8*16+15`+	li		r11,`$FRAME+8*16+31`+	stvx		v20,r10,$sp		# ABI says so+	addi		r10,r10,32+	stvx		v21,r11,$sp+	addi		r11,r11,32+	stvx		v22,r10,$sp+	addi		r10,r10,32+	stvx		v23,r11,$sp+	addi		r11,r11,32+	stvx		v24,r10,$sp+	addi		r10,r10,32+	stvx		v25,r11,$sp+	addi		r11,r11,32+	stvx		v26,r10,$sp+	addi		r10,r10,32+	stvx		v27,r11,$sp+	addi		r11,r11,32+	stvx		v28,r10,$sp+	addi		r10,r10,32+	stvx		v29,r11,$sp+	addi		r11,r11,32+	stvx		v30,r10,$sp+	stvx		v31,r11,$sp+	li		r0,-1+	stw		$vrsave,`$FRAME+21*16-4`($sp)	# save vrsave+	li		$x10,0x10+	$PUSH		r26,`$FRAME+21*16+0*$SIZE_T`($sp)+	li		$x20,0x20+	$PUSH		r27,`$FRAME+21*16+1*$SIZE_T`($sp)+	li		$x30,0x30+	$PUSH		r28,`$FRAME+21*16+2*$SIZE_T`($sp)+	li		$x40,0x40+	$PUSH		r29,`$FRAME+21*16+3*$SIZE_T`($sp)+	li		$x50,0x50+	$PUSH		r30,`$FRAME+21*16+4*$SIZE_T`($sp)+	li		$x60,0x60+	$PUSH		r31,`$FRAME+21*16+5*$SIZE_T`($sp)+	li		$x70,0x70+	mtspr		256,r0++	subi		$rounds,$rounds,3	# -4 in total+	subi		$len,$len,128		# bias++	lvx		$rndkey0,$x00,$key	# load key schedule+	lvx		v30,$x10,$key+	addi		$key,$key,0x20+	lvx		v31,$x00,$key+	?vperm		$rndkey0,$rndkey0,v30,$keyperm+	addi		$key_,$sp,$FRAME+15+	mtctr		$rounds++Load_cbc_dec_key:+	?vperm		v24,v30,v31,$keyperm+	lvx		v30,$x10,$key+	addi		$key,$key,0x20+	stvx		v24,$x00,$key_		# off-load round[1]+	?vperm		v25,v31,v30,$keyperm+	lvx		v31,$x00,$key+	stvx		v25,$x10,$key_		# off-load round[2]+	addi		$key_,$key_,0x20+	bdnz		Load_cbc_dec_key++	lvx		v26,$x10,$key+	?vperm		v24,v30,v31,$keyperm+	lvx		v27,$x20,$key+	stvx		v24,$x00,$key_		# off-load round[3]+	?vperm		v25,v31,v26,$keyperm+	lvx		v28,$x30,$key+	stvx		v25,$x10,$key_		# off-load round[4]+	addi		$key_,$sp,$FRAME+15	# rewind $key_+	?vperm		v26,v26,v27,$keyperm+	lvx		v29,$x40,$key+	?vperm		v27,v27,v28,$keyperm+	lvx		v30,$x50,$key+	?vperm		v28,v28,v29,$keyperm+	lvx		v31,$x60,$key+	?vperm		v29,v29,v30,$keyperm+	lvx		$out0,$x70,$key		# borrow $out0+	?vperm		v30,v30,v31,$keyperm+	lvx		v24,$x00,$key_		# pre-load round[1]+	?vperm		v31,v31,$out0,$keyperm+	lvx		v25,$x10,$key_		# pre-load round[2]++	#lvx		$inptail,0,$inp		# "caller" already did this+	#addi		$inp,$inp,15		# 15 is not typo+	subi		$inp,$inp,15		# undo "caller"++	 le?li		$idx,8+	lvx_u		$in0,$x00,$inp		# load first 8 "words"+	 le?lvsl	$inpperm,0,$idx+	 le?vspltisb	$tmp,0x0f+	lvx_u		$in1,$x10,$inp+	 le?vxor	$inpperm,$inpperm,$tmp	# transform for lvx_u/stvx_u+	lvx_u		$in2,$x20,$inp+	 le?vperm	$in0,$in0,$in0,$inpperm+	lvx_u		$in3,$x30,$inp+	 le?vperm	$in1,$in1,$in1,$inpperm+	lvx_u		$in4,$x40,$inp+	 le?vperm	$in2,$in2,$in2,$inpperm+	vxor		$out0,$in0,$rndkey0+	lvx_u		$in5,$x50,$inp+	 le?vperm	$in3,$in3,$in3,$inpperm+	vxor		$out1,$in1,$rndkey0+	lvx_u		$in6,$x60,$inp+	 le?vperm	$in4,$in4,$in4,$inpperm+	vxor		$out2,$in2,$rndkey0+	lvx_u		$in7,$x70,$inp+	addi		$inp,$inp,0x80+	 le?vperm	$in5,$in5,$in5,$inpperm+	vxor		$out3,$in3,$rndkey0+	 le?vperm	$in6,$in6,$in6,$inpperm+	vxor		$out4,$in4,$rndkey0+	 le?vperm	$in7,$in7,$in7,$inpperm+	vxor		$out5,$in5,$rndkey0+	vxor		$out6,$in6,$rndkey0+	vxor		$out7,$in7,$rndkey0++	mtctr		$rounds+	b		Loop_cbc_dec8x+.align	5+Loop_cbc_dec8x:+	vncipher	$out0,$out0,v24+	vncipher	$out1,$out1,v24+	vncipher	$out2,$out2,v24+	vncipher	$out3,$out3,v24+	vncipher	$out4,$out4,v24+	vncipher	$out5,$out5,v24+	vncipher	$out6,$out6,v24+	vncipher	$out7,$out7,v24+	lvx		v24,$x20,$key_		# round[3]+	addi		$key_,$key_,0x20++	vncipher	$out0,$out0,v25+	vncipher	$out1,$out1,v25+	vncipher	$out2,$out2,v25+	vncipher	$out3,$out3,v25+	vncipher	$out4,$out4,v25+	vncipher	$out5,$out5,v25+	vncipher	$out6,$out6,v25+	vncipher	$out7,$out7,v25+	lvx		v25,$x10,$key_		# round[4]+	bdnz		Loop_cbc_dec8x++	subic		$len,$len,128		# $len-=128+	vncipher	$out0,$out0,v24+	vncipher	$out1,$out1,v24+	vncipher	$out2,$out2,v24+	vncipher	$out3,$out3,v24+	vncipher	$out4,$out4,v24+	vncipher	$out5,$out5,v24+	vncipher	$out6,$out6,v24+	vncipher	$out7,$out7,v24++	subfe.		r0,r0,r0		# borrow?-1:0+	vncipher	$out0,$out0,v25+	vncipher	$out1,$out1,v25+	vncipher	$out2,$out2,v25+	vncipher	$out3,$out3,v25+	vncipher	$out4,$out4,v25+	vncipher	$out5,$out5,v25+	vncipher	$out6,$out6,v25+	vncipher	$out7,$out7,v25++	and		r0,r0,$len+	vncipher	$out0,$out0,v26+	vncipher	$out1,$out1,v26+	vncipher	$out2,$out2,v26+	vncipher	$out3,$out3,v26+	vncipher	$out4,$out4,v26+	vncipher	$out5,$out5,v26+	vncipher	$out6,$out6,v26+	vncipher	$out7,$out7,v26++	add		$inp,$inp,r0		# $inp is adjusted in such+						# way that at exit from the+						# loop inX-in7 are loaded+						# with last "words"+	vncipher	$out0,$out0,v27+	vncipher	$out1,$out1,v27+	vncipher	$out2,$out2,v27+	vncipher	$out3,$out3,v27+	vncipher	$out4,$out4,v27+	vncipher	$out5,$out5,v27+	vncipher	$out6,$out6,v27+	vncipher	$out7,$out7,v27++	addi		$key_,$sp,$FRAME+15	# rewind $key_+	vncipher	$out0,$out0,v28+	vncipher	$out1,$out1,v28+	vncipher	$out2,$out2,v28+	vncipher	$out3,$out3,v28+	vncipher	$out4,$out4,v28+	vncipher	$out5,$out5,v28+	vncipher	$out6,$out6,v28+	vncipher	$out7,$out7,v28+	lvx		v24,$x00,$key_		# re-pre-load round[1]++	vncipher	$out0,$out0,v29+	vncipher	$out1,$out1,v29+	vncipher	$out2,$out2,v29+	vncipher	$out3,$out3,v29+	vncipher	$out4,$out4,v29+	vncipher	$out5,$out5,v29+	vncipher	$out6,$out6,v29+	vncipher	$out7,$out7,v29+	lvx		v25,$x10,$key_		# re-pre-load round[2]++	vncipher	$out0,$out0,v30+	 vxor		$ivec,$ivec,v31		# xor with last round key+	vncipher	$out1,$out1,v30+	 vxor		$in0,$in0,v31+	vncipher	$out2,$out2,v30+	 vxor		$in1,$in1,v31+	vncipher	$out3,$out3,v30+	 vxor		$in2,$in2,v31+	vncipher	$out4,$out4,v30+	 vxor		$in3,$in3,v31+	vncipher	$out5,$out5,v30+	 vxor		$in4,$in4,v31+	vncipher	$out6,$out6,v30+	 vxor		$in5,$in5,v31+	vncipher	$out7,$out7,v30+	 vxor		$in6,$in6,v31++	vncipherlast	$out0,$out0,$ivec+	vncipherlast	$out1,$out1,$in0+	 lvx_u		$in0,$x00,$inp		# load next input block+	vncipherlast	$out2,$out2,$in1+	 lvx_u		$in1,$x10,$inp+	vncipherlast	$out3,$out3,$in2+	 le?vperm	$in0,$in0,$in0,$inpperm+	 lvx_u		$in2,$x20,$inp+	vncipherlast	$out4,$out4,$in3+	 le?vperm	$in1,$in1,$in1,$inpperm+	 lvx_u		$in3,$x30,$inp+	vncipherlast	$out5,$out5,$in4+	 le?vperm	$in2,$in2,$in2,$inpperm+	 lvx_u		$in4,$x40,$inp+	vncipherlast	$out6,$out6,$in5+	 le?vperm	$in3,$in3,$in3,$inpperm+	 lvx_u		$in5,$x50,$inp+	vncipherlast	$out7,$out7,$in6+	 le?vperm	$in4,$in4,$in4,$inpperm+	 lvx_u		$in6,$x60,$inp+	vmr		$ivec,$in7+	 le?vperm	$in5,$in5,$in5,$inpperm+	 lvx_u		$in7,$x70,$inp+	 addi		$inp,$inp,0x80++	le?vperm	$out0,$out0,$out0,$inpperm+	le?vperm	$out1,$out1,$out1,$inpperm+	stvx_u		$out0,$x00,$out+	 le?vperm	$in6,$in6,$in6,$inpperm+	 vxor		$out0,$in0,$rndkey0+	le?vperm	$out2,$out2,$out2,$inpperm+	stvx_u		$out1,$x10,$out+	 le?vperm	$in7,$in7,$in7,$inpperm+	 vxor		$out1,$in1,$rndkey0+	le?vperm	$out3,$out3,$out3,$inpperm+	stvx_u		$out2,$x20,$out+	 vxor		$out2,$in2,$rndkey0+	le?vperm	$out4,$out4,$out4,$inpperm+	stvx_u		$out3,$x30,$out+	 vxor		$out3,$in3,$rndkey0+	le?vperm	$out5,$out5,$out5,$inpperm+	stvx_u		$out4,$x40,$out+	 vxor		$out4,$in4,$rndkey0+	le?vperm	$out6,$out6,$out6,$inpperm+	stvx_u		$out5,$x50,$out+	 vxor		$out5,$in5,$rndkey0+	le?vperm	$out7,$out7,$out7,$inpperm+	stvx_u		$out6,$x60,$out+	 vxor		$out6,$in6,$rndkey0+	stvx_u		$out7,$x70,$out+	addi		$out,$out,0x80+	 vxor		$out7,$in7,$rndkey0++	mtctr		$rounds+	beq		Loop_cbc_dec8x		# did $len-=128 borrow?++	addic.		$len,$len,128+	beq		Lcbc_dec8x_done+	nop+	nop++Loop_cbc_dec8x_tail:				# up to 7 "words" tail...+	vncipher	$out1,$out1,v24+	vncipher	$out2,$out2,v24+	vncipher	$out3,$out3,v24+	vncipher	$out4,$out4,v24+	vncipher	$out5,$out5,v24+	vncipher	$out6,$out6,v24+	vncipher	$out7,$out7,v24+	lvx		v24,$x20,$key_		# round[3]+	addi		$key_,$key_,0x20++	vncipher	$out1,$out1,v25+	vncipher	$out2,$out2,v25+	vncipher	$out3,$out3,v25+	vncipher	$out4,$out4,v25+	vncipher	$out5,$out5,v25+	vncipher	$out6,$out6,v25+	vncipher	$out7,$out7,v25+	lvx		v25,$x10,$key_		# round[4]+	bdnz		Loop_cbc_dec8x_tail++	vncipher	$out1,$out1,v24+	vncipher	$out2,$out2,v24+	vncipher	$out3,$out3,v24+	vncipher	$out4,$out4,v24+	vncipher	$out5,$out5,v24+	vncipher	$out6,$out6,v24+	vncipher	$out7,$out7,v24++	vncipher	$out1,$out1,v25+	vncipher	$out2,$out2,v25+	vncipher	$out3,$out3,v25+	vncipher	$out4,$out4,v25+	vncipher	$out5,$out5,v25+	vncipher	$out6,$out6,v25+	vncipher	$out7,$out7,v25++	vncipher	$out1,$out1,v26+	vncipher	$out2,$out2,v26+	vncipher	$out3,$out3,v26+	vncipher	$out4,$out4,v26+	vncipher	$out5,$out5,v26+	vncipher	$out6,$out6,v26+	vncipher	$out7,$out7,v26++	vncipher	$out1,$out1,v27+	vncipher	$out2,$out2,v27+	vncipher	$out3,$out3,v27+	vncipher	$out4,$out4,v27+	vncipher	$out5,$out5,v27+	vncipher	$out6,$out6,v27+	vncipher	$out7,$out7,v27++	vncipher	$out1,$out1,v28+	vncipher	$out2,$out2,v28+	vncipher	$out3,$out3,v28+	vncipher	$out4,$out4,v28+	vncipher	$out5,$out5,v28+	vncipher	$out6,$out6,v28+	vncipher	$out7,$out7,v28++	vncipher	$out1,$out1,v29+	vncipher	$out2,$out2,v29+	vncipher	$out3,$out3,v29+	vncipher	$out4,$out4,v29+	vncipher	$out5,$out5,v29+	vncipher	$out6,$out6,v29+	vncipher	$out7,$out7,v29++	vncipher	$out1,$out1,v30+	 vxor		$ivec,$ivec,v31		# last round key+	vncipher	$out2,$out2,v30+	 vxor		$in1,$in1,v31+	vncipher	$out3,$out3,v30+	 vxor		$in2,$in2,v31+	vncipher	$out4,$out4,v30+	 vxor		$in3,$in3,v31+	vncipher	$out5,$out5,v30+	 vxor		$in4,$in4,v31+	vncipher	$out6,$out6,v30+	 vxor		$in5,$in5,v31+	vncipher	$out7,$out7,v30+	 vxor		$in6,$in6,v31++	cmplwi		$len,32			# switch($len)+	blt		Lcbc_dec8x_one+	nop+	beq		Lcbc_dec8x_two+	cmplwi		$len,64+	blt		Lcbc_dec8x_three+	nop+	beq		Lcbc_dec8x_four+	cmplwi		$len,96+	blt		Lcbc_dec8x_five+	nop+	beq		Lcbc_dec8x_six++Lcbc_dec8x_seven:+	vncipherlast	$out1,$out1,$ivec+	vncipherlast	$out2,$out2,$in1+	vncipherlast	$out3,$out3,$in2+	vncipherlast	$out4,$out4,$in3+	vncipherlast	$out5,$out5,$in4+	vncipherlast	$out6,$out6,$in5+	vncipherlast	$out7,$out7,$in6+	vmr		$ivec,$in7++	le?vperm	$out1,$out1,$out1,$inpperm+	le?vperm	$out2,$out2,$out2,$inpperm+	stvx_u		$out1,$x00,$out+	le?vperm	$out3,$out3,$out3,$inpperm+	stvx_u		$out2,$x10,$out+	le?vperm	$out4,$out4,$out4,$inpperm+	stvx_u		$out3,$x20,$out+	le?vperm	$out5,$out5,$out5,$inpperm+	stvx_u		$out4,$x30,$out+	le?vperm	$out6,$out6,$out6,$inpperm+	stvx_u		$out5,$x40,$out+	le?vperm	$out7,$out7,$out7,$inpperm+	stvx_u		$out6,$x50,$out+	stvx_u		$out7,$x60,$out+	addi		$out,$out,0x70+	b		Lcbc_dec8x_done++.align	5+Lcbc_dec8x_six:+	vncipherlast	$out2,$out2,$ivec+	vncipherlast	$out3,$out3,$in2+	vncipherlast	$out4,$out4,$in3+	vncipherlast	$out5,$out5,$in4+	vncipherlast	$out6,$out6,$in5+	vncipherlast	$out7,$out7,$in6+	vmr		$ivec,$in7++	le?vperm	$out2,$out2,$out2,$inpperm+	le?vperm	$out3,$out3,$out3,$inpperm+	stvx_u		$out2,$x00,$out+	le?vperm	$out4,$out4,$out4,$inpperm+	stvx_u		$out3,$x10,$out+	le?vperm	$out5,$out5,$out5,$inpperm+	stvx_u		$out4,$x20,$out+	le?vperm	$out6,$out6,$out6,$inpperm+	stvx_u		$out5,$x30,$out+	le?vperm	$out7,$out7,$out7,$inpperm+	stvx_u		$out6,$x40,$out+	stvx_u		$out7,$x50,$out+	addi		$out,$out,0x60+	b		Lcbc_dec8x_done++.align	5+Lcbc_dec8x_five:+	vncipherlast	$out3,$out3,$ivec+	vncipherlast	$out4,$out4,$in3+	vncipherlast	$out5,$out5,$in4+	vncipherlast	$out6,$out6,$in5+	vncipherlast	$out7,$out7,$in6+	vmr		$ivec,$in7++	le?vperm	$out3,$out3,$out3,$inpperm+	le?vperm	$out4,$out4,$out4,$inpperm+	stvx_u		$out3,$x00,$out+	le?vperm	$out5,$out5,$out5,$inpperm+	stvx_u		$out4,$x10,$out+	le?vperm	$out6,$out6,$out6,$inpperm+	stvx_u		$out5,$x20,$out+	le?vperm	$out7,$out7,$out7,$inpperm+	stvx_u		$out6,$x30,$out+	stvx_u		$out7,$x40,$out+	addi		$out,$out,0x50+	b		Lcbc_dec8x_done++.align	5+Lcbc_dec8x_four:+	vncipherlast	$out4,$out4,$ivec+	vncipherlast	$out5,$out5,$in4+	vncipherlast	$out6,$out6,$in5+	vncipherlast	$out7,$out7,$in6+	vmr		$ivec,$in7++	le?vperm	$out4,$out4,$out4,$inpperm+	le?vperm	$out5,$out5,$out5,$inpperm+	stvx_u		$out4,$x00,$out+	le?vperm	$out6,$out6,$out6,$inpperm+	stvx_u		$out5,$x10,$out+	le?vperm	$out7,$out7,$out7,$inpperm+	stvx_u		$out6,$x20,$out+	stvx_u		$out7,$x30,$out+	addi		$out,$out,0x40+	b		Lcbc_dec8x_done++.align	5+Lcbc_dec8x_three:+	vncipherlast	$out5,$out5,$ivec+	vncipherlast	$out6,$out6,$in5+	vncipherlast	$out7,$out7,$in6+	vmr		$ivec,$in7++	le?vperm	$out5,$out5,$out5,$inpperm+	le?vperm	$out6,$out6,$out6,$inpperm+	stvx_u		$out5,$x00,$out+	le?vperm	$out7,$out7,$out7,$inpperm+	stvx_u		$out6,$x10,$out+	stvx_u		$out7,$x20,$out+	addi		$out,$out,0x30+	b		Lcbc_dec8x_done++.align	5+Lcbc_dec8x_two:+	vncipherlast	$out6,$out6,$ivec+	vncipherlast	$out7,$out7,$in6+	vmr		$ivec,$in7++	le?vperm	$out6,$out6,$out6,$inpperm+	le?vperm	$out7,$out7,$out7,$inpperm+	stvx_u		$out6,$x00,$out+	stvx_u		$out7,$x10,$out+	addi		$out,$out,0x20+	b		Lcbc_dec8x_done++.align	5+Lcbc_dec8x_one:+	vncipherlast	$out7,$out7,$ivec+	vmr		$ivec,$in7++	le?vperm	$out7,$out7,$out7,$inpperm+	stvx_u		$out7,0,$out+	addi		$out,$out,0x10++Lcbc_dec8x_done:+	le?vperm	$ivec,$ivec,$ivec,$inpperm+	stvx_u		$ivec,0,$ivp		# write [unaligned] iv++	li		r10,`$FRAME+15`+	li		r11,`$FRAME+31`+	stvx		$inpperm,r10,$sp	# wipe copies of round keys+	addi		r10,r10,32+	stvx		$inpperm,r11,$sp+	addi		r11,r11,32+	stvx		$inpperm,r10,$sp+	addi		r10,r10,32+	stvx		$inpperm,r11,$sp+	addi		r11,r11,32+	stvx		$inpperm,r10,$sp+	addi		r10,r10,32+	stvx		$inpperm,r11,$sp+	addi		r11,r11,32+	stvx		$inpperm,r10,$sp+	addi		r10,r10,32+	stvx		$inpperm,r11,$sp+	addi		r11,r11,32++	mtspr		256,$vrsave+	lvx		v20,r10,$sp		# ABI says so+	addi		r10,r10,32+	lvx		v21,r11,$sp+	addi		r11,r11,32+	lvx		v22,r10,$sp+	addi		r10,r10,32+	lvx		v23,r11,$sp+	addi		r11,r11,32+	lvx		v24,r10,$sp+	addi		r10,r10,32+	lvx		v25,r11,$sp+	addi		r11,r11,32+	lvx		v26,r10,$sp+	addi		r10,r10,32+	lvx		v27,r11,$sp+	addi		r11,r11,32+	lvx		v28,r10,$sp+	addi		r10,r10,32+	lvx		v29,r11,$sp+	addi		r11,r11,32+	lvx		v30,r10,$sp+	lvx		v31,r11,$sp+	$POP		r26,`$FRAME+21*16+0*$SIZE_T`($sp)+	$POP		r27,`$FRAME+21*16+1*$SIZE_T`($sp)+	$POP		r28,`$FRAME+21*16+2*$SIZE_T`($sp)+	$POP		r29,`$FRAME+21*16+3*$SIZE_T`($sp)+	$POP		r30,`$FRAME+21*16+4*$SIZE_T`($sp)+	$POP		r31,`$FRAME+21*16+5*$SIZE_T`($sp)+	addi		$sp,$sp,`$FRAME+21*16+6*$SIZE_T`+	blr+	.long		0+	.byte		0,12,0x04,0,0x80,6,6,0+	.long		0+.size	.${prefix}_cbc_encrypt,.-.${prefix}_cbc_encrypt+___+}}	}}}++#########################################################################+{{{	# CTR procedure[s]						#+my ($inp,$out,$len,$key,$ivp,$x10,$rounds,$idx)=map("r$_",(3..10));+my ($rndkey0,$rndkey1,$inout,$tmp)=		map("v$_",(0..3));+my ($ivec,$inptail,$inpperm,$outhead,$outperm,$outmask,$keyperm,$one)=+						map("v$_",(4..11));+my $dat=$tmp;++$code.=<<___;+.globl	.${prefix}_ctr32_encrypt_blocks+.align	5+.${prefix}_ctr32_encrypt_blocks:+	${UCMP}i	$len,1+	bltlr-++	lis		r0,0xfff0+	mfspr		$vrsave,256+	mtspr		256,r0++	li		$idx,15+	vxor		$rndkey0,$rndkey0,$rndkey0+	le?vspltisb	$tmp,0x0f++	lvx		$ivec,0,$ivp		# load [unaligned] iv+	lvsl		$inpperm,0,$ivp+	lvx		$inptail,$idx,$ivp+	 vspltisb	$one,1+	le?vxor		$inpperm,$inpperm,$tmp+	vperm		$ivec,$ivec,$inptail,$inpperm+	 vsldoi		$one,$rndkey0,$one,1++	neg		r11,$inp+	?lvsl		$keyperm,0,$key		# prepare for unaligned key+	lwz		$rounds,240($key)++	lvsr		$inpperm,0,r11		# prepare for unaligned load+	lvx		$inptail,0,$inp+	addi		$inp,$inp,15		# 15 is not typo+	le?vxor		$inpperm,$inpperm,$tmp++	srwi		$rounds,$rounds,1+	li		$idx,16+	subi		$rounds,$rounds,1++	${UCMP}i	$len,8+	bge		_aesp8_ctr32_encrypt8x++	?lvsr		$outperm,0,$out		# prepare for unaligned store+	vspltisb	$outmask,-1+	lvx		$outhead,0,$out+	?vperm		$outmask,$rndkey0,$outmask,$outperm+	le?vxor		$outperm,$outperm,$tmp++	lvx		$rndkey0,0,$key+	mtctr		$rounds+	lvx		$rndkey1,$idx,$key+	addi		$idx,$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vxor		$inout,$ivec,$rndkey0+	lvx		$rndkey0,$idx,$key+	addi		$idx,$idx,16+	b		Loop_ctr32_enc++.align	5+Loop_ctr32_enc:+	?vperm		$rndkey1,$rndkey1,$rndkey0,$keyperm+	vcipher		$inout,$inout,$rndkey1+	lvx		$rndkey1,$idx,$key+	addi		$idx,$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vcipher		$inout,$inout,$rndkey0+	lvx		$rndkey0,$idx,$key+	addi		$idx,$idx,16+	bdnz		Loop_ctr32_enc++	vadduwm		$ivec,$ivec,$one+	 vmr		$dat,$inptail+	 lvx		$inptail,0,$inp+	 addi		$inp,$inp,16+	 subic.		$len,$len,1		# blocks--++	?vperm		$rndkey1,$rndkey1,$rndkey0,$keyperm+	vcipher		$inout,$inout,$rndkey1+	lvx		$rndkey1,$idx,$key+	 vperm		$dat,$dat,$inptail,$inpperm+	 li		$idx,16+	?vperm		$rndkey1,$rndkey0,$rndkey1,$keyperm+	 lvx		$rndkey0,0,$key+	vxor		$dat,$dat,$rndkey1	# last round key+	vcipherlast	$inout,$inout,$dat++	 lvx		$rndkey1,$idx,$key+	 addi		$idx,$idx,16+	vperm		$inout,$inout,$inout,$outperm+	vsel		$dat,$outhead,$inout,$outmask+	 mtctr		$rounds+	 ?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vmr		$outhead,$inout+	 vxor		$inout,$ivec,$rndkey0+	 lvx		$rndkey0,$idx,$key+	 addi		$idx,$idx,16+	stvx		$dat,0,$out+	addi		$out,$out,16+	bne		Loop_ctr32_enc++	addi		$out,$out,-1+	lvx		$inout,0,$out		# redundant in aligned case+	vsel		$inout,$outhead,$inout,$outmask+	stvx		$inout,0,$out++	mtspr		256,$vrsave+	blr+	.long		0+	.byte		0,12,0x14,0,0,0,6,0+	.long		0+___+#########################################################################+{{	# Optimized CTR procedure					#+my $key_="r11";+my ($x00,$x10,$x20,$x30,$x40,$x50,$x60,$x70)=map("r$_",(0,8,26..31));+    $x00=0 if ($flavour =~ /osx/);+my ($in0, $in1, $in2, $in3, $in4, $in5, $in6, $in7 )=map("v$_",(0..3,10,12..14));+my ($out0,$out1,$out2,$out3,$out4,$out5,$out6,$out7)=map("v$_",(15..22));+my $rndkey0="v23";	# v24-v25 rotating buffer for first found keys+			# v26-v31 last 6 round keys+my ($tmp,$keyperm)=($in3,$in4);	# aliases with "caller", redundant assignment+my ($two,$three,$four)=($outhead,$outperm,$outmask);++$code.=<<___;+.align	5+_aesp8_ctr32_encrypt8x:+	$STU		$sp,-`($FRAME+21*16+6*$SIZE_T)`($sp)+	li		r10,`$FRAME+8*16+15`+	li		r11,`$FRAME+8*16+31`+	stvx		v20,r10,$sp		# ABI says so+	addi		r10,r10,32+	stvx		v21,r11,$sp+	addi		r11,r11,32+	stvx		v22,r10,$sp+	addi		r10,r10,32+	stvx		v23,r11,$sp+	addi		r11,r11,32+	stvx		v24,r10,$sp+	addi		r10,r10,32+	stvx		v25,r11,$sp+	addi		r11,r11,32+	stvx		v26,r10,$sp+	addi		r10,r10,32+	stvx		v27,r11,$sp+	addi		r11,r11,32+	stvx		v28,r10,$sp+	addi		r10,r10,32+	stvx		v29,r11,$sp+	addi		r11,r11,32+	stvx		v30,r10,$sp+	stvx		v31,r11,$sp+	li		r0,-1+	stw		$vrsave,`$FRAME+21*16-4`($sp)	# save vrsave+	li		$x10,0x10+	$PUSH		r26,`$FRAME+21*16+0*$SIZE_T`($sp)+	li		$x20,0x20+	$PUSH		r27,`$FRAME+21*16+1*$SIZE_T`($sp)+	li		$x30,0x30+	$PUSH		r28,`$FRAME+21*16+2*$SIZE_T`($sp)+	li		$x40,0x40+	$PUSH		r29,`$FRAME+21*16+3*$SIZE_T`($sp)+	li		$x50,0x50+	$PUSH		r30,`$FRAME+21*16+4*$SIZE_T`($sp)+	li		$x60,0x60+	$PUSH		r31,`$FRAME+21*16+5*$SIZE_T`($sp)+	li		$x70,0x70+	mtspr		256,r0++	subi		$rounds,$rounds,3	# -4 in total++	lvx		$rndkey0,$x00,$key	# load key schedule+	lvx		v30,$x10,$key+	addi		$key,$key,0x20+	lvx		v31,$x00,$key+	?vperm		$rndkey0,$rndkey0,v30,$keyperm+	addi		$key_,$sp,$FRAME+15+	mtctr		$rounds++Load_ctr32_enc_key:+	?vperm		v24,v30,v31,$keyperm+	lvx		v30,$x10,$key+	addi		$key,$key,0x20+	stvx		v24,$x00,$key_		# off-load round[1]+	?vperm		v25,v31,v30,$keyperm+	lvx		v31,$x00,$key+	stvx		v25,$x10,$key_		# off-load round[2]+	addi		$key_,$key_,0x20+	bdnz		Load_ctr32_enc_key++	lvx		v26,$x10,$key+	?vperm		v24,v30,v31,$keyperm+	lvx		v27,$x20,$key+	stvx		v24,$x00,$key_		# off-load round[3]+	?vperm		v25,v31,v26,$keyperm+	lvx		v28,$x30,$key+	stvx		v25,$x10,$key_		# off-load round[4]+	addi		$key_,$sp,$FRAME+15	# rewind $key_+	?vperm		v26,v26,v27,$keyperm+	lvx		v29,$x40,$key+	?vperm		v27,v27,v28,$keyperm+	lvx		v30,$x50,$key+	?vperm		v28,v28,v29,$keyperm+	lvx		v31,$x60,$key+	?vperm		v29,v29,v30,$keyperm+	lvx		$out0,$x70,$key		# borrow $out0+	?vperm		v30,v30,v31,$keyperm+	lvx		v24,$x00,$key_		# pre-load round[1]+	?vperm		v31,v31,$out0,$keyperm+	lvx		v25,$x10,$key_		# pre-load round[2]++	vadduwm		$two,$one,$one+	subi		$inp,$inp,15		# undo "caller"+	$SHL		$len,$len,4++	vadduwm		$out1,$ivec,$one	# counter values ...+	vadduwm		$out2,$ivec,$two+	vxor		$out0,$ivec,$rndkey0	# ... xored with rndkey[0]+	 le?li		$idx,8+	vadduwm		$out3,$out1,$two+	vxor		$out1,$out1,$rndkey0+	 le?lvsl	$inpperm,0,$idx+	vadduwm		$out4,$out2,$two+	vxor		$out2,$out2,$rndkey0+	 le?vspltisb	$tmp,0x0f+	vadduwm		$out5,$out3,$two+	vxor		$out3,$out3,$rndkey0+	 le?vxor	$inpperm,$inpperm,$tmp	# transform for lvx_u/stvx_u+	vadduwm		$out6,$out4,$two+	vxor		$out4,$out4,$rndkey0+	vadduwm		$out7,$out5,$two+	vxor		$out5,$out5,$rndkey0+	vadduwm		$ivec,$out6,$two	# next counter value+	vxor		$out6,$out6,$rndkey0+	vxor		$out7,$out7,$rndkey0++	mtctr		$rounds+	b		Loop_ctr32_enc8x+.align	5+Loop_ctr32_enc8x:+	vcipher 	$out0,$out0,v24+	vcipher 	$out1,$out1,v24+	vcipher 	$out2,$out2,v24+	vcipher 	$out3,$out3,v24+	vcipher 	$out4,$out4,v24+	vcipher 	$out5,$out5,v24+	vcipher 	$out6,$out6,v24+	vcipher 	$out7,$out7,v24+Loop_ctr32_enc8x_middle:+	lvx		v24,$x20,$key_		# round[3]+	addi		$key_,$key_,0x20++	vcipher 	$out0,$out0,v25+	vcipher 	$out1,$out1,v25+	vcipher 	$out2,$out2,v25+	vcipher 	$out3,$out3,v25+	vcipher 	$out4,$out4,v25+	vcipher 	$out5,$out5,v25+	vcipher 	$out6,$out6,v25+	vcipher 	$out7,$out7,v25+	lvx		v25,$x10,$key_		# round[4]+	bdnz		Loop_ctr32_enc8x++	subic		r11,$len,256		# $len-256, borrow $key_+	vcipher 	$out0,$out0,v24+	vcipher 	$out1,$out1,v24+	vcipher 	$out2,$out2,v24+	vcipher 	$out3,$out3,v24+	vcipher 	$out4,$out4,v24+	vcipher 	$out5,$out5,v24+	vcipher 	$out6,$out6,v24+	vcipher 	$out7,$out7,v24++	subfe		r0,r0,r0		# borrow?-1:0+	vcipher 	$out0,$out0,v25+	vcipher 	$out1,$out1,v25+	vcipher 	$out2,$out2,v25+	vcipher 	$out3,$out3,v25+	vcipher 	$out4,$out4,v25+	vcipher		$out5,$out5,v25+	vcipher		$out6,$out6,v25+	vcipher		$out7,$out7,v25++	and		r0,r0,r11+	addi		$key_,$sp,$FRAME+15	# rewind $key_+	vcipher		$out0,$out0,v26+	vcipher		$out1,$out1,v26+	vcipher		$out2,$out2,v26+	vcipher		$out3,$out3,v26+	vcipher		$out4,$out4,v26+	vcipher		$out5,$out5,v26+	vcipher		$out6,$out6,v26+	vcipher		$out7,$out7,v26+	lvx		v24,$x00,$key_		# re-pre-load round[1]++	subic		$len,$len,129		# $len-=129+	vcipher		$out0,$out0,v27+	addi		$len,$len,1		# $len-=128 really+	vcipher		$out1,$out1,v27+	vcipher		$out2,$out2,v27+	vcipher		$out3,$out3,v27+	vcipher		$out4,$out4,v27+	vcipher		$out5,$out5,v27+	vcipher		$out6,$out6,v27+	vcipher		$out7,$out7,v27+	lvx		v25,$x10,$key_		# re-pre-load round[2]++	vcipher		$out0,$out0,v28+	 lvx_u		$in0,$x00,$inp		# load input+	vcipher		$out1,$out1,v28+	 lvx_u		$in1,$x10,$inp+	vcipher		$out2,$out2,v28+	 lvx_u		$in2,$x20,$inp+	vcipher		$out3,$out3,v28+	 lvx_u		$in3,$x30,$inp+	vcipher		$out4,$out4,v28+	 lvx_u		$in4,$x40,$inp+	vcipher		$out5,$out5,v28+	 lvx_u		$in5,$x50,$inp+	vcipher		$out6,$out6,v28+	 lvx_u		$in6,$x60,$inp+	vcipher		$out7,$out7,v28+	 lvx_u		$in7,$x70,$inp+	 addi		$inp,$inp,0x80++	vcipher		$out0,$out0,v29+	 le?vperm	$in0,$in0,$in0,$inpperm+	vcipher		$out1,$out1,v29+	 le?vperm	$in1,$in1,$in1,$inpperm+	vcipher		$out2,$out2,v29+	 le?vperm	$in2,$in2,$in2,$inpperm+	vcipher		$out3,$out3,v29+	 le?vperm	$in3,$in3,$in3,$inpperm+	vcipher		$out4,$out4,v29+	 le?vperm	$in4,$in4,$in4,$inpperm+	vcipher		$out5,$out5,v29+	 le?vperm	$in5,$in5,$in5,$inpperm+	vcipher		$out6,$out6,v29+	 le?vperm	$in6,$in6,$in6,$inpperm+	vcipher		$out7,$out7,v29+	 le?vperm	$in7,$in7,$in7,$inpperm++	add		$inp,$inp,r0		# $inp is adjusted in such+						# way that at exit from the+						# loop inX-in7 are loaded+						# with last "words"+	subfe.		r0,r0,r0		# borrow?-1:0+	vcipher		$out0,$out0,v30+	 vxor		$in0,$in0,v31		# xor with last round key+	vcipher		$out1,$out1,v30+	 vxor		$in1,$in1,v31+	vcipher		$out2,$out2,v30+	 vxor		$in2,$in2,v31+	vcipher		$out3,$out3,v30+	 vxor		$in3,$in3,v31+	vcipher		$out4,$out4,v30+	 vxor		$in4,$in4,v31+	vcipher		$out5,$out5,v30+	 vxor		$in5,$in5,v31+	vcipher		$out6,$out6,v30+	 vxor		$in6,$in6,v31+	vcipher		$out7,$out7,v30+	 vxor		$in7,$in7,v31++	bne		Lctr32_enc8x_break	# did $len-129 borrow?++	vcipherlast	$in0,$out0,$in0+	vcipherlast	$in1,$out1,$in1+	 vadduwm	$out1,$ivec,$one	# counter values ...+	vcipherlast	$in2,$out2,$in2+	 vadduwm	$out2,$ivec,$two+	 vxor		$out0,$ivec,$rndkey0	# ... xored with rndkey[0]+	vcipherlast	$in3,$out3,$in3+	 vadduwm	$out3,$out1,$two+	 vxor		$out1,$out1,$rndkey0+	vcipherlast	$in4,$out4,$in4+	 vadduwm	$out4,$out2,$two+	 vxor		$out2,$out2,$rndkey0+	vcipherlast	$in5,$out5,$in5+	 vadduwm	$out5,$out3,$two+	 vxor		$out3,$out3,$rndkey0+	vcipherlast	$in6,$out6,$in6+	 vadduwm	$out6,$out4,$two+	 vxor		$out4,$out4,$rndkey0+	vcipherlast	$in7,$out7,$in7+	 vadduwm	$out7,$out5,$two+	 vxor		$out5,$out5,$rndkey0+	le?vperm	$in0,$in0,$in0,$inpperm+	 vadduwm	$ivec,$out6,$two	# next counter value+	 vxor		$out6,$out6,$rndkey0+	le?vperm	$in1,$in1,$in1,$inpperm+	 vxor		$out7,$out7,$rndkey0+	mtctr		$rounds++	 vcipher	$out0,$out0,v24+	stvx_u		$in0,$x00,$out+	le?vperm	$in2,$in2,$in2,$inpperm+	 vcipher	$out1,$out1,v24+	stvx_u		$in1,$x10,$out+	le?vperm	$in3,$in3,$in3,$inpperm+	 vcipher	$out2,$out2,v24+	stvx_u		$in2,$x20,$out+	le?vperm	$in4,$in4,$in4,$inpperm+	 vcipher	$out3,$out3,v24+	stvx_u		$in3,$x30,$out+	le?vperm	$in5,$in5,$in5,$inpperm+	 vcipher	$out4,$out4,v24+	stvx_u		$in4,$x40,$out+	le?vperm	$in6,$in6,$in6,$inpperm+	 vcipher	$out5,$out5,v24+	stvx_u		$in5,$x50,$out+	le?vperm	$in7,$in7,$in7,$inpperm+	 vcipher	$out6,$out6,v24+	stvx_u		$in6,$x60,$out+	 vcipher	$out7,$out7,v24+	stvx_u		$in7,$x70,$out+	addi		$out,$out,0x80++	b		Loop_ctr32_enc8x_middle++.align	5+Lctr32_enc8x_break:+	cmpwi		$len,-0x60+	blt		Lctr32_enc8x_one+	nop+	beq		Lctr32_enc8x_two+	cmpwi		$len,-0x40+	blt		Lctr32_enc8x_three+	nop+	beq		Lctr32_enc8x_four+	cmpwi		$len,-0x20+	blt		Lctr32_enc8x_five+	nop+	beq		Lctr32_enc8x_six+	cmpwi		$len,0x00+	blt		Lctr32_enc8x_seven++Lctr32_enc8x_eight:+	vcipherlast	$out0,$out0,$in0+	vcipherlast	$out1,$out1,$in1+	vcipherlast	$out2,$out2,$in2+	vcipherlast	$out3,$out3,$in3+	vcipherlast	$out4,$out4,$in4+	vcipherlast	$out5,$out5,$in5+	vcipherlast	$out6,$out6,$in6+	vcipherlast	$out7,$out7,$in7++	le?vperm	$out0,$out0,$out0,$inpperm+	le?vperm	$out1,$out1,$out1,$inpperm+	stvx_u		$out0,$x00,$out+	le?vperm	$out2,$out2,$out2,$inpperm+	stvx_u		$out1,$x10,$out+	le?vperm	$out3,$out3,$out3,$inpperm+	stvx_u		$out2,$x20,$out+	le?vperm	$out4,$out4,$out4,$inpperm+	stvx_u		$out3,$x30,$out+	le?vperm	$out5,$out5,$out5,$inpperm+	stvx_u		$out4,$x40,$out+	le?vperm	$out6,$out6,$out6,$inpperm+	stvx_u		$out5,$x50,$out+	le?vperm	$out7,$out7,$out7,$inpperm+	stvx_u		$out6,$x60,$out+	stvx_u		$out7,$x70,$out+	addi		$out,$out,0x80+	b		Lctr32_enc8x_done++.align	5+Lctr32_enc8x_seven:+	vcipherlast	$out0,$out0,$in1+	vcipherlast	$out1,$out1,$in2+	vcipherlast	$out2,$out2,$in3+	vcipherlast	$out3,$out3,$in4+	vcipherlast	$out4,$out4,$in5+	vcipherlast	$out5,$out5,$in6+	vcipherlast	$out6,$out6,$in7++	le?vperm	$out0,$out0,$out0,$inpperm+	le?vperm	$out1,$out1,$out1,$inpperm+	stvx_u		$out0,$x00,$out+	le?vperm	$out2,$out2,$out2,$inpperm+	stvx_u		$out1,$x10,$out+	le?vperm	$out3,$out3,$out3,$inpperm+	stvx_u		$out2,$x20,$out+	le?vperm	$out4,$out4,$out4,$inpperm+	stvx_u		$out3,$x30,$out+	le?vperm	$out5,$out5,$out5,$inpperm+	stvx_u		$out4,$x40,$out+	le?vperm	$out6,$out6,$out6,$inpperm+	stvx_u		$out5,$x50,$out+	stvx_u		$out6,$x60,$out+	addi		$out,$out,0x70+	b		Lctr32_enc8x_done++.align	5+Lctr32_enc8x_six:+	vcipherlast	$out0,$out0,$in2+	vcipherlast	$out1,$out1,$in3+	vcipherlast	$out2,$out2,$in4+	vcipherlast	$out3,$out3,$in5+	vcipherlast	$out4,$out4,$in6+	vcipherlast	$out5,$out5,$in7++	le?vperm	$out0,$out0,$out0,$inpperm+	le?vperm	$out1,$out1,$out1,$inpperm+	stvx_u		$out0,$x00,$out+	le?vperm	$out2,$out2,$out2,$inpperm+	stvx_u		$out1,$x10,$out+	le?vperm	$out3,$out3,$out3,$inpperm+	stvx_u		$out2,$x20,$out+	le?vperm	$out4,$out4,$out4,$inpperm+	stvx_u		$out3,$x30,$out+	le?vperm	$out5,$out5,$out5,$inpperm+	stvx_u		$out4,$x40,$out+	stvx_u		$out5,$x50,$out+	addi		$out,$out,0x60+	b		Lctr32_enc8x_done++.align	5+Lctr32_enc8x_five:+	vcipherlast	$out0,$out0,$in3+	vcipherlast	$out1,$out1,$in4+	vcipherlast	$out2,$out2,$in5+	vcipherlast	$out3,$out3,$in6+	vcipherlast	$out4,$out4,$in7++	le?vperm	$out0,$out0,$out0,$inpperm+	le?vperm	$out1,$out1,$out1,$inpperm+	stvx_u		$out0,$x00,$out+	le?vperm	$out2,$out2,$out2,$inpperm+	stvx_u		$out1,$x10,$out+	le?vperm	$out3,$out3,$out3,$inpperm+	stvx_u		$out2,$x20,$out+	le?vperm	$out4,$out4,$out4,$inpperm+	stvx_u		$out3,$x30,$out+	stvx_u		$out4,$x40,$out+	addi		$out,$out,0x50+	b		Lctr32_enc8x_done++.align	5+Lctr32_enc8x_four:+	vcipherlast	$out0,$out0,$in4+	vcipherlast	$out1,$out1,$in5+	vcipherlast	$out2,$out2,$in6+	vcipherlast	$out3,$out3,$in7++	le?vperm	$out0,$out0,$out0,$inpperm+	le?vperm	$out1,$out1,$out1,$inpperm+	stvx_u		$out0,$x00,$out+	le?vperm	$out2,$out2,$out2,$inpperm+	stvx_u		$out1,$x10,$out+	le?vperm	$out3,$out3,$out3,$inpperm+	stvx_u		$out2,$x20,$out+	stvx_u		$out3,$x30,$out+	addi		$out,$out,0x40+	b		Lctr32_enc8x_done++.align	5+Lctr32_enc8x_three:+	vcipherlast	$out0,$out0,$in5+	vcipherlast	$out1,$out1,$in6+	vcipherlast	$out2,$out2,$in7++	le?vperm	$out0,$out0,$out0,$inpperm+	le?vperm	$out1,$out1,$out1,$inpperm+	stvx_u		$out0,$x00,$out+	le?vperm	$out2,$out2,$out2,$inpperm+	stvx_u		$out1,$x10,$out+	stvx_u		$out2,$x20,$out+	addi		$out,$out,0x30+	b		Lctr32_enc8x_done++.align	5+Lctr32_enc8x_two:+	vcipherlast	$out0,$out0,$in6+	vcipherlast	$out1,$out1,$in7++	le?vperm	$out0,$out0,$out0,$inpperm+	le?vperm	$out1,$out1,$out1,$inpperm+	stvx_u		$out0,$x00,$out+	stvx_u		$out1,$x10,$out+	addi		$out,$out,0x20+	b		Lctr32_enc8x_done++.align	5+Lctr32_enc8x_one:+	vcipherlast	$out0,$out0,$in7++	le?vperm	$out0,$out0,$out0,$inpperm+	stvx_u		$out0,0,$out+	addi		$out,$out,0x10++Lctr32_enc8x_done:+	li		r10,`$FRAME+15`+	li		r11,`$FRAME+31`+	stvx		$inpperm,r10,$sp	# wipe copies of round keys+	addi		r10,r10,32+	stvx		$inpperm,r11,$sp+	addi		r11,r11,32+	stvx		$inpperm,r10,$sp+	addi		r10,r10,32+	stvx		$inpperm,r11,$sp+	addi		r11,r11,32+	stvx		$inpperm,r10,$sp+	addi		r10,r10,32+	stvx		$inpperm,r11,$sp+	addi		r11,r11,32+	stvx		$inpperm,r10,$sp+	addi		r10,r10,32+	stvx		$inpperm,r11,$sp+	addi		r11,r11,32++	mtspr		256,$vrsave+	lvx		v20,r10,$sp		# ABI says so+	addi		r10,r10,32+	lvx		v21,r11,$sp+	addi		r11,r11,32+	lvx		v22,r10,$sp+	addi		r10,r10,32+	lvx		v23,r11,$sp+	addi		r11,r11,32+	lvx		v24,r10,$sp+	addi		r10,r10,32+	lvx		v25,r11,$sp+	addi		r11,r11,32+	lvx		v26,r10,$sp+	addi		r10,r10,32+	lvx		v27,r11,$sp+	addi		r11,r11,32+	lvx		v28,r10,$sp+	addi		r10,r10,32+	lvx		v29,r11,$sp+	addi		r11,r11,32+	lvx		v30,r10,$sp+	lvx		v31,r11,$sp+	$POP		r26,`$FRAME+21*16+0*$SIZE_T`($sp)+	$POP		r27,`$FRAME+21*16+1*$SIZE_T`($sp)+	$POP		r28,`$FRAME+21*16+2*$SIZE_T`($sp)+	$POP		r29,`$FRAME+21*16+3*$SIZE_T`($sp)+	$POP		r30,`$FRAME+21*16+4*$SIZE_T`($sp)+	$POP		r31,`$FRAME+21*16+5*$SIZE_T`($sp)+	addi		$sp,$sp,`$FRAME+21*16+6*$SIZE_T`+	blr+	.long		0+	.byte		0,12,0x04,0,0x80,6,6,0+	.long		0+.size	.${prefix}_ctr32_encrypt_blocks,.-.${prefix}_ctr32_encrypt_blocks+___+}}	}}}++#########################################################################+{{{	# XTS procedures						#+# int aes_p8_xts_[en|de]crypt(const char *inp, char *out, size_t len,	#+#                             const AES_KEY *key1, const AES_KEY *key2,	#+#                             [const] unsigned char iv[16]);		#+# If $key2 is NULL, then a "tweak chaining" mode is engaged, in which	#+# input tweak value is assumed to be encrypted already, and last tweak	#+# value, one suitable for consecutive call on same chunk of data, is	#+# written back to original buffer. In addition, in "tweak chaining"	#+# mode only complete input blocks are processed.			#++my ($inp,$out,$len,$key1,$key2,$ivp,$rounds,$idx) =	map("r$_",(3..10));+my ($rndkey0,$rndkey1,$inout) =				map("v$_",(0..2));+my ($output,$inptail,$inpperm,$leperm,$keyperm) =	map("v$_",(3..7));+my ($tweak,$seven,$eighty7,$tmp,$tweak1) =		map("v$_",(8..12));+my $taillen = $key2;++   ($inp,$idx) = ($idx,$inp);				# reassign++$code.=<<___;+.globl	.${prefix}_xts_encrypt+.align	5+.${prefix}_xts_encrypt:+	mr		$inp,r3				# reassign+	li		r3,-1+	${UCMP}i	$len,16+	bltlr-++	lis		r0,0xfff0+	mfspr		r12,256				# save vrsave+	li		r11,0+	mtspr		256,r0++	vspltisb	$seven,0x07			# 0x070707..07+	le?lvsl		$leperm,r11,r11+	le?vspltisb	$tmp,0x0f+	le?vxor		$leperm,$leperm,$seven++	li		$idx,15+	lvx		$tweak,0,$ivp			# load [unaligned] iv+	lvsl		$inpperm,0,$ivp+	lvx		$inptail,$idx,$ivp+	le?vxor		$inpperm,$inpperm,$tmp+	vperm		$tweak,$tweak,$inptail,$inpperm++	neg		r11,$inp+	lvsr		$inpperm,0,r11			# prepare for unaligned load+	lvx		$inout,0,$inp+	addi		$inp,$inp,15			# 15 is not typo+	le?vxor		$inpperm,$inpperm,$tmp++	${UCMP}i	$key2,0				# key2==NULL?+	beq		Lxts_enc_no_key2++	?lvsl		$keyperm,0,$key2		# prepare for unaligned key+	lwz		$rounds,240($key2)+	srwi		$rounds,$rounds,1+	subi		$rounds,$rounds,1+	li		$idx,16++	lvx		$rndkey0,0,$key2+	lvx		$rndkey1,$idx,$key2+	addi		$idx,$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vxor		$tweak,$tweak,$rndkey0+	lvx		$rndkey0,$idx,$key2+	addi		$idx,$idx,16+	mtctr		$rounds++Ltweak_xts_enc:+	?vperm		$rndkey1,$rndkey1,$rndkey0,$keyperm+	vcipher		$tweak,$tweak,$rndkey1+	lvx		$rndkey1,$idx,$key2+	addi		$idx,$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vcipher		$tweak,$tweak,$rndkey0+	lvx		$rndkey0,$idx,$key2+	addi		$idx,$idx,16+	bdnz		Ltweak_xts_enc++	?vperm		$rndkey1,$rndkey1,$rndkey0,$keyperm+	vcipher		$tweak,$tweak,$rndkey1+	lvx		$rndkey1,$idx,$key2+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vcipherlast	$tweak,$tweak,$rndkey0++	li		$ivp,0				# don't chain the tweak+	b		Lxts_enc++Lxts_enc_no_key2:+	li		$idx,-16+	and		$len,$len,$idx			# in "tweak chaining"+							# mode only complete+							# blocks are processed+Lxts_enc:+	lvx		$inptail,0,$inp+	addi		$inp,$inp,16++	?lvsl		$keyperm,0,$key1		# prepare for unaligned key+	lwz		$rounds,240($key1)+	srwi		$rounds,$rounds,1+	subi		$rounds,$rounds,1+	li		$idx,16++	vslb		$eighty7,$seven,$seven		# 0x808080..80+	vor		$eighty7,$eighty7,$seven	# 0x878787..87+	vspltisb	$tmp,1				# 0x010101..01+	vsldoi		$eighty7,$eighty7,$tmp,15	# 0x870101..01++	${UCMP}i	$len,96+	bge		_aesp8_xts_encrypt6x++	andi.		$taillen,$len,15+	subic		r0,$len,32+	subi		$taillen,$taillen,16+	subfe		r0,r0,r0+	and		r0,r0,$taillen+	add		$inp,$inp,r0++	lvx		$rndkey0,0,$key1+	lvx		$rndkey1,$idx,$key1+	addi		$idx,$idx,16+	vperm		$inout,$inout,$inptail,$inpperm+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vxor		$inout,$inout,$tweak+	vxor		$inout,$inout,$rndkey0+	lvx		$rndkey0,$idx,$key1+	addi		$idx,$idx,16+	mtctr		$rounds+	b		Loop_xts_enc++.align	5+Loop_xts_enc:+	?vperm		$rndkey1,$rndkey1,$rndkey0,$keyperm+	vcipher		$inout,$inout,$rndkey1+	lvx		$rndkey1,$idx,$key1+	addi		$idx,$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vcipher		$inout,$inout,$rndkey0+	lvx		$rndkey0,$idx,$key1+	addi		$idx,$idx,16+	bdnz		Loop_xts_enc++	?vperm		$rndkey1,$rndkey1,$rndkey0,$keyperm+	vcipher		$inout,$inout,$rndkey1+	lvx		$rndkey1,$idx,$key1+	li		$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vxor		$rndkey0,$rndkey0,$tweak+	vcipherlast	$output,$inout,$rndkey0++	le?vperm	$tmp,$output,$output,$leperm+	be?nop+	le?stvx_u	$tmp,0,$out+	be?stvx_u	$output,0,$out+	addi		$out,$out,16++	subic.		$len,$len,16+	beq		Lxts_enc_done++	vmr		$inout,$inptail+	lvx		$inptail,0,$inp+	addi		$inp,$inp,16+	lvx		$rndkey0,0,$key1+	lvx		$rndkey1,$idx,$key1+	addi		$idx,$idx,16++	subic		r0,$len,32+	subfe		r0,r0,r0+	and		r0,r0,$taillen+	add		$inp,$inp,r0++	vsrab		$tmp,$tweak,$seven		# next tweak value+	vaddubm		$tweak,$tweak,$tweak+	vsldoi		$tmp,$tmp,$tmp,15+	vand		$tmp,$tmp,$eighty7+	vxor		$tweak,$tweak,$tmp++	vperm		$inout,$inout,$inptail,$inpperm+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vxor		$inout,$inout,$tweak+	vxor		$output,$output,$rndkey0	# just in case $len<16+	vxor		$inout,$inout,$rndkey0+	lvx		$rndkey0,$idx,$key1+	addi		$idx,$idx,16++	mtctr		$rounds+	${UCMP}i	$len,16+	bge		Loop_xts_enc++	vxor		$output,$output,$tweak+	lvsr		$inpperm,0,$len			# $inpperm is no longer needed+	vxor		$inptail,$inptail,$inptail	# $inptail is no longer needed+	vspltisb	$tmp,-1+	vperm		$inptail,$inptail,$tmp,$inpperm+	vsel		$inout,$inout,$output,$inptail++	subi		r11,$out,17+	subi		$out,$out,16+	mtctr		$len+	li		$len,16+Loop_xts_enc_steal:+	lbzu		r0,1(r11)+	stb		r0,16(r11)+	bdnz		Loop_xts_enc_steal++	mtctr		$rounds+	b		Loop_xts_enc			# one more time...++Lxts_enc_done:+	${UCMP}i	$ivp,0+	beq		Lxts_enc_ret++	vsrab		$tmp,$tweak,$seven		# next tweak value+	vaddubm		$tweak,$tweak,$tweak+	vsldoi		$tmp,$tmp,$tmp,15+	vand		$tmp,$tmp,$eighty7+	vxor		$tweak,$tweak,$tmp++	le?vperm	$tweak,$tweak,$tweak,$leperm+	stvx_u		$tweak,0,$ivp++Lxts_enc_ret:+	mtspr		256,r12				# restore vrsave+	li		r3,0+	blr+	.long		0+	.byte		0,12,0x04,0,0x80,6,6,0+	.long		0+.size	.${prefix}_xts_encrypt,.-.${prefix}_xts_encrypt++.globl	.${prefix}_xts_decrypt+.align	5+.${prefix}_xts_decrypt:+	mr		$inp,r3				# reassign+	li		r3,-1+	${UCMP}i	$len,16+	bltlr-++	lis		r0,0xfff8+	mfspr		r12,256				# save vrsave+	li		r11,0+	mtspr		256,r0++	andi.		r0,$len,15+	neg		r0,r0+	andi.		r0,r0,16+	sub		$len,$len,r0++	vspltisb	$seven,0x07			# 0x070707..07+	le?lvsl		$leperm,r11,r11+	le?vspltisb	$tmp,0x0f+	le?vxor		$leperm,$leperm,$seven++	li		$idx,15+	lvx		$tweak,0,$ivp			# load [unaligned] iv+	lvsl		$inpperm,0,$ivp+	lvx		$inptail,$idx,$ivp+	le?vxor		$inpperm,$inpperm,$tmp+	vperm		$tweak,$tweak,$inptail,$inpperm++	neg		r11,$inp+	lvsr		$inpperm,0,r11			# prepare for unaligned load+	lvx		$inout,0,$inp+	addi		$inp,$inp,15			# 15 is not typo+	le?vxor		$inpperm,$inpperm,$tmp++	${UCMP}i	$key2,0				# key2==NULL?+	beq		Lxts_dec_no_key2++	?lvsl		$keyperm,0,$key2		# prepare for unaligned key+	lwz		$rounds,240($key2)+	srwi		$rounds,$rounds,1+	subi		$rounds,$rounds,1+	li		$idx,16++	lvx		$rndkey0,0,$key2+	lvx		$rndkey1,$idx,$key2+	addi		$idx,$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vxor		$tweak,$tweak,$rndkey0+	lvx		$rndkey0,$idx,$key2+	addi		$idx,$idx,16+	mtctr		$rounds++Ltweak_xts_dec:+	?vperm		$rndkey1,$rndkey1,$rndkey0,$keyperm+	vcipher		$tweak,$tweak,$rndkey1+	lvx		$rndkey1,$idx,$key2+	addi		$idx,$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vcipher		$tweak,$tweak,$rndkey0+	lvx		$rndkey0,$idx,$key2+	addi		$idx,$idx,16+	bdnz		Ltweak_xts_dec++	?vperm		$rndkey1,$rndkey1,$rndkey0,$keyperm+	vcipher		$tweak,$tweak,$rndkey1+	lvx		$rndkey1,$idx,$key2+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vcipherlast	$tweak,$tweak,$rndkey0++	li		$ivp,0				# don't chain the tweak+	b		Lxts_dec++Lxts_dec_no_key2:+	neg		$idx,$len+	andi.		$idx,$idx,15+	add		$len,$len,$idx			# in "tweak chaining"+							# mode only complete+							# blocks are processed+Lxts_dec:+	lvx		$inptail,0,$inp+	addi		$inp,$inp,16++	?lvsl		$keyperm,0,$key1		# prepare for unaligned key+	lwz		$rounds,240($key1)+	srwi		$rounds,$rounds,1+	subi		$rounds,$rounds,1+	li		$idx,16++	vslb		$eighty7,$seven,$seven		# 0x808080..80+	vor		$eighty7,$eighty7,$seven	# 0x878787..87+	vspltisb	$tmp,1				# 0x010101..01+	vsldoi		$eighty7,$eighty7,$tmp,15	# 0x870101..01++	${UCMP}i	$len,96+	bge		_aesp8_xts_decrypt6x++	lvx		$rndkey0,0,$key1+	lvx		$rndkey1,$idx,$key1+	addi		$idx,$idx,16+	vperm		$inout,$inout,$inptail,$inpperm+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vxor		$inout,$inout,$tweak+	vxor		$inout,$inout,$rndkey0+	lvx		$rndkey0,$idx,$key1+	addi		$idx,$idx,16+	mtctr		$rounds++	${UCMP}i	$len,16+	blt		Ltail_xts_dec+	be?b		Loop_xts_dec++.align	5+Loop_xts_dec:+	?vperm		$rndkey1,$rndkey1,$rndkey0,$keyperm+	vncipher	$inout,$inout,$rndkey1+	lvx		$rndkey1,$idx,$key1+	addi		$idx,$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vncipher	$inout,$inout,$rndkey0+	lvx		$rndkey0,$idx,$key1+	addi		$idx,$idx,16+	bdnz		Loop_xts_dec++	?vperm		$rndkey1,$rndkey1,$rndkey0,$keyperm+	vncipher	$inout,$inout,$rndkey1+	lvx		$rndkey1,$idx,$key1+	li		$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vxor		$rndkey0,$rndkey0,$tweak+	vncipherlast	$output,$inout,$rndkey0++	le?vperm	$tmp,$output,$output,$leperm+	be?nop+	le?stvx_u	$tmp,0,$out+	be?stvx_u	$output,0,$out+	addi		$out,$out,16++	subic.		$len,$len,16+	beq		Lxts_dec_done++	vmr		$inout,$inptail+	lvx		$inptail,0,$inp+	addi		$inp,$inp,16+	lvx		$rndkey0,0,$key1+	lvx		$rndkey1,$idx,$key1+	addi		$idx,$idx,16++	vsrab		$tmp,$tweak,$seven		# next tweak value+	vaddubm		$tweak,$tweak,$tweak+	vsldoi		$tmp,$tmp,$tmp,15+	vand		$tmp,$tmp,$eighty7+	vxor		$tweak,$tweak,$tmp++	vperm		$inout,$inout,$inptail,$inpperm+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vxor		$inout,$inout,$tweak+	vxor		$inout,$inout,$rndkey0+	lvx		$rndkey0,$idx,$key1+	addi		$idx,$idx,16++	mtctr		$rounds+	${UCMP}i	$len,16+	bge		Loop_xts_dec++Ltail_xts_dec:+	vsrab		$tmp,$tweak,$seven		# next tweak value+	vaddubm		$tweak1,$tweak,$tweak+	vsldoi		$tmp,$tmp,$tmp,15+	vand		$tmp,$tmp,$eighty7+	vxor		$tweak1,$tweak1,$tmp++	subi		$inp,$inp,16+	add		$inp,$inp,$len++	vxor		$inout,$inout,$tweak		# :-(+	vxor		$inout,$inout,$tweak1		# :-)++Loop_xts_dec_short:+	?vperm		$rndkey1,$rndkey1,$rndkey0,$keyperm+	vncipher	$inout,$inout,$rndkey1+	lvx		$rndkey1,$idx,$key1+	addi		$idx,$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vncipher	$inout,$inout,$rndkey0+	lvx		$rndkey0,$idx,$key1+	addi		$idx,$idx,16+	bdnz		Loop_xts_dec_short++	?vperm		$rndkey1,$rndkey1,$rndkey0,$keyperm+	vncipher	$inout,$inout,$rndkey1+	lvx		$rndkey1,$idx,$key1+	li		$idx,16+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm+	vxor		$rndkey0,$rndkey0,$tweak1+	vncipherlast	$output,$inout,$rndkey0++	le?vperm	$tmp,$output,$output,$leperm+	be?nop+	le?stvx_u	$tmp,0,$out+	be?stvx_u	$output,0,$out++	vmr		$inout,$inptail+	lvx		$inptail,0,$inp+	#addi		$inp,$inp,16+	lvx		$rndkey0,0,$key1+	lvx		$rndkey1,$idx,$key1+	addi		$idx,$idx,16+	vperm		$inout,$inout,$inptail,$inpperm+	?vperm		$rndkey0,$rndkey0,$rndkey1,$keyperm++	lvsr		$inpperm,0,$len			# $inpperm is no longer needed+	vxor		$inptail,$inptail,$inptail	# $inptail is no longer needed+	vspltisb	$tmp,-1+	vperm		$inptail,$inptail,$tmp,$inpperm+	vsel		$inout,$inout,$output,$inptail++	vxor		$rndkey0,$rndkey0,$tweak+	vxor		$inout,$inout,$rndkey0+	lvx		$rndkey0,$idx,$key1+	addi		$idx,$idx,16++	subi		r11,$out,1+	mtctr		$len+	li		$len,16+Loop_xts_dec_steal:+	lbzu		r0,1(r11)+	stb		r0,16(r11)+	bdnz		Loop_xts_dec_steal++	mtctr		$rounds+	b		Loop_xts_dec			# one more time...++Lxts_dec_done:+	${UCMP}i	$ivp,0+	beq		Lxts_dec_ret++	vsrab		$tmp,$tweak,$seven		# next tweak value+	vaddubm		$tweak,$tweak,$tweak+	vsldoi		$tmp,$tmp,$tmp,15+	vand		$tmp,$tmp,$eighty7+	vxor		$tweak,$tweak,$tmp++	le?vperm	$tweak,$tweak,$tweak,$leperm+	stvx_u		$tweak,0,$ivp++Lxts_dec_ret:+	mtspr		256,r12				# restore vrsave+	li		r3,0+	blr+	.long		0+	.byte		0,12,0x04,0,0x80,6,6,0+	.long		0+.size	.${prefix}_xts_decrypt,.-.${prefix}_xts_decrypt+___+#########################################################################+{{	# Optimized XTS procedures					#+my $key_=$key2;+my ($x00,$x10,$x20,$x30,$x40,$x50,$x60,$x70)=map("r$_",(0,3,26..31));+    $x00=0 if ($flavour =~ /osx/);+my ($in0,  $in1,  $in2,  $in3,  $in4,  $in5 )=map("v$_",(0..5));+my ($out0, $out1, $out2, $out3, $out4, $out5)=map("v$_",(7,12..16));+my ($twk0, $twk1, $twk2, $twk3, $twk4, $twk5)=map("v$_",(17..22));+my $rndkey0="v23";	# v24-v25 rotating buffer for first found keys+			# v26-v31 last 6 round keys+my ($keyperm)=($out0);	# aliases with "caller", redundant assignment+my $taillen=$x70;++$code.=<<___;+.align	5+_aesp8_xts_encrypt6x:+	$STU		$sp,-`($FRAME+21*16+6*$SIZE_T)`($sp)+	mflr		r11+	li		r7,`$FRAME+8*16+15`+	li		r3,`$FRAME+8*16+31`+	$PUSH		r11,`$FRAME+21*16+6*$SIZE_T+$LRSAVE`($sp)+	stvx		v20,r7,$sp		# ABI says so+	addi		r7,r7,32+	stvx		v21,r3,$sp+	addi		r3,r3,32+	stvx		v22,r7,$sp+	addi		r7,r7,32+	stvx		v23,r3,$sp+	addi		r3,r3,32+	stvx		v24,r7,$sp+	addi		r7,r7,32+	stvx		v25,r3,$sp+	addi		r3,r3,32+	stvx		v26,r7,$sp+	addi		r7,r7,32+	stvx		v27,r3,$sp+	addi		r3,r3,32+	stvx		v28,r7,$sp+	addi		r7,r7,32+	stvx		v29,r3,$sp+	addi		r3,r3,32+	stvx		v30,r7,$sp+	stvx		v31,r3,$sp+	li		r0,-1+	stw		$vrsave,`$FRAME+21*16-4`($sp)	# save vrsave+	li		$x10,0x10+	$PUSH		r26,`$FRAME+21*16+0*$SIZE_T`($sp)+	li		$x20,0x20+	$PUSH		r27,`$FRAME+21*16+1*$SIZE_T`($sp)+	li		$x30,0x30+	$PUSH		r28,`$FRAME+21*16+2*$SIZE_T`($sp)+	li		$x40,0x40+	$PUSH		r29,`$FRAME+21*16+3*$SIZE_T`($sp)+	li		$x50,0x50+	$PUSH		r30,`$FRAME+21*16+4*$SIZE_T`($sp)+	li		$x60,0x60+	$PUSH		r31,`$FRAME+21*16+5*$SIZE_T`($sp)+	li		$x70,0x70+	mtspr		256,r0++	subi		$rounds,$rounds,3	# -4 in total++	lvx		$rndkey0,$x00,$key1	# load key schedule+	lvx		v30,$x10,$key1+	addi		$key1,$key1,0x20+	lvx		v31,$x00,$key1+	?vperm		$rndkey0,$rndkey0,v30,$keyperm+	addi		$key_,$sp,$FRAME+15+	mtctr		$rounds++Load_xts_enc_key:+	?vperm		v24,v30,v31,$keyperm+	lvx		v30,$x10,$key1+	addi		$key1,$key1,0x20+	stvx		v24,$x00,$key_		# off-load round[1]+	?vperm		v25,v31,v30,$keyperm+	lvx		v31,$x00,$key1+	stvx		v25,$x10,$key_		# off-load round[2]+	addi		$key_,$key_,0x20+	bdnz		Load_xts_enc_key++	lvx		v26,$x10,$key1+	?vperm		v24,v30,v31,$keyperm+	lvx		v27,$x20,$key1+	stvx		v24,$x00,$key_		# off-load round[3]+	?vperm		v25,v31,v26,$keyperm+	lvx		v28,$x30,$key1+	stvx		v25,$x10,$key_		# off-load round[4]+	addi		$key_,$sp,$FRAME+15	# rewind $key_+	?vperm		v26,v26,v27,$keyperm+	lvx		v29,$x40,$key1+	?vperm		v27,v27,v28,$keyperm+	lvx		v30,$x50,$key1+	?vperm		v28,v28,v29,$keyperm+	lvx		v31,$x60,$key1+	?vperm		v29,v29,v30,$keyperm+	lvx		$twk5,$x70,$key1	# borrow $twk5+	?vperm		v30,v30,v31,$keyperm+	lvx		v24,$x00,$key_		# pre-load round[1]+	?vperm		v31,v31,$twk5,$keyperm+	lvx		v25,$x10,$key_		# pre-load round[2]++	 vperm		$in0,$inout,$inptail,$inpperm+	 subi		$inp,$inp,31		# undo "caller"+	vxor		$twk0,$tweak,$rndkey0+	vsrab		$tmp,$tweak,$seven	# next tweak value+	vaddubm		$tweak,$tweak,$tweak+	vsldoi		$tmp,$tmp,$tmp,15+	vand		$tmp,$tmp,$eighty7+	 vxor		$out0,$in0,$twk0+	vxor		$tweak,$tweak,$tmp++	 lvx_u		$in1,$x10,$inp+	vxor		$twk1,$tweak,$rndkey0+	vsrab		$tmp,$tweak,$seven	# next tweak value+	vaddubm		$tweak,$tweak,$tweak+	vsldoi		$tmp,$tmp,$tmp,15+	 le?vperm	$in1,$in1,$in1,$leperm+	vand		$tmp,$tmp,$eighty7+	 vxor		$out1,$in1,$twk1+	vxor		$tweak,$tweak,$tmp++	 lvx_u		$in2,$x20,$inp+	 andi.		$taillen,$len,15+	vxor		$twk2,$tweak,$rndkey0+	vsrab		$tmp,$tweak,$seven	# next tweak value+	vaddubm		$tweak,$tweak,$tweak+	vsldoi		$tmp,$tmp,$tmp,15+	 le?vperm	$in2,$in2,$in2,$leperm+	vand		$tmp,$tmp,$eighty7+	 vxor		$out2,$in2,$twk2+	vxor		$tweak,$tweak,$tmp++	 lvx_u		$in3,$x30,$inp+	 sub		$len,$len,$taillen+	vxor		$twk3,$tweak,$rndkey0+	vsrab		$tmp,$tweak,$seven	# next tweak value+	vaddubm		$tweak,$tweak,$tweak+	vsldoi		$tmp,$tmp,$tmp,15+	 le?vperm	$in3,$in3,$in3,$leperm+	vand		$tmp,$tmp,$eighty7+	 vxor		$out3,$in3,$twk3+	vxor		$tweak,$tweak,$tmp++	 lvx_u		$in4,$x40,$inp+	 subi		$len,$len,0x60+	vxor		$twk4,$tweak,$rndkey0+	vsrab		$tmp,$tweak,$seven	# next tweak value+	vaddubm		$tweak,$tweak,$tweak+	vsldoi		$tmp,$tmp,$tmp,15+	 le?vperm	$in4,$in4,$in4,$leperm+	vand		$tmp,$tmp,$eighty7+	 vxor		$out4,$in4,$twk4+	vxor		$tweak,$tweak,$tmp++	 lvx_u		$in5,$x50,$inp+	 addi		$inp,$inp,0x60+	vxor		$twk5,$tweak,$rndkey0+	vsrab		$tmp,$tweak,$seven	# next tweak value+	vaddubm		$tweak,$tweak,$tweak+	vsldoi		$tmp,$tmp,$tmp,15+	 le?vperm	$in5,$in5,$in5,$leperm+	vand		$tmp,$tmp,$eighty7+	 vxor		$out5,$in5,$twk5+	vxor		$tweak,$tweak,$tmp++	vxor		v31,v31,$rndkey0+	mtctr		$rounds+	b		Loop_xts_enc6x++.align	5+Loop_xts_enc6x:+	vcipher		$out0,$out0,v24+	vcipher		$out1,$out1,v24+	vcipher		$out2,$out2,v24+	vcipher		$out3,$out3,v24+	vcipher		$out4,$out4,v24+	vcipher		$out5,$out5,v24+	lvx		v24,$x20,$key_		# round[3]+	addi		$key_,$key_,0x20++	vcipher		$out0,$out0,v25+	vcipher		$out1,$out1,v25+	vcipher		$out2,$out2,v25+	vcipher		$out3,$out3,v25+	vcipher		$out4,$out4,v25+	vcipher		$out5,$out5,v25+	lvx		v25,$x10,$key_		# round[4]+	bdnz		Loop_xts_enc6x++	subic		$len,$len,96		# $len-=96+	 vxor		$in0,$twk0,v31		# xor with last round key+	vcipher		$out0,$out0,v24+	vcipher		$out1,$out1,v24+	 vsrab		$tmp,$tweak,$seven	# next tweak value+	 vxor		$twk0,$tweak,$rndkey0+	 vaddubm	$tweak,$tweak,$tweak+	vcipher		$out2,$out2,v24+	vcipher		$out3,$out3,v24+	 vsldoi		$tmp,$tmp,$tmp,15+	vcipher		$out4,$out4,v24+	vcipher		$out5,$out5,v24++	subfe.		r0,r0,r0		# borrow?-1:0+	 vand		$tmp,$tmp,$eighty7+	vcipher		$out0,$out0,v25+	vcipher		$out1,$out1,v25+	 vxor		$tweak,$tweak,$tmp+	vcipher		$out2,$out2,v25+	vcipher		$out3,$out3,v25+	 vxor		$in1,$twk1,v31+	 vsrab		$tmp,$tweak,$seven	# next tweak value+	 vxor		$twk1,$tweak,$rndkey0+	vcipher		$out4,$out4,v25+	vcipher		$out5,$out5,v25++	and		r0,r0,$len+	 vaddubm	$tweak,$tweak,$tweak+	 vsldoi		$tmp,$tmp,$tmp,15+	vcipher		$out0,$out0,v26+	vcipher		$out1,$out1,v26+	 vand		$tmp,$tmp,$eighty7+	vcipher		$out2,$out2,v26+	vcipher		$out3,$out3,v26+	 vxor		$tweak,$tweak,$tmp+	vcipher		$out4,$out4,v26+	vcipher		$out5,$out5,v26++	add		$inp,$inp,r0		# $inp is adjusted in such+						# way that at exit from the+						# loop inX-in5 are loaded+						# with last "words"+	 vxor		$in2,$twk2,v31+	 vsrab		$tmp,$tweak,$seven	# next tweak value+	 vxor		$twk2,$tweak,$rndkey0+	 vaddubm	$tweak,$tweak,$tweak+	vcipher		$out0,$out0,v27+	vcipher		$out1,$out1,v27+	 vsldoi		$tmp,$tmp,$tmp,15+	vcipher		$out2,$out2,v27+	vcipher		$out3,$out3,v27+	 vand		$tmp,$tmp,$eighty7+	vcipher		$out4,$out4,v27+	vcipher		$out5,$out5,v27++	addi		$key_,$sp,$FRAME+15	# rewind $key_+	 vxor		$tweak,$tweak,$tmp+	vcipher		$out0,$out0,v28+	vcipher		$out1,$out1,v28+	 vxor		$in3,$twk3,v31+	 vsrab		$tmp,$tweak,$seven	# next tweak value+	 vxor		$twk3,$tweak,$rndkey0+	vcipher		$out2,$out2,v28+	vcipher		$out3,$out3,v28+	 vaddubm	$tweak,$tweak,$tweak+	 vsldoi		$tmp,$tmp,$tmp,15+	vcipher		$out4,$out4,v28+	vcipher		$out5,$out5,v28+	lvx		v24,$x00,$key_		# re-pre-load round[1]+	 vand		$tmp,$tmp,$eighty7++	vcipher		$out0,$out0,v29+	vcipher		$out1,$out1,v29+	 vxor		$tweak,$tweak,$tmp+	vcipher		$out2,$out2,v29+	vcipher		$out3,$out3,v29+	 vxor		$in4,$twk4,v31+	 vsrab		$tmp,$tweak,$seven	# next tweak value+	 vxor		$twk4,$tweak,$rndkey0+	vcipher		$out4,$out4,v29+	vcipher		$out5,$out5,v29+	lvx		v25,$x10,$key_		# re-pre-load round[2]+	 vaddubm	$tweak,$tweak,$tweak+	 vsldoi		$tmp,$tmp,$tmp,15++	vcipher		$out0,$out0,v30+	vcipher		$out1,$out1,v30+	 vand		$tmp,$tmp,$eighty7+	vcipher		$out2,$out2,v30+	vcipher		$out3,$out3,v30+	 vxor		$tweak,$tweak,$tmp+	vcipher		$out4,$out4,v30+	vcipher		$out5,$out5,v30+	 vxor		$in5,$twk5,v31+	 vsrab		$tmp,$tweak,$seven	# next tweak value+	 vxor		$twk5,$tweak,$rndkey0++	vcipherlast	$out0,$out0,$in0+	 lvx_u		$in0,$x00,$inp		# load next input block+	 vaddubm	$tweak,$tweak,$tweak+	 vsldoi		$tmp,$tmp,$tmp,15+	vcipherlast	$out1,$out1,$in1+	 lvx_u		$in1,$x10,$inp+	vcipherlast	$out2,$out2,$in2+	 le?vperm	$in0,$in0,$in0,$leperm+	 lvx_u		$in2,$x20,$inp+	 vand		$tmp,$tmp,$eighty7+	vcipherlast	$out3,$out3,$in3+	 le?vperm	$in1,$in1,$in1,$leperm+	 lvx_u		$in3,$x30,$inp+	vcipherlast	$out4,$out4,$in4+	 le?vperm	$in2,$in2,$in2,$leperm+	 lvx_u		$in4,$x40,$inp+	 vxor		$tweak,$tweak,$tmp+	vcipherlast	$tmp,$out5,$in5		# last block might be needed+						# in stealing mode+	 le?vperm	$in3,$in3,$in3,$leperm+	 lvx_u		$in5,$x50,$inp+	 addi		$inp,$inp,0x60+	 le?vperm	$in4,$in4,$in4,$leperm+	 le?vperm	$in5,$in5,$in5,$leperm++	le?vperm	$out0,$out0,$out0,$leperm+	le?vperm	$out1,$out1,$out1,$leperm+	stvx_u		$out0,$x00,$out		# store output+	 vxor		$out0,$in0,$twk0+	le?vperm	$out2,$out2,$out2,$leperm+	stvx_u		$out1,$x10,$out+	 vxor		$out1,$in1,$twk1+	le?vperm	$out3,$out3,$out3,$leperm+	stvx_u		$out2,$x20,$out+	 vxor		$out2,$in2,$twk2+	le?vperm	$out4,$out4,$out4,$leperm+	stvx_u		$out3,$x30,$out+	 vxor		$out3,$in3,$twk3+	le?vperm	$out5,$tmp,$tmp,$leperm+	stvx_u		$out4,$x40,$out+	 vxor		$out4,$in4,$twk4+	le?stvx_u	$out5,$x50,$out+	be?stvx_u	$tmp, $x50,$out+	 vxor		$out5,$in5,$twk5+	addi		$out,$out,0x60++	mtctr		$rounds+	beq		Loop_xts_enc6x		# did $len-=96 borrow?++	addic.		$len,$len,0x60+	beq		Lxts_enc6x_zero+	cmpwi		$len,0x20+	blt		Lxts_enc6x_one+	nop+	beq		Lxts_enc6x_two+	cmpwi		$len,0x40+	blt		Lxts_enc6x_three+	nop+	beq		Lxts_enc6x_four++Lxts_enc6x_five:+	vxor		$out0,$in1,$twk0+	vxor		$out1,$in2,$twk1+	vxor		$out2,$in3,$twk2+	vxor		$out3,$in4,$twk3+	vxor		$out4,$in5,$twk4++	bl		_aesp8_xts_enc5x++	le?vperm	$out0,$out0,$out0,$leperm+	vmr		$twk0,$twk5		# unused tweak+	le?vperm	$out1,$out1,$out1,$leperm+	stvx_u		$out0,$x00,$out		# store output+	le?vperm	$out2,$out2,$out2,$leperm+	stvx_u		$out1,$x10,$out+	le?vperm	$out3,$out3,$out3,$leperm+	stvx_u		$out2,$x20,$out+	vxor		$tmp,$out4,$twk5	# last block prep for stealing+	le?vperm	$out4,$out4,$out4,$leperm+	stvx_u		$out3,$x30,$out+	stvx_u		$out4,$x40,$out+	addi		$out,$out,0x50+	bne		Lxts_enc6x_steal+	b		Lxts_enc6x_done++.align	4+Lxts_enc6x_four:+	vxor		$out0,$in2,$twk0+	vxor		$out1,$in3,$twk1+	vxor		$out2,$in4,$twk2+	vxor		$out3,$in5,$twk3+	vxor		$out4,$out4,$out4++	bl		_aesp8_xts_enc5x++	le?vperm	$out0,$out0,$out0,$leperm+	vmr		$twk0,$twk4		# unused tweak+	le?vperm	$out1,$out1,$out1,$leperm+	stvx_u		$out0,$x00,$out		# store output+	le?vperm	$out2,$out2,$out2,$leperm+	stvx_u		$out1,$x10,$out+	vxor		$tmp,$out3,$twk4	# last block prep for stealing+	le?vperm	$out3,$out3,$out3,$leperm+	stvx_u		$out2,$x20,$out+	stvx_u		$out3,$x30,$out+	addi		$out,$out,0x40+	bne		Lxts_enc6x_steal+	b		Lxts_enc6x_done++.align	4+Lxts_enc6x_three:+	vxor		$out0,$in3,$twk0+	vxor		$out1,$in4,$twk1+	vxor		$out2,$in5,$twk2+	vxor		$out3,$out3,$out3+	vxor		$out4,$out4,$out4++	bl		_aesp8_xts_enc5x++	le?vperm	$out0,$out0,$out0,$leperm+	vmr		$twk0,$twk3		# unused tweak+	le?vperm	$out1,$out1,$out1,$leperm+	stvx_u		$out0,$x00,$out		# store output+	vxor		$tmp,$out2,$twk3	# last block prep for stealing+	le?vperm	$out2,$out2,$out2,$leperm+	stvx_u		$out1,$x10,$out+	stvx_u		$out2,$x20,$out+	addi		$out,$out,0x30+	bne		Lxts_enc6x_steal+	b		Lxts_enc6x_done++.align	4+Lxts_enc6x_two:+	vxor		$out0,$in4,$twk0+	vxor		$out1,$in5,$twk1+	vxor		$out2,$out2,$out2+	vxor		$out3,$out3,$out3+	vxor		$out4,$out4,$out4++	bl		_aesp8_xts_enc5x++	le?vperm	$out0,$out0,$out0,$leperm+	vmr		$twk0,$twk2		# unused tweak+	vxor		$tmp,$out1,$twk2	# last block prep for stealing+	le?vperm	$out1,$out1,$out1,$leperm+	stvx_u		$out0,$x00,$out		# store output+	stvx_u		$out1,$x10,$out+	addi		$out,$out,0x20+	bne		Lxts_enc6x_steal+	b		Lxts_enc6x_done++.align	4+Lxts_enc6x_one:+	vxor		$out0,$in5,$twk0+	nop+Loop_xts_enc1x:+	vcipher		$out0,$out0,v24+	lvx		v24,$x20,$key_		# round[3]+	addi		$key_,$key_,0x20++	vcipher		$out0,$out0,v25+	lvx		v25,$x10,$key_		# round[4]+	bdnz		Loop_xts_enc1x++	add		$inp,$inp,$taillen+	cmpwi		$taillen,0+	vcipher		$out0,$out0,v24++	subi		$inp,$inp,16+	vcipher		$out0,$out0,v25++	lvsr		$inpperm,0,$taillen+	vcipher		$out0,$out0,v26++	lvx_u		$in0,0,$inp+	vcipher		$out0,$out0,v27++	addi		$key_,$sp,$FRAME+15	# rewind $key_+	vcipher		$out0,$out0,v28+	lvx		v24,$x00,$key_		# re-pre-load round[1]++	vcipher		$out0,$out0,v29+	lvx		v25,$x10,$key_		# re-pre-load round[2]+	 vxor		$twk0,$twk0,v31++	le?vperm	$in0,$in0,$in0,$leperm+	vcipher		$out0,$out0,v30++	vperm		$in0,$in0,$in0,$inpperm+	vcipherlast	$out0,$out0,$twk0++	vmr		$twk0,$twk1		# unused tweak+	vxor		$tmp,$out0,$twk1	# last block prep for stealing+	le?vperm	$out0,$out0,$out0,$leperm+	stvx_u		$out0,$x00,$out		# store output+	addi		$out,$out,0x10+	bne		Lxts_enc6x_steal+	b		Lxts_enc6x_done++.align	4+Lxts_enc6x_zero:+	cmpwi		$taillen,0+	beq		Lxts_enc6x_done++	add		$inp,$inp,$taillen+	subi		$inp,$inp,16+	lvx_u		$in0,0,$inp+	lvsr		$inpperm,0,$taillen	# $in5 is no more+	le?vperm	$in0,$in0,$in0,$leperm+	vperm		$in0,$in0,$in0,$inpperm+	vxor		$tmp,$tmp,$twk0+Lxts_enc6x_steal:+	vxor		$in0,$in0,$twk0+	vxor		$out0,$out0,$out0+	vspltisb	$out1,-1+	vperm		$out0,$out0,$out1,$inpperm+	vsel		$out0,$in0,$tmp,$out0	# $tmp is last block, remember?++	subi		r30,$out,17+	subi		$out,$out,16+	mtctr		$taillen+Loop_xts_enc6x_steal:+	lbzu		r0,1(r30)+	stb		r0,16(r30)+	bdnz		Loop_xts_enc6x_steal++	li		$taillen,0+	mtctr		$rounds+	b		Loop_xts_enc1x		# one more time...++.align	4+Lxts_enc6x_done:+	${UCMP}i	$ivp,0+	beq		Lxts_enc6x_ret++	vxor		$tweak,$twk0,$rndkey0+	le?vperm	$tweak,$tweak,$tweak,$leperm+	stvx_u		$tweak,0,$ivp++Lxts_enc6x_ret:+	mtlr		r11+	li		r10,`$FRAME+15`+	li		r11,`$FRAME+31`+	stvx		$seven,r10,$sp		# wipe copies of round keys+	addi		r10,r10,32+	stvx		$seven,r11,$sp+	addi		r11,r11,32+	stvx		$seven,r10,$sp+	addi		r10,r10,32+	stvx		$seven,r11,$sp+	addi		r11,r11,32+	stvx		$seven,r10,$sp+	addi		r10,r10,32+	stvx		$seven,r11,$sp+	addi		r11,r11,32+	stvx		$seven,r10,$sp+	addi		r10,r10,32+	stvx		$seven,r11,$sp+	addi		r11,r11,32++	mtspr		256,$vrsave+	lvx		v20,r10,$sp		# ABI says so+	addi		r10,r10,32+	lvx		v21,r11,$sp+	addi		r11,r11,32+	lvx		v22,r10,$sp+	addi		r10,r10,32+	lvx		v23,r11,$sp+	addi		r11,r11,32+	lvx		v24,r10,$sp+	addi		r10,r10,32+	lvx		v25,r11,$sp+	addi		r11,r11,32+	lvx		v26,r10,$sp+	addi		r10,r10,32+	lvx		v27,r11,$sp+	addi		r11,r11,32+	lvx		v28,r10,$sp+	addi		r10,r10,32+	lvx		v29,r11,$sp+	addi		r11,r11,32+	lvx		v30,r10,$sp+	lvx		v31,r11,$sp+	$POP		r26,`$FRAME+21*16+0*$SIZE_T`($sp)+	$POP		r27,`$FRAME+21*16+1*$SIZE_T`($sp)+	$POP		r28,`$FRAME+21*16+2*$SIZE_T`($sp)+	$POP		r29,`$FRAME+21*16+3*$SIZE_T`($sp)+	$POP		r30,`$FRAME+21*16+4*$SIZE_T`($sp)+	$POP		r31,`$FRAME+21*16+5*$SIZE_T`($sp)+	addi		$sp,$sp,`$FRAME+21*16+6*$SIZE_T`+	blr+	.long		0+	.byte		0,12,0x04,1,0x80,6,6,0+	.long		0++.align	5+_aesp8_xts_enc5x:+	vcipher		$out0,$out0,v24+	vcipher		$out1,$out1,v24+	vcipher		$out2,$out2,v24+	vcipher		$out3,$out3,v24+	vcipher		$out4,$out4,v24+	lvx		v24,$x20,$key_		# round[3]+	addi		$key_,$key_,0x20++	vcipher		$out0,$out0,v25+	vcipher		$out1,$out1,v25+	vcipher		$out2,$out2,v25+	vcipher		$out3,$out3,v25+	vcipher		$out4,$out4,v25+	lvx		v25,$x10,$key_		# round[4]+	bdnz		_aesp8_xts_enc5x++	add		$inp,$inp,$taillen+	cmpwi		$taillen,0+	vcipher		$out0,$out0,v24+	vcipher		$out1,$out1,v24+	vcipher		$out2,$out2,v24+	vcipher		$out3,$out3,v24+	vcipher		$out4,$out4,v24++	subi		$inp,$inp,16+	vcipher		$out0,$out0,v25+	vcipher		$out1,$out1,v25+	vcipher		$out2,$out2,v25+	vcipher		$out3,$out3,v25+	vcipher		$out4,$out4,v25+	 vxor		$twk0,$twk0,v31++	vcipher		$out0,$out0,v26+	lvsr		$inpperm,0,$taillen	# $in5 is no more+	vcipher		$out1,$out1,v26+	vcipher		$out2,$out2,v26+	vcipher		$out3,$out3,v26+	vcipher		$out4,$out4,v26+	 vxor		$in1,$twk1,v31++	vcipher		$out0,$out0,v27+	lvx_u		$in0,0,$inp+	vcipher		$out1,$out1,v27+	vcipher		$out2,$out2,v27+	vcipher		$out3,$out3,v27+	vcipher		$out4,$out4,v27+	 vxor		$in2,$twk2,v31++	addi		$key_,$sp,$FRAME+15	# rewind $key_+	vcipher		$out0,$out0,v28+	vcipher		$out1,$out1,v28+	vcipher		$out2,$out2,v28+	vcipher		$out3,$out3,v28+	vcipher		$out4,$out4,v28+	lvx		v24,$x00,$key_		# re-pre-load round[1]+	 vxor		$in3,$twk3,v31++	vcipher		$out0,$out0,v29+	le?vperm	$in0,$in0,$in0,$leperm+	vcipher		$out1,$out1,v29+	vcipher		$out2,$out2,v29+	vcipher		$out3,$out3,v29+	vcipher		$out4,$out4,v29+	lvx		v25,$x10,$key_		# re-pre-load round[2]+	 vxor		$in4,$twk4,v31++	vcipher		$out0,$out0,v30+	vperm		$in0,$in0,$in0,$inpperm+	vcipher		$out1,$out1,v30+	vcipher		$out2,$out2,v30+	vcipher		$out3,$out3,v30+	vcipher		$out4,$out4,v30++	vcipherlast	$out0,$out0,$twk0+	vcipherlast	$out1,$out1,$in1+	vcipherlast	$out2,$out2,$in2+	vcipherlast	$out3,$out3,$in3+	vcipherlast	$out4,$out4,$in4+	blr+        .long   	0+        .byte   	0,12,0x14,0,0,0,0,0++.align	5+_aesp8_xts_decrypt6x:+	$STU		$sp,-`($FRAME+21*16+6*$SIZE_T)`($sp)+	mflr		r11+	li		r7,`$FRAME+8*16+15`+	li		r3,`$FRAME+8*16+31`+	$PUSH		r11,`$FRAME+21*16+6*$SIZE_T+$LRSAVE`($sp)+	stvx		v20,r7,$sp		# ABI says so+	addi		r7,r7,32+	stvx		v21,r3,$sp+	addi		r3,r3,32+	stvx		v22,r7,$sp+	addi		r7,r7,32+	stvx		v23,r3,$sp+	addi		r3,r3,32+	stvx		v24,r7,$sp+	addi		r7,r7,32+	stvx		v25,r3,$sp+	addi		r3,r3,32+	stvx		v26,r7,$sp+	addi		r7,r7,32+	stvx		v27,r3,$sp+	addi		r3,r3,32+	stvx		v28,r7,$sp+	addi		r7,r7,32+	stvx		v29,r3,$sp+	addi		r3,r3,32+	stvx		v30,r7,$sp+	stvx		v31,r3,$sp+	li		r0,-1+	stw		$vrsave,`$FRAME+21*16-4`($sp)	# save vrsave+	li		$x10,0x10+	$PUSH		r26,`$FRAME+21*16+0*$SIZE_T`($sp)+	li		$x20,0x20+	$PUSH		r27,`$FRAME+21*16+1*$SIZE_T`($sp)+	li		$x30,0x30+	$PUSH		r28,`$FRAME+21*16+2*$SIZE_T`($sp)+	li		$x40,0x40+	$PUSH		r29,`$FRAME+21*16+3*$SIZE_T`($sp)+	li		$x50,0x50+	$PUSH		r30,`$FRAME+21*16+4*$SIZE_T`($sp)+	li		$x60,0x60+	$PUSH		r31,`$FRAME+21*16+5*$SIZE_T`($sp)+	li		$x70,0x70+	mtspr		256,r0++	subi		$rounds,$rounds,3	# -4 in total++	lvx		$rndkey0,$x00,$key1	# load key schedule+	lvx		v30,$x10,$key1+	addi		$key1,$key1,0x20+	lvx		v31,$x00,$key1+	?vperm		$rndkey0,$rndkey0,v30,$keyperm+	addi		$key_,$sp,$FRAME+15+	mtctr		$rounds++Load_xts_dec_key:+	?vperm		v24,v30,v31,$keyperm+	lvx		v30,$x10,$key1+	addi		$key1,$key1,0x20+	stvx		v24,$x00,$key_		# off-load round[1]+	?vperm		v25,v31,v30,$keyperm+	lvx		v31,$x00,$key1+	stvx		v25,$x10,$key_		# off-load round[2]+	addi		$key_,$key_,0x20+	bdnz		Load_xts_dec_key++	lvx		v26,$x10,$key1+	?vperm		v24,v30,v31,$keyperm+	lvx		v27,$x20,$key1+	stvx		v24,$x00,$key_		# off-load round[3]+	?vperm		v25,v31,v26,$keyperm+	lvx		v28,$x30,$key1+	stvx		v25,$x10,$key_		# off-load round[4]+	addi		$key_,$sp,$FRAME+15	# rewind $key_+	?vperm		v26,v26,v27,$keyperm+	lvx		v29,$x40,$key1+	?vperm		v27,v27,v28,$keyperm+	lvx		v30,$x50,$key1+	?vperm		v28,v28,v29,$keyperm+	lvx		v31,$x60,$key1+	?vperm		v29,v29,v30,$keyperm+	lvx		$twk5,$x70,$key1	# borrow $twk5+	?vperm		v30,v30,v31,$keyperm+	lvx		v24,$x00,$key_		# pre-load round[1]+	?vperm		v31,v31,$twk5,$keyperm+	lvx		v25,$x10,$key_		# pre-load round[2]++	 vperm		$in0,$inout,$inptail,$inpperm+	 subi		$inp,$inp,31		# undo "caller"+	vxor		$twk0,$tweak,$rndkey0+	vsrab		$tmp,$tweak,$seven	# next tweak value+	vaddubm		$tweak,$tweak,$tweak+	vsldoi		$tmp,$tmp,$tmp,15+	vand		$tmp,$tmp,$eighty7+	 vxor		$out0,$in0,$twk0+	vxor		$tweak,$tweak,$tmp++	 lvx_u		$in1,$x10,$inp+	vxor		$twk1,$tweak,$rndkey0+	vsrab		$tmp,$tweak,$seven	# next tweak value+	vaddubm		$tweak,$tweak,$tweak+	vsldoi		$tmp,$tmp,$tmp,15+	 le?vperm	$in1,$in1,$in1,$leperm+	vand		$tmp,$tmp,$eighty7+	 vxor		$out1,$in1,$twk1+	vxor		$tweak,$tweak,$tmp++	 lvx_u		$in2,$x20,$inp+	 andi.		$taillen,$len,15+	vxor		$twk2,$tweak,$rndkey0+	vsrab		$tmp,$tweak,$seven	# next tweak value+	vaddubm		$tweak,$tweak,$tweak+	vsldoi		$tmp,$tmp,$tmp,15+	 le?vperm	$in2,$in2,$in2,$leperm+	vand		$tmp,$tmp,$eighty7+	 vxor		$out2,$in2,$twk2+	vxor		$tweak,$tweak,$tmp++	 lvx_u		$in3,$x30,$inp+	 sub		$len,$len,$taillen+	vxor		$twk3,$tweak,$rndkey0+	vsrab		$tmp,$tweak,$seven	# next tweak value+	vaddubm		$tweak,$tweak,$tweak+	vsldoi		$tmp,$tmp,$tmp,15+	 le?vperm	$in3,$in3,$in3,$leperm+	vand		$tmp,$tmp,$eighty7+	 vxor		$out3,$in3,$twk3+	vxor		$tweak,$tweak,$tmp++	 lvx_u		$in4,$x40,$inp+	 subi		$len,$len,0x60+	vxor		$twk4,$tweak,$rndkey0+	vsrab		$tmp,$tweak,$seven	# next tweak value+	vaddubm		$tweak,$tweak,$tweak+	vsldoi		$tmp,$tmp,$tmp,15+	 le?vperm	$in4,$in4,$in4,$leperm+	vand		$tmp,$tmp,$eighty7+	 vxor		$out4,$in4,$twk4+	vxor		$tweak,$tweak,$tmp++	 lvx_u		$in5,$x50,$inp+	 addi		$inp,$inp,0x60+	vxor		$twk5,$tweak,$rndkey0+	vsrab		$tmp,$tweak,$seven	# next tweak value+	vaddubm		$tweak,$tweak,$tweak+	vsldoi		$tmp,$tmp,$tmp,15+	 le?vperm	$in5,$in5,$in5,$leperm+	vand		$tmp,$tmp,$eighty7+	 vxor		$out5,$in5,$twk5+	vxor		$tweak,$tweak,$tmp++	vxor		v31,v31,$rndkey0+	mtctr		$rounds+	b		Loop_xts_dec6x++.align	5+Loop_xts_dec6x:+	vncipher	$out0,$out0,v24+	vncipher	$out1,$out1,v24+	vncipher	$out2,$out2,v24+	vncipher	$out3,$out3,v24+	vncipher	$out4,$out4,v24+	vncipher	$out5,$out5,v24+	lvx		v24,$x20,$key_		# round[3]+	addi		$key_,$key_,0x20++	vncipher	$out0,$out0,v25+	vncipher	$out1,$out1,v25+	vncipher	$out2,$out2,v25+	vncipher	$out3,$out3,v25+	vncipher	$out4,$out4,v25+	vncipher	$out5,$out5,v25+	lvx		v25,$x10,$key_		# round[4]+	bdnz		Loop_xts_dec6x++	subic		$len,$len,96		# $len-=96+	 vxor		$in0,$twk0,v31		# xor with last round key+	vncipher	$out0,$out0,v24+	vncipher	$out1,$out1,v24+	 vsrab		$tmp,$tweak,$seven	# next tweak value+	 vxor		$twk0,$tweak,$rndkey0+	 vaddubm	$tweak,$tweak,$tweak+	vncipher	$out2,$out2,v24+	vncipher	$out3,$out3,v24+	 vsldoi		$tmp,$tmp,$tmp,15+	vncipher	$out4,$out4,v24+	vncipher	$out5,$out5,v24++	subfe.		r0,r0,r0		# borrow?-1:0+	 vand		$tmp,$tmp,$eighty7+	vncipher	$out0,$out0,v25+	vncipher	$out1,$out1,v25+	 vxor		$tweak,$tweak,$tmp+	vncipher	$out2,$out2,v25+	vncipher	$out3,$out3,v25+	 vxor		$in1,$twk1,v31+	 vsrab		$tmp,$tweak,$seven	# next tweak value+	 vxor		$twk1,$tweak,$rndkey0+	vncipher	$out4,$out4,v25+	vncipher	$out5,$out5,v25++	and		r0,r0,$len+	 vaddubm	$tweak,$tweak,$tweak+	 vsldoi		$tmp,$tmp,$tmp,15+	vncipher	$out0,$out0,v26+	vncipher	$out1,$out1,v26+	 vand		$tmp,$tmp,$eighty7+	vncipher	$out2,$out2,v26+	vncipher	$out3,$out3,v26+	 vxor		$tweak,$tweak,$tmp+	vncipher	$out4,$out4,v26+	vncipher	$out5,$out5,v26++	add		$inp,$inp,r0		# $inp is adjusted in such+						# way that at exit from the+						# loop inX-in5 are loaded+						# with last "words"+	 vxor		$in2,$twk2,v31+	 vsrab		$tmp,$tweak,$seven	# next tweak value+	 vxor		$twk2,$tweak,$rndkey0+	 vaddubm	$tweak,$tweak,$tweak+	vncipher	$out0,$out0,v27+	vncipher	$out1,$out1,v27+	 vsldoi		$tmp,$tmp,$tmp,15+	vncipher	$out2,$out2,v27+	vncipher	$out3,$out3,v27+	 vand		$tmp,$tmp,$eighty7+	vncipher	$out4,$out4,v27+	vncipher	$out5,$out5,v27++	addi		$key_,$sp,$FRAME+15	# rewind $key_+	 vxor		$tweak,$tweak,$tmp+	vncipher	$out0,$out0,v28+	vncipher	$out1,$out1,v28+	 vxor		$in3,$twk3,v31+	 vsrab		$tmp,$tweak,$seven	# next tweak value+	 vxor		$twk3,$tweak,$rndkey0+	vncipher	$out2,$out2,v28+	vncipher	$out3,$out3,v28+	 vaddubm	$tweak,$tweak,$tweak+	 vsldoi		$tmp,$tmp,$tmp,15+	vncipher	$out4,$out4,v28+	vncipher	$out5,$out5,v28+	lvx		v24,$x00,$key_		# re-pre-load round[1]+	 vand		$tmp,$tmp,$eighty7++	vncipher	$out0,$out0,v29+	vncipher	$out1,$out1,v29+	 vxor		$tweak,$tweak,$tmp+	vncipher	$out2,$out2,v29+	vncipher	$out3,$out3,v29+	 vxor		$in4,$twk4,v31+	 vsrab		$tmp,$tweak,$seven	# next tweak value+	 vxor		$twk4,$tweak,$rndkey0+	vncipher	$out4,$out4,v29+	vncipher	$out5,$out5,v29+	lvx		v25,$x10,$key_		# re-pre-load round[2]+	 vaddubm	$tweak,$tweak,$tweak+	 vsldoi		$tmp,$tmp,$tmp,15++	vncipher	$out0,$out0,v30+	vncipher	$out1,$out1,v30+	 vand		$tmp,$tmp,$eighty7+	vncipher	$out2,$out2,v30+	vncipher	$out3,$out3,v30+	 vxor		$tweak,$tweak,$tmp+	vncipher	$out4,$out4,v30+	vncipher	$out5,$out5,v30+	 vxor		$in5,$twk5,v31+	 vsrab		$tmp,$tweak,$seven	# next tweak value+	 vxor		$twk5,$tweak,$rndkey0++	vncipherlast	$out0,$out0,$in0+	 lvx_u		$in0,$x00,$inp		# load next input block+	 vaddubm	$tweak,$tweak,$tweak+	 vsldoi		$tmp,$tmp,$tmp,15+	vncipherlast	$out1,$out1,$in1+	 lvx_u		$in1,$x10,$inp+	vncipherlast	$out2,$out2,$in2+	 le?vperm	$in0,$in0,$in0,$leperm+	 lvx_u		$in2,$x20,$inp+	 vand		$tmp,$tmp,$eighty7+	vncipherlast	$out3,$out3,$in3+	 le?vperm	$in1,$in1,$in1,$leperm+	 lvx_u		$in3,$x30,$inp+	vncipherlast	$out4,$out4,$in4+	 le?vperm	$in2,$in2,$in2,$leperm+	 lvx_u		$in4,$x40,$inp+	 vxor		$tweak,$tweak,$tmp+	vncipherlast	$out5,$out5,$in5+	 le?vperm	$in3,$in3,$in3,$leperm+	 lvx_u		$in5,$x50,$inp+	 addi		$inp,$inp,0x60+	 le?vperm	$in4,$in4,$in4,$leperm+	 le?vperm	$in5,$in5,$in5,$leperm++	le?vperm	$out0,$out0,$out0,$leperm+	le?vperm	$out1,$out1,$out1,$leperm+	stvx_u		$out0,$x00,$out		# store output+	 vxor		$out0,$in0,$twk0+	le?vperm	$out2,$out2,$out2,$leperm+	stvx_u		$out1,$x10,$out+	 vxor		$out1,$in1,$twk1+	le?vperm	$out3,$out3,$out3,$leperm+	stvx_u		$out2,$x20,$out+	 vxor		$out2,$in2,$twk2+	le?vperm	$out4,$out4,$out4,$leperm+	stvx_u		$out3,$x30,$out+	 vxor		$out3,$in3,$twk3+	le?vperm	$out5,$out5,$out5,$leperm+	stvx_u		$out4,$x40,$out+	 vxor		$out4,$in4,$twk4+	stvx_u		$out5,$x50,$out+	 vxor		$out5,$in5,$twk5+	addi		$out,$out,0x60++	mtctr		$rounds+	beq		Loop_xts_dec6x		# did $len-=96 borrow?++	addic.		$len,$len,0x60+	beq		Lxts_dec6x_zero+	cmpwi		$len,0x20+	blt		Lxts_dec6x_one+	nop+	beq		Lxts_dec6x_two+	cmpwi		$len,0x40+	blt		Lxts_dec6x_three+	nop+	beq		Lxts_dec6x_four++Lxts_dec6x_five:+	vxor		$out0,$in1,$twk0+	vxor		$out1,$in2,$twk1+	vxor		$out2,$in3,$twk2+	vxor		$out3,$in4,$twk3+	vxor		$out4,$in5,$twk4++	bl		_aesp8_xts_dec5x++	le?vperm	$out0,$out0,$out0,$leperm+	vmr		$twk0,$twk5		# unused tweak+	vxor		$twk1,$tweak,$rndkey0+	le?vperm	$out1,$out1,$out1,$leperm+	stvx_u		$out0,$x00,$out		# store output+	vxor		$out0,$in0,$twk1+	le?vperm	$out2,$out2,$out2,$leperm+	stvx_u		$out1,$x10,$out+	le?vperm	$out3,$out3,$out3,$leperm+	stvx_u		$out2,$x20,$out+	le?vperm	$out4,$out4,$out4,$leperm+	stvx_u		$out3,$x30,$out+	stvx_u		$out4,$x40,$out+	addi		$out,$out,0x50+	bne		Lxts_dec6x_steal+	b		Lxts_dec6x_done++.align	4+Lxts_dec6x_four:+	vxor		$out0,$in2,$twk0+	vxor		$out1,$in3,$twk1+	vxor		$out2,$in4,$twk2+	vxor		$out3,$in5,$twk3+	vxor		$out4,$out4,$out4++	bl		_aesp8_xts_dec5x++	le?vperm	$out0,$out0,$out0,$leperm+	vmr		$twk0,$twk4		# unused tweak+	vmr		$twk1,$twk5+	le?vperm	$out1,$out1,$out1,$leperm+	stvx_u		$out0,$x00,$out		# store output+	vxor		$out0,$in0,$twk5+	le?vperm	$out2,$out2,$out2,$leperm+	stvx_u		$out1,$x10,$out+	le?vperm	$out3,$out3,$out3,$leperm+	stvx_u		$out2,$x20,$out+	stvx_u		$out3,$x30,$out+	addi		$out,$out,0x40+	bne		Lxts_dec6x_steal+	b		Lxts_dec6x_done++.align	4+Lxts_dec6x_three:+	vxor		$out0,$in3,$twk0+	vxor		$out1,$in4,$twk1+	vxor		$out2,$in5,$twk2+	vxor		$out3,$out3,$out3+	vxor		$out4,$out4,$out4++	bl		_aesp8_xts_dec5x++	le?vperm	$out0,$out0,$out0,$leperm+	vmr		$twk0,$twk3		# unused tweak+	vmr		$twk1,$twk4+	le?vperm	$out1,$out1,$out1,$leperm+	stvx_u		$out0,$x00,$out		# store output+	vxor		$out0,$in0,$twk4+	le?vperm	$out2,$out2,$out2,$leperm+	stvx_u		$out1,$x10,$out+	stvx_u		$out2,$x20,$out+	addi		$out,$out,0x30+	bne		Lxts_dec6x_steal+	b		Lxts_dec6x_done++.align	4+Lxts_dec6x_two:+	vxor		$out0,$in4,$twk0+	vxor		$out1,$in5,$twk1+	vxor		$out2,$out2,$out2+	vxor		$out3,$out3,$out3+	vxor		$out4,$out4,$out4++	bl		_aesp8_xts_dec5x++	le?vperm	$out0,$out0,$out0,$leperm+	vmr		$twk0,$twk2		# unused tweak+	vmr		$twk1,$twk3+	le?vperm	$out1,$out1,$out1,$leperm+	stvx_u		$out0,$x00,$out		# store output+	vxor		$out0,$in0,$twk3+	stvx_u		$out1,$x10,$out+	addi		$out,$out,0x20+	bne		Lxts_dec6x_steal+	b		Lxts_dec6x_done++.align	4+Lxts_dec6x_one:+	vxor		$out0,$in5,$twk0+	nop+Loop_xts_dec1x:+	vncipher	$out0,$out0,v24+	lvx		v24,$x20,$key_		# round[3]+	addi		$key_,$key_,0x20++	vncipher	$out0,$out0,v25+	lvx		v25,$x10,$key_		# round[4]+	bdnz		Loop_xts_dec1x++	subi		r0,$taillen,1+	vncipher	$out0,$out0,v24++	andi.		r0,r0,16+	cmpwi		$taillen,0+	vncipher	$out0,$out0,v25++	sub		$inp,$inp,r0+	vncipher	$out0,$out0,v26++	lvx_u		$in0,0,$inp+	vncipher	$out0,$out0,v27++	addi		$key_,$sp,$FRAME+15	# rewind $key_+	vncipher	$out0,$out0,v28+	lvx		v24,$x00,$key_		# re-pre-load round[1]++	vncipher	$out0,$out0,v29+	lvx		v25,$x10,$key_		# re-pre-load round[2]+	 vxor		$twk0,$twk0,v31++	le?vperm	$in0,$in0,$in0,$leperm+	vncipher	$out0,$out0,v30++	mtctr		$rounds+	vncipherlast	$out0,$out0,$twk0++	vmr		$twk0,$twk1		# unused tweak+	vmr		$twk1,$twk2+	le?vperm	$out0,$out0,$out0,$leperm+	stvx_u		$out0,$x00,$out		# store output+	addi		$out,$out,0x10+	vxor		$out0,$in0,$twk2+	bne		Lxts_dec6x_steal+	b		Lxts_dec6x_done++.align	4+Lxts_dec6x_zero:+	cmpwi		$taillen,0+	beq		Lxts_dec6x_done++	lvx_u		$in0,0,$inp+	le?vperm	$in0,$in0,$in0,$leperm+	vxor		$out0,$in0,$twk1+Lxts_dec6x_steal:+	vncipher	$out0,$out0,v24+	lvx		v24,$x20,$key_		# round[3]+	addi		$key_,$key_,0x20++	vncipher	$out0,$out0,v25+	lvx		v25,$x10,$key_		# round[4]+	bdnz		Lxts_dec6x_steal++	add		$inp,$inp,$taillen+	vncipher	$out0,$out0,v24++	cmpwi		$taillen,0+	vncipher	$out0,$out0,v25++	lvx_u		$in0,0,$inp+	vncipher	$out0,$out0,v26++	lvsr		$inpperm,0,$taillen	# $in5 is no more+	vncipher	$out0,$out0,v27++	addi		$key_,$sp,$FRAME+15	# rewind $key_+	vncipher	$out0,$out0,v28+	lvx		v24,$x00,$key_		# re-pre-load round[1]++	vncipher	$out0,$out0,v29+	lvx		v25,$x10,$key_		# re-pre-load round[2]+	 vxor		$twk1,$twk1,v31++	le?vperm	$in0,$in0,$in0,$leperm+	vncipher	$out0,$out0,v30++	vperm		$in0,$in0,$in0,$inpperm+	vncipherlast	$tmp,$out0,$twk1++	le?vperm	$out0,$tmp,$tmp,$leperm+	le?stvx_u	$out0,0,$out+	be?stvx_u	$tmp,0,$out++	vxor		$out0,$out0,$out0+	vspltisb	$out1,-1+	vperm		$out0,$out0,$out1,$inpperm+	vsel		$out0,$in0,$tmp,$out0+	vxor		$out0,$out0,$twk0++	subi		r30,$out,1+	mtctr		$taillen+Loop_xts_dec6x_steal:+	lbzu		r0,1(r30)+	stb		r0,16(r30)+	bdnz		Loop_xts_dec6x_steal++	li		$taillen,0+	mtctr		$rounds+	b		Loop_xts_dec1x		# one more time...++.align	4+Lxts_dec6x_done:+	${UCMP}i	$ivp,0+	beq		Lxts_dec6x_ret++	vxor		$tweak,$twk0,$rndkey0+	le?vperm	$tweak,$tweak,$tweak,$leperm+	stvx_u		$tweak,0,$ivp++Lxts_dec6x_ret:+	mtlr		r11+	li		r10,`$FRAME+15`+	li		r11,`$FRAME+31`+	stvx		$seven,r10,$sp		# wipe copies of round keys+	addi		r10,r10,32+	stvx		$seven,r11,$sp+	addi		r11,r11,32+	stvx		$seven,r10,$sp+	addi		r10,r10,32+	stvx		$seven,r11,$sp+	addi		r11,r11,32+	stvx		$seven,r10,$sp+	addi		r10,r10,32+	stvx		$seven,r11,$sp+	addi		r11,r11,32+	stvx		$seven,r10,$sp+	addi		r10,r10,32+	stvx		$seven,r11,$sp+	addi		r11,r11,32++	mtspr		256,$vrsave+	lvx		v20,r10,$sp		# ABI says so+	addi		r10,r10,32+	lvx		v21,r11,$sp+	addi		r11,r11,32+	lvx		v22,r10,$sp+	addi		r10,r10,32+	lvx		v23,r11,$sp+	addi		r11,r11,32+	lvx		v24,r10,$sp+	addi		r10,r10,32+	lvx		v25,r11,$sp+	addi		r11,r11,32+	lvx		v26,r10,$sp+	addi		r10,r10,32+	lvx		v27,r11,$sp+	addi		r11,r11,32+	lvx		v28,r10,$sp+	addi		r10,r10,32+	lvx		v29,r11,$sp+	addi		r11,r11,32+	lvx		v30,r10,$sp+	lvx		v31,r11,$sp+	$POP		r26,`$FRAME+21*16+0*$SIZE_T`($sp)+	$POP		r27,`$FRAME+21*16+1*$SIZE_T`($sp)+	$POP		r28,`$FRAME+21*16+2*$SIZE_T`($sp)+	$POP		r29,`$FRAME+21*16+3*$SIZE_T`($sp)+	$POP		r30,`$FRAME+21*16+4*$SIZE_T`($sp)+	$POP		r31,`$FRAME+21*16+5*$SIZE_T`($sp)+	addi		$sp,$sp,`$FRAME+21*16+6*$SIZE_T`+	blr+	.long		0+	.byte		0,12,0x04,1,0x80,6,6,0+	.long		0++.align	5+_aesp8_xts_dec5x:+	vncipher	$out0,$out0,v24+	vncipher	$out1,$out1,v24+	vncipher	$out2,$out2,v24+	vncipher	$out3,$out3,v24+	vncipher	$out4,$out4,v24+	lvx		v24,$x20,$key_		# round[3]+	addi		$key_,$key_,0x20++	vncipher	$out0,$out0,v25+	vncipher	$out1,$out1,v25+	vncipher	$out2,$out2,v25+	vncipher	$out3,$out3,v25+	vncipher	$out4,$out4,v25+	lvx		v25,$x10,$key_		# round[4]+	bdnz		_aesp8_xts_dec5x++	subi		r0,$taillen,1+	vncipher	$out0,$out0,v24+	vncipher	$out1,$out1,v24+	vncipher	$out2,$out2,v24+	vncipher	$out3,$out3,v24+	vncipher	$out4,$out4,v24++	andi.		r0,r0,16+	cmpwi		$taillen,0+	vncipher	$out0,$out0,v25+	vncipher	$out1,$out1,v25+	vncipher	$out2,$out2,v25+	vncipher	$out3,$out3,v25+	vncipher	$out4,$out4,v25+	 vxor		$twk0,$twk0,v31++	sub		$inp,$inp,r0+	vncipher	$out0,$out0,v26+	vncipher	$out1,$out1,v26+	vncipher	$out2,$out2,v26+	vncipher	$out3,$out3,v26+	vncipher	$out4,$out4,v26+	 vxor		$in1,$twk1,v31++	vncipher	$out0,$out0,v27+	lvx_u		$in0,0,$inp+	vncipher	$out1,$out1,v27+	vncipher	$out2,$out2,v27+	vncipher	$out3,$out3,v27+	vncipher	$out4,$out4,v27+	 vxor		$in2,$twk2,v31++	addi		$key_,$sp,$FRAME+15	# rewind $key_+	vncipher	$out0,$out0,v28+	vncipher	$out1,$out1,v28+	vncipher	$out2,$out2,v28+	vncipher	$out3,$out3,v28+	vncipher	$out4,$out4,v28+	lvx		v24,$x00,$key_		# re-pre-load round[1]+	 vxor		$in3,$twk3,v31++	vncipher	$out0,$out0,v29+	le?vperm	$in0,$in0,$in0,$leperm+	vncipher	$out1,$out1,v29+	vncipher	$out2,$out2,v29+	vncipher	$out3,$out3,v29+	vncipher	$out4,$out4,v29+	lvx		v25,$x10,$key_		# re-pre-load round[2]+	 vxor		$in4,$twk4,v31++	vncipher	$out0,$out0,v30+	vncipher	$out1,$out1,v30+	vncipher	$out2,$out2,v30+	vncipher	$out3,$out3,v30+	vncipher	$out4,$out4,v30++	vncipherlast	$out0,$out0,$twk0+	vncipherlast	$out1,$out1,$in1+	vncipherlast	$out2,$out2,$in2+	vncipherlast	$out3,$out3,$in3+	vncipherlast	$out4,$out4,$in4+	mtctr		$rounds+	blr+        .long   	0+        .byte   	0,12,0x14,0,0,0,0,0+___+}}	}}}++my $consts=1;+foreach(split("\n",$code)) {+        s/\`([^\`]*)\`/eval($1)/geo;++	# constants table endian-specific conversion+	if ($consts && m/\.(long|byte)\s+(.+)\s+(\?[a-z]*)$/o) {+	    my $conv=$3;+	    my @bytes=();++	    # convert to endian-agnostic format+	    if ($1 eq "long") {+	      foreach (split(/,\s*/,$2)) {+		my $l = /^0/?oct:int;+		push @bytes,($l>>24)&0xff,($l>>16)&0xff,($l>>8)&0xff,$l&0xff;+	      }+	    } else {+		@bytes = map(/^0/?oct:int,split(/,\s*/,$2));+	    }++	    # little-endian conversion+	    if ($flavour =~ /le$/o) {+		SWITCH: for($conv)  {+		    /\?inv/ && do   { @bytes=map($_^0xf,@bytes); last; };+		    /\?rev/ && do   { @bytes=reverse(@bytes);    last; };+		}+	    }++	    #emit+	    print ".byte\t",join(',',map (sprintf("0x%02x",$_),@bytes)),"\n";+	    next;+	}+	$consts=0 if (m/Lconsts:/o);	# end of table++	# instructions prefixed with '?' are endian-specific and need+	# to be adjusted accordingly...+	if ($flavour =~ /le$/o) {	# little-endian+	    s/le\?//o		or+	    s/be\?/#be#/o	or+	    s/\?lvsr/lvsl/o	or+	    s/\?lvsl/lvsr/o	or+	    s/\?(vperm\s+v[0-9]+,\s*)(v[0-9]+,\s*)(v[0-9]+,\s*)(v[0-9]+)/$1$3$2$4/o or+	    s/\?(vsldoi\s+v[0-9]+,\s*)(v[0-9]+,)\s*(v[0-9]+,\s*)([0-9]+)/$1$3$2 16-$4/o or+	    s/\?(vspltw\s+v[0-9]+,\s*)(v[0-9]+,)\s*([0-9])/$1$2 3-$3/o;+	} else {			# big-endian+	    s/le\?/#le#/o	or+	    s/be\?//o		or+	    s/\?([a-z]+)/$1/o;+	}++        print $_,"\n";+}++close STDOUT;
cbits/asm/generate.sh view
@@ -12,6 +12,8 @@ #     arm/sha1-armv8.pl		arm/sha512-armv8.pl #     arm/keccak1600-armv8.pl	arm/arm-xlate.pl #     arm/arm_arch.h+#     ppc/aesp8-ppc.pl		ppc/ghashp8-ppc.pl+#     ppc/ppc-xlate.pl # # The .pl files are the generator, not the product: each one emits # assembly for a given "flavour", which is the calling convention and the@@ -150,4 +152,25 @@  	.section	.note.GNU-stack,"",%progbits 	NOTE+done++# POWER8's AES and GHASH, little-endian only.  The generator emits a+# big-endian flavour from the same source, and crypton does not build for+# big-endian POWER: nothing here has been able to run that code, and an AES+# path that has never been executed is not one to check in.  Adding+# "linux64" to the list below is what it would take.+#+# These two read no capability word of their own -- unlike the ARM and x86-64+# modules above -- so the entry points are all that is renamed.+for flavour in linux64le; do+	perl aesp8-ppc.pl $flavour tmp-$flavour.S+	sed -e 's/\baes_p8_/crypton_aes_p8_/g' \+	    tmp-$flavour.S > aesp8-ppc-$flavour.S++	perl ghashp8-ppc.pl $flavour tmp-$flavour.S+	sed -e 's/\bgcm_init_p8/crypton_gcm_init_p8/g' \+	    -e 's/\bgcm_gmult_p8/crypton_gcm_gmult_p8/g' \+	    -e 's/\bgcm_ghash_p8/crypton_gcm_ghash_p8/g' \+	    tmp-$flavour.S > ghashp8-ppc-$flavour.S+	rm -f tmp-$flavour.S done
+ cbits/asm/ghashp8-ppc-linux64le.S view
@@ -0,0 +1,575 @@+.machine	"any"++.abiversion	2+.text++.globl	crypton_gcm_init_p8+.type	crypton_gcm_init_p8,@function+.align	5+crypton_gcm_init_p8:+.localentry	crypton_gcm_init_p8,0++	li	0,-4096+	li	8,0x10+	li	12,-1+	li	9,0x20+	or	0,0,0+	li	10,0x30+	.long	0x7D202699++	vspltisb	8,-16+	vspltisb	5,1+	vaddubm	8,8,8+	vxor	4,4,4+	vor	8,8,5+	vsldoi	8,8,4,15+	vsldoi	6,4,5,1+	vaddubm	8,8,8+	vspltisb	7,7+	vor	8,8,6+	vspltb	6,9,0+	vsl	9,9,5+	vsrab	6,6,7+	vand	6,6,8+	vxor	3,9,6++	vsldoi	9,3,3,8+	vsldoi	8,4,8,8+	vsldoi	11,4,9,8+	vsldoi	10,9,4,8++	.long	0x7D001F99+	.long	0x7D681F99+	li	8,0x40+	.long	0x7D291F99+	li	9,0x50+	.long	0x7D4A1F99+	li	10,0x60++	.long	0x10035CC8+	.long	0x10234CC8+	.long	0x104354C8++	.long	0x10E044C8++	vsldoi	5,1,4,8+	vsldoi	6,4,1,8+	vxor	0,0,5+	vxor	2,2,6++	vsldoi	0,0,0,8+	vxor	0,0,7++	vsldoi	6,0,0,8+	.long	0x100044C8+	vxor	6,6,2+	vxor	16,0,6++	vsldoi	17,16,16,8+	vsldoi	19,4,17,8+	vsldoi	18,17,4,8++	.long	0x7E681F99+	li	8,0x70+	.long	0x7E291F99+	li	9,0x80+	.long	0x7E4A1F99+	li	10,0x90+	.long	0x10039CC8+	.long	0x11B09CC8+	.long	0x10238CC8+	.long	0x11D08CC8+	.long	0x104394C8+	.long	0x11F094C8++	.long	0x10E044C8+	.long	0x114D44C8++	vsldoi	5,1,4,8+	vsldoi	6,4,1,8+	vsldoi	11,14,4,8+	vsldoi	9,4,14,8+	vxor	0,0,5+	vxor	2,2,6+	vxor	13,13,11+	vxor	15,15,9++	vsldoi	0,0,0,8+	vsldoi	13,13,13,8+	vxor	0,0,7+	vxor	13,13,10++	vsldoi	6,0,0,8+	vsldoi	9,13,13,8+	.long	0x100044C8+	.long	0x11AD44C8+	vxor	6,6,2+	vxor	9,9,15+	vxor	0,0,6+	vxor	13,13,9++	vsldoi	9,0,0,8+	vsldoi	17,13,13,8+	vsldoi	11,4,9,8+	vsldoi	10,9,4,8+	vsldoi	19,4,17,8+	vsldoi	18,17,4,8++	.long	0x7D681F99+	li	8,0xa0+	.long	0x7D291F99+	li	9,0xb0+	.long	0x7D4A1F99+	li	10,0xc0+	.long	0x7E681F99+	.long	0x7E291F99+	.long	0x7E4A1F99++	or	12,12,12+	blr	+.long	0+.byte	0,12,0x14,0,0,0,2,0+.long	0+.size	crypton_gcm_init_p8,.-crypton_gcm_init_p8+.globl	crypton_gcm_gmult_p8+.type	crypton_gcm_gmult_p8,@function+.align	5+crypton_gcm_gmult_p8:+.localentry	crypton_gcm_gmult_p8,0++	lis	0,0xfff8+	li	8,0x10+	li	12,-1+	li	9,0x20+	or	0,0,0+	li	10,0x30+	.long	0x7C601E99++	.long	0x7D682699+	lvsl	12,0,0+	.long	0x7D292699+	vspltisb	5,0x07+	.long	0x7D4A2699+	vxor	12,12,5+	.long	0x7D002699+	vperm	3,3,3,12+	vxor	4,4,4++	.long	0x10035CC8+	.long	0x10234CC8+	.long	0x104354C8++	.long	0x10E044C8++	vsldoi	5,1,4,8+	vsldoi	6,4,1,8+	vxor	0,0,5+	vxor	2,2,6++	vsldoi	0,0,0,8+	vxor	0,0,7++	vsldoi	6,0,0,8+	.long	0x100044C8+	vxor	6,6,2+	vxor	0,0,6++	vperm	0,0,0,12+	.long	0x7C001F99++	or	12,12,12+	blr	+.long	0+.byte	0,12,0x14,0,0,0,2,0+.long	0+.size	crypton_gcm_gmult_p8,.-crypton_gcm_gmult_p8++.globl	crypton_gcm_ghash_p8+.type	crypton_gcm_ghash_p8,@function+.align	5+crypton_gcm_ghash_p8:+.localentry	crypton_gcm_ghash_p8,0++	li	0,-4096+	li	8,0x10+	li	12,-1+	li	9,0x20+	or	0,0,0+	li	10,0x30+	.long	0x7C001E99++	.long	0x7D682699+	li	8,0x40+	lvsl	12,0,0+	.long	0x7D292699+	li	9,0x50+	vspltisb	5,0x07+	.long	0x7D4A2699+	li	10,0x60+	vxor	12,12,5+	.long	0x7D002699+	vperm	0,0,0,12+	vxor	4,4,4++	cmpldi	6,64+	bge	.Lgcm_ghash_p8_4x++	.long	0x7C602E99+	addi	5,5,16+	subic.	6,6,16+	vperm	3,3,3,12+	vxor	3,3,0+	beq	.Lshort++	.long	0x7E682699+	li	8,16+	.long	0x7E292699+	add	9,5,6+	.long	0x7E4A2699+++.align	5+.Loop_2x:+	.long	0x7E002E99+	vperm	16,16,16,12++	subic	6,6,32+	.long	0x10039CC8+	.long	0x11B05CC8+	subfe	0,0,0+	.long	0x10238CC8+	.long	0x11D04CC8+	and	0,0,6+	.long	0x104394C8+	.long	0x11F054C8+	add	5,5,0++	vxor	0,0,13+	vxor	1,1,14++	.long	0x10E044C8++	vsldoi	5,1,4,8+	vsldoi	6,4,1,8+	vxor	2,2,15+	vxor	0,0,5+	vxor	2,2,6++	vsldoi	0,0,0,8+	vxor	0,0,7+	.long	0x7C682E99+	addi	5,5,32++	vsldoi	6,0,0,8+	.long	0x100044C8+	vperm	3,3,3,12+	vxor	6,6,2+	vxor	3,3,6+	vxor	3,3,0+	cmpld	9,5+	bgt	.Loop_2x++	cmplwi	6,0+	bne	.Leven++.Lshort:+	.long	0x10035CC8+	.long	0x10234CC8+	.long	0x104354C8++	.long	0x10E044C8++	vsldoi	5,1,4,8+	vsldoi	6,4,1,8+	vxor	0,0,5+	vxor	2,2,6++	vsldoi	0,0,0,8+	vxor	0,0,7++	vsldoi	6,0,0,8+	.long	0x100044C8+	vxor	6,6,2++.Leven:+	vxor	0,0,6+	vperm	0,0,0,12+	.long	0x7C001F99++	or	12,12,12+	blr	+.long	0+.byte	0,12,0x14,0,0,0,4,0+.long	0+.align	5+.crypton_gcm_ghash_p8_4x:+.Lgcm_ghash_p8_4x:+	stdu	1,-256(1)+	li	10,63+	li	11,79+	stvx	20,10,1+	addi	10,10,32+	stvx	21,11,1+	addi	11,11,32+	stvx	22,10,1+	addi	10,10,32+	stvx	23,11,1+	addi	11,11,32+	stvx	24,10,1+	addi	10,10,32+	stvx	25,11,1+	addi	11,11,32+	stvx	26,10,1+	addi	10,10,32+	stvx	27,11,1+	addi	11,11,32+	stvx	28,10,1+	addi	10,10,32+	stvx	29,11,1+	addi	11,11,32+	stvx	30,10,1+	li	10,0x60+	stvx	31,11,1+	li	0,-1+	stw	12,252(1)+	or	0,0,0++	lvsl	5,0,8++	li	8,0x70+	.long	0x7E292699+	li	9,0x80+	vspltisb	6,8++	li	10,0x90+	.long	0x7EE82699+	li	8,0xa0+	.long	0x7F092699+	li	9,0xb0+	.long	0x7F2A2699+	li	10,0xc0+	.long	0x7FA82699+	li	8,0x10+	.long	0x7FC92699+	li	9,0x20+	.long	0x7FEA2699+	li	10,0x30++	vsldoi	7,4,6,8+	vaddubm	18,5,7+	vaddubm	19,6,18++	srdi	6,6,4++	.long	0x7C602E99+	.long	0x7E082E99+	subic.	6,6,8+	.long	0x7EC92E99+	.long	0x7F8A2E99+	addi	5,5,0x40+	vperm	3,3,3,12+	vperm	16,16,16,12+	vperm	22,22,22,12+	vperm	28,28,28,12++	vxor	2,3,0++	.long	0x11B0BCC8+	.long	0x11D0C4C8+	.long	0x11F0CCC8++	vperm	11,17,9,18+	vperm	5,22,28,19+	vperm	10,17,9,19+	vperm	6,22,28,18+	.long	0x12B68CC8+	.long	0x12855CC8+	.long	0x137C4CC8+	.long	0x134654C8++	vxor	21,21,14+	vxor	20,20,13+	vxor	27,27,21+	vxor	26,26,15++	blt	.Ltail_4x++.Loop_4x:+	.long	0x7C602E99+	.long	0x7E082E99+	subic.	6,6,4+	.long	0x7EC92E99+	.long	0x7F8A2E99+	addi	5,5,0x40+	vperm	16,16,16,12+	vperm	22,22,22,12+	vperm	28,28,28,12+	vperm	3,3,3,12++	.long	0x1002ECC8+	.long	0x1022F4C8+	.long	0x1042FCC8+	.long	0x11B0BCC8+	.long	0x11D0C4C8+	.long	0x11F0CCC8++	vxor	0,0,20+	vxor	1,1,27+	vxor	2,2,26+	vperm	5,22,28,19+	vperm	6,22,28,18++	.long	0x10E044C8+	.long	0x12855CC8+	.long	0x134654C8++	vsldoi	5,1,4,8+	vsldoi	6,4,1,8+	vxor	0,0,5+	vxor	2,2,6++	vsldoi	0,0,0,8+	vxor	0,0,7++	vsldoi	6,0,0,8+	.long	0x12B68CC8+	.long	0x137C4CC8+	.long	0x100044C8++	vxor	20,20,13+	vxor	26,26,15+	vxor	2,2,3+	vxor	21,21,14+	vxor	2,2,6+	vxor	27,27,21+	vxor	2,2,0+	bge	.Loop_4x++.Ltail_4x:+	.long	0x1002ECC8+	.long	0x1022F4C8+	.long	0x1042FCC8++	vxor	0,0,20+	vxor	1,1,27++	.long	0x10E044C8++	vsldoi	5,1,4,8+	vsldoi	6,4,1,8+	vxor	2,2,26+	vxor	0,0,5+	vxor	2,2,6++	vsldoi	0,0,0,8+	vxor	0,0,7++	vsldoi	6,0,0,8+	.long	0x100044C8+	vxor	6,6,2+	vxor	0,0,6++	addic.	6,6,4+	beq	.Ldone_4x++	.long	0x7C602E99+	cmpldi	6,2+	li	6,-4+	blt	.Lone+	.long	0x7E082E99+	beq	.Ltwo++.Lthree:+	.long	0x7EC92E99+	vperm	3,3,3,12+	vperm	16,16,16,12+	vperm	22,22,22,12++	vxor	2,3,0+	vor	29,23,23+	vor	30,24,24+	vor	31,25,25++	vperm	5,16,22,19+	vperm	6,16,22,18+	.long	0x12B08CC8+	.long	0x13764CC8+	.long	0x12855CC8+	.long	0x134654C8++	vxor	27,27,21+	b	.Ltail_4x++.align	4+.Ltwo:+	vperm	3,3,3,12+	vperm	16,16,16,12++	vxor	2,3,0+	vperm	5,4,16,19+	vperm	6,4,16,18++	vsldoi	29,4,17,8+	vor	30,17,17+	vsldoi	31,17,4,8++	.long	0x12855CC8+	.long	0x13704CC8+	.long	0x134654C8++	b	.Ltail_4x++.align	4+.Lone:+	vperm	3,3,3,12++	vsldoi	29,4,9,8+	vor	30,9,9+	vsldoi	31,9,4,8++	vxor	2,3,0+	vxor	20,20,20+	vxor	27,27,27+	vxor	26,26,26++	b	.Ltail_4x++.Ldone_4x:+	vperm	0,0,0,12+	.long	0x7C001F99++	li	10,63+	li	11,79+	or	12,12,12+	lvx	20,10,1+	addi	10,10,32+	lvx	21,11,1+	addi	11,11,32+	lvx	22,10,1+	addi	10,10,32+	lvx	23,11,1+	addi	11,11,32+	lvx	24,10,1+	addi	10,10,32+	lvx	25,11,1+	addi	11,11,32+	lvx	26,10,1+	addi	10,10,32+	lvx	27,11,1+	addi	11,11,32+	lvx	28,10,1+	addi	10,10,32+	lvx	29,11,1+	addi	11,11,32+	lvx	30,10,1+	lvx	31,11,1+	addi	1,1,256+	blr	+.long	0+.byte	0,12,0x04,0,0x80,0,4,0+.long	0+.size	crypton_gcm_ghash_p8,.-crypton_gcm_ghash_p8++.byte	71,72,65,83,72,32,102,111,114,32,80,111,119,101,114,73,83,65,32,50,46,48,55,44,67,82,89,80,84,79,71,65,77,83,32,98,121,32,60,97,112,112,114,111,64,111,112,101,110,115,115,108,46,111,114,103,62,0+.align	2+.align	2
+ cbits/asm/ghashp8-ppc.pl view
@@ -0,0 +1,664 @@+#!/usr/bin/env perl+#+# ====================================================================+# Written by Andy Polyakov, @dot-asm, initially for use in the OpenSSL+# project. The module is dual licensed under OpenSSL and CRYPTOGAMS+# licenses depending on where you obtain it. For further details see+# https://github.com/dot-asm/cryptogams/.+# ====================================================================+#+# GHASH for for PowerISA v2.07.+#+# July 2014+#+# Accurate performance measurements are problematic, because it's+# always virtualized setup with possibly throttled processor.+# Relative comparison is therefore more informative. This initial+# version is ~2.1x slower than hardware-assisted AES-128-CTR, ~12x+# faster than "4-bit" integer-only compiler-generated 64-bit code.+# "Initial version" means that there is room for further improvement.++# May 2016+#+# 2x aggregated reduction improves performance by 50% (resulting+# performance on POWER8 is 1 cycle per processed byte), and 4x+# aggregated reduction - by 170% or 2.7x (resulting in 0.55 cpb).+# POWER9 delivers 0.51 cpb.++$flavour=shift;+$output =shift;++if ($flavour =~ /64/) {+	$SIZE_T=8;+	$LRSAVE=2*$SIZE_T;+	$STU="stdu";+	$POP="ld";+	$PUSH="std";+	$UCMP="cmpld";+	$SHRI="srdi";+} elsif ($flavour =~ /32/) {+	$SIZE_T=4;+	$LRSAVE=$SIZE_T;+	$STU="stwu";+	$POP="lwz";+	$PUSH="stw";+	$UCMP="cmplw";+	$SHRI="srwi";+} else { die "nonsense $flavour"; }++$sp="r1";+$FRAME=6*$SIZE_T+13*16;	# 13*16 is for v20-v31 offload++$0 =~ m/(.*[\/\\])[^\/\\]+$/; $dir=$1;+( $xlate="${dir}ppc-xlate.pl" and -f $xlate ) or+( $xlate="${dir}../../perlasm/ppc-xlate.pl" and -f $xlate) or+die "can't locate ppc-xlate.pl";++open STDOUT,"| $^X $xlate $flavour $output" || die "can't call $xlate: $!";++my ($Xip,$Htbl,$inp,$len)=map("r$_",(3..6));	# argument block++my ($Xl,$Xm,$Xh,$IN)=map("v$_",(0..3));+my ($zero,$t0,$t1,$t2,$xC2,$H,$Hh,$Hl,$lemask)=map("v$_",(4..12));+my ($Xl1,$Xm1,$Xh1,$IN1,$H2,$H2h,$H2l)=map("v$_",(13..19));+my $vrsave="r12";++$code=<<___;+.machine	"any"++.text++.globl	.gcm_init_p8+.align	5+.gcm_init_p8:+	li		r0,-4096+	li		r8,0x10+	mfspr		$vrsave,256+	li		r9,0x20+	mtspr		256,r0+	li		r10,0x30+	lvx_u		$H,0,r4			# load H++	vspltisb	$xC2,-16		# 0xf0+	vspltisb	$t0,1			# one+	vaddubm		$xC2,$xC2,$xC2		# 0xe0+	vxor		$zero,$zero,$zero+	vor		$xC2,$xC2,$t0		# 0xe1+	vsldoi		$xC2,$xC2,$zero,15	# 0xe1...+	vsldoi		$t1,$zero,$t0,1		# ...1+	vaddubm		$xC2,$xC2,$xC2		# 0xc2...+	vspltisb	$t2,7+	vor		$xC2,$xC2,$t1		# 0xc2....01+	vspltb		$t1,$H,0		# most significant byte+	vsl		$H,$H,$t0		# H<<=1+	vsrab		$t1,$t1,$t2		# broadcast carry bit+	vand		$t1,$t1,$xC2+	vxor		$IN,$H,$t1		# twisted H++	vsldoi		$H,$IN,$IN,8		# twist even more ...+	vsldoi		$xC2,$zero,$xC2,8	# 0xc2.0+	vsldoi		$Hl,$zero,$H,8		# ... and split+	vsldoi		$Hh,$H,$zero,8++	stvx_u		$xC2,0,r3		# save pre-computed table+	stvx_u		$Hl,r8,r3+	li		r8,0x40+	stvx_u		$H, r9,r3+	li		r9,0x50+	stvx_u		$Hh,r10,r3+	li		r10,0x60++	vpmsumd		$Xl,$IN,$Hl		# H.lo·H.lo+	vpmsumd		$Xm,$IN,$H		# H.hi·H.lo+H.lo·H.hi+	vpmsumd		$Xh,$IN,$Hh		# H.hi·H.hi++	vpmsumd		$t2,$Xl,$xC2		# 1st reduction phase++	vsldoi		$t0,$Xm,$zero,8+	vsldoi		$t1,$zero,$Xm,8+	vxor		$Xl,$Xl,$t0+	vxor		$Xh,$Xh,$t1++	vsldoi		$Xl,$Xl,$Xl,8+	vxor		$Xl,$Xl,$t2++	vsldoi		$t1,$Xl,$Xl,8		# 2nd reduction phase+	vpmsumd		$Xl,$Xl,$xC2+	vxor		$t1,$t1,$Xh+	vxor		$IN1,$Xl,$t1++	vsldoi		$H2,$IN1,$IN1,8+	vsldoi		$H2l,$zero,$H2,8+	vsldoi		$H2h,$H2,$zero,8++	stvx_u		$H2l,r8,r3		# save H^2+	li		r8,0x70+	stvx_u		$H2,r9,r3+	li		r9,0x80+	stvx_u		$H2h,r10,r3+	li		r10,0x90+___+{+my ($t4,$t5,$t6) = ($Hl,$H,$Hh);+$code.=<<___;+	vpmsumd		$Xl,$IN,$H2l		# H.lo·H^2.lo+	 vpmsumd	$Xl1,$IN1,$H2l		# H^2.lo·H^2.lo+	vpmsumd		$Xm,$IN,$H2		# H.hi·H^2.lo+H.lo·H^2.hi+	 vpmsumd	$Xm1,$IN1,$H2		# H^2.hi·H^2.lo+H^2.lo·H^2.hi+	vpmsumd		$Xh,$IN,$H2h		# H.hi·H^2.hi+	 vpmsumd	$Xh1,$IN1,$H2h		# H^2.hi·H^2.hi++	vpmsumd		$t2,$Xl,$xC2		# 1st reduction phase+	 vpmsumd	$t6,$Xl1,$xC2		# 1st reduction phase++	vsldoi		$t0,$Xm,$zero,8+	vsldoi		$t1,$zero,$Xm,8+	 vsldoi		$t4,$Xm1,$zero,8+	 vsldoi		$t5,$zero,$Xm1,8+	vxor		$Xl,$Xl,$t0+	vxor		$Xh,$Xh,$t1+	 vxor		$Xl1,$Xl1,$t4+	 vxor		$Xh1,$Xh1,$t5++	vsldoi		$Xl,$Xl,$Xl,8+	 vsldoi		$Xl1,$Xl1,$Xl1,8+	vxor		$Xl,$Xl,$t2+	 vxor		$Xl1,$Xl1,$t6++	vsldoi		$t1,$Xl,$Xl,8		# 2nd reduction phase+	 vsldoi		$t5,$Xl1,$Xl1,8		# 2nd reduction phase+	vpmsumd		$Xl,$Xl,$xC2+	 vpmsumd	$Xl1,$Xl1,$xC2+	vxor		$t1,$t1,$Xh+	 vxor		$t5,$t5,$Xh1+	vxor		$Xl,$Xl,$t1+	 vxor		$Xl1,$Xl1,$t5++	vsldoi		$H,$Xl,$Xl,8+	 vsldoi		$H2,$Xl1,$Xl1,8+	vsldoi		$Hl,$zero,$H,8+	vsldoi		$Hh,$H,$zero,8+	 vsldoi		$H2l,$zero,$H2,8+	 vsldoi		$H2h,$H2,$zero,8++	stvx_u		$Hl,r8,r3		# save H^3+	li		r8,0xa0+	stvx_u		$H,r9,r3+	li		r9,0xb0+	stvx_u		$Hh,r10,r3+	li		r10,0xc0+	 stvx_u		$H2l,r8,r3		# save H^4+	 stvx_u		$H2,r9,r3+	 stvx_u		$H2h,r10,r3++	mtspr		256,$vrsave+	blr+	.long		0+	.byte		0,12,0x14,0,0,0,2,0+	.long		0+.size	.gcm_init_p8,.-.gcm_init_p8+___+}+$code.=<<___;+.globl	.gcm_gmult_p8+.align	5+.gcm_gmult_p8:+	lis		r0,0xfff8+	li		r8,0x10+	mfspr		$vrsave,256+	li		r9,0x20+	mtspr		256,r0+	li		r10,0x30+	lvx_u		$IN,0,$Xip		# load Xi++	lvx_u		$Hl,r8,$Htbl		# load pre-computed table+	 le?lvsl	$lemask,r0,r0+	lvx_u		$H, r9,$Htbl+	 le?vspltisb	$t0,0x07+	lvx_u		$Hh,r10,$Htbl+	 le?vxor	$lemask,$lemask,$t0+	lvx_u		$xC2,0,$Htbl+	 le?vperm	$IN,$IN,$IN,$lemask+	vxor		$zero,$zero,$zero++	vpmsumd		$Xl,$IN,$Hl		# H.lo·Xi.lo+	vpmsumd		$Xm,$IN,$H		# H.hi·Xi.lo+H.lo·Xi.hi+	vpmsumd		$Xh,$IN,$Hh		# H.hi·Xi.hi++	vpmsumd		$t2,$Xl,$xC2		# 1st reduction phase++	vsldoi		$t0,$Xm,$zero,8+	vsldoi		$t1,$zero,$Xm,8+	vxor		$Xl,$Xl,$t0+	vxor		$Xh,$Xh,$t1++	vsldoi		$Xl,$Xl,$Xl,8+	vxor		$Xl,$Xl,$t2++	vsldoi		$t1,$Xl,$Xl,8		# 2nd reduction phase+	vpmsumd		$Xl,$Xl,$xC2+	vxor		$t1,$t1,$Xh+	vxor		$Xl,$Xl,$t1++	le?vperm	$Xl,$Xl,$Xl,$lemask+	stvx_u		$Xl,0,$Xip		# write out Xi++	mtspr		256,$vrsave+	blr+	.long		0+	.byte		0,12,0x14,0,0,0,2,0+	.long		0+.size	.gcm_gmult_p8,.-.gcm_gmult_p8++.globl	.gcm_ghash_p8+.align	5+.gcm_ghash_p8:+	li		r0,-4096+	li		r8,0x10+	mfspr		$vrsave,256+	li		r9,0x20+	mtspr		256,r0+	li		r10,0x30+	lvx_u		$Xl,0,$Xip		# load Xi++	lvx_u		$Hl,r8,$Htbl		# load pre-computed table+	li		r8,0x40+	 le?lvsl	$lemask,r0,r0+	lvx_u		$H, r9,$Htbl+	li		r9,0x50+	 le?vspltisb	$t0,0x07+	lvx_u		$Hh,r10,$Htbl+	li		r10,0x60+	 le?vxor	$lemask,$lemask,$t0+	lvx_u		$xC2,0,$Htbl+	 le?vperm	$Xl,$Xl,$Xl,$lemask+	vxor		$zero,$zero,$zero++	${UCMP}i	$len,64+	bge		Lgcm_ghash_p8_4x++	lvx_u		$IN,0,$inp+	addi		$inp,$inp,16+	subic.		$len,$len,16+	 le?vperm	$IN,$IN,$IN,$lemask+	vxor		$IN,$IN,$Xl+	beq		Lshort++	lvx_u		$H2l,r8,$Htbl		# load H^2+	li		r8,16+	lvx_u		$H2, r9,$Htbl+	add		r9,$inp,$len		# end of input+	lvx_u		$H2h,r10,$Htbl+	be?b		Loop_2x++.align	5+Loop_2x:+	lvx_u		$IN1,0,$inp+	le?vperm	$IN1,$IN1,$IN1,$lemask++	 subic		$len,$len,32+	vpmsumd		$Xl,$IN,$H2l		# H^2.lo·Xi.lo+	 vpmsumd	$Xl1,$IN1,$Hl		# H.lo·Xi+1.lo+	 subfe		r0,r0,r0		# borrow?-1:0+	vpmsumd		$Xm,$IN,$H2		# H^2.hi·Xi.lo+H^2.lo·Xi.hi+	 vpmsumd	$Xm1,$IN1,$H		# H.hi·Xi+1.lo+H.lo·Xi+1.hi+	 and		r0,r0,$len+	vpmsumd		$Xh,$IN,$H2h		# H^2.hi·Xi.hi+	 vpmsumd	$Xh1,$IN1,$Hh		# H.hi·Xi+1.hi+	 add		$inp,$inp,r0++	vxor		$Xl,$Xl,$Xl1+	vxor		$Xm,$Xm,$Xm1++	vpmsumd		$t2,$Xl,$xC2		# 1st reduction phase++	vsldoi		$t0,$Xm,$zero,8+	vsldoi		$t1,$zero,$Xm,8+	 vxor		$Xh,$Xh,$Xh1+	vxor		$Xl,$Xl,$t0+	vxor		$Xh,$Xh,$t1++	vsldoi		$Xl,$Xl,$Xl,8+	vxor		$Xl,$Xl,$t2+	 lvx_u		$IN,r8,$inp+	 addi		$inp,$inp,32++	vsldoi		$t1,$Xl,$Xl,8		# 2nd reduction phase+	vpmsumd		$Xl,$Xl,$xC2+	 le?vperm	$IN,$IN,$IN,$lemask+	vxor		$t1,$t1,$Xh+	vxor		$IN,$IN,$t1+	vxor		$IN,$IN,$Xl+	$UCMP		r9,$inp+	bgt		Loop_2x			# done yet?++	cmplwi		$len,0+	bne		Leven++Lshort:+	vpmsumd		$Xl,$IN,$Hl		# H.lo·Xi.lo+	vpmsumd		$Xm,$IN,$H		# H.hi·Xi.lo+H.lo·Xi.hi+	vpmsumd		$Xh,$IN,$Hh		# H.hi·Xi.hi++	vpmsumd		$t2,$Xl,$xC2		# 1st reduction phase++	vsldoi		$t0,$Xm,$zero,8+	vsldoi		$t1,$zero,$Xm,8+	vxor		$Xl,$Xl,$t0+	vxor		$Xh,$Xh,$t1++	vsldoi		$Xl,$Xl,$Xl,8+	vxor		$Xl,$Xl,$t2++	vsldoi		$t1,$Xl,$Xl,8		# 2nd reduction phase+	vpmsumd		$Xl,$Xl,$xC2+	vxor		$t1,$t1,$Xh++Leven:+	vxor		$Xl,$Xl,$t1+	le?vperm	$Xl,$Xl,$Xl,$lemask+	stvx_u		$Xl,0,$Xip		# write out Xi++	mtspr		256,$vrsave+	blr+	.long		0+	.byte		0,12,0x14,0,0,0,4,0+	.long		0+___+{+my ($Xl3,$Xm2,$IN2,$H3l,$H3,$H3h,+    $Xh3,$Xm3,$IN3,$H4l,$H4,$H4h) = map("v$_",(20..31));+my $IN0=$IN;+my ($H21l,$H21h,$loperm,$hiperm) = ($Hl,$Hh,$H2l,$H2h);++$code.=<<___;+.align	5+.gcm_ghash_p8_4x:+Lgcm_ghash_p8_4x:+	$STU		$sp,-$FRAME($sp)+	li		r10,`15+6*$SIZE_T`+	li		r11,`31+6*$SIZE_T`+	stvx		v20,r10,$sp+	addi		r10,r10,32+	stvx		v21,r11,$sp+	addi		r11,r11,32+	stvx		v22,r10,$sp+	addi		r10,r10,32+	stvx		v23,r11,$sp+	addi		r11,r11,32+	stvx		v24,r10,$sp+	addi		r10,r10,32+	stvx		v25,r11,$sp+	addi		r11,r11,32+	stvx		v26,r10,$sp+	addi		r10,r10,32+	stvx		v27,r11,$sp+	addi		r11,r11,32+	stvx		v28,r10,$sp+	addi		r10,r10,32+	stvx		v29,r11,$sp+	addi		r11,r11,32+	stvx		v30,r10,$sp+	li		r10,0x60+	stvx		v31,r11,$sp+	li		r0,-1+	stw		$vrsave,`$FRAME-4`($sp)	# save vrsave+	mtspr		256,r0			# preserve all AltiVec registers++	lvsl		$t0,0,r8		# 0x0001..0e0f+	#lvx_u		$H2l,r8,$Htbl		# load H^2+	li		r8,0x70+	lvx_u		$H2, r9,$Htbl+	li		r9,0x80+	vspltisb	$t1,8			# 0x0808..0808+	#lvx_u		$H2h,r10,$Htbl+	li		r10,0x90+	lvx_u		$H3l,r8,$Htbl		# load H^3+	li		r8,0xa0+	lvx_u		$H3, r9,$Htbl+	li		r9,0xb0+	lvx_u		$H3h,r10,$Htbl+	li		r10,0xc0+	lvx_u		$H4l,r8,$Htbl		# load H^4+	li		r8,0x10+	lvx_u		$H4, r9,$Htbl+	li		r9,0x20+	lvx_u		$H4h,r10,$Htbl+	li		r10,0x30++	vsldoi		$t2,$zero,$t1,8		# 0x0000..0808+	vaddubm		$hiperm,$t0,$t2		# 0x0001..1617+	vaddubm		$loperm,$t1,$hiperm	# 0x0809..1e1f++	$SHRI		$len,$len,4		# this allows to use sign bit+						# as carry+	lvx_u		$IN0,0,$inp		# load input+	lvx_u		$IN1,r8,$inp+	subic.		$len,$len,8+	lvx_u		$IN2,r9,$inp+	lvx_u		$IN3,r10,$inp+	addi		$inp,$inp,0x40+	le?vperm	$IN0,$IN0,$IN0,$lemask+	le?vperm	$IN1,$IN1,$IN1,$lemask+	le?vperm	$IN2,$IN2,$IN2,$lemask+	le?vperm	$IN3,$IN3,$IN3,$lemask++	vxor		$Xh,$IN0,$Xl++	 vpmsumd	$Xl1,$IN1,$H3l+	 vpmsumd	$Xm1,$IN1,$H3+	 vpmsumd	$Xh1,$IN1,$H3h++	 vperm		$H21l,$H2,$H,$hiperm+	 vperm		$t0,$IN2,$IN3,$loperm+	 vperm		$H21h,$H2,$H,$loperm+	 vperm		$t1,$IN2,$IN3,$hiperm+	 vpmsumd	$Xm2,$IN2,$H2		# H^2.lo·Xi+2.hi+H^2.hi·Xi+2.lo+	 vpmsumd	$Xl3,$t0,$H21l		# H^2.lo·Xi+2.lo+H.lo·Xi+3.lo+	 vpmsumd	$Xm3,$IN3,$H		# H.hi·Xi+3.lo  +H.lo·Xi+3.hi+	 vpmsumd	$Xh3,$t1,$H21h		# H^2.hi·Xi+2.hi+H.hi·Xi+3.hi++	 vxor		$Xm2,$Xm2,$Xm1+	 vxor		$Xl3,$Xl3,$Xl1+	 vxor		$Xm3,$Xm3,$Xm2+	 vxor		$Xh3,$Xh3,$Xh1++	blt		Ltail_4x++Loop_4x:+	lvx_u		$IN0,0,$inp+	lvx_u		$IN1,r8,$inp+	subic.		$len,$len,4+	lvx_u		$IN2,r9,$inp+	lvx_u		$IN3,r10,$inp+	addi		$inp,$inp,0x40+	le?vperm	$IN1,$IN1,$IN1,$lemask+	le?vperm	$IN2,$IN2,$IN2,$lemask+	le?vperm	$IN3,$IN3,$IN3,$lemask+	le?vperm	$IN0,$IN0,$IN0,$lemask++	vpmsumd		$Xl,$Xh,$H4l		# H^4.lo·Xi.lo+	vpmsumd		$Xm,$Xh,$H4		# H^4.hi·Xi.lo+H^4.lo·Xi.hi+	vpmsumd		$Xh,$Xh,$H4h		# H^4.hi·Xi.hi+	 vpmsumd	$Xl1,$IN1,$H3l+	 vpmsumd	$Xm1,$IN1,$H3+	 vpmsumd	$Xh1,$IN1,$H3h++	vxor		$Xl,$Xl,$Xl3+	vxor		$Xm,$Xm,$Xm3+	vxor		$Xh,$Xh,$Xh3+	 vperm		$t0,$IN2,$IN3,$loperm+	 vperm		$t1,$IN2,$IN3,$hiperm++	vpmsumd		$t2,$Xl,$xC2		# 1st reduction phase+	 vpmsumd	$Xl3,$t0,$H21l		# H.lo·Xi+3.lo  +H^2.lo·Xi+2.lo+	 vpmsumd	$Xh3,$t1,$H21h		# H.hi·Xi+3.hi  +H^2.hi·Xi+2.hi++	vsldoi		$t0,$Xm,$zero,8+	vsldoi		$t1,$zero,$Xm,8+	vxor		$Xl,$Xl,$t0+	vxor		$Xh,$Xh,$t1++	vsldoi		$Xl,$Xl,$Xl,8+	vxor		$Xl,$Xl,$t2++	vsldoi		$t1,$Xl,$Xl,8		# 2nd reduction phase+	 vpmsumd	$Xm2,$IN2,$H2		# H^2.hi·Xi+2.lo+H^2.lo·Xi+2.hi+	 vpmsumd	$Xm3,$IN3,$H		# H.hi·Xi+3.lo  +H.lo·Xi+3.hi+	vpmsumd		$Xl,$Xl,$xC2++	 vxor		$Xl3,$Xl3,$Xl1+	 vxor		$Xh3,$Xh3,$Xh1+	vxor		$Xh,$Xh,$IN0+	 vxor		$Xm2,$Xm2,$Xm1+	vxor		$Xh,$Xh,$t1+	 vxor		$Xm3,$Xm3,$Xm2+	vxor		$Xh,$Xh,$Xl+	bge		Loop_4x++Ltail_4x:+	vpmsumd		$Xl,$Xh,$H4l		# H^4.lo·Xi.lo+	vpmsumd		$Xm,$Xh,$H4		# H^4.hi·Xi.lo+H^4.lo·Xi.hi+	vpmsumd		$Xh,$Xh,$H4h		# H^4.hi·Xi.hi++	vxor		$Xl,$Xl,$Xl3+	vxor		$Xm,$Xm,$Xm3++	vpmsumd		$t2,$Xl,$xC2		# 1st reduction phase++	vsldoi		$t0,$Xm,$zero,8+	vsldoi		$t1,$zero,$Xm,8+	 vxor		$Xh,$Xh,$Xh3+	vxor		$Xl,$Xl,$t0+	vxor		$Xh,$Xh,$t1++	vsldoi		$Xl,$Xl,$Xl,8+	vxor		$Xl,$Xl,$t2++	vsldoi		$t1,$Xl,$Xl,8		# 2nd reduction phase+	vpmsumd		$Xl,$Xl,$xC2+	vxor		$t1,$t1,$Xh+	vxor		$Xl,$Xl,$t1++	addic.		$len,$len,4+	beq		Ldone_4x++	lvx_u		$IN0,0,$inp+	${UCMP}i	$len,2+	li		$len,-4+	blt		Lone+	lvx_u		$IN1,r8,$inp+	beq		Ltwo++Lthree:+	lvx_u		$IN2,r9,$inp+	le?vperm	$IN0,$IN0,$IN0,$lemask+	le?vperm	$IN1,$IN1,$IN1,$lemask+	le?vperm	$IN2,$IN2,$IN2,$lemask++	vxor		$Xh,$IN0,$Xl+	vmr		$H4l,$H3l+	vmr		$H4, $H3+	vmr		$H4h,$H3h++	vperm		$t0,$IN1,$IN2,$loperm+	vperm		$t1,$IN1,$IN2,$hiperm+	vpmsumd		$Xm2,$IN1,$H2		# H^2.lo·Xi+1.hi+H^2.hi·Xi+1.lo+	vpmsumd		$Xm3,$IN2,$H		# H.hi·Xi+2.lo  +H.lo·Xi+2.hi+	vpmsumd		$Xl3,$t0,$H21l		# H^2.lo·Xi+1.lo+H.lo·Xi+2.lo+	vpmsumd		$Xh3,$t1,$H21h		# H^2.hi·Xi+1.hi+H.hi·Xi+2.hi++	vxor		$Xm3,$Xm3,$Xm2+	b		Ltail_4x++.align	4+Ltwo:+	le?vperm	$IN0,$IN0,$IN0,$lemask+	le?vperm	$IN1,$IN1,$IN1,$lemask++	vxor		$Xh,$IN0,$Xl+	vperm		$t0,$zero,$IN1,$loperm+	vperm		$t1,$zero,$IN1,$hiperm++	vsldoi		$H4l,$zero,$H2,8+	vmr		$H4, $H2+	vsldoi		$H4h,$H2,$zero,8++	vpmsumd		$Xl3,$t0, $H21l		# H.lo·Xi+1.lo+	vpmsumd		$Xm3,$IN1,$H		# H.hi·Xi+1.lo+H.lo·Xi+2.hi+	vpmsumd		$Xh3,$t1, $H21h		# H.hi·Xi+1.hi++	b		Ltail_4x++.align	4+Lone:+	le?vperm	$IN0,$IN0,$IN0,$lemask++	vsldoi		$H4l,$zero,$H,8+	vmr		$H4, $H+	vsldoi		$H4h,$H,$zero,8++	vxor		$Xh,$IN0,$Xl+	vxor		$Xl3,$Xl3,$Xl3+	vxor		$Xm3,$Xm3,$Xm3+	vxor		$Xh3,$Xh3,$Xh3++	b		Ltail_4x++Ldone_4x:+	le?vperm	$Xl,$Xl,$Xl,$lemask+	stvx_u		$Xl,0,$Xip		# write out Xi++	li		r10,`15+6*$SIZE_T`+	li		r11,`31+6*$SIZE_T`+	mtspr		256,$vrsave+	lvx		v20,r10,$sp+	addi		r10,r10,32+	lvx		v21,r11,$sp+	addi		r11,r11,32+	lvx		v22,r10,$sp+	addi		r10,r10,32+	lvx		v23,r11,$sp+	addi		r11,r11,32+	lvx		v24,r10,$sp+	addi		r10,r10,32+	lvx		v25,r11,$sp+	addi		r11,r11,32+	lvx		v26,r10,$sp+	addi		r10,r10,32+	lvx		v27,r11,$sp+	addi		r11,r11,32+	lvx		v28,r10,$sp+	addi		r10,r10,32+	lvx		v29,r11,$sp+	addi		r11,r11,32+	lvx		v30,r10,$sp+	lvx		v31,r11,$sp+	addi		$sp,$sp,$FRAME+	blr+	.long		0+	.byte		0,12,0x04,0,0x80,0,4,0+	.long		0+___+}+$code.=<<___;+.size	.gcm_ghash_p8,.-.gcm_ghash_p8++.asciz  "GHASH for PowerISA 2.07, CRYPTOGAMS by <appro\@openssl.org>"+.align  2+___++foreach (split("\n",$code)) {+	s/\`([^\`]*)\`/eval $1/geo;++	if ($flavour =~ /le$/o) {	# little-endian+	    s/le\?//o		or+	    s/be\?/#be#/o;+	} else {+	    s/le\?/#le#/o	or+	    s/be\?//o;+	}+	print $_,"\n";+}++close STDOUT; # enforce flush
+ cbits/asm/ppc-xlate.pl view
@@ -0,0 +1,352 @@+#!/usr/bin/env perl++# PowerPC assembler distiller by \@dot-asm.++################################################################+# Recognized "flavour"-s are:+#+# linux{32|64}[le]  GNU assembler and ELF symbol decorations,+#                   with little-endian option+# linux64v2         GNU asssembler and big-endian instantiation+#                   of latest ELF specification+# aix{32|64}        AIX assembler and symbol decorations+# osx{32|64}        Mac OS X assembler and symbol decoratons++my $flavour = shift;+my $output = shift;+open STDOUT,">$output" || die "can't open $output: $!";++my %GLOBALS;+my %TYPES;+my $dotinlocallabels=($flavour=~/linux/)?1:0;++################################################################+# directives which need special treatment on different platforms+################################################################+my $type = sub {+    my ($dir,$name,$type) = @_;++    $TYPES{$name} = $type;+    if ($flavour =~ /linux/) {+	$name =~ s|^\.||;+	".type	$name,$type";+    } else {+	"";+    }+};+my $globl = sub {+    my $junk = shift;+    my $name = shift;+    my $global = \$GLOBALS{$name};+    my $type = \$TYPES{$name};+    my $ret;++    $name =~ s|^\.||;++    SWITCH: for ($flavour) {+	/aix/		&& do { if (!$$type) {+				    $$type = "\@function";+				}+				if ($$type =~ /function/) {+				    $name = ".$name";+				}+				last;+			      };+	/osx/		&& do { $name = "_$name";+				last;+			      };+	/linux.*(32|64(le|v2))/+			&& do {	$ret .= ".globl	$name";+				if (!$$type) {+				    $ret .= "\n.type	$name,\@function";+				    $$type = "\@function";+				}+				last;+			      };+	/linux.*64/	&& do {	$ret .= ".globl	$name";+				if (!$$type) {+				    $ret .= "\n.type	$name,\@function";+				    $$type = "\@function";+				}+				if ($$type =~ /function/) {+				    $ret .= "\n.section	\".opd\",\"aw\"";+				    $ret .= "\n.align	3";+				    $ret .= "\n$name:";+				    $ret .= "\n.quad	.$name,.TOC.\@tocbase,0";+				    $ret .= "\n.previous";+				    $name = ".$name";+				}+				last;+			      };+    }++    $ret = ".globl	$name" if (!$ret);+    $$global = $name;+    $ret;+};+my $text = sub {+    my $ret = ($flavour =~ /aix/) ? ".csect\t.text[PR],7" : ".text";+    $ret = ".abiversion	2\n".$ret	if ($flavour =~ /linux.*64(le|v2)/);+    $ret;+};+my $machine = sub {+    my $junk = shift;+    my $arch = shift;+    if ($flavour =~ /osx/)+    {	$arch =~ s/\"//g;+	$arch = ($flavour=~/64/) ? "ppc970-64" : "ppc970" if ($arch eq "any");+    }+    ".machine	$arch";+};+my $size = sub {+    if ($flavour =~ /linux/)+    {	shift;+	my $name = shift;+	my $real = $GLOBALS{$name} ? \$GLOBALS{$name} : \$name;+	my $ret  = ".size	$$real,.-$$real";+	$name =~ s|^\.||;+	if ($$real ne $name) {+	    $ret .= "\n.size	$name,.-$$real";+	}+	$ret;+    }+    else+    {	"";	}+};+my $asciz = sub {+    shift;+    my $line = join(",",@_);+    if ($line =~ /^"(.*)"$/)+    {	".byte	" . join(",",unpack("C*",$1),0) . "\n.align	2";	}+    else+    {	"";	}+};+my $quad = sub {+    shift;+    my @ret;+    my ($hi,$lo);+    for (@_) {+	if (/^0x([0-9a-f]*?)([0-9a-f]{1,8})$/io)+	{  $hi=$1?"0x$1":"0"; $lo="0x$2";  }+	elsif (/^([0-9]+)$/o)+	{  $hi=$1>>32; $lo=$1&0xffffffff;  } # error-prone with 32-bit perl+	else+	{  $hi=undef; $lo=$_; }++	if (defined($hi))+	{  push(@ret,$flavour=~/le$/o?".long\t$lo,$hi":".long\t$hi,$lo");  }+	else+	{  push(@ret,".quad	$lo");  }+    }+    join("\n",@ret);+};++################################################################+# simplified mnemonics not handled by at least one assembler+################################################################+my $cmplw = sub {+    my $f = shift;+    my $cr = 0; $cr = shift if ($#_>1);+    # Some out-of-date 32-bit GNU assembler just can't handle cmplw...+    ($flavour =~ /linux.*32/) ?+	"	.long	".sprintf "0x%x",31<<26|$cr<<23|$_[0]<<16|$_[1]<<11|64 :+	"	cmplw	".join(',',$cr,@_);+};+my $bdnz = sub {+    my $f = shift;+    my $bo = $f=~/[\+\-]/ ? 16+9 : 16;	# optional "to be taken" hint+    "	bc	$bo,0,".shift;+} if ($flavour!~/linux/);+my $bltlr = sub {+    my $f = shift;+    my $bo = $f=~/\-/ ? 12+2 : 12;	# optional "not to be taken" hint+    ($flavour =~ /linux/) ?		# GNU as doesn't allow most recent hints+	"	.long	".sprintf "0x%x",19<<26|$bo<<21|16<<1 :+	"	bclr	$bo,0";+};+my $bnelr = sub {+    my $f = shift;+    my $bo = $f=~/\-/ ? 4+2 : 4;	# optional "not to be taken" hint+    ($flavour =~ /linux/) ?		# GNU as doesn't allow most recent hints+	"	.long	".sprintf "0x%x",19<<26|$bo<<21|2<<16|16<<1 :+	"	bclr	$bo,2";+};+my $beqlr = sub {+    my $f = shift;+    my $bo = $f=~/-/ ? 12+2 : 12;	# optional "not to be taken" hint+    ($flavour =~ /linux/) ?		# GNU as doesn't allow most recent hints+	"	.long	".sprintf "0x%X",19<<26|$bo<<21|2<<16|16<<1 :+	"	bclr	$bo,2";+};+# GNU assembler can't handle extrdi rA,rS,16,48, or when sum of last two+# arguments is 64, with "operand out of range" error.+my $extrdi = sub {+    my ($f,$ra,$rs,$n,$b) = @_;+    $b = ($b+$n)&63; $n = 64-$n;+    "	rldicl	$ra,$rs,$b,$n";+};+my $vmr = sub {+    my ($f,$vx,$vy) = @_;+    "	vor	$vx,$vy,$vy";+};++# Some ABIs specify vrsave, special-purpose register #256, as reserved+# for system use.+my $no_vrsave = ($flavour =~ /aix|linux64(le|v2)/);+my $mtspr = sub {+    my ($f,$idx,$ra) = @_;+    if ($idx == 256 && $no_vrsave) {+	"	or	$ra,$ra,$ra";+    } else {+	"	mtspr	$idx,$ra";+    }+};+my $mfspr = sub {+    my ($f,$rd,$idx) = @_;+    if ($idx == 256 && $no_vrsave) {+	"	li	$rd,-1";+    } else {+	"	mfspr	$rd,$idx";+    }+};++# PowerISA 2.06 stuff+sub vsxmem_op {+    my ($f, $vrt, $ra, $rb, $op) = @_;+    "	.long	".sprintf "0x%X",(31<<26)|($vrt<<21)|($ra<<16)|($rb<<11)|($op*2+1);+}+# made-up unaligned memory reference AltiVec/VMX instructions+my $lvx_u	= sub {	vsxmem_op(@_, 844); };	# lxvd2x+my $stvx_u	= sub {	vsxmem_op(@_, 972); };	# stxvd2x+my $lvdx_u	= sub {	vsxmem_op(@_, 588); };	# lxsdx+my $stvdx_u	= sub {	vsxmem_op(@_, 716); };	# stxsdx+my $lvx_4w	= sub { vsxmem_op(@_, 780); };	# lxvw4x+my $stvx_4w	= sub { vsxmem_op(@_, 908); };	# stxvw4x+my $lvx_splt	= sub { vsxmem_op(@_, 332); };	# lxvdsx+# VSX instruction[s] masqueraded as made-up AltiVec/VMX+my $vpermdi	= sub {				# xxpermdi+    my ($f, $vrt, $vra, $vrb, $dm) = @_;+    $dm = oct($dm) if ($dm =~ /^0/);+    "	.long	".sprintf "0x%X",(60<<26)|($vrt<<21)|($vra<<16)|($vrb<<11)|($dm<<8)|(10<<3)|7;+};++# PowerISA 2.07 stuff+sub vcrypto_op {+    my ($f, $vrt, $vra, $vrb, $op) = @_;+    "	.long	".sprintf "0x%X",(4<<26)|($vrt<<21)|($vra<<16)|($vrb<<11)|$op;+}+sub vfour {+    my ($f, $vrt, $vra, $vrb, $vrc, $op) = @_;+    "	.long	".sprintf "0x%X",(4<<26)|($vrt<<21)|($vra<<16)|($vrb<<11)|($vrc<<6)|$op;+};+my $vcipher	= sub { vcrypto_op(@_, 1288); };+my $vcipherlast	= sub { vcrypto_op(@_, 1289); };+my $vncipher	= sub { vcrypto_op(@_, 1352); };+my $vncipherlast= sub { vcrypto_op(@_, 1353); };+my $vsbox	= sub { vcrypto_op(@_, 0, 1480); };+my $vshasigmad	= sub { my ($st,$six)=splice(@_,-2); vcrypto_op(@_, $st<<4|$six, 1730); };+my $vshasigmaw	= sub { my ($st,$six)=splice(@_,-2); vcrypto_op(@_, $st<<4|$six, 1666); };+my $vpmsumb	= sub { vcrypto_op(@_, 1032); };+my $vpmsumd	= sub { vcrypto_op(@_, 1224); };+my $vpmsubh	= sub { vcrypto_op(@_, 1096); };+my $vpmsumw	= sub { vcrypto_op(@_, 1160); };+# These are not really crypto, but vcrypto_op template works+my $vaddudm	= sub { vcrypto_op(@_, 192);  };+my $vadduqm	= sub { vcrypto_op(@_, 256);  };+my $vmuleuw	= sub { vcrypto_op(@_, 648);  };+my $vmulouw	= sub { vcrypto_op(@_, 136);  };+my $vrld	= sub { vcrypto_op(@_, 196);  };+my $vsld	= sub { vcrypto_op(@_, 1476); };+my $vsrd	= sub { vcrypto_op(@_, 1732); };+my $vsubudm	= sub { vcrypto_op(@_, 1216); };+my $vaddcuq	= sub { vcrypto_op(@_, 320);  };+my $vaddeuqm	= sub { vfour(@_,60); };+my $vaddecuq	= sub { vfour(@_,61); };+my $vmrgew	= sub { vfour(@_,0,1932); };+my $vmrgow	= sub { vfour(@_,0,1676); };++my $mtsle	= sub {+    my ($f, $arg) = @_;+    "	.long	".sprintf "0x%X",(31<<26)|($arg<<21)|(147*2);+};++# VSX instructions masqueraded as AltiVec/VMX+my $mtvrd	= sub {+    my ($f, $vrt, $ra) = @_;+    "	.long	".sprintf "0x%X",(31<<26)|($vrt<<21)|($ra<<16)|(179<<1)|1;+};+my $mtvrwz	= sub {+    my ($f, $vrt, $ra) = @_;+    "	.long	".sprintf "0x%X",(31<<26)|($vrt<<21)|($ra<<16)|(243<<1)|1;+};+my $lvwzx_u	= sub { vsxmem_op(@_, 12); };	# lxsiwzx+my $stvwx_u	= sub { vsxmem_op(@_, 140); };	# stxsiwx++# PowerISA 3.0 stuff+my $maddhdu	= sub { vfour(@_,49); };+my $maddld	= sub { vfour(@_,51); };+my $darn = sub {+    my ($f, $rt, $l) = @_;+    "	.long	".sprintf "0x%X",(31<<26)|($rt<<21)|($l<<16)|(755<<1);+};+my $iseleq = sub {+    my ($f, $rt, $ra, $rb) = @_;+    "	.long	".sprintf "0x%X",(31<<26)|($rt<<21)|($ra<<16)|($rb<<11)|(2<<6)|30;+};+# VSX instruction[s] masqueraded as made-up AltiVec/VMX+my $vspltib	= sub {				# xxspltib+    my ($f, $vrt, $imm8) = @_;+    $imm8 = oct($imm8) if ($imm8 =~ /^0/);+    $imm8 &= 0xff;+    "	.long	".sprintf "0x%X",(60<<26)|($vrt<<21)|($imm8<<11)|(360<<1)|1;+};++# PowerISA 3.0B stuff+my $addex = sub {+    my ($f, $rt, $ra, $rb, $cy) = @_;	# only cy==0 is specified in 3.0B+    "	.long	".sprintf "0x%X",(31<<26)|($rt<<21)|($ra<<16)|($rb<<11)|($cy<<9)|(170<<1);+};+my $vmsumudm	= sub { vfour(@_,35); };++while($line=<>) {++    $line =~ s|[#!;].*$||;	# get rid of asm-style comments...+    $line =~ s|/\*.*\*/||;	# ... and C-style comments...+    $line =~ s|^\s+||;		# ... and skip white spaces in beginning...+    $line =~ s|\s+$||;		# ... and at the end++    {+	$line =~ s|\.L(\w+)|L$1|g;	# common denominator for Locallabel+	$line =~ s|\bL(\w+)|\.L$1|g	if ($dotinlocallabels);+    }++    {+	$line =~ s|(^[\.\w]+)\:\s*||;+	my $label = $1;+	if ($label) {+	    my $xlated = ($GLOBALS{$label} or $label);+	    print "$xlated:";+	    if ($flavour =~ /linux.*64(le|v2)/) {+		if ($TYPES{$label} =~ /function/) {+		    printf "\n.localentry	%s,0\n",$xlated;+		}+	    }+	}+    }++    {+	$line =~ s|^\s*(\.?)(\w+)([\.\+\-]?)\s*||;+	my $c = $1; $c = "\t" if ($c eq "");+	my $mnemonic = $2;+	my $f = $3;+	my $opcode = eval("\$$mnemonic");+	$line =~ s/\b(c?[rf]|v|vs)([0-9]+)\b/$2/g if ($c ne "." and $flavour !~ /osx/);+	if (ref($opcode) eq 'CODE') { $line = &$opcode($f,split(/,\s*/,$line)); }+	elsif ($mnemonic)           { $line = $c.$mnemonic.$f."\t".$line; }+    }++    print $line if ($line);+    print "\n";+}++close STDOUT;
+ cbits/bearssl/LICENSE view
@@ -0,0 +1,21 @@+Copyright (c) 2016 Thomas Pornin <pornin@bolet.org>++Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the+"Software"), to deal in the Software without restriction, including+without limitation the rights to use, copy, modify, merge, publish,+distribute, sublicense, and/or sell copies of the Software, and to+permit persons to whom the Software is furnished to do so, subject to+the following conditions:++The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS+BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN+ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.
+ cbits/bearssl/README.md view
@@ -0,0 +1,57 @@+# BearSSL++Constant-time AES and GHASH from [BearSSL](https://bearssl.org/), vendored+here.  `VERSION` holds the upstream release these files came from, and+`import.sh` fetches them again.++## Why++crypton's portable AES and GHASH are table-driven, and both index their+tables with a secret: `cbits/aes/generic.c` with a byte of the state, in+every round and in the key expansion, and `cbits/aes/gf.c` with a nibble of+the GHASH accumulator.  Both are variable-time by construction, and the+cache-timing attacks on that shape of code are old and well documented.++The processors that have AES instructions do not run any of it -- crypton+asks them first.  What runs it is everything else: ppc64le and s390x, where+the instructions exist but crypton has no path to them; 32-bit ARM, where+the same is true; riscv64 and loongarch64; and the boards that genuinely+have no AES instructions at all, of which the Raspberry Pi 3 and 4 are by+some distance the largest population.++BearSSL's answer is bitslicing.  `aes_ct64` holds four blocks interleaved+across eight 64-bit words and computes the S-box as boolean algebra, so+there is no table and no address derived from a secret; `ghash_ctmul64`+builds the GF(2^128) multiply out of shifts, masks and integer multiplies+rather than a table of H.  Both are plain C99 and assume nothing beyond+`uint64_t`.++## What is here++| file | from |+| --- | --- |+| `aes_ct64.c` | `src/symcipher/aes_ct64.c` |+| `aes_ct64_enc.c` | `src/symcipher/aes_ct64_enc.c` |+| `aes_ct64_dec.c` | `src/symcipher/aes_ct64_dec.c` |+| `ghash_ctmul64.c` | `src/hash/ghash_ctmul64.c` |+| `dec32le.c` | `src/codec/dec32le.c`, for `br_range_dec32le` |+| `LICENSE` | `LICENSE.txt` -- MIT, (c) 2016 Thomas Pornin |++`inner.h` is **crypton's, not upstream's**.  Each of those five opens with+`#include "inner.h"`, and upstream's is some two thousand lines declaring+the whole library; these five want six things from it.  So this one gives+those six and nothing else, which is what lets the five stay byte for byte+what upstream ships.  `import.sh` does not overwrite it.++The byte-order helpers in it are written rather than copied, in their plain+portable form without upstream's unaligned-access fast paths, so that no+platform configuration comes with them.++## Keeping it honest++`cbits/tests/bearssl_diff.c` checks this against the implementation it+replaces: the FIPS-197 vectors first, so that agreement means AES and not+merely that both sides compute the same wrong thing, then a run of random+keys and blocks, then GHASH against the 4-bit table.  Given any argument it+corrupts three results on purpose and the comparison has to notice -- a+differential test that cannot fail has said nothing.
+ cbits/bearssl/VERSION view
@@ -0,0 +1,1 @@+0.6
+ cbits/bearssl/aes_ct64.c view
@@ -0,0 +1,398 @@+/*+ * Copyright (c) 2016 Thomas Pornin <pornin@bolet.org>+ *+ * Permission is hereby granted, free of charge, to any person obtaining + * a copy of this software and associated documentation files (the+ * "Software"), to deal in the Software without restriction, including+ * without limitation the rights to use, copy, modify, merge, publish,+ * distribute, sublicense, and/or sell copies of the Software, and to+ * permit persons to whom the Software is furnished to do so, subject to+ * the following conditions:+ *+ * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software.+ *+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+ * SOFTWARE.+ */++#include "inner.h"++/* see inner.h */+void+br_aes_ct64_bitslice_Sbox(uint64_t *q)+{+	/*+	 * This S-box implementation is a straightforward translation of+	 * the circuit described by Boyar and Peralta in "A new+	 * combinational logic minimization technique with applications+	 * to cryptology" (https://eprint.iacr.org/2009/191.pdf).+	 *+	 * Note that variables x* (input) and s* (output) are numbered+	 * in "reverse" order (x0 is the high bit, x7 is the low bit).+	 */++	uint64_t x0, x1, x2, x3, x4, x5, x6, x7;+	uint64_t y1, y2, y3, y4, y5, y6, y7, y8, y9;+	uint64_t y10, y11, y12, y13, y14, y15, y16, y17, y18, y19;+	uint64_t y20, y21;+	uint64_t z0, z1, z2, z3, z4, z5, z6, z7, z8, z9;+	uint64_t z10, z11, z12, z13, z14, z15, z16, z17;+	uint64_t t0, t1, t2, t3, t4, t5, t6, t7, t8, t9;+	uint64_t t10, t11, t12, t13, t14, t15, t16, t17, t18, t19;+	uint64_t t20, t21, t22, t23, t24, t25, t26, t27, t28, t29;+	uint64_t t30, t31, t32, t33, t34, t35, t36, t37, t38, t39;+	uint64_t t40, t41, t42, t43, t44, t45, t46, t47, t48, t49;+	uint64_t t50, t51, t52, t53, t54, t55, t56, t57, t58, t59;+	uint64_t t60, t61, t62, t63, t64, t65, t66, t67;+	uint64_t s0, s1, s2, s3, s4, s5, s6, s7;++	x0 = q[7];+	x1 = q[6];+	x2 = q[5];+	x3 = q[4];+	x4 = q[3];+	x5 = q[2];+	x6 = q[1];+	x7 = q[0];++	/*+	 * Top linear transformation.+	 */+	y14 = x3 ^ x5;+	y13 = x0 ^ x6;+	y9 = x0 ^ x3;+	y8 = x0 ^ x5;+	t0 = x1 ^ x2;+	y1 = t0 ^ x7;+	y4 = y1 ^ x3;+	y12 = y13 ^ y14;+	y2 = y1 ^ x0;+	y5 = y1 ^ x6;+	y3 = y5 ^ y8;+	t1 = x4 ^ y12;+	y15 = t1 ^ x5;+	y20 = t1 ^ x1;+	y6 = y15 ^ x7;+	y10 = y15 ^ t0;+	y11 = y20 ^ y9;+	y7 = x7 ^ y11;+	y17 = y10 ^ y11;+	y19 = y10 ^ y8;+	y16 = t0 ^ y11;+	y21 = y13 ^ y16;+	y18 = x0 ^ y16;++	/*+	 * Non-linear section.+	 */+	t2 = y12 & y15;+	t3 = y3 & y6;+	t4 = t3 ^ t2;+	t5 = y4 & x7;+	t6 = t5 ^ t2;+	t7 = y13 & y16;+	t8 = y5 & y1;+	t9 = t8 ^ t7;+	t10 = y2 & y7;+	t11 = t10 ^ t7;+	t12 = y9 & y11;+	t13 = y14 & y17;+	t14 = t13 ^ t12;+	t15 = y8 & y10;+	t16 = t15 ^ t12;+	t17 = t4 ^ t14;+	t18 = t6 ^ t16;+	t19 = t9 ^ t14;+	t20 = t11 ^ t16;+	t21 = t17 ^ y20;+	t22 = t18 ^ y19;+	t23 = t19 ^ y21;+	t24 = t20 ^ y18;++	t25 = t21 ^ t22;+	t26 = t21 & t23;+	t27 = t24 ^ t26;+	t28 = t25 & t27;+	t29 = t28 ^ t22;+	t30 = t23 ^ t24;+	t31 = t22 ^ t26;+	t32 = t31 & t30;+	t33 = t32 ^ t24;+	t34 = t23 ^ t33;+	t35 = t27 ^ t33;+	t36 = t24 & t35;+	t37 = t36 ^ t34;+	t38 = t27 ^ t36;+	t39 = t29 & t38;+	t40 = t25 ^ t39;++	t41 = t40 ^ t37;+	t42 = t29 ^ t33;+	t43 = t29 ^ t40;+	t44 = t33 ^ t37;+	t45 = t42 ^ t41;+	z0 = t44 & y15;+	z1 = t37 & y6;+	z2 = t33 & x7;+	z3 = t43 & y16;+	z4 = t40 & y1;+	z5 = t29 & y7;+	z6 = t42 & y11;+	z7 = t45 & y17;+	z8 = t41 & y10;+	z9 = t44 & y12;+	z10 = t37 & y3;+	z11 = t33 & y4;+	z12 = t43 & y13;+	z13 = t40 & y5;+	z14 = t29 & y2;+	z15 = t42 & y9;+	z16 = t45 & y14;+	z17 = t41 & y8;++	/*+	 * Bottom linear transformation.+	 */+	t46 = z15 ^ z16;+	t47 = z10 ^ z11;+	t48 = z5 ^ z13;+	t49 = z9 ^ z10;+	t50 = z2 ^ z12;+	t51 = z2 ^ z5;+	t52 = z7 ^ z8;+	t53 = z0 ^ z3;+	t54 = z6 ^ z7;+	t55 = z16 ^ z17;+	t56 = z12 ^ t48;+	t57 = t50 ^ t53;+	t58 = z4 ^ t46;+	t59 = z3 ^ t54;+	t60 = t46 ^ t57;+	t61 = z14 ^ t57;+	t62 = t52 ^ t58;+	t63 = t49 ^ t58;+	t64 = z4 ^ t59;+	t65 = t61 ^ t62;+	t66 = z1 ^ t63;+	s0 = t59 ^ t63;+	s6 = t56 ^ ~t62;+	s7 = t48 ^ ~t60;+	t67 = t64 ^ t65;+	s3 = t53 ^ t66;+	s4 = t51 ^ t66;+	s5 = t47 ^ t65;+	s1 = t64 ^ ~s3;+	s2 = t55 ^ ~t67;++	q[7] = s0;+	q[6] = s1;+	q[5] = s2;+	q[4] = s3;+	q[3] = s4;+	q[2] = s5;+	q[1] = s6;+	q[0] = s7;+}++/* see inner.h */+void+br_aes_ct64_ortho(uint64_t *q)+{+#define SWAPN(cl, ch, s, x, y)   do { \+		uint64_t a, b; \+		a = (x); \+		b = (y); \+		(x) = (a & (uint64_t)cl) | ((b & (uint64_t)cl) << (s)); \+		(y) = ((a & (uint64_t)ch) >> (s)) | (b & (uint64_t)ch); \+	} while (0)++#define SWAP2(x, y)    SWAPN(0x5555555555555555, 0xAAAAAAAAAAAAAAAA,  1, x, y)+#define SWAP4(x, y)    SWAPN(0x3333333333333333, 0xCCCCCCCCCCCCCCCC,  2, x, y)+#define SWAP8(x, y)    SWAPN(0x0F0F0F0F0F0F0F0F, 0xF0F0F0F0F0F0F0F0,  4, x, y)++	SWAP2(q[0], q[1]);+	SWAP2(q[2], q[3]);+	SWAP2(q[4], q[5]);+	SWAP2(q[6], q[7]);++	SWAP4(q[0], q[2]);+	SWAP4(q[1], q[3]);+	SWAP4(q[4], q[6]);+	SWAP4(q[5], q[7]);++	SWAP8(q[0], q[4]);+	SWAP8(q[1], q[5]);+	SWAP8(q[2], q[6]);+	SWAP8(q[3], q[7]);+}++/* see inner.h */+void+br_aes_ct64_interleave_in(uint64_t *q0, uint64_t *q1, const uint32_t *w)+{+	uint64_t x0, x1, x2, x3;++	x0 = w[0];+	x1 = w[1];+	x2 = w[2];+	x3 = w[3];+	x0 |= (x0 << 16);+	x1 |= (x1 << 16);+	x2 |= (x2 << 16);+	x3 |= (x3 << 16);+	x0 &= (uint64_t)0x0000FFFF0000FFFF;+	x1 &= (uint64_t)0x0000FFFF0000FFFF;+	x2 &= (uint64_t)0x0000FFFF0000FFFF;+	x3 &= (uint64_t)0x0000FFFF0000FFFF;+	x0 |= (x0 << 8);+	x1 |= (x1 << 8);+	x2 |= (x2 << 8);+	x3 |= (x3 << 8);+	x0 &= (uint64_t)0x00FF00FF00FF00FF;+	x1 &= (uint64_t)0x00FF00FF00FF00FF;+	x2 &= (uint64_t)0x00FF00FF00FF00FF;+	x3 &= (uint64_t)0x00FF00FF00FF00FF;+	*q0 = x0 | (x2 << 8);+	*q1 = x1 | (x3 << 8);+}++/* see inner.h */+void+br_aes_ct64_interleave_out(uint32_t *w, uint64_t q0, uint64_t q1)+{+	uint64_t x0, x1, x2, x3;++	x0 = q0 & (uint64_t)0x00FF00FF00FF00FF;+	x1 = q1 & (uint64_t)0x00FF00FF00FF00FF;+	x2 = (q0 >> 8) & (uint64_t)0x00FF00FF00FF00FF;+	x3 = (q1 >> 8) & (uint64_t)0x00FF00FF00FF00FF;+	x0 |= (x0 >> 8);+	x1 |= (x1 >> 8);+	x2 |= (x2 >> 8);+	x3 |= (x3 >> 8);+	x0 &= (uint64_t)0x0000FFFF0000FFFF;+	x1 &= (uint64_t)0x0000FFFF0000FFFF;+	x2 &= (uint64_t)0x0000FFFF0000FFFF;+	x3 &= (uint64_t)0x0000FFFF0000FFFF;+	w[0] = (uint32_t)x0 | (uint32_t)(x0 >> 16);+	w[1] = (uint32_t)x1 | (uint32_t)(x1 >> 16);+	w[2] = (uint32_t)x2 | (uint32_t)(x2 >> 16);+	w[3] = (uint32_t)x3 | (uint32_t)(x3 >> 16);+}++static const unsigned char Rcon[] = {+	0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1B, 0x36+};++static uint32_t+sub_word(uint32_t x)+{+	uint64_t q[8];++	memset(q, 0, sizeof q);+	q[0] = x;+	br_aes_ct64_ortho(q);+	br_aes_ct64_bitslice_Sbox(q);+	br_aes_ct64_ortho(q);+	return (uint32_t)q[0];+}++/* see inner.h */+unsigned+br_aes_ct64_keysched(uint64_t *comp_skey, const void *key, size_t key_len)+{+	unsigned num_rounds;+	int i, j, k, nk, nkf;+	uint32_t tmp;+	uint32_t skey[60];++	switch (key_len) {+	case 16:+		num_rounds = 10;+		break;+	case 24:+		num_rounds = 12;+		break;+	case 32:+		num_rounds = 14;+		break;+	default:+		/* abort(); */+		return 0;+	}+	nk = (int)(key_len >> 2);+	nkf = (int)((num_rounds + 1) << 2);+	br_range_dec32le(skey, (key_len >> 2), key);+	tmp = skey[(key_len >> 2) - 1];+	for (i = nk, j = 0, k = 0; i < nkf; i ++) {+		if (j == 0) {+			tmp = (tmp << 24) | (tmp >> 8);+			tmp = sub_word(tmp) ^ Rcon[k];+		} else if (nk > 6 && j == 4) {+			tmp = sub_word(tmp);+		}+		tmp ^= skey[i - nk];+		skey[i] = tmp;+		if (++ j == nk) {+			j = 0;+			k ++;+		}+	}++	for (i = 0, j = 0; i < nkf; i += 4, j += 2) {+		uint64_t q[8];++		br_aes_ct64_interleave_in(&q[0], &q[4], skey + i);+		q[1] = q[0];+		q[2] = q[0];+		q[3] = q[0];+		q[5] = q[4];+		q[6] = q[4];+		q[7] = q[4];+		br_aes_ct64_ortho(q);+		comp_skey[j + 0] =+			  (q[0] & (uint64_t)0x1111111111111111)+			| (q[1] & (uint64_t)0x2222222222222222)+			| (q[2] & (uint64_t)0x4444444444444444)+			| (q[3] & (uint64_t)0x8888888888888888);+		comp_skey[j + 1] =+			  (q[4] & (uint64_t)0x1111111111111111)+			| (q[5] & (uint64_t)0x2222222222222222)+			| (q[6] & (uint64_t)0x4444444444444444)+			| (q[7] & (uint64_t)0x8888888888888888);+	}+	return num_rounds;+}++/* see inner.h */+void+br_aes_ct64_skey_expand(uint64_t *skey,+	unsigned num_rounds, const uint64_t *comp_skey)+{+	unsigned u, v, n;++	n = (num_rounds + 1) << 1;+	for (u = 0, v = 0; u < n; u ++, v += 4) {+		uint64_t x0, x1, x2, x3;++		x0 = x1 = x2 = x3 = comp_skey[u];+		x0 &= (uint64_t)0x1111111111111111;+		x1 &= (uint64_t)0x2222222222222222;+		x2 &= (uint64_t)0x4444444444444444;+		x3 &= (uint64_t)0x8888888888888888;+		x1 >>= 1;+		x2 >>= 2;+		x3 >>= 3;+		skey[v + 0] = (x0 << 4) - x0;+		skey[v + 1] = (x1 << 4) - x1;+		skey[v + 2] = (x2 << 4) - x2;+		skey[v + 3] = (x3 << 4) - x3;+	}+}
+ cbits/bearssl/aes_ct64_dec.c view
@@ -0,0 +1,159 @@+/*+ * Copyright (c) 2016 Thomas Pornin <pornin@bolet.org>+ *+ * Permission is hereby granted, free of charge, to any person obtaining + * a copy of this software and associated documentation files (the+ * "Software"), to deal in the Software without restriction, including+ * without limitation the rights to use, copy, modify, merge, publish,+ * distribute, sublicense, and/or sell copies of the Software, and to+ * permit persons to whom the Software is furnished to do so, subject to+ * the following conditions:+ *+ * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software.+ *+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+ * SOFTWARE.+ */++#include "inner.h"++/* see inner.h */+void+br_aes_ct64_bitslice_invSbox(uint64_t *q)+{+	/*+	 * See br_aes_ct_bitslice_invSbox(). This is the natural extension+	 * to 64-bit registers.+	 */+	uint64_t q0, q1, q2, q3, q4, q5, q6, q7;++	q0 = ~q[0];+	q1 = ~q[1];+	q2 = q[2];+	q3 = q[3];+	q4 = q[4];+	q5 = ~q[5];+	q6 = ~q[6];+	q7 = q[7];+	q[7] = q1 ^ q4 ^ q6;+	q[6] = q0 ^ q3 ^ q5;+	q[5] = q7 ^ q2 ^ q4;+	q[4] = q6 ^ q1 ^ q3;+	q[3] = q5 ^ q0 ^ q2;+	q[2] = q4 ^ q7 ^ q1;+	q[1] = q3 ^ q6 ^ q0;+	q[0] = q2 ^ q5 ^ q7;++	br_aes_ct64_bitslice_Sbox(q);++	q0 = ~q[0];+	q1 = ~q[1];+	q2 = q[2];+	q3 = q[3];+	q4 = q[4];+	q5 = ~q[5];+	q6 = ~q[6];+	q7 = q[7];+	q[7] = q1 ^ q4 ^ q6;+	q[6] = q0 ^ q3 ^ q5;+	q[5] = q7 ^ q2 ^ q4;+	q[4] = q6 ^ q1 ^ q3;+	q[3] = q5 ^ q0 ^ q2;+	q[2] = q4 ^ q7 ^ q1;+	q[1] = q3 ^ q6 ^ q0;+	q[0] = q2 ^ q5 ^ q7;+}++static void+add_round_key(uint64_t *q, const uint64_t *sk)+{+	int i;++	for (i = 0; i < 8; i ++) {+		q[i] ^= sk[i];+	}+}++static void+inv_shift_rows(uint64_t *q)+{+	int i;++	for (i = 0; i < 8; i ++) {+		uint64_t x;++		x = q[i];+		q[i] = (x & (uint64_t)0x000000000000FFFF)+			| ((x & (uint64_t)0x000000000FFF0000) << 4)+			| ((x & (uint64_t)0x00000000F0000000) >> 12)+			| ((x & (uint64_t)0x000000FF00000000) << 8)+			| ((x & (uint64_t)0x0000FF0000000000) >> 8)+			| ((x & (uint64_t)0x000F000000000000) << 12)+			| ((x & (uint64_t)0xFFF0000000000000) >> 4);+	}+}++static inline uint64_t+rotr32(uint64_t x)+{+	return (x << 32) | (x >> 32);+}++static void+inv_mix_columns(uint64_t *q)+{+	uint64_t q0, q1, q2, q3, q4, q5, q6, q7;+	uint64_t r0, r1, r2, r3, r4, r5, r6, r7;++	q0 = q[0];+	q1 = q[1];+	q2 = q[2];+	q3 = q[3];+	q4 = q[4];+	q5 = q[5];+	q6 = q[6];+	q7 = q[7];+	r0 = (q0 >> 16) | (q0 << 48);+	r1 = (q1 >> 16) | (q1 << 48);+	r2 = (q2 >> 16) | (q2 << 48);+	r3 = (q3 >> 16) | (q3 << 48);+	r4 = (q4 >> 16) | (q4 << 48);+	r5 = (q5 >> 16) | (q5 << 48);+	r6 = (q6 >> 16) | (q6 << 48);+	r7 = (q7 >> 16) | (q7 << 48);++	q[0] = q5 ^ q6 ^ q7 ^ r0 ^ r5 ^ r7 ^ rotr32(q0 ^ q5 ^ q6 ^ r0 ^ r5);+	q[1] = q0 ^ q5 ^ r0 ^ r1 ^ r5 ^ r6 ^ r7 ^ rotr32(q1 ^ q5 ^ q7 ^ r1 ^ r5 ^ r6);+	q[2] = q0 ^ q1 ^ q6 ^ r1 ^ r2 ^ r6 ^ r7 ^ rotr32(q0 ^ q2 ^ q6 ^ r2 ^ r6 ^ r7);+	q[3] = q0 ^ q1 ^ q2 ^ q5 ^ q6 ^ r0 ^ r2 ^ r3 ^ r5 ^ rotr32(q0 ^ q1 ^ q3 ^ q5 ^ q6 ^ q7 ^ r0 ^ r3 ^ r5 ^ r7);+	q[4] = q1 ^ q2 ^ q3 ^ q5 ^ r1 ^ r3 ^ r4 ^ r5 ^ r6 ^ r7 ^ rotr32(q1 ^ q2 ^ q4 ^ q5 ^ q7 ^ r1 ^ r4 ^ r5 ^ r6);+	q[5] = q2 ^ q3 ^ q4 ^ q6 ^ r2 ^ r4 ^ r5 ^ r6 ^ r7 ^ rotr32(q2 ^ q3 ^ q5 ^ q6 ^ r2 ^ r5 ^ r6 ^ r7);+	q[6] = q3 ^ q4 ^ q5 ^ q7 ^ r3 ^ r5 ^ r6 ^ r7 ^ rotr32(q3 ^ q4 ^ q6 ^ q7 ^ r3 ^ r6 ^ r7);+	q[7] = q4 ^ q5 ^ q6 ^ r4 ^ r6 ^ r7 ^ rotr32(q4 ^ q5 ^ q7 ^ r4 ^ r7);+}++/* see inner.h */+void+br_aes_ct64_bitslice_decrypt(unsigned num_rounds,+	const uint64_t *skey, uint64_t *q)+{+	unsigned u;++	add_round_key(q, skey + (num_rounds << 3));+	for (u = num_rounds - 1; u > 0; u --) {+		inv_shift_rows(q);+		br_aes_ct64_bitslice_invSbox(q);+		add_round_key(q, skey + (u << 3));+		inv_mix_columns(q);+	}+	inv_shift_rows(q);+	br_aes_ct64_bitslice_invSbox(q);+	add_round_key(q, skey);+}
+ cbits/bearssl/aes_ct64_enc.c view
@@ -0,0 +1,115 @@+/*+ * Copyright (c) 2016 Thomas Pornin <pornin@bolet.org>+ *+ * Permission is hereby granted, free of charge, to any person obtaining + * a copy of this software and associated documentation files (the+ * "Software"), to deal in the Software without restriction, including+ * without limitation the rights to use, copy, modify, merge, publish,+ * distribute, sublicense, and/or sell copies of the Software, and to+ * permit persons to whom the Software is furnished to do so, subject to+ * the following conditions:+ *+ * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software.+ *+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+ * SOFTWARE.+ */++#include "inner.h"++static inline void+add_round_key(uint64_t *q, const uint64_t *sk)+{+	q[0] ^= sk[0];+	q[1] ^= sk[1];+	q[2] ^= sk[2];+	q[3] ^= sk[3];+	q[4] ^= sk[4];+	q[5] ^= sk[5];+	q[6] ^= sk[6];+	q[7] ^= sk[7];+}++static inline void+shift_rows(uint64_t *q)+{+	int i;++	for (i = 0; i < 8; i ++) {+		uint64_t x;++		x = q[i];+		q[i] = (x & (uint64_t)0x000000000000FFFF)+			| ((x & (uint64_t)0x00000000FFF00000) >> 4)+			| ((x & (uint64_t)0x00000000000F0000) << 12)+			| ((x & (uint64_t)0x0000FF0000000000) >> 8)+			| ((x & (uint64_t)0x000000FF00000000) << 8)+			| ((x & (uint64_t)0xF000000000000000) >> 12)+			| ((x & (uint64_t)0x0FFF000000000000) << 4);+	}+}++static inline uint64_t+rotr32(uint64_t x)+{+	return (x << 32) | (x >> 32);+}++static inline void+mix_columns(uint64_t *q)+{+	uint64_t q0, q1, q2, q3, q4, q5, q6, q7;+	uint64_t r0, r1, r2, r3, r4, r5, r6, r7;++	q0 = q[0];+	q1 = q[1];+	q2 = q[2];+	q3 = q[3];+	q4 = q[4];+	q5 = q[5];+	q6 = q[6];+	q7 = q[7];+	r0 = (q0 >> 16) | (q0 << 48);+	r1 = (q1 >> 16) | (q1 << 48);+	r2 = (q2 >> 16) | (q2 << 48);+	r3 = (q3 >> 16) | (q3 << 48);+	r4 = (q4 >> 16) | (q4 << 48);+	r5 = (q5 >> 16) | (q5 << 48);+	r6 = (q6 >> 16) | (q6 << 48);+	r7 = (q7 >> 16) | (q7 << 48);++	q[0] = q7 ^ r7 ^ r0 ^ rotr32(q0 ^ r0);+	q[1] = q0 ^ r0 ^ q7 ^ r7 ^ r1 ^ rotr32(q1 ^ r1);+	q[2] = q1 ^ r1 ^ r2 ^ rotr32(q2 ^ r2);+	q[3] = q2 ^ r2 ^ q7 ^ r7 ^ r3 ^ rotr32(q3 ^ r3);+	q[4] = q3 ^ r3 ^ q7 ^ r7 ^ r4 ^ rotr32(q4 ^ r4);+	q[5] = q4 ^ r4 ^ r5 ^ rotr32(q5 ^ r5);+	q[6] = q5 ^ r5 ^ r6 ^ rotr32(q6 ^ r6);+	q[7] = q6 ^ r6 ^ r7 ^ rotr32(q7 ^ r7);+}++/* see inner.h */+void+br_aes_ct64_bitslice_encrypt(unsigned num_rounds,+	const uint64_t *skey, uint64_t *q)+{+	unsigned u;++	add_round_key(q, skey);+	for (u = 1; u < num_rounds; u ++) {+		br_aes_ct64_bitslice_Sbox(q);+		shift_rows(q);+		mix_columns(q);+		add_round_key(q, skey + (u << 3));+	}+	br_aes_ct64_bitslice_Sbox(q);+	shift_rows(q);+	add_round_key(q, skey + (num_rounds << 3));+}
+ cbits/bearssl/dec32le.c view
@@ -0,0 +1,38 @@+/*+ * Copyright (c) 2016 Thomas Pornin <pornin@bolet.org>+ *+ * Permission is hereby granted, free of charge, to any person obtaining + * a copy of this software and associated documentation files (the+ * "Software"), to deal in the Software without restriction, including+ * without limitation the rights to use, copy, modify, merge, publish,+ * distribute, sublicense, and/or sell copies of the Software, and to+ * permit persons to whom the Software is furnished to do so, subject to+ * the following conditions:+ *+ * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software.+ *+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+ * SOFTWARE.+ */++#include "inner.h"++/* see inner.h */+void+br_range_dec32le(uint32_t *v, size_t num, const void *src)+{+	const unsigned char *buf;++	buf = src;+	while (num -- > 0) {+		*v ++ = br_dec32le(buf);+		buf += 4;+	}+}
+ cbits/bearssl/ghash_ctmul64.c view
@@ -0,0 +1,154 @@+/*+ * Copyright (c) 2016 Thomas Pornin <pornin@bolet.org>+ *+ * Permission is hereby granted, free of charge, to any person obtaining + * a copy of this software and associated documentation files (the+ * "Software"), to deal in the Software without restriction, including+ * without limitation the rights to use, copy, modify, merge, publish,+ * distribute, sublicense, and/or sell copies of the Software, and to+ * permit persons to whom the Software is furnished to do so, subject to+ * the following conditions:+ *+ * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software.+ *+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+ * SOFTWARE.+ */++#include "inner.h"++/*+ * This is the 64-bit variant of br_ghash_ctmul32(), with 64-bit operands+ * and bit reversal of 64-bit words.+ */++static inline uint64_t+bmul64(uint64_t x, uint64_t y)+{+	uint64_t x0, x1, x2, x3;+	uint64_t y0, y1, y2, y3;+	uint64_t z0, z1, z2, z3;++	x0 = x & (uint64_t)0x1111111111111111;+	x1 = x & (uint64_t)0x2222222222222222;+	x2 = x & (uint64_t)0x4444444444444444;+	x3 = x & (uint64_t)0x8888888888888888;+	y0 = y & (uint64_t)0x1111111111111111;+	y1 = y & (uint64_t)0x2222222222222222;+	y2 = y & (uint64_t)0x4444444444444444;+	y3 = y & (uint64_t)0x8888888888888888;+	z0 = (x0 * y0) ^ (x1 * y3) ^ (x2 * y2) ^ (x3 * y1);+	z1 = (x0 * y1) ^ (x1 * y0) ^ (x2 * y3) ^ (x3 * y2);+	z2 = (x0 * y2) ^ (x1 * y1) ^ (x2 * y0) ^ (x3 * y3);+	z3 = (x0 * y3) ^ (x1 * y2) ^ (x2 * y1) ^ (x3 * y0);+	z0 &= (uint64_t)0x1111111111111111;+	z1 &= (uint64_t)0x2222222222222222;+	z2 &= (uint64_t)0x4444444444444444;+	z3 &= (uint64_t)0x8888888888888888;+	return z0 | z1 | z2 | z3;+}++static uint64_t+rev64(uint64_t x)+{+#define RMS(m, s)   do { \+		x = ((x & (uint64_t)(m)) << (s)) \+			| ((x >> (s)) & (uint64_t)(m)); \+	} while (0)++	RMS(0x5555555555555555,  1);+	RMS(0x3333333333333333,  2);+	RMS(0x0F0F0F0F0F0F0F0F,  4);+	RMS(0x00FF00FF00FF00FF,  8);+	RMS(0x0000FFFF0000FFFF, 16);+	return (x << 32) | (x >> 32);++#undef RMS+}++/* see bearssl_ghash.h */+void+br_ghash_ctmul64(void *y, const void *h, const void *data, size_t len)+{+	const unsigned char *buf, *hb;+	unsigned char *yb;+	uint64_t y0, y1;+	uint64_t h0, h1, h2, h0r, h1r, h2r;++	buf = data;+	yb = y;+	hb = h;+	y1 = br_dec64be(yb);+	y0 = br_dec64be(yb + 8);+	h1 = br_dec64be(hb);+	h0 = br_dec64be(hb + 8);+	h0r = rev64(h0);+	h1r = rev64(h1);+	h2 = h0 ^ h1;+	h2r = h0r ^ h1r;+	while (len > 0) {+		const unsigned char *src;+		unsigned char tmp[16];+		uint64_t y0r, y1r, y2, y2r;+		uint64_t z0, z1, z2, z0h, z1h, z2h;+		uint64_t v0, v1, v2, v3;++		if (len >= 16) {+			src = buf;+			buf += 16;+			len -= 16;+		} else {+			memcpy(tmp, buf, len);+			memset(tmp + len, 0, (sizeof tmp) - len);+			src = tmp;+			len = 0;+		}+		y1 ^= br_dec64be(src);+		y0 ^= br_dec64be(src + 8);++		y0r = rev64(y0);+		y1r = rev64(y1);+		y2 = y0 ^ y1;+		y2r = y0r ^ y1r;++		z0 = bmul64(y0, h0);+		z1 = bmul64(y1, h1);+		z2 = bmul64(y2, h2);+		z0h = bmul64(y0r, h0r);+		z1h = bmul64(y1r, h1r);+		z2h = bmul64(y2r, h2r);+		z2 ^= z0 ^ z1;+		z2h ^= z0h ^ z1h;+		z0h = rev64(z0h) >> 1;+		z1h = rev64(z1h) >> 1;+		z2h = rev64(z2h) >> 1;++		v0 = z0;+		v1 = z0h ^ z2;+		v2 = z1 ^ z2h;+		v3 = z1h;++		v3 = (v3 << 1) | (v2 >> 63);+		v2 = (v2 << 1) | (v1 >> 63);+		v1 = (v1 << 1) | (v0 >> 63);+		v0 = (v0 << 1);++		v2 ^= v0 ^ (v0 >> 1) ^ (v0 >> 2) ^ (v0 >> 7);+		v1 ^= (v0 << 63) ^ (v0 << 62) ^ (v0 << 57);+		v3 ^= v1 ^ (v1 >> 1) ^ (v1 >> 2) ^ (v1 >> 7);+		v2 ^= (v1 << 63) ^ (v1 << 62) ^ (v1 << 57);++		y0 = v2;+		y1 = v3;+	}++	br_enc64be(yb, y1);+	br_enc64be(yb + 8, y0);+}
+ cbits/bearssl/import.sh view
@@ -0,0 +1,34 @@+#!/bin/sh+# Re-import the vendored parts of BearSSL.+#+# Only the five files crypton calls are kept, and they are kept unmodified,+# so that `diff` against a new release is readable.  What they want from+# upstream's two-thousand-line src/inner.h is supplied by the inner.h here,+# which is crypton's and is NOT overwritten by this script.  Run this from+# cbits/bearssl:+#+#     ./import.sh [version]+#+# and commit the result together with the VERSION line it writes, so that+# the tree always says which upstream release it holds.+set -eu++VER=${1:-0.6}+HERE=$(cd "$(dirname "$0")" && pwd)+TMP=$(mktemp -d)+trap 'rm -rf "$TMP"' EXIT++curl -sSL "https://bearssl.org/bearssl-$VER.tar.gz" -o "$TMP/b.tgz"+tar xzf "$TMP/b.tgz" -C "$TMP"+SRC="$TMP/bearssl-$VER"++cp "$SRC/LICENSE.txt"                    "$HERE/LICENSE"+cp "$SRC/src/symcipher/aes_ct64.c"       "$HERE/"+cp "$SRC/src/symcipher/aes_ct64_enc.c"   "$HERE/"+cp "$SRC/src/symcipher/aes_ct64_dec.c"   "$HERE/"+cp "$SRC/src/hash/ghash_ctmul64.c"       "$HERE/"+cp "$SRC/src/codec/dec32le.c"            "$HERE/"+echo "$VER" > "$HERE/VERSION"++echo "imported BearSSL $VER"+echo "now run the differential test in cbits/tests/bearssl_diff.c"
+ cbits/bearssl/inner.h view
@@ -0,0 +1,92 @@+/*+ * Stands in for BearSSL's own src/inner.h.+ *+ * The five .c files beside this one are upstream's, byte for byte, and each+ * of them opens with #include "inner.h".  Upstream's is some two thousand+ * lines and declares the whole library; these five want six things from it.+ * So this header gives those six and nothing else, and the upstream files+ * stay unmodified -- which is what makes `diff` against a new BearSSL+ * release readable.  See README.md.+ *+ * The byte-order helpers are written here rather than copied, in their plain+ * portable form without upstream's unaligned-access fast paths, so that they+ * carry no platform configuration with them.  cbits/tests/bearssl_diff.c+ * checks the whole thing against crypton's existing implementation.+ */+#ifndef CRYPTON_BEARSSL_INNER_H+#define CRYPTON_BEARSSL_INNER_H++#include <stddef.h>+#include <stdint.h>+#include <string.h>++static inline uint32_t+br_dec32le(const void *src)+{+	const unsigned char *b = src;++	return (uint32_t)b[0]+	    | ((uint32_t)b[1] << 8)+	    | ((uint32_t)b[2] << 16)+	    | ((uint32_t)b[3] << 24);+}++static inline void+br_enc32le(void *dst, uint32_t x)+{+	unsigned char *b = dst;++	b[0] = (unsigned char)x;+	b[1] = (unsigned char)(x >> 8);+	b[2] = (unsigned char)(x >> 16);+	b[3] = (unsigned char)(x >> 24);+}++static inline uint64_t+br_dec64be(const void *src)+{+	const unsigned char *b = src;+	uint64_t x = 0;+	int i;++	for (i = 0; i < 8; i++)+		x = (x << 8) | (uint64_t)b[i];+	return x;+}++static inline void+br_enc64be(void *dst, uint64_t x)+{+	unsigned char *b = dst;+	int i;++	for (i = 7; i >= 0; i--) {+		b[i] = (unsigned char)(x & 0xff);+		x >>= 8;+	}+}++/* dec32le.c */+void br_range_dec32le(uint32_t *v, size_t num, const void *src);++/* aes_ct64.c */+void br_aes_ct64_bitslice_Sbox(uint64_t *q);+void br_aes_ct64_ortho(uint64_t *q);+void br_aes_ct64_interleave_in(uint64_t *q0, uint64_t *q1, const uint32_t *w);+void br_aes_ct64_interleave_out(uint32_t *w, uint64_t q0, uint64_t q1);+unsigned br_aes_ct64_keysched(uint64_t *comp_skey, const void *key,+	size_t key_len);+void br_aes_ct64_skey_expand(uint64_t *skey, unsigned num_rounds,+	const uint64_t *comp_skey);++/* aes_ct64_enc.c, aes_ct64_dec.c */+void br_aes_ct64_bitslice_encrypt(unsigned num_rounds, const uint64_t *skey,+	uint64_t *q);+void br_aes_ct64_bitslice_invSbox(uint64_t *q);+void br_aes_ct64_bitslice_decrypt(unsigned num_rounds, const uint64_t *skey,+	uint64_t *q);++/* ghash_ctmul64.c */+void br_ghash_ctmul64(void *y, const void *h, const void *data, size_t len);++#endif
cbits/crypton_aes.c view
@@ -38,6 +38,12 @@ #include <aes/generic.h> #include <aes/gf.h> #include <aes/x86ni.h>+#ifdef WITH_PPC8_CRYPTO+#include <aes/ppc8.h>+#endif+#ifdef WITH_GCM_FUSED+#include <aes/gcm_fused_x86.h>+#endif  void crypton_aes_generic_encrypt_ecb(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks); void crypton_aes_generic_decrypt_ecb(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks);@@ -51,6 +57,8 @@                              uint32_t spoint, aes_block *input, uint32_t nb_blocks); void crypton_aes_generic_gcm_encrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length); void crypton_aes_generic_gcm_decrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length);+void crypton_aes_bitsliced_gcm_encrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length);+void crypton_aes_bitsliced_gcm_decrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length); void crypton_aes_generic_ocb_encrypt(uint8_t *output, aes_ocb *ocb, aes_key *key, uint8_t *input, uint32_t length); void crypton_aes_generic_ocb_decrypt(uint8_t *output, aes_ocb *ocb, aes_key *key, uint8_t *input, uint32_t length); void crypton_aes_generic_ccm_encrypt(uint8_t *output, aes_ccm *ccm, aes_key *key, uint8_t *input, uint32_t length);@@ -58,6 +66,18 @@  #ifdef WITH_ARMV8_CRYPTO void crypton_aes_armv8_init(aes_key *key, uint8_t *origkey, uint8_t size);+int crypton_aes_armv8_gcm_fused_dec(uint8_t *out, const block128 *ht,+                                    aes_key *key, const uint8_t *nonce,+                                    const uint8_t *aad, uint32_t aadlen,+                                    const uint8_t *in, uint32_t inlen,+                                    const uint8_t *tag, uint32_t taglen,+                                    uint8_t *outtag);+void crypton_aes_armv8_gcm_fused(uint8_t *out, const block128 *ht,+                                 aes_key *key, const uint8_t *nonce,+                                 const uint8_t *aad, uint32_t aadlen,+                                 const uint8_t *in, uint32_t inlen,+                                 uint32_t taglen, aes_key *hpkey,+                                 uint32_t sampleoff, uint8_t *mask); #define ARMV8_DECLS(sz) \ 	void crypton_aes_armv8_encrypt_block##sz(aes_block *output, aes_key *key, aes_block *input); \ 	void crypton_aes_armv8_decrypt_block##sz(aes_block *output, aes_key *key, aes_block *input); \@@ -193,7 +213,7 @@ typedef void (*gf_mul_f)(block128 *a, const table_4bit htable); typedef void (*gf_mul4_f)(block128 *a, const block128 *blocks, const table_4bit htable); -#if defined(WITH_AESNI) || defined(WITH_ARMV8_CRYPTO)+#if defined(WITH_AESNI) || defined(WITH_ARMV8_CRYPTO) || defined(WITH_PPC8_CRYPTO) #define GET_INIT(strength) \ 	((init_f) (crypton_aes_branch_table[INIT_128 + strength])) #define GET_ECB_ENCRYPT(strength) \@@ -240,12 +260,16 @@ #define GET_ECB_DECRYPT(strength) crypton_aes_generic_decrypt_ecb #define GET_CBC_ENCRYPT(strength) crypton_aes_generic_encrypt_cbc #define GET_CBC_DECRYPT(strength) crypton_aes_generic_decrypt_cbc-#define GET_CTR_ENCRYPT(strength) crypton_aes_generic_encrypt_ctr-#define GET_C32_ENCRYPT(strength) crypton_aes_generic_encrypt_c32+/* No accelerator is compiled in, so the portable schedule is the only one+ * a key can hold and the four-block CTR is named here rather than installed+ * at run time.  This is the build ppc64le, s390x, riscv64, 32-bit ARM and+ * -f-support_aesni all get, and nothing in it reads the branch table. */+#define GET_CTR_ENCRYPT(strength) crypton_aes_bitsliced_encrypt_ctr+#define GET_C32_ENCRYPT(strength) crypton_aes_bitsliced_encrypt_c32 #define GET_XTS_ENCRYPT(strength) crypton_aes_generic_encrypt_xts #define GET_XTS_DECRYPT(strength) crypton_aes_generic_decrypt_xts-#define GET_GCM_ENCRYPT(strength) crypton_aes_generic_gcm_encrypt-#define GET_GCM_DECRYPT(strength) crypton_aes_generic_gcm_decrypt+#define GET_GCM_ENCRYPT(strength) crypton_aes_bitsliced_gcm_encrypt+#define GET_GCM_DECRYPT(strength) crypton_aes_bitsliced_gcm_decrypt #define GET_OCB_ENCRYPT(strength) crypton_aes_generic_ocb_encrypt #define GET_OCB_DECRYPT(strength) crypton_aes_generic_ocb_decrypt #define GET_CCM_ENCRYPT(strength) crypton_aes_generic_ccm_encrypt@@ -257,10 +281,6 @@ #define crypton_gf_mul4(a,b,t) crypton_aes_generic_gf_mul4(a,b,t) #endif -#define CPU_AESNI        0-#define CPU_PCLMUL       1-#define CPU_OPTION_COUNT 2- static uint8_t crypton_aes_cpu_options[CPU_OPTION_COUNT] = {};  #if defined(ARCH_X86) && defined(WITH_AESNI)@@ -410,14 +430,112 @@ } #endif -uint8_t *crypton_aes_cpu_init(void)+/* Which implementation each entry of the branch table names is decided once,+ * before anything else runs.+ *+ * It used to be decided again on every crypton_aes_initkey, which meant two+ * threads taking a key at the same time were writing the whole table at the+ * same time -- a data race for as long as the program used AES, not just at+ * the start.  ThreadSanitizer reports forty-two of them for eight threads+ * doing nothing but taking keys.  The values written are the same ones every+ * time and the table starts out holding valid generic implementations, so+ * nothing has ever come of it; it is a race the standard gives no meaning to+ * all the same.+ *+ * A constructor runs while there is one thread, which is the cheapest way to+ * have no race at all: no flag to test, no lock to take, and one less thing+ * for crypton_aes_initkey to do per key. */+/*+ * The two CTR entries, where the portable implementation is the one that+ * will run.  This is for the build that has an accelerator compiled in and+ * did not find it on the processor; a build with no accelerator at all+ * names them directly in the GET_ macros above and never reads this table.+ *+ * They cannot be the defaults.  crypton_aes_generic_encrypt_c32 is not+ * overridden on AArch64 -- the ARMv8 table has no C32 of its own -- and+ * neither CCM nor OCB is overridden anywhere, so those reach the block+ * function through the branch table and work whatever is installed.  A CTR+ * that read the portable schedule directly would be wrong on exactly those+ * machines.  So these go in only once nothing has claimed AES.+ */+static void initialize_table_bitsliced(void) {+	crypton_aes_branch_table[ENCRYPT_CTR_128] = crypton_aes_bitsliced_encrypt_ctr;+	crypton_aes_branch_table[ENCRYPT_CTR_192] = crypton_aes_bitsliced_encrypt_ctr;+	crypton_aes_branch_table[ENCRYPT_CTR_256] = crypton_aes_bitsliced_encrypt_ctr;+	crypton_aes_branch_table[ENCRYPT_C32_128] = crypton_aes_bitsliced_encrypt_c32;+	crypton_aes_branch_table[ENCRYPT_C32_192] = crypton_aes_bitsliced_encrypt_c32;+	crypton_aes_branch_table[ENCRYPT_C32_256] = crypton_aes_bitsliced_encrypt_c32;+	crypton_aes_branch_table[ENCRYPT_GCM_128] = crypton_aes_bitsliced_gcm_encrypt;+	crypton_aes_branch_table[ENCRYPT_GCM_192] = crypton_aes_bitsliced_gcm_encrypt;+	crypton_aes_branch_table[ENCRYPT_GCM_256] = crypton_aes_bitsliced_gcm_encrypt;+	crypton_aes_branch_table[DECRYPT_GCM_128] = crypton_aes_bitsliced_gcm_decrypt;+	crypton_aes_branch_table[DECRYPT_GCM_192] = crypton_aes_bitsliced_gcm_decrypt;+	crypton_aes_branch_table[DECRYPT_GCM_256] = crypton_aes_bitsliced_gcm_decrypt;+}++#ifdef WITH_PPC8_CRYPTO+/*+ * POWER8.  One function per operation rather than three: the assembly reads+ * the round count out of the key, so the same entry serves every key size.+ *+ * GCM and the 32-bit counter are left where they are.  Neither has an entry+ * in the assembly, and the generic loops reach the block function and the+ * GHASH through this table, so both get the instructions anyway.+ */+static void initialize_table_ppc8(void)+{+	int sz;++	if (!crypton_aes_ppc8_available())+		return;+	/* what stops initialize_table_bitsliced below from putting its own CTR+	 * in, which would read this key as a bitsliced schedule */+	crypton_aes_cpu_options[CPU_AESNI] = 1;+	crypton_aes_cpu_options[CPU_PCLMUL] = 1;++	for (sz = 0; sz < 3; sz++) {+		crypton_aes_branch_table[INIT_128 + sz] = crypton_aes_ppc8_init;+		crypton_aes_branch_table[ENCRYPT_BLOCK_128 + sz] = crypton_aes_ppc8_encrypt_block;+		crypton_aes_branch_table[DECRYPT_BLOCK_128 + sz] = crypton_aes_ppc8_decrypt_block;+		crypton_aes_branch_table[ENCRYPT_ECB_128 + sz] = crypton_aes_ppc8_encrypt_ecb;+		crypton_aes_branch_table[DECRYPT_ECB_128 + sz] = crypton_aes_ppc8_decrypt_ecb;+		crypton_aes_branch_table[ENCRYPT_CBC_128 + sz] = crypton_aes_ppc8_encrypt_cbc;+		crypton_aes_branch_table[DECRYPT_CBC_128 + sz] = crypton_aes_ppc8_decrypt_cbc;+		crypton_aes_branch_table[ENCRYPT_CTR_128 + sz] = crypton_aes_ppc8_encrypt_ctr;+		crypton_aes_branch_table[ENCRYPT_XTS_128 + sz] = crypton_aes_ppc8_encrypt_xts;+		crypton_aes_branch_table[DECRYPT_XTS_128 + sz] = crypton_aes_ppc8_decrypt_xts;+	}++	crypton_aes_branch_table[GHASH_HINIT]   = crypton_aes_ppc8_hinit;+	crypton_aes_branch_table[GHASH_GF_MUL]  = crypton_aes_ppc8_gf_mul;+	crypton_aes_branch_table[GHASH_GF_MUL4] = crypton_aes_ppc8_gf_mul4;+}+#endif++static void crypton_aes_cpu_setup(void)+{ #if defined(ARCH_X86) && defined(WITH_AESNI) 	crypton_aesni_initialize_hw(initialize_table_ni); #endif #ifdef WITH_ARMV8_CRYPTO 	initialize_table_armv8(); #endif+#ifdef WITH_PPC8_CRYPTO+	initialize_table_ppc8();+#endif+	if (crypton_aes_cpu_options[CPU_AESNI] == 0)+		initialize_table_bitsliced();+}++__attribute__((constructor))+static void crypton_aes_cpu_ctor(void)+{+	crypton_aes_cpu_setup();+}++uint8_t *crypton_aes_cpu_init(void)+{ 	return crypton_aes_cpu_options; } @@ -428,7 +546,6 @@ 	case 24: key->nbr = 12; key->strength = 1; break; 	case 32: key->nbr = 14; key->strength = 2; break; 	}-	crypton_aes_cpu_init(); 	init_f _init = GET_INIT(key->strength); 	_init(key, origkey, size); }@@ -532,20 +649,33 @@ 	crypton_gf_mul4(&gcm->tag, b, gcm->htable); } -void crypton_aes_gcm_init(aes_gcm *gcm, aes_key *key, uint8_t *iv, uint32_t len)+/* The part of the state that depends on the key alone: H = encrypt_K(0^128)+ * and the table of its multiples.  It is 256 of the 320 bytes, and a caller+ * that keeps a key can compute it once instead of once per message. */+void crypton_aes_gcm_key_init(aes_gcm_key *gk, aes_key *key) { 	block128 h;++	block128_zero(&h);+	crypton_aes_encrypt_block(&h, key, &h);+	crypton_hinit(gk->gcm.htable, &h);+#ifdef WITH_GCM_FUSED+	if (crypton_aes_cpu_options[CPU_AESNI]+	    && crypton_aes_cpu_options[CPU_PCLMUL])+		crypton_gcm_fused_key_init(&gk->fused, key);+#endif+}++/* Everything else: what the nonce and the message determine.  Leaves htable+ * alone, so it runs on a state whose key part is already there. */+static void gcm_message_init(aes_gcm *gcm, uint8_t *iv, uint32_t len)+{ 	gcm->length_aad = 0; 	gcm->length_input = 0; -	block128_zero(&h); 	block128_zero(&gcm->tag); 	block128_zero(&gcm->iv); -	/* prepare H : encrypt_K(0^128) */-	crypton_aes_encrypt_block(&h, key, &h);-	crypton_hinit(gcm->htable, &h);- 	if (len == 12) { 		block128_copy_bytes(&gcm->iv, iv, 12); 		gcm->iv.b[15] = 0x01;@@ -568,6 +698,16 @@ 	block128_copy_aligned(&gcm->civ, &gcm->iv); } +void crypton_aes_gcm_init(aes_gcm *gcm, aes_key *key, uint8_t *iv, uint32_t len)+{+	block128 h;++	block128_zero(&h);+	crypton_aes_encrypt_block(&h, key, &h);+	crypton_hinit(gcm->htable, &h);+	gcm_message_init(gcm, iv, len);+}+ void crypton_aes_gcm_aad(aes_gcm *gcm, uint8_t *input, uint32_t length) { 	gcm->length_aad += length;@@ -604,6 +744,195 @@ 	} } +/* One message, one call.  The key part of the state comes in already built,+ * the rest is set up on the stack, and the additional data, the encryption+ * and the tag all happen before returning, so nothing crosses a language+ * boundary between them and no intermediate state is copied out.  The output+ * buffer takes the ciphertext and then the tag, so it wants length + taglen+ * bytes. */+void crypton_aes_gcm_full_encrypt(uint8_t *output, const aes_gcm_key *gcmkey, aes_key *key,+                                  uint8_t *iv, uint32_t ivlen,+                                  uint8_t *aad, uint32_t aadlen,+                                  uint8_t *input, uint32_t length, uint32_t taglen)+{+	aes_gcm gcm;+	uint8_t tag[16];++#ifdef WITH_GCM_FUSED+	/* Short messages go the other way: the assembly below will not start+	 * on anything under 288 bytes, and under about 1.5 KB the fused path+	 * is ahead of it even where it does. */+	if (ivlen == 12 && length <= CRYPTON_GCM_FUSED_MAX_MESSAGE+	    && crypton_aes_cpu_options[CPU_AESNI]+	    && crypton_aes_cpu_options[CPU_PCLMUL]) {+		crypton_gcm_fused_encrypt(output, &gcmkey->fused, key, iv,+		                          aad, aadlen, input, length, taglen,+		                          NULL, 0, NULL);+		return;+	}+#endif+#ifdef WITH_ARMV8_CRYPTO+	/* The same on AArch64, where the framing is what costs: composing the+	 * additional data, the encryption and the tag reaches each through the+	 * branch table, so the running state goes back to memory between them+	 * and a one-block header pays a reduction of its own.  Measured on an+	 * Apple M4, a 100-byte packet is 3.0x faster taken in one call.+	 *+	 * No length limit, unlike x86: there is no vendored assembly on this+	 * side for a long message to be handed to instead, and measured+	 * against the path this replaces it is never slower -- 1.25x at 1440+	 * bytes, level from about 6 KB up. */+	if (ivlen == 12+	    && crypton_aes_cpu_options[CPU_AESNI]+	    && crypton_aes_cpu_options[CPU_PCLMUL]) {+		crypton_aes_armv8_gcm_fused(output, gcmkey->gcm.htable, key, iv,+		                            aad, aadlen, input, length, taglen,+		                            NULL, 0, NULL);+		return;+	}+#endif+	memcpy(gcm.htable, gcmkey->gcm.htable, sizeof(gcm.htable));+	gcm_message_init(&gcm, iv, ivlen);+	if (aadlen)+		crypton_aes_gcm_aad(&gcm, aad, aadlen);+	if (length)+		crypton_aes_gcm_encrypt(output, &gcm, key, input, length);+	crypton_aes_gcm_finish(tag, &gcm, key);+	memcpy(output + length, tag, taglen);+}++/* The same, and then the header protection mask.  QUIC takes its sample from+ * the ciphertext, so the mask cannot be had before the encryption -- but it+ * can be had before returning, which saves a second crossing for one AES+ * block.  The block itself is about a nanosecond; what it saves is the call.+ * sampleoff is where the sixteen bytes of sample start in the output. */+void crypton_aes_gcm_full_encrypt_mask(uint8_t *output, const aes_gcm_key *gcmkey, aes_key *key,+                                       uint8_t *iv, uint32_t ivlen,+                                       uint8_t *aad, uint32_t aadlen,+                                       uint8_t *input, uint32_t length, uint32_t taglen,+                                       aes_key *hpkey, uint32_t sampleoff, uint8_t *mask)+{+	block128 sample, m;++#ifdef WITH_GCM_FUSED+	/* Here the mask rides in a lane of the AES pipeline that the message+	 * length leaves idle, so it costs very nearly nothing on top of the+	 * encryption rather than a block of its own. */+	if (ivlen == 12 && length <= CRYPTON_GCM_FUSED_MAX_MESSAGE+	    && crypton_aes_cpu_options[CPU_AESNI]+	    && crypton_aes_cpu_options[CPU_PCLMUL]) {+		crypton_gcm_fused_encrypt(output, &gcmkey->fused, key, iv,+		                          aad, aadlen, input, length, taglen,+		                          hpkey, sampleoff, mask);+		return;+	}+#endif+#ifdef WITH_ARMV8_CRYPTO+	/* The same on AArch64, where the framing is what costs: composing the+	 * additional data, the encryption and the tag reaches each through the+	 * branch table, so the running state goes back to memory between them+	 * and a one-block header pays a reduction of its own.  Measured on an+	 * Apple M4, a 100-byte packet is 3.3x faster taken in one call. */+	if (ivlen == 12+	    && crypton_aes_cpu_options[CPU_AESNI]+	    && crypton_aes_cpu_options[CPU_PCLMUL]) {+		crypton_aes_armv8_gcm_fused(output, gcmkey->gcm.htable, key, iv,+		                            aad, aadlen, input, length, taglen,+		                            hpkey, sampleoff, mask);+		return;+	}+#endif+	crypton_aes_gcm_full_encrypt(output, gcmkey, key, iv, ivlen, aad, aadlen,+	                             input, length, taglen);+	/* copied rather than cast: the sample lands wherever the header put it+	 * and a block128 is read as 64-bit words */+	memcpy(&sample, output + sampleoff, 16);+	crypton_aes_encrypt_block(&m, hpkey, &sample);+	memcpy(mask, &m, 16);+}++/* The same the other way, with the tag checked here rather than by the+ * caller: returns 1 when it matches and 0 when it does not, comparing every+ * byte either way.  The plaintext is written whatever the answer, so a caller+ * that gets 0 must not use it. */+/* Decrypt, and either compare the tag or hand it back.+ *+ * With outtag NULL this is the verifying form: the tag is compared here, a+ * byte at a time over its whole length whichever way the answer goes, and the+ * answer is the return value.  With outtag not NULL the computed tag is+ * written there instead and the return value is 1 -- for a caller that holds+ * the expected tag in a form of its own and will compare it itself. */+static int gcm_full_decrypt(uint8_t *output, const aes_gcm_key *gcmkey, aes_key *key,+                            uint8_t *iv, uint32_t ivlen,+                            uint8_t *aad, uint32_t aadlen,+                            uint8_t *input, uint32_t length,+                            const uint8_t *tag, uint32_t taglen,+                            uint8_t *outtag)+{+	aes_gcm gcm;+	uint8_t expected[16];+	uint32_t i;+	uint8_t diff = 0;++#ifdef WITH_GCM_FUSED+	/* The same as the encryption side, and simpler: what GHASH absorbs+	 * here is the ciphertext, which is the input, so the multiplies need+	 * not wait for anything.  Measured on an Intel Haswell, a 100-byte+	 * packet was three times the cost of encrypting one before this. */+	if (ivlen == 12 && length <= CRYPTON_GCM_FUSED_MAX_MESSAGE+	    && crypton_aes_cpu_options[CPU_AESNI]+	    && crypton_aes_cpu_options[CPU_PCLMUL])+		return crypton_gcm_fused_decrypt(output, &gcmkey->fused, key,+		                                 iv, aad, aadlen, input,+		                                 length, tag, taglen, outtag);+#endif+#ifdef WITH_ARMV8_CRYPTO+	if (ivlen == 12+	    && crypton_aes_cpu_options[CPU_AESNI]+	    && crypton_aes_cpu_options[CPU_PCLMUL])+		return crypton_aes_armv8_gcm_fused_dec(output, gcmkey->gcm.htable,+		                                       key, iv, aad, aadlen,+		                                       input, length, tag, taglen,+		                                       outtag);+#endif+	memcpy(gcm.htable, gcmkey->gcm.htable, sizeof(gcm.htable));+	gcm_message_init(&gcm, iv, ivlen);+	if (aadlen)+		crypton_aes_gcm_aad(&gcm, aad, aadlen);+	if (length)+		crypton_aes_gcm_decrypt(output, &gcm, key, input, length);+	crypton_aes_gcm_finish(expected, &gcm, key);++	if (outtag) {+		memcpy(outtag, expected, taglen);+		return 1;+	}+	for (i = 0; i < taglen; i++)+		diff |= (uint8_t) (expected[i] ^ tag[i]);+	return diff == 0;+}++int crypton_aes_gcm_full_decrypt(uint8_t *output, const aes_gcm_key *gcmkey, aes_key *key,+                                 uint8_t *iv, uint32_t ivlen,+                                 uint8_t *aad, uint32_t aadlen,+                                 uint8_t *input, uint32_t length,+                                 const uint8_t *tag, uint32_t taglen)+{+	return gcm_full_decrypt(output, gcmkey, key, iv, ivlen, aad, aadlen,+	                        input, length, tag, taglen, NULL);+}++void crypton_aes_gcm_full_decrypt_tag(uint8_t *output, uint8_t *outtag,+                                      const aes_gcm_key *gcmkey, aes_key *key,+                                      uint8_t *iv, uint32_t ivlen,+                                      uint8_t *aad, uint32_t aadlen,+                                      uint8_t *input, uint32_t length,+                                      uint32_t taglen)+{+	(void) gcm_full_decrypt(output, gcmkey, key, iv, ivlen, aad, aadlen,+	                        input, length, NULL, taglen, outtag);+}+ static inline uint8_t ccm_b0_flags(uint32_t has_adata, uint32_t m, uint32_t l) { 	return 8*m + l + (has_adata? 64: 0);@@ -885,18 +1214,30 @@ 	block128_xor((block128 *) tag, &ocb->sum_aad); } +/*+ * These three reach cbits/aes/generic.c directly rather than through the+ * branch table, so they run only where the portable implementation was the+ * one installed -- which is what lets them read its schedule and hand its+ * core four blocks at a time.  The CTR entries below cannot do the same:+ * they go through the branch table for the block, and so run on accelerated+ * machines too.+ */ void crypton_aes_generic_encrypt_ecb(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks) {-	for ( ; nb_blocks-- > 0; input++, output++) {-		crypton_aes_generic_encrypt_block(output, key, input);-	}+	aes_sched sched;++	crypton_aes_generic_schedule(&sched, key);+	crypton_aes_generic_blocks((uint8_t *) output, (const uint8_t *) input,+	                           nb_blocks, &sched, 0); }  void crypton_aes_generic_decrypt_ecb(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks) {-	for ( ; nb_blocks-- > 0; input++, output++) {-		crypton_aes_generic_decrypt_block(output, key, input);-	}+	aes_sched sched;++	crypton_aes_generic_schedule(&sched, key);+	crypton_aes_generic_blocks((uint8_t *) output, (const uint8_t *) input,+	                           nb_blocks, &sched, 1); }  void crypton_aes_generic_encrypt_cbc(aes_block *output, aes_key *key, aes_block *iv, aes_block *input, uint32_t nb_blocks)@@ -912,18 +1253,35 @@ 	} } +/*+ * Decryption, unlike encryption, does not wait for the block before it:+ * every ciphertext block is ready at once and the chaining is a XOR+ * afterwards.  So four at a pass, with the ciphertext copied aside first,+ * since the caller is allowed to decrypt in place and the next group's IV+ * is the last ciphertext block of this one.+ */ void crypton_aes_generic_decrypt_cbc(aes_block *output, aes_key *key, aes_block *ivini, aes_block *input, uint32_t nb_blocks) {-	aes_block block, blocko;+	aes_sched sched; 	aes_block iv;+	uint8_t ct[64], plain[64]; -	/* preload IV in block */+	crypton_aes_generic_schedule(&sched, key); 	block128_copy(&iv, ivini);-	for ( ; nb_blocks-- > 0; input++, output++) {-		block128_copy(&block, (block128 *) input);-		crypton_aes_generic_decrypt_block(&blocko, key, &block);-		block128_vxor((block128 *) output, &blocko, &iv);-		block128_copy(&iv, &block);+	while (nb_blocks > 0) {+		uint32_t n = nb_blocks < 4 ? nb_blocks : 4;+		uint32_t i;++		memcpy(ct, input, n * 16);+		crypton_aes_generic_blocks(plain, ct, n, &sched, 1);+		for (i = 0; i < n; i++) {+			block128_vxor((block128 *) (output + i),+			              (block128 *) (plain + 16 * i), &iv);+			block128_copy(&iv, (block128 *) (ct + 16 * i));+		}+		input += n;+		output += n;+		nb_blocks -= n; 	} } @@ -978,7 +1336,9 @@ void crypton_aes_generic_encrypt_xts(aes_block *output, aes_key *k1, aes_key *k2, aes_block *dataunit,                              uint32_t spoint, aes_block *input, uint32_t nb_blocks) {-	aes_block block, tweak;+	aes_sched sched;+	aes_block tweak;+	uint8_t buf[64], tw[64];  	/* load IV and encrypt it using k2 as the tweak */ 	block128_copy(&tweak, dataunit);@@ -988,17 +1348,34 @@ 	while (spoint-- > 0) 		crypton_aes_generic_gf_mulx(&tweak); -	for ( ; nb_blocks-- > 0; input++, output++, crypton_aes_generic_gf_mulx(&tweak)) {-		block128_vxor(&block, input, &tweak);-		crypton_aes_encrypt_block(&block, k1, &block);-		block128_vxor(output, &block, &tweak);+	crypton_aes_generic_schedule(&sched, k1);+	while (nb_blocks > 0) {+		uint32_t n = nb_blocks < 4 ? nb_blocks : 4;+		uint32_t i;++		/* the tweaks for a group depend on nothing but each other, so+		 * all four are known before any block is enciphered */+		for (i = 0; i < n; i++) {+			block128_copy((block128 *) (tw + 16 * i), &tweak);+			block128_vxor((block128 *) (buf + 16 * i), input + i, &tweak);+			crypton_aes_generic_gf_mulx(&tweak);+		}+		crypton_aes_generic_blocks(buf, buf, n, &sched, 0);+		for (i = 0; i < n; i++)+			block128_vxor(output + i, (block128 *) (buf + 16 * i),+			              (block128 *) (tw + 16 * i));+		input += n;+		output += n;+		nb_blocks -= n; 	} }  void crypton_aes_generic_decrypt_xts(aes_block *output, aes_key *k1, aes_key *k2, aes_block *dataunit,                              uint32_t spoint, aes_block *input, uint32_t nb_blocks) {-	aes_block block, tweak;+	aes_sched sched;+	aes_block tweak;+	uint8_t buf[64], tw[64];  	/* load IV and encrypt it using k2 as the tweak */ 	block128_copy(&tweak, dataunit);@@ -1008,10 +1385,25 @@ 	while (spoint-- > 0) 		crypton_aes_generic_gf_mulx(&tweak); -	for ( ; nb_blocks-- > 0; input++, output++, crypton_aes_generic_gf_mulx(&tweak)) {-		block128_vxor(&block, input, &tweak);-		crypton_aes_decrypt_block(&block, k1, &block);-		block128_vxor(output, &block, &tweak);+	crypton_aes_generic_schedule(&sched, k1);+	while (nb_blocks > 0) {+		uint32_t n = nb_blocks < 4 ? nb_blocks : 4;+		uint32_t i;++		/* the tweaks for a group depend on nothing but each other, so+		 * all four are known before any block is enciphered */+		for (i = 0; i < n; i++) {+			block128_copy((block128 *) (tw + 16 * i), &tweak);+			block128_vxor((block128 *) (buf + 16 * i), input + i, &tweak);+			crypton_aes_generic_gf_mulx(&tweak);+		}+		crypton_aes_generic_blocks(buf, buf, n, &sched, 1);+		for (i = 0; i < n; i++)+			block128_vxor(output + i, (block128 *) (buf + 16 * i),+			              (block128 *) (tw + 16 * i));+		input += n;+		output += n;+		nb_blocks -= n; 	} } @@ -1240,6 +1632,116 @@ 		block128_zero(&tmp); 		block128_copy_bytes(&tmp, input, length); 		ccm_cbcmac_add(ccm, key, &tmp);+	}+}++/*+ * GCM where the portable implementation is the one that will run.+ *+ * The same shape as the two above, with two differences: the four counter+ * blocks of a group are encrypted together, which is what the bitsliced+ * core is for, and the schedule is expanded once for the message rather+ * than once per block.+ *+ * These cannot replace the generic pair.  That pair also runs on an x86+ * machine that has AES-NI and no carry-less multiply, where crypton_aes.c+ * installs the AES-NI block function and leaves GCM alone -- and there the+ * key holds the AES-NI schedule, not this one.  So the choice is made where+ * the rest of the portable entries are chosen.+ */+void crypton_aes_bitsliced_gcm_encrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length)+{+	aes_sched sched;+	aes_block out;++	crypton_aes_generic_schedule(&sched, key);+	gcm->length_input += length;+	for (; length >= 64; input += 64, output += 64, length -= 64) {+		aes_block buf[4];+		int i;++		for (i = 0; i < 4; i++) {+			block128_inc32_be(&gcm->civ);+			block128_copy(&buf[i], &gcm->civ);+		}+		crypton_aes_generic_blocks((uint8_t *) buf, (const uint8_t *) buf,+		                           4, &sched, 0);+		for (i = 0; i < 4; i++)+			block128_xor(&buf[i], (block128 *) (input + 16 * i));+		gcm_ghash_add4(gcm, buf);+		for (i = 0; i < 4; i++)+			block128_copy((block128 *) (output + 16 * i), &buf[i]);+	}+	for (; length >= 16; input += 16, output += 16, length -= 16) {+		block128_inc32_be(&gcm->civ);+		crypton_aes_generic_blocks((uint8_t *) &out,+		                           (const uint8_t *) &gcm->civ, 1, &sched, 0);+		block128_xor(&out, (block128 *) input);+		gcm_ghash_add(gcm, &out);+		block128_copy((block128 *) output, &out);+	}+	if (length > 0) {+		aes_block tmp;+		uint32_t i;++		block128_inc32_be(&gcm->civ);+		crypton_aes_generic_blocks((uint8_t *) &out,+		                           (const uint8_t *) &gcm->civ, 1, &sched, 0);+		block128_zero(&tmp);+		block128_copy_bytes(&tmp, input, length);+		block128_xor_bytes(&tmp, out.b, length);+		gcm_ghash_add(gcm, &tmp);+		for (i = 0; i < length; i++)+			output[i] = tmp.b[i];+	}+}++void crypton_aes_bitsliced_gcm_decrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length)+{+	aes_sched sched;+	aes_block out;++	crypton_aes_generic_schedule(&sched, key);+	gcm->length_input += length;+	/* GHASH all four ciphertext blocks before writing any plaintext, since+	 * output may be input */+	for (; length >= 64; input += 64, output += 64, length -= 64) {+		aes_block buf[4];+		int i;++		gcm_ghash_add4(gcm, (const block128 *) input);+		for (i = 0; i < 4; i++) {+			block128_inc32_be(&gcm->civ);+			block128_copy(&buf[i], &gcm->civ);+		}+		crypton_aes_generic_blocks((uint8_t *) buf, (const uint8_t *) buf,+		                           4, &sched, 0);+		for (i = 0; i < 4; i++) {+			block128_xor(&buf[i], (block128 *) (input + 16 * i));+			block128_copy((block128 *) (output + 16 * i), &buf[i]);+		}+	}+	for (; length >= 16; input += 16, output += 16, length -= 16) {+		block128_inc32_be(&gcm->civ);+		crypton_aes_generic_blocks((uint8_t *) &out,+		                           (const uint8_t *) &gcm->civ, 1, &sched, 0);+		gcm_ghash_add(gcm, (block128 *) input);+		block128_xor(&out, (block128 *) input);+		block128_copy((block128 *) output, &out);+	}+	if (length > 0) {+		aes_block tmp;+		uint32_t i;++		block128_inc32_be(&gcm->civ);+		crypton_aes_generic_blocks((uint8_t *) &out,+		                           (const uint8_t *) &gcm->civ, 1, &sched, 0);+		block128_zero(&tmp);+		block128_copy_bytes(&tmp, input, length);+		gcm_ghash_add(gcm, &tmp);+		block128_xor_bytes(&tmp, out.b, length);+		for (i = 0; i < length; i++)+			output[i] = tmp.b[i]; 	} }
cbits/crypton_aes.h view
@@ -55,6 +55,55 @@ 	uint64_t length_input; } aes_gcm; +/*+ * How many powers of H a key keeps for the fused path in+ * cbits/aes/gcm_fused_x86.c.  A power for every block of the message would+ * fold its whole GHASH into one reduction, which is what picotls does, but+ * then the state grows with the longest message a caller might send and a+ * server holding many keys pays it for each.  A fixed count costs one+ * reduction per this many blocks and keeps the state one size.  Sixteen was+ * measured against 6, 8, 32, 64, 96 and 256: above eight the choice is worth+ * about two per cent, since only messages short enough to take this path at+ * all reach a second batch.  Six is worth avoiding -- at 1440 bytes it is+ * slower than not taking the path.+ */+#define CRYPTON_GCM_FUSED_POWERS 16++/*+ * Beyond this many bytes the stitched assembly in cbits/asm is faster than+ * the fused path, so longer messages go there instead.  Measured on an Intel+ * Haswell: even at 1440 bytes, the assembly ahead by 12 per cent at 3 KB and+ * 20 per cent at 16 KB, and the fused path ahead by 1.9x at 100 bytes and+ * 1.16x at 1200.  QUIC packets fall below this; TLS records do not.+ */+#define CRYPTON_GCM_FUSED_MAX_MESSAGE 1536++/*+ * The powers themselves, each shifted up by one bit, and beside each the+ * halves of it added together for the Karatsuba term.  The two are kept+ * adjacent rather than in two arrays: a multiply wants both, and two arrays+ * put them 256 bytes apart, which is two cache lines where this is one.+ *+ * Defined on every platform so that the key state below is one size+ * everywhere; filled only where the fused path is compiled in.+ */+typedef struct {+	struct {+		aes_block h;+		aes_block r;+	} p[CRYPTON_GCM_FUSED_POWERS];+} aes_gcm_fused;++/*+ * Everything a key determines, built once by crypton_aes_gcm_key_init and+ * read by every message sent under that key: the key half of a GCM state,+ * and the powers of H the fused path reads.  832 bytes.+ */+typedef struct {+	aes_gcm gcm;+	aes_gcm_fused fused;+} aes_gcm_key;+ /* size = 4*16+4*4= 80 */ typedef struct { 	aes_block xi;@@ -104,6 +153,27 @@                      uint32_t spoint, aes_block *input, uint32_t nb_blocks);  void crypton_aes_gcm_init(aes_gcm *gcm, aes_key *key, uint8_t *iv, uint32_t len);+void crypton_aes_gcm_key_init(aes_gcm_key *gk, aes_key *key);+void crypton_aes_gcm_full_encrypt(uint8_t *output, const aes_gcm_key *gcmkey, aes_key *key,+                                  uint8_t *iv, uint32_t ivlen,+                                  uint8_t *aad, uint32_t aadlen,+                                  uint8_t *input, uint32_t length, uint32_t taglen);+void crypton_aes_gcm_full_encrypt_mask(uint8_t *output, const aes_gcm_key *gcmkey, aes_key *key,+                                       uint8_t *iv, uint32_t ivlen,+                                       uint8_t *aad, uint32_t aadlen,+                                       uint8_t *input, uint32_t length, uint32_t taglen,+                                       aes_key *hpkey, uint32_t sampleoff, uint8_t *mask);+int crypton_aes_gcm_full_decrypt(uint8_t *output, const aes_gcm_key *gcmkey, aes_key *key,+                                 uint8_t *iv, uint32_t ivlen,+                                 uint8_t *aad, uint32_t aadlen,+                                 uint8_t *input, uint32_t length,+                                 const uint8_t *tag, uint32_t taglen);+void crypton_aes_gcm_full_decrypt_tag(uint8_t *output, uint8_t *outtag,+                                      const aes_gcm_key *gcmkey, aes_key *key,+                                      uint8_t *iv, uint32_t ivlen,+                                      uint8_t *aad, uint32_t aadlen,+                                      uint8_t *input, uint32_t length,+                                      uint32_t taglen); void crypton_aes_gcm_aad(aes_gcm *gcm, uint8_t *input, uint32_t length); void crypton_aes_gcm_encrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length); void crypton_aes_gcm_decrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length);
cbits/crypton_align.h view
@@ -3,6 +3,8 @@  #include "crypton_bitfn.h" +#include <string.h>+ #if (defined(__i386__)) # define UNALIGNED_ACCESS_OK #elif defined(__x86_64__)@@ -34,107 +36,119 @@ #define need_alignment(p,n) IS_ALIGNED(p,n) #endif -static inline uint32_t load_le32_aligned(const uint8_t *p)-{-	return le32_to_cpu(*((uint32_t *) p));		-}+/*+ * Reading and writing a 32- or 64-bit word at a byte pointer.+ *+ * Through memcpy, not a cast to uint32_t * or uint64_t *.  A cast is two+ * things the standard does not allow -- a read of the value through the+ * wrong type, and a read at an address that type is not aligned for -- and+ * this file used to do both wherever UNALIGNED_ACCESS_OK is defined, which+ * is i386 and x86-64.  UndefinedBehaviorSanitizer reported seventy-eight+ * lines of it.+ *+ * Every compiler crypton is built with turns a memcpy of four or eight bytes+ * into the one load or store the cast used to be, so this is the same code+ * with none of the licence.  Where the target cannot do an unaligned load,+ * the compiler is the one that knows, and it emits what the target needs --+ * which is what the byte-at-a-time versions this replaces were for.+ *+ * The _aligned names stay because nineteen files use them.  They no longer+ * ask anything of the pointer.+ */ -static inline void store_le32_aligned(uint8_t *dst, const uint32_t v)+static inline uint32_t load_le32(const uint8_t *p) {-	*((uint32_t *) dst) = cpu_to_le32(v);-}+	uint32_t v; -static inline void xor_le32_aligned(uint8_t *dst, const uint32_t v)-{-	*((uint32_t *) dst) ^= cpu_to_le32(v);+	memcpy(&v, p, sizeof(v));+	return le32_to_cpu(v); } -static inline void store_be32_aligned(uint8_t *dst, const uint32_t v)+static inline uint64_t load_le64(const uint8_t *p) {-	*((uint32_t *) dst) = cpu_to_be32(v);-}+	uint64_t v; -static inline void xor_be32_aligned(uint8_t *dst, const uint32_t v)-{-	*((uint32_t *) dst) ^= cpu_to_be32(v);+	memcpy(&v, p, sizeof(v));+	return le64_to_cpu(v); } -static inline void store_le64_aligned(uint8_t *dst, const uint64_t v)+static inline uint32_t load_be32(const uint8_t *p) {-	*((uint64_t *) dst) = cpu_to_le64(v);-}+	uint32_t v; -static inline void store_be64_aligned(uint8_t *dst, const uint64_t v)-{-	*((uint64_t *) dst) = cpu_to_be64(v);+	memcpy(&v, p, sizeof(v));+	return be32_to_cpu(v); } -static inline void xor_be64_aligned(uint8_t *dst, const uint64_t v)+static inline uint64_t load_be64(const uint8_t *p) {-	*((uint64_t *) dst) ^= cpu_to_be64(v);-}+	uint64_t v; -#ifdef UNALIGNED_ACCESS_OK-#define load_le32(a) load_le32_aligned(a)-#else-static inline uint32_t load_le32(const uint8_t *p)-{-	return ((uint32_t)p[0]) | ((uint32_t)p[1] <<  8) | ((uint32_t)p[2] << 16) | ((uint32_t)p[3] << 24);+	memcpy(&v, p, sizeof(v));+	return be64_to_cpu(v); }-#endif -#ifdef UNALIGNED_ACCESS_OK-#define store_le32(a, b) store_le32_aligned(a, b)-#define xor_le32(a, b) xor_le32_aligned(a, b)-#else static inline void store_le32(uint8_t *dst, const uint32_t v) {-	dst[0] = v; dst[1] = v >> 8; dst[2] = v >> 16; dst[3] = v >> 24;+	uint32_t w = cpu_to_le32(v);++	memcpy(dst, &w, sizeof(w)); }+ static inline void xor_le32(uint8_t *dst, const uint32_t v) {-	dst[0] ^= v; dst[1] ^= v >> 8; dst[2] ^= v >> 16; dst[3] ^= v >> 24;+	store_le32(dst, le32_to_cpu(load_le32(dst)) ^ v); }-#endif -#ifdef UNALIGNED_ACCESS_OK-#define store_be32(a, b) store_be32_aligned(a, b)-#define xor_be32(a, b) xor_be32_aligned(a, b)-#else static inline void store_be32(uint8_t *dst, const uint32_t v) {-	dst[3] = v; dst[2] = v >> 8; dst[1] = v >> 16; dst[0] = v >> 24;+	uint32_t w = cpu_to_be32(v);++	memcpy(dst, &w, sizeof(w)); }+ static inline void xor_be32(uint8_t *dst, const uint32_t v) {-	dst[3] ^= v; dst[2] ^= v >> 8; dst[1] ^= v >> 16; dst[0] ^= v >> 24;+	uint32_t w;++	memcpy(&w, dst, sizeof(w));+	w ^= cpu_to_be32(v);+	memcpy(dst, &w, sizeof(w)); }-#endif -#ifdef UNALIGNED_ACCESS_OK-#define store_le64(a, b) store_le64_aligned(a, b)-#else static inline void store_le64(uint8_t *dst, const uint64_t v) {-	dst[0] = v      ; dst[1] = v >> 8 ; dst[2] = v >> 16; dst[3] = v >> 24;-	dst[4] = v >> 32; dst[5] = v >> 40; dst[6] = v >> 48; dst[7] = v >> 56;+	uint64_t w = cpu_to_le64(v);++	memcpy(dst, &w, sizeof(w)); }-#endif -#ifdef UNALIGNED_ACCESS_OK-#define store_be64(a, b) store_be64_aligned(a, b)-#define xor_be64(a, b) xor_be64_aligned(a, b)-#else static inline void store_be64(uint8_t *dst, const uint64_t v) {-	dst[7] = v      ; dst[6] = v >> 8 ; dst[5] = v >> 16; dst[4] = v >> 24;-	dst[3] = v >> 32; dst[2] = v >> 40; dst[1] = v >> 48; dst[0] = v >> 56;+	uint64_t w = cpu_to_be64(v);++	memcpy(dst, &w, sizeof(w)); }+ static inline void xor_be64(uint8_t *dst, const uint64_t v) {-	dst[7] ^= v      ; dst[6] ^= v >> 8 ; dst[5] ^= v >> 16; dst[4] ^= v >> 24;-	dst[3] ^= v >> 32; dst[2] ^= v >> 40; dst[1] ^= v >> 48; dst[0] ^= v >> 56;+	uint64_t w;++	memcpy(&w, dst, sizeof(w));+	w ^= cpu_to_be64(v);+	memcpy(dst, &w, sizeof(w)); }-#endif++#define load_le32_aligned(p)     load_le32(p)+#define load_le64_aligned(p)     load_le64(p)+#define load_be32_aligned(p)     load_be32(p)+#define load_be64_aligned(p)     load_be64(p)+#define store_le32_aligned(d, v) store_le32(d, v)+#define xor_le32_aligned(d, v)   xor_le32(d, v)+#define store_be32_aligned(d, v) store_be32(d, v)+#define xor_be32_aligned(d, v)   xor_be32(d, v)+#define store_le64_aligned(d, v) store_le64(d, v)+#define store_be64_aligned(d, v) store_be64(d, v)+#define xor_be64_aligned(d, v)   xor_be64(d, v)  #endif
+ cbits/crypton_armv8_target.h view
@@ -0,0 +1,52 @@+/*+ * Asking for an AArch64 extension on one function.+ *+ * The instructions these files use are extensions, so a translation unit+ * compiled for the baseline may not emit them, and the function that does has+ * to say which extension it needs.  The two compilers spell that differently+ * and did not always:+ *+ *   clang          target("+crypto")            for as long as it matters+ *   GCC 13 and up  target("+crypto")            as well+ *   GCC before 13  target("arch=armv8-a+crypto")  -- the bare "+feature" form+ *                  is not understood, and the extension never reaches the+ *                  function, so an always_inline intrinsic that needs it+ *                  fails to inline and the build stops+ *+ * That last case is #273: gcc 12.2 on an aarch64 Linux could not build+ * cbits/sha3_armv8.c at all.  Naming the architecture as well as the+ * extension is understood by every version of both compilers, so GCC is given+ * that spelling and clang keeps the shorter one, which leaves whatever+ * baseline the caller chose alone.+ */+#ifndef CRYPTON_ARMV8_TARGET_H+#define CRYPTON_ARMV8_TARGET_H++#ifdef WITH_TARGET_ATTRIBUTES+#if defined(__arm__)+/*+ * AArch32 asks for an FPU rather than an architecture extension, and the+ * spelling differs between the compilers in ways that cannot be checked from+ * here.  Rather than guess, the cabal file passes -mfpu=crypto-neon-fp-armv8+ * for the whole component on this architecture whatever this flag says, and+ * these expand to nothing.  A global target option is safe because a+ * compiler emits these instructions where an intrinsic asks for them and+ * nowhere else.+ */+#define CRYPTON_TARGET_ARMV8_CRYPTO+#define CRYPTON_TARGET_ARMV8_SHA3+#elif defined(__clang__)+#define CRYPTON_TARGET_ARMV8_CRYPTO __attribute__((target("+crypto")))+#define CRYPTON_TARGET_ARMV8_SHA3 __attribute__((target("+sha3")))+#else+#define CRYPTON_TARGET_ARMV8_CRYPTO \+	__attribute__((target("arch=armv8-a+crypto")))+#define CRYPTON_TARGET_ARMV8_SHA3 \+	__attribute__((target("arch=armv8.2-a+sha3")))+#endif+#else+#define CRYPTON_TARGET_ARMV8_CRYPTO+#define CRYPTON_TARGET_ARMV8_SHA3+#endif++#endif
cbits/crypton_bignum.h view
@@ -29,11 +29,6 @@  #define LIMB_BYTES (LIMB_BITS / 8) -/* four bits of exponent per window, so a table of sixteen and no leftover- * bits: a byte holds exactly two windows */-#define WINDOW_BITS 4-#define TABLE_SIZE (1 << WINDOW_BITS)- /* r = a - b, returning the borrow out of the top */ static inline limb_t sub_n(limb_t *r, const limb_t *a, const limb_t *b, uint32_t n) {@@ -112,9 +107,196 @@ 	return (limb_t) 0 - inv; } -/* t += a * b over n limbs, returning the carry.  This is where nearly all of- * the time goes, so the limbs are taken eight at a time; what is left over at- * the end is taken one at a time. */+/*+ * t += a * b over n limbs, returning the carry.  This is where nearly all of+ * the time goes.+ *+ * The C below takes the limbs eight at a time; what is left over at the end+ * is taken one at a time.  On AArch64 the compiler writes each limb as+ * `mul`, `umulh`, `adds`, `cset`, `adds`, `adc`: the carry out of one+ * 128-bit addition leaves the flags for a general register and is added back+ * in the next, because in C each addition is a statement of its own.  Two of+ * those instructions are that round trip.+ *+ * Three attempts to take them back measured worse than the C, and are+ * written down here so that they are not tried again -- one RSA-2048 CRT+ * private operation on an Apple M4, best of many:+ *+ *     this loop in inline assembly, a carry chain per limb       654.7 us+ *     the same with the loads hoisted out of the chain           644.5+ *     the multiply interleaved with its reduction (CIOS)         604.1+ *     the C below                                                590.1+ *     what the AArch64 block does, four limbs and two chains     514.9+ *     the same, with the ragged end of the row written out too   503.4+ *+ * The first two lose because a chain per limb serialises what the spare+ * `cset` lets overlap: `adds`, `adc`, `adds`, `adc` is four dependent steps+ * per limb, and a wide out-of-order core would rather have the extra+ * instruction than the dependency.  The third loses because shifting the+ * accumulator down a limb each round costs more than the round trip through+ * 2n limbs that it saves.  What works is neither: four limbs to an+ * iteration with the flags carrying through two long chains, one for the low+ * halves of the products and one for the high halves a place up.+ *+ * There is more still there.  OpenSSL's armv8-mont.pl runs a 16-limb+ * Montgomery multiplication at about 0.98 multiply-accumulates per cycle;+ * this file was at 0.55 and the block below brings it to 0.64, where 1.0 is+ * the ceiling -- a multiply-accumulate is two instructions and the machine+ * issues two multiplies a cycle.  That code cannot be borrowed: it is in+ * OpenSSL's tree only, under Apache-2.0, and CRYPTOGAMS, which this library+ * does vendor from, publishes no Montgomery generator at all.  BearSSL's+ * only ARM assembly is 32-bit Thumb for Cortex-M0 to M3, with fifteen-bit+ * limbs for cores that have no fast multiplier, and Botan's AArch64 inline+ * assembly is the `mul`/`umulh`/`adds`/`adc` primitive the compiler already+ * emits.+ */+#if defined(__aarch64__) && LIMB_BITS == 64 \+    && (defined(__GNUC__) || defined(__clang__))+/*+ * Four limbs to an iteration, accumulated in two chains rather than one per+ * limb: the low halves of the four products, with the carry coming in, are+ * one run of `adds` and `adcs`, and the high halves shifted up a place are+ * another.  The flags carry the whole way through each, which is what the C+ * above cannot say and what it pays for in `cset` and an extra add.+ *+ * The arrangement is the one in Go's crypto/internal/fips140/bigmod+ * (nat_arm64.s, addMulVVWx), which is BSD-3-Clause like this library --+ * cbits/LICENSE.go carries its notice.  Written out here in the assembler+ * this file's compiler speaks.+ */+static inline limb_t addmul_1(limb_t *t, const limb_t *a, uint32_t n, limb_t b)+{+	uint64_t carry = 0;+	uint64_t x0, x1, x2, x3, z0, z1, z2, z3;+	uint64_t l0, l1, l2, l3, h0, h1, h2, h3;+	uint64_t blocks = n / 4, left = n % 4;++	if (blocks) {+		__asm__ volatile(+		"1:\n\t"+		"ldp	%[x0], %[x1], [%[a]], #16\n\t"+		"ldp	%[x2], %[x3], [%[a]], #16\n\t"+		"ldp	%[z0], %[z1], [%[t]]\n\t"+		/* the low halves, one place up from the second chain, with+		 * the carry that came in */+		"adds	%[z0], %[z0], %[c]\n\t"+		"mul	%[l1], %[x1], %[b]\n\t"+		"adcs	%[z1], %[z1], %[l1]\n\t"+		"mul	%[l2], %[x2], %[b]\n\t"+		"ldp	%[z2], %[z3], [%[t], #16]\n\t"+		"adcs	%[z2], %[z2], %[l2]\n\t"+		"mul	%[l3], %[x3], %[b]\n\t"+		"adcs	%[z3], %[z3], %[l3]\n\t"+		"umulh	%[h3], %[x3], %[b]\n\t"+		"adc	%[h3], %[h3], xzr\n\t"+		/* and the high halves, which is where this block's own carry+		 * ends up */+		"mul	%[l0], %[x0], %[b]\n\t"+		"adds	%[z0], %[z0], %[l0]\n\t"+		"umulh	%[h0], %[x0], %[b]\n\t"+		"adcs	%[z1], %[z1], %[h0]\n\t"+		"umulh	%[h1], %[x1], %[b]\n\t"+		"stp	%[z0], %[z1], [%[t]], #16\n\t"+		"adcs	%[z2], %[z2], %[h1]\n\t"+		"umulh	%[h2], %[x2], %[b]\n\t"+		"adcs	%[z3], %[z3], %[h2]\n\t"+		"stp	%[z2], %[z3], [%[t]], #16\n\t"+		"adc	%[c], %[h3], xzr\n\t"+		"subs	%[k], %[k], #1\n\t"+		"b.ne	1b\n\t"+		: [a] "+r"(a), [t] "+r"(t), [c] "+r"(carry), [k] "+r"(blocks),+		  [x0] "=&r"(x0), [x1] "=&r"(x1), [x2] "=&r"(x2),+		  [x3] "=&r"(x3), [z0] "=&r"(z0), [z1] "=&r"(z1),+		  [z2] "=&r"(z2), [z3] "=&r"(z3), [l0] "=&r"(l0),+		  [l1] "=&r"(l1), [l2] "=&r"(l2), [l3] "=&r"(l3),+		  [h0] "=&r"(h0), [h1] "=&r"(h1), [h2] "=&r"(h2),+		  [h3] "=&r"(h3)+		: [b] "r"(b)+		: "cc", "memory");+	}+	/* What is left of the row: three limbs, two, or one, each the same two+	 * chains cut short.  This is not a rare case to be handed back to C --+	 * mont_sqr asks for every length from n-1 down to 1, so three rows in+	 * four end ragged.  Only 4.7% of the limbs in a 1024-bit exponentiation+	 * arrive here, but they were the dearer ones, and writing them out is+	 * worth the last two per cent in the table above.  The three lengths+	 * are spelled out rather than run as 2+1, because chaining two short+	 * blocks makes the second wait on the first: that costs two thirds of+	 * the gain.+	 */+	if (left == 3) {+		__asm__ volatile(+		"ldp	%[x0], %[x1], [%[a]]\n\t"+		"ldr	%[x2], [%[a], #16]\n\t"+		"add	%[a], %[a], #24\n\t"+		"ldp	%[z0], %[z1], [%[t]]\n\t"+		"ldr	%[z2], [%[t], #16]\n\t"+		"adds	%[z0], %[z0], %[c]\n\t"+		"mul	%[l1], %[x1], %[b]\n\t"+		"adcs	%[z1], %[z1], %[l1]\n\t"+		"mul	%[l2], %[x2], %[b]\n\t"+		"adcs	%[z2], %[z2], %[l2]\n\t"+		"umulh	%[h2], %[x2], %[b]\n\t"+		"adc	%[h2], %[h2], xzr\n\t"+		"mul	%[l0], %[x0], %[b]\n\t"+		"adds	%[z0], %[z0], %[l0]\n\t"+		"umulh	%[h0], %[x0], %[b]\n\t"+		"adcs	%[z1], %[z1], %[h0]\n\t"+		"umulh	%[h1], %[x1], %[b]\n\t"+		"stp	%[z0], %[z1], [%[t]], #16\n\t"+		"adcs	%[z2], %[z2], %[h1]\n\t"+		"str	%[z2], [%[t]], #8\n\t"+		"adc	%[c], %[h2], xzr\n\t"+		: [a] "+r"(a), [t] "+r"(t), [c] "+r"(carry),+		  [x0] "=&r"(x0), [x1] "=&r"(x1), [x2] "=&r"(x2),+		  [z0] "=&r"(z0), [z1] "=&r"(z1), [z2] "=&r"(z2),+		  [l0] "=&r"(l0), [l1] "=&r"(l1), [l2] "=&r"(l2),+		  [h0] "=&r"(h0), [h1] "=&r"(h1), [h2] "=&r"(h2)+		: [b] "r"(b)+		: "cc", "memory");+	} else if (left == 2) {+		__asm__ volatile(+		"ldp	%[x0], %[x1], [%[a]], #16\n\t"+		"ldp	%[z0], %[z1], [%[t]]\n\t"+		"adds	%[z0], %[z0], %[c]\n\t"+		"mul	%[l1], %[x1], %[b]\n\t"+		"adcs	%[z1], %[z1], %[l1]\n\t"+		"umulh	%[h1], %[x1], %[b]\n\t"+		"adc	%[h1], %[h1], xzr\n\t"+		"mul	%[l0], %[x0], %[b]\n\t"+		"adds	%[z0], %[z0], %[l0]\n\t"+		"umulh	%[h0], %[x0], %[b]\n\t"+		"adcs	%[z1], %[z1], %[h0]\n\t"+		"stp	%[z0], %[z1], [%[t]], #16\n\t"+		"adc	%[c], %[h1], xzr\n\t"+		: [a] "+r"(a), [t] "+r"(t), [c] "+r"(carry),+		  [x0] "=&r"(x0), [x1] "=&r"(x1), [z0] "=&r"(z0),+		  [z1] "=&r"(z1), [l0] "=&r"(l0), [l1] "=&r"(l1),+		  [h0] "=&r"(h0), [h1] "=&r"(h1)+		: [b] "r"(b)+		: "cc", "memory");+	} else if (left == 1) {+		__asm__ volatile(+		"ldr	%[x0], [%[a]], #8\n\t"+		"ldr	%[z0], [%[t]]\n\t"+		"mul	%[l0], %[x0], %[b]\n\t"+		"adds	%[z0], %[z0], %[c]\n\t"+		"umulh	%[h0], %[x0], %[b]\n\t"+		"adc	%[h0], %[h0], xzr\n\t"+		"adds	%[z0], %[z0], %[l0]\n\t"+		"str	%[z0], [%[t]], #8\n\t"+		"adc	%[c], %[h0], xzr\n\t"+		: [a] "+r"(a), [t] "+r"(t), [c] "+r"(carry),+		  [x0] "=&r"(x0), [z0] "=&r"(z0), [l0] "=&r"(l0),+		  [h0] "=&r"(h0)+		: [b] "r"(b)+		: "cc", "memory");+	}+	return carry;+}+#else+/* one limb of it, so that the loops below can say how many they take at a+ * time without saying the rest of it four times over */ #define ADDMUL_STEP(k)                                                  \ 	p = (dlimb_t) a[i + (k)] * b + t[i + (k)] + carry;                  \ 	t[i + (k)] = (limb_t) p;                                            \@@ -141,6 +323,7 @@ 	} 	return carry; }+#endif  /* r = t * R^-1 mod m, with t of 2n limbs and destroyed on the way */ static inline void mont_reduce(limb_t *r, limb_t *t, const limb_t *m, limb_t n0,@@ -165,14 +348,33 @@ 	select_n(r, r, t + n, take & 1, n); } +/* t = a * b, the low n limbs, returning the limb above them: addmul_1 with+ * nothing to add to, for the row of a product that lands on empty space. */+static inline limb_t mul_1(limb_t *t, const limb_t *a, uint32_t n, limb_t b)+{+	limb_t carry = 0;+	uint32_t i;++	for (i = 0; i < n; i++) {+		dlimb_t p = (dlimb_t) a[i] * b + carry;++		t[i] = (limb_t) p;+		carry = (limb_t) (p >> LIMB_BITS);+	}+	return carry;+}+ /* r = a * b * R^-1 mod m, with t of 2n limbs */ static inline void mont_mul(limb_t *r, const limb_t *a, const limb_t *b,                      const limb_t *m, limb_t n0, uint32_t n, limb_t *t) { 	uint32_t i; -	memset(t, 0, 2 * n * sizeof(limb_t));-	for (i = 0; i < n; i++)+	/* Nothing has to be cleared first.  The first row lands on empty space+	 * and writes t[0 .. n], and every row after it reads t[i .. i+n-1],+	 * whose top limb is the one the row before it wrote. */+	t[n] = mul_1(t, a, n, b[0]);+	for (i = 1; i < n; i++) 		t[n + i] = addmul_1(t + i, a, n, b[i]); 	mont_reduce(r, t, m, n0, n); }@@ -186,8 +388,14 @@ 	limb_t carry = 0; 	uint32_t i; -	memset(t, 0, 2 * n * sizeof(limb_t));-	for (i = 0; i + 1 < n; i++)+	/* The first row lands on empty space here too, on t[1 .. n], and the+	 * rows between them write t[1 .. 2n-2] before any of it is read.  The+	 * diagonal below is the only reader of the two ends. */+	t[0] = 0;+	t[2 * n - 1] = 0;+	if (n > 1)+		t[n] = mul_1(t + 1, a + 1, n - 1, a[0]);+	for (i = 1; i + 1 < n; i++) 		t[n + i] = addmul_1(t + i + i + 1, a + i + 1, n - 1 - i, a[i]); 	shl1(t, 2 * n); /* their sum is under half of what 2n limbs hold */ 	for (i = 0; i < n; i++) {@@ -201,15 +409,54 @@ 	mont_reduce(r, t, m, n0, n); } -/* r2 = R^2 mod m, by doubling+/* a = 2a mod m, for an a already under m */+static inline void dbl_mod(limb_t *a, const limb_t *m, uint32_t n, limb_t *tmp)+{+	limb_t carry = shl1(a, n);+	limb_t borrow = sub_n(tmp, a, m, n);++	select_n(a, tmp, a, (carry | (borrow ^ 1)) & 1, n);+}++/* r2 = R^2 mod m, where R is 2^(n * LIMB_BITS)  *+ * Doubling the whole way there is 2n * LIMB_BITS steps, and at RSA sizes+ * that is a tenth of the exponentiation it is setting up for.  Only the+ * first half of it has to be done a bit at a time.+ *+ * Write a value as 2^(lgR + d) mod m.  A Montgomery squaring divides by R,+ * so it takes that to 2(lgR + d) - lgR = lgR + 2d: it doubles d.  So double+ * up to R mod m, where d is zero, take one more step to make d one, and then+ * climb to d = lgR by the binary expansion of lgR -- a squaring for each bit+ * and one more doubling where the bit is set.  For a 1024-bit modulus that+ * is ten squarings in place of a thousand and twenty-five doublings, and on+ * an Apple M4 that is 2.1 microseconds against 24.7.+ *  * Doubling starts at the highest power of two under the modulus rather than  * at one, since everything below that power is where doubling would go- * anyway: for a modulus that fills its limbs that is half the steps.+ * anyway: for a modulus that fills its limbs, that first half is one step.+ *+ * What the trip counts depend on is the modulus' length, which is what the+ * doubling loop showed as well, and nothing else about it.  */-static inline void mont_r2(limb_t *r2, const limb_t *m, uint32_t n, limb_t *tmp)+/* s2n-bignum's bignum_montifier computes the same constant in assembly, on+ * both architectures, and is the faster of the two by a good margin -- the+ * C cannot form the carry chains it does.  It is used where it is built,+ * and the C below is what everywhere else gets and what it is checked+ * against.  It needs a scratch buffer of n limbs, which every caller+ * already passes. */+#if defined(CRYPTON_S2N_BIGNUM) && !defined(CRYPTON_NO_MONTIFIER) \+    && (defined(__x86_64__) || defined(__aarch64__)) && LIMB_BITS == 64+#define CRYPTON_MONTIFIER 1+extern void bignum_montifier(uint64_t k, uint64_t *z, const uint64_t *m,+                             uint64_t *t);+#endif++static inline void mont_r2_c(limb_t *r2, const limb_t *m, limb_t n0, uint32_t n,+                    limb_t *t) {-	uint32_t i, k = 0, steps;+	uint32_t lgr = n * LIMB_BITS;+	uint32_t i, k = 0, msb = 0;  	for (i = n; i > 0 && k == 0; i--) 		if (m[i - 1] != 0) {@@ -225,12 +472,28 @@ 	if (k == 0) 		return; /* a modulus of nothing, which the caller rules out */ 	r2[(k - 1) / LIMB_BITS] = (limb_t) 1 << ((k - 1) % LIMB_BITS);-	steps = 2 * n * LIMB_BITS - (k - 1);-	for (i = 0; i < steps; i++) {-		limb_t carry = shl1(r2, n);-		limb_t borrow = sub_n(tmp, r2, m, n);-		select_n(r2, tmp, r2, (carry | (borrow ^ 1)) & 1, n);+	for (i = k - 1; i < lgr; i++)+		dbl_mod(r2, m, n, t);+	/* r2 is R mod m, so d is zero and squaring would leave it there */+	dbl_mod(r2, m, n, t);+	while ((lgr >> (msb + 1)) != 0)+		msb++;+	for (i = msb; i > 0; i--) {+		mont_sqr(r2, r2, m, n0, n, t);+		if ((lgr >> (i - 1)) & 1)+			dbl_mod(r2, m, n, t); 	}+}++static inline void mont_r2(limb_t *r2, const limb_t *m, limb_t n0, uint32_t n,+                    limb_t *t)+{+#ifdef CRYPTON_MONTIFIER+	(void) n0;+	bignum_montifier(n, r2, m, t);+#else+	mont_r2_c(r2, m, n0, n, t);+#endif }  /* big-endian bytes into limbs, least significant limb first; anything above
cbits/crypton_bitfn.h view
@@ -127,23 +127,6 @@ } #endif -#ifndef ARCH_HAS_MEMORY_ZERO-static inline void memory_zero(void *ptr, uint32_t len)-{-	uint32_t *ptr32 = ptr;-	uint8_t *ptr8;-	int i;--	for (i = 0; i < len / 4; i++)-		*ptr32++ = 0;-	if (len % 4) {-		ptr8 = (uint8_t *) ptr32;-		for (i = len % 4; i >= 0; i--)-			ptr8[i] = 0;-	}-}-#endif- #ifndef ARCH_HAS_ARRAY_COPY32 static inline void array_copy32(uint32_t *d, uint32_t *s, uint32_t nb) {
+ cbits/crypton_bzero.h view
@@ -0,0 +1,27 @@+/*+ * Erasing memory that the compiler is entitled to decide nobody reads.+ *+ * memset on an object that is about to die -- freed, or a local going out of+ * scope -- is a store to memory nothing can observe, and an optimizer may+ * drop it.  That is the whole reason explicit_bzero and memset_s exist.+ * Neither is everywhere, so this writes through a volatile pointer, which the+ * standard says cannot be elided.+ *+ * Use it wherever key material stops being needed.  Plain memset is still+ * right for memory that is about to be read again.+ */+#ifndef CRYPTON_BZERO_H+#define CRYPTON_BZERO_H++#include <stddef.h>+#include <stdint.h>++static inline void crypton_bzero(void *p, size_t n)+{+	volatile uint8_t *q = (volatile uint8_t *)p;++	while (n--)+		*q++ = 0;+}++#endif
cbits/crypton_chacha.c view
@@ -51,7 +51,7 @@                                  const crypton_chacha_state *in); void crypton_chacha_simd_generate(int rounds, uint8_t *dst,                                   const crypton_chacha_state *in);-/* The counters in a group must not carry into d[13], which the block loop+/* The counters in a group must not carry into d[13], which the crypton_chacha_block loop  * below handles and the vector one does not; that is one run in 2^29. */ #define CHACHA_SIMD_OK(st, n) ((st)->d[12] <= 0xffffffffU - (uint32_t) (n)) #endif@@ -89,7 +89,7 @@  * How much is worth handing over.  On AArch64 the module's vector path  * starts at three blocks and below that its scalar path measures level with  * the C here, so there is nothing to gain; on x86-64 it is ahead from one- * block, the C there having no vector path until eight.+ * crypton_chacha_block, the C there having no vector path until eight.  */ #ifndef CHACHA_ASM_MIN_BLOCKS #ifdef WITH_X86_CHACHA_ASM@@ -109,10 +109,11 @@ #define ALIGNED64(PTR) \ 	(((uintptr_t)(const void *)(PTR)) % 8 == 0) -static const uint8_t sigma[16] = "expand 32-byte k";-static const uint8_t tau[16] = "expand 16-byte k";+/* sixteen bytes are read; the seventeenth is the string's terminator */+static const uint8_t sigma[17] = "expand 32-byte k";+static const uint8_t tau[17] = "expand 16-byte k"; -static void chacha_core(int rounds, block *out, const crypton_chacha_state *in)+static void chacha_core(int rounds, crypton_chacha_block *out, const crypton_chacha_state *in) { 	uint32_t x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13, x14, x15; 	int i;@@ -296,7 +297,7 @@  void crypton_chacha_combine(uint8_t *dst, crypton_chacha_context *ctx, const uint8_t *src, uint32_t bytes) {-	block out;+	crypton_chacha_block out; 	crypton_chacha_state *st; 	int i; @@ -384,7 +385,7 @@ 		}  		/* xor as much as needed */-		for (i = 0; i < bytes; i++)+		for (i = 0; i < (int) bytes; i++) 			dst[i] = src[i] ^ out.b[i];  		/* copy the left over in the buffer */@@ -434,7 +435,7 @@ void crypton_chacha_generate(uint8_t *dst, crypton_chacha_context *ctx, uint32_t bytes) { 	crypton_chacha_state *st;-	block out;+	crypton_chacha_block out; 	int i;  	if (!bytes)@@ -474,7 +475,7 @@ 		/* xor new 64-bytes chunks and store the left over if any */ 		for (; bytes >= 64; bytes -= 64, dst += 64) { 			/* generate new chunk and update state */-			chacha_core(ctx->nb_rounds, (block *) dst, st);+			chacha_core(ctx->nb_rounds, (crypton_chacha_block *) dst, st); 			uint32_t t0 = le32_to_cpu(st->d[12]); 			st->d[12] = cpu_to_le32(t0 + 1); 			if (st->d[12] == 0) {@@ -510,7 +511,7 @@ 		}  		/* xor as much as needed */-		for (i = 0; i < bytes; i++)+		for (i = 0; i < (int) bytes; i++) 			dst[i] = out.b[i];  		/* copy the left over in the buffer */@@ -524,9 +525,9 @@ void crypton_chacha_generate_simple_block(uint8_t *dst, crypton_chacha_state *st, uint8_t rounds) { 	if (ALIGNED64(dst)) {-		chacha_core(rounds, (block *) dst, st);+		chacha_core(rounds, (crypton_chacha_block *) dst, st); 	} else {-		block out;+		crypton_chacha_block out; 		int i; 		chacha_core(rounds, &out, st); 		for (i = 0; i < 64; ++i) {@@ -544,7 +545,7 @@  void crypton_chacha_random(uint32_t rounds, uint8_t *dst, crypton_chacha_state *st, uint32_t bytes) {-	block out;+	crypton_chacha_block out;  	if (!bytes) 		return;
cbits/crypton_chacha.h view
@@ -34,9 +34,9 @@ 	uint64_t q[8]; 	uint32_t d[16]; 	uint8_t  b[64];-} block;+} crypton_chacha_block; -typedef block crypton_chacha_state;+typedef crypton_chacha_block crypton_chacha_state;  typedef struct { 	crypton_chacha_state st;
+ cbits/crypton_chachapoly.c view
@@ -0,0 +1,156 @@+/*+ * Copyright (c) 2026 Kazu Yamamoto+ *+ * Redistribution and use in source and binary forms, with or without+ * modification, are permitted provided that the following conditions+ * are met:+ * 1. Redistributions of source code must retain the above copyright+ *    notice, this list of conditions and the following disclaimer.+ * 2. Redistributions in binary form must reproduce the above copyright+ *    notice, this list of conditions and the following disclaimer in the+ *    documentation and/or other materials provided with the distribution.+ *+ * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE+ * POSSIBILITY OF SUCH DAMAGE.+ */++#include <stdint.h>+#include <string.h>++#include "crypton_chacha.h"+#include "crypton_chachapoly.h"+#include "crypton_poly1305.h"++/* RFC 8439.  The one-time Poly1305 key is the first 32 bytes of the ChaCha20+ * keystream at counter 0; a whole 64-byte block is generated so the counter+ * lands on 1, which is where the message starts. */+static void chachapoly_start(crypton_chacha_context *cctx, poly1305_ctx *pctx,+                             const uint8_t *key,+                             const uint8_t *nonce, uint32_t noncelen)+{+	uint8_t block[64];++	crypton_chacha_init(cctx, 20, 32, key, noncelen, nonce);+	crypton_chacha_generate(block, cctx, sizeof(block));+	crypton_poly1305_init(pctx, (poly1305_key *) block);+	memset(block, 0, sizeof(block));+}++/* Poly1305 over an associated or encrypted part, then zeros up to the next+ * multiple of sixteen. */+static void absorb_padded(poly1305_ctx *pctx, const uint8_t *p, uint32_t len)+{+	static const uint8_t zeros[16] = {0};+	uint32_t rem;++	if (len)+		crypton_poly1305_update(pctx, (uint8_t *) p, len);+	rem = len % 16;+	if (rem)+		crypton_poly1305_update(pctx, (uint8_t *) zeros, 16 - rem);+}++/* The two lengths, little endian, eight bytes each, which is what the tag+ * ends on. */+static void absorb_lengths(poly1305_ctx *pctx, uint32_t aadlen, uint32_t inlen)+{+	uint8_t lens[16];+	int i;++	for (i = 0; i < 8; i++)+		lens[i] = (uint8_t) (((uint64_t) aadlen) >> (8 * i));+	for (i = 0; i < 8; i++)+		lens[8 + i] = (uint8_t) (((uint64_t) inlen) >> (8 * i));+	crypton_poly1305_update(pctx, lens, sizeof(lens));+}++void crypton_chachapoly_encrypt(uint8_t *out, uint8_t *tag, uint32_t taglen,+                                const uint8_t *key,+                                const uint8_t *nonce, uint32_t noncelen,+                                const uint8_t *aad, uint32_t aadlen,+                                const uint8_t *input, uint32_t inlen)+{+	crypton_chacha_context cctx;+	poly1305_ctx pctx;+	poly1305_mac mac;++	chachapoly_start(&cctx, &pctx, key, nonce, noncelen);+	absorb_padded(&pctx, aad, aadlen);+	if (inlen)+		crypton_chacha_combine(out, &cctx, input, inlen);+	/* what the tag covers is the ciphertext, which is now in out */+	absorb_padded(&pctx, out, inlen);+	absorb_lengths(&pctx, aadlen, inlen);+	crypton_poly1305_finalize(mac, &pctx);+	memcpy(tag, mac, taglen);++	memset(&cctx, 0, sizeof(cctx));+	memset(&pctx, 0, sizeof(pctx));+}++/* Shared by the two decrypting entry points: with outtag NULL the tag is+ * compared here and the answer returned, otherwise it is written there. */+static int chachapoly_decrypt(uint8_t *out, const uint8_t *tag, uint32_t taglen,+                              uint8_t *outtag, const uint8_t *key,+                              const uint8_t *nonce, uint32_t noncelen,+                              const uint8_t *aad, uint32_t aadlen,+                              const uint8_t *input, uint32_t inlen)+{+	crypton_chacha_context cctx;+	poly1305_ctx pctx;+	poly1305_mac mac;+	uint8_t diff = 0;+	uint32_t i;++	chachapoly_start(&cctx, &pctx, key, nonce, noncelen);+	absorb_padded(&pctx, aad, aadlen);+	/* here the ciphertext is the input, so the tag can be taken before the+	 * plaintext is written and out may alias input */+	absorb_padded(&pctx, input, inlen);+	absorb_lengths(&pctx, aadlen, inlen);+	crypton_poly1305_finalize(mac, &pctx);++	if (inlen)+		crypton_chacha_combine(out, &cctx, input, inlen);++	memset(&cctx, 0, sizeof(cctx));+	memset(&pctx, 0, sizeof(pctx));++	if (outtag) {+		memcpy(outtag, mac, taglen);+		return 1;+	}+	for (i = 0; i < taglen; i++)+		diff |= (uint8_t) (mac[i] ^ tag[i]);+	return diff == 0;+}++int crypton_chachapoly_decrypt(uint8_t *out,+                               const uint8_t *tag, uint32_t taglen,+                               const uint8_t *key,+                               const uint8_t *nonce, uint32_t noncelen,+                               const uint8_t *aad, uint32_t aadlen,+                               const uint8_t *input, uint32_t inlen)+{+	return chachapoly_decrypt(out, tag, taglen, NULL, key, nonce, noncelen,+	                          aad, aadlen, input, inlen);+}++void crypton_chachapoly_decrypt_tag(uint8_t *out, uint8_t *outtag,+                                    uint32_t taglen, const uint8_t *key,+                                    const uint8_t *nonce, uint32_t noncelen,+                                    const uint8_t *aad, uint32_t aadlen,+                                    const uint8_t *input, uint32_t inlen)+{+	(void) chachapoly_decrypt(out, NULL, taglen, outtag, key, nonce,+	                          noncelen, aad, aadlen, input, inlen);+}
+ cbits/crypton_chachapoly.h view
@@ -0,0 +1,62 @@+/*+ * Copyright (c) 2026 Kazu Yamamoto+ *+ * Redistribution and use in source and binary forms, with or without+ * modification, are permitted provided that the following conditions+ * are met:+ * 1. Redistributions of source code must retain the above copyright+ *    notice, this list of conditions and the following disclaimer.+ * 2. Redistributions in binary form must reproduce the above copyright+ *    notice, this list of conditions and the following disclaimer in the+ *    documentation and/or other materials provided with the distribution.+ *+ * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR+ * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE+ * POSSIBILITY OF SUCH DAMAGE.+ */++#ifndef CRYPTON_CHACHAPOLY_H+#define CRYPTON_CHACHAPOLY_H++#include <stdint.h>++/* ChaCha20-Poly1305 (RFC 8439) as one call.+ *+ * The pieces are the ChaCha20 and Poly1305 already here; what these do is+ * hold them together, which the Haskell above used to do at the cost of eight+ * foreign calls and the allocations between them.+ *+ * The nonce is the twelve bytes RFC 8439 defines.  taglen is at most 16.+ */++void crypton_chachapoly_encrypt(uint8_t *out, uint8_t *tag, uint32_t taglen,+                                const uint8_t *key,+                                const uint8_t *nonce, uint32_t noncelen,+                                const uint8_t *aad, uint32_t aadlen,+                                const uint8_t *input, uint32_t inlen);++/* Decrypt and compare, a byte at a time over the whole tag whichever way the+ * answer goes.  Returns non-zero when the tag matched. */+int crypton_chachapoly_decrypt(uint8_t *out,+                               const uint8_t *tag, uint32_t taglen,+                               const uint8_t *key,+                               const uint8_t *nonce, uint32_t noncelen,+                               const uint8_t *aad, uint32_t aadlen,+                               const uint8_t *input, uint32_t inlen);++/* Decrypt and hand the computed tag back rather than comparing it. */+void crypton_chachapoly_decrypt_tag(uint8_t *out, uint8_t *outtag,+                                    uint32_t taglen, const uint8_t *key,+                                    const uint8_t *nonce, uint32_t noncelen,+                                    const uint8_t *aad, uint32_t aadlen,+                                    const uint8_t *input, uint32_t inlen);++#endif
cbits/crypton_cpu.c view
@@ -32,6 +32,23 @@ #include <stdint.h>  /*+ * PE has no way to say "hidden": every symbol in an object is local to the+ * image unless something exports it, which is what hidden asks for+ * elsewhere, so nothing is lost by dropping the attribute here.  Saying it+ * anyway is not harmless -- the gcc that GHC 9.2 ships for Windows parses+ * the attribute, discards it and warns, and it is the only warning crypton's+ * own code produces anywhere in the CI matrix.  Measured on mingw gcc 13.2.0+ * and clang 14.0.6 (the compiler GHC 9.4 and later ship): gcc warns for+ * "hidden" and is silent for "default", clang is silent for both, which is+ * why the vendored decaf and argon2 headers ask for "default" unnoticed.+ */+#if defined(_WIN32) || defined(__CYGWIN__)+#define CRYPTON_HIDDEN+#else+#define CRYPTON_HIDDEN __attribute__((visibility("hidden")))+#endif++/*  * The word the assembly reads; crypton_cpu.h says what is in it.  Hidden,  * so that the reference to it from the assembly resolves at link time in a  * shared object as well as a static one.  The SHA-256 bit is set by@@ -39,10 +56,226 @@  * instructions.  */ #ifdef CRYPTON_ARM_ASM-__attribute__((visibility("hidden"))) unsigned int crypton_armcap_P =+CRYPTON_HIDDEN unsigned int crypton_armcap_P =     CRYPTON_ARMCAP_NEON; #endif +#if defined(__aarch64__) || defined(__arm__)+#if defined(__APPLE__)+#include <sys/sysctl.h>+#elif defined(__linux__)+#include <sys/auxv.h>+#include <asm/hwcap.h>+#elif defined(__FreeBSD__)+#include <sys/auxv.h>+#endif++/*+ * The bit positions are the ARM ELF ABI's, so every system that reports+ * through the auxiliary vector agrees on them.  What differs is the tag --+ * AT_HWCAP is 16 on Linux and 25 on FreeBSD -- and that comes from each+ * system's own header, which is why no tag is written out here.  Defined+ * only where the system's headers did not define them, the way compiler-rt+ * does it, so that a system which reports through the auxiliary vector+ * without shipping the ARM names still compiles.+ *+ * The two execution states do not share a word or an order.  AArch64 puts+ * these in AT_HWCAP; AArch32 has filled that word with older features and+ * puts the cryptographic ones in AT_HWCAP2, starting again from bit zero.+ */+#if defined(__aarch64__)+#ifndef HWCAP_AES+#define HWCAP_AES    (1 << 3)+#endif+#ifndef HWCAP_PMULL+#define HWCAP_PMULL  (1 << 4)+#endif+#ifndef HWCAP_SHA1+#define HWCAP_SHA1   (1 << 5)+#endif+#ifndef HWCAP_SHA2+#define HWCAP_SHA2   (1 << 6)+#endif+#ifndef HWCAP_SHA3+#define HWCAP_SHA3   (1 << 17)+#endif+#ifndef HWCAP_SHA512+#define HWCAP_SHA512 (1 << 21)+#endif+#else+#ifndef HWCAP2_AES+#define HWCAP2_AES   (1 << 0)+#endif+#ifndef HWCAP2_PMULL+#define HWCAP2_PMULL (1 << 1)+#endif+#ifndef HWCAP2_SHA1+#define HWCAP2_SHA1  (1 << 2)+#endif+#ifndef HWCAP2_SHA2+#define HWCAP2_SHA2  (1 << 3)+#endif+#endif++#if defined(__APPLE__)+static int apple_has(const char *name)+{+	int v = 0;+	size_t n = sizeof(v);++	if (sysctlbyname(name, &v, &n, NULL, 0) != 0)+		return 0;+	return v != 0;+}+#endif++/*+ * Systems still answering zero, and what each would need:+ *+ *   OpenBSD, NetBSD   sysctl on machdep.id_aa64isar0, which hands out the+ *                     ID_AA64ISAR0_EL1 fields rather than a HWCAP word, so+ *                     it is a different shape of answer rather than another+ *                     tag.+ *   Windows on ARM    IsProcessorFeaturePresent with+ *                     PF_ARM_V8_CRYPTO_INSTRUCTIONS_AVAILABLE, which covers+ *                     AES, PMULL, SHA-1 and SHA-256 as one bit and says+ *                     nothing about SHA-512 or SHA-3.  An ARM64 Windows+ *                     guest answers 1 to it, but GHC has no native ARM64+ *                     Windows target: there it builds x86-64 and the+ *                     emulator reports AES-NI, so this file is not reached.+ *+ * Neither is written here because neither can be built and run to see it+ * work, and an untested answer about whether a machine has AES is worse+ * than the honest zero it replaces.+ */+unsigned int crypton_arm_features(void)+{+	static unsigned int features;+	static int resolved;++	if (!resolved) {+		unsigned int f = 0;+#if defined(__APPLE__)+		/* AES, PMULL, SHA-1 and SHA-256 are not optional on Apple+		 * silicon.  The two later ones are. */+		f = CRYPTON_ARM_AES | CRYPTON_ARM_PMULL+		  | CRYPTON_ARM_SHA1 | CRYPTON_ARM_SHA2;+		if (apple_has("hw.optional.arm.FEAT_SHA512"))+			f |= CRYPTON_ARM_SHA512;+		if (apple_has("hw.optional.arm.FEAT_SHA3"))+			f |= CRYPTON_ARM_SHA3;+#elif defined(__aarch64__)+		unsigned long cap = 0;++#if defined(__linux__)+		cap = getauxval(AT_HWCAP);+#elif defined(__FreeBSD__)+		if (elf_aux_info(AT_HWCAP, &cap, sizeof(cap)) != 0)+			cap = 0;+#endif+		if (cap & HWCAP_AES)    f |= CRYPTON_ARM_AES;+		if (cap & HWCAP_PMULL)  f |= CRYPTON_ARM_PMULL;+		if (cap & HWCAP_SHA1)   f |= CRYPTON_ARM_SHA1;+		if (cap & HWCAP_SHA2)   f |= CRYPTON_ARM_SHA2;+		if (cap & HWCAP_SHA512) f |= CRYPTON_ARM_SHA512;+		if (cap & HWCAP_SHA3)   f |= CRYPTON_ARM_SHA3;+#else+		/* AArch32: the second word, and nothing later than SHA-256 to+		 * ask about */+		unsigned long cap = 0;++#if defined(__linux__)+		cap = getauxval(AT_HWCAP2);+#elif defined(__FreeBSD__)+		if (elf_aux_info(AT_HWCAP2, &cap, sizeof(cap)) != 0)+			cap = 0;+#endif+		if (cap & HWCAP2_AES)   f |= CRYPTON_ARM_AES;+		if (cap & HWCAP2_PMULL) f |= CRYPTON_ARM_PMULL;+		if (cap & HWCAP2_SHA1)  f |= CRYPTON_ARM_SHA1;+		if (cap & HWCAP2_SHA2)  f |= CRYPTON_ARM_SHA2;+#endif+		features = f;+		resolved = 1;+	}+	return features;+}+#endif /* __aarch64__ || __arm__ */++#if defined(__powerpc64__) || defined(__PPC64__)+#if defined(__linux__)+#include <sys/auxv.h>+#elif defined(__FreeBSD__)+#include <sys/auxv.h>++#endif++/*+ * PPC_FEATURE2_VEC_CRYPTO, from the Power ELF ABI.  Defined here only where+ * the system's headers did not, so that a system reporting through the+ * auxiliary vector without shipping the name still compiles.+ */+#ifndef PPC_FEATURE2_VEC_CRYPTO+#define PPC_FEATURE2_VEC_CRYPTO 0x02000000+#endif++unsigned int crypton_ppc_features(void)+{+	static unsigned int features;+	static int resolved;++	if (!resolved) {+		unsigned long cap = 0;++#if defined(__linux__)+		cap = getauxval(AT_HWCAP2);+#elif defined(__FreeBSD__)+		if (elf_aux_info(AT_HWCAP2, &cap, sizeof(cap)) != 0)+			cap = 0;+#endif+		features = (cap & PPC_FEATURE2_VEC_CRYPTO)+		         ? CRYPTON_PPC_VCRYPTO : 0;+		resolved = 1;+	}+	return features;+}+#endif /* __powerpc64__ */++/*+ * For testing on a machine whose CPUID says less than its core executes, and+ * for nothing else: each of these makes the detection below report a+ * feature it did not find.+ *+ *   CRYPTON_CPU_TESTING_ADX     BMI2 and ADX+ *   CRYPTON_CPU_TESTING_SHA_NI  the SHA extensions+ *   CRYPTON_CPU_TESTING_VAES    VAES and VPCLMULQDQ, 256-bit+ *+ * A hypervisor may show a guest an older processor model than the host's so+ * that the guest can be moved to another host, and the instructions it hides+ * then still run.  That is the only place this is for: forcing a path that+ * CPUID would not choose, to measure it or to test it, on a machine where the+ * instructions have been seen to execute.  Anywhere they do not, the first+ * use is a SIGILL -- which is the bug #317 fixed, made on purpose.+ *+ * What the operating system has to agree to is still asked.  VAES works in+ * the wide registers, so it is assumed only where AVX2 was found, which+ * means XCR0 says those are saved; the SHA extensions still want SSSE3 and+ * SSE4.1 to have been found, as the detection asks.  Nothing 512-bit can be+ * assumed: whether the operating system saves that state is not something a+ * build can promise.+ *+ * The library is never built with these, and a build that has one says so.+ */+#if defined(CRYPTON_CPU_TESTING_ADX) || defined(CRYPTON_CPU_TESTING_SHA_NI) \+    || defined(CRYPTON_CPU_TESTING_VAES)+/* The code below asks for each switch by its own name, which comes from the+ * command line and so is seen everywhere in the file.  It once asked for one+ * umbrella name defined beside the first function that read it -- and the+ * capability vector the assembly reads is filled earlier in the file than+ * that, so the SHA bit never reached it, without a word from the compiler. */+#warning "CRYPTON_CPU_TESTING_*: features assumed, not detected"+#endif+ #ifdef ARCH_X86 static void cpuid(uint32_t info, uint32_t *eax, uint32_t *ebx, uint32_t *ecx, uint32_t *edx) {@@ -101,7 +334,7 @@ }  #ifdef CRYPTON_X86_ASM-__attribute__((visibility("hidden"))) unsigned int crypton_ia32cap_P[4];+CRYPTON_HIDDEN unsigned int crypton_ia32cap_P[4];  /*  * The AVX-512 bits of leaf 7 EBX -- F, DQ, IFMA, PF, ER, CD, BW and VL,@@ -169,6 +402,16 @@ 		} 		if (!(f & CRYPTON_X86_AVX2)) 			leaf7_ebx &= ~(1u << 5);+		/* what the testing switches assumed, told to the assembly as+		 * well: its SHA-256 reads the SHA bit here, not the word */+#ifdef CRYPTON_CPU_TESTING_SHA_NI+		if (f & CRYPTON_X86_SHA_NI)+			leaf7_ebx |= 1u << 29;+#endif+#ifdef CRYPTON_CPU_TESTING_ADX+		if (f & CRYPTON_X86_ADX)+			leaf7_ebx |= (1u << 8) | (1u << 19);+#endif 		crypton_ia32cap_P[2] = leaf7_ebx & ~IA32CAP_AVX512;  		resolved = 1;@@ -182,13 +425,23 @@ 	static uint32_t features = 0;  	if (!resolved) {-		uint32_t eax, ebx, ecx, edx, leaf1, maxleaf, f = 0;+		uint32_t eax, ebx, ecx, edx, leaf1, maxleaf, family, f = 0;+		int amd;  		cpuid(0, &eax, &ebx, &ecx, &edx); 		maxleaf = eax;+		/* "AuthenticAMD" arrives as EBX, EDX, ECX in that order */+		amd = (ebx == 0x68747541 && edx == 0x69746e65+		       && ecx == 0x444d4163);  		cpuid(1, &eax, &ebx, &ecx, &edx); 		leaf1 = ecx;+		/* the family is the base one, and the extended field is+		 * added to it only when the base reads 0xf, which is how+		 * every AMD Zen part reports */+		family = (eax >> 8) & 0xf;+		if (family == 0xf)+			family += (eax >> 20) & 0xff; 		if (leaf1 & (1 << 9)) 			f |= CRYPTON_X86_SSSE3; 		if (leaf1 & (1 << 1))@@ -223,7 +476,62 @@ 			if ((ebx & (1 << 5)) && (leaf1 & (1 << 27)) 			    && (leaf1 & (1 << 28)) && ((xcr0() & 6) == 6)) 				f |= CRYPTON_X86_AVX2;+			/* BMI2 for MULX and ADX for ADCX/ADOX.  Both are+			 * wanted together and neither touches vector state,+			 * so there is nothing to ask the operating system */+			if ((ebx & (1 << 8)) && (ebx & (1 << 19)))+				f |= CRYPTON_X86_ADX;+			/* VAES and VPCLMULQDQ, leaf 7 ECX bits 9 and 10.+			 * They are wanted together -- one without the other+			 * leaves half of AES-GCM narrow -- and they need the+			 * wide registers, so AVX2 has to have answered first,+			 * which settles the operating system's part. */+			if ((ecx & (1 << 9)) && (ecx & (1 << 10))+			    && (f & CRYPTON_X86_AVX2))+				f |= CRYPTON_X86_VAES;+			/* The same two instructions in their 512-bit form,+			 * which wants AVX-512 F, BW and VL as well -- and+			 * three more bits of XCR0, for the mask registers and+			 * the two upper halves of the vector state.  A+			 * machine can report the instructions and still fault+			 * on them when the operating system has not said it+			 * saves that state, which is what those bits are. */+			if ((f & CRYPTON_X86_VAES)+			    && (ebx & (1 << 16)) && (ebx & (1u << 30))+			    && (ebx & (1u << 31))+			    && ((xcr0() & 0xe6) == 0xe6)+			    /* Not on Zen 4, which is AMD family 19h with+			     * AVX-512: there the 512-bit instructions are two+			     * 256-bit passes through a 256-bit datapath, so+			     * they carry the wider encoding for none of the+			     * throughput, and AES-GCM measures 0.6 to 3+			     * per cent slower than the 256-bit path.  Zen 5+			     * is family 1Ah and does have the wide datapath,+			     * where the same code is half as fast again;+			     * Zen 3, the other family 19h part, has no+			     * AVX-512 at all and never reaches here. */+			    && !(amd && family == 0x19))+				f |= CRYPTON_X86_VAES512;+			/* IFMA (bit 21) with F (16) and VL (31), and the same+			 * AVX-512 state.  Zen 4 is not left out here: RSA's+			 * multiplication uses the 256-bit form. */+			if ((f & CRYPTON_X86_AVX2)+			    && (ebx & (1 << 16)) && (ebx & (1 << 21))+			    && (ebx & (1u << 31))+			    && ((xcr0() & 0xe6) == 0xe6))+				f |= CRYPTON_X86_IFMA; 		}+#ifdef CRYPTON_CPU_TESTING_ADX+		f |= CRYPTON_X86_ADX;+#endif+#ifdef CRYPTON_CPU_TESTING_SHA_NI+		if ((leaf1 & (1 << 9)) && (leaf1 & (1 << 19)))+			f |= CRYPTON_X86_SHA_NI;+#endif+#ifdef CRYPTON_CPU_TESTING_VAES+		if (f & CRYPTON_X86_AVX2)+			f |= CRYPTON_X86_VAES;+#endif 		features = f; 		resolved = 1; 	}@@ -250,3 +558,110 @@ #endif  #endif++/*+ * What Crypto.System.CPU reports.+ *+ * The numbers are that module's, and this answers for them.  The old shape+ * was the other way round -- Haskell read crypton_aes.c's two-entry array by+ * the Enum index of its own constructors -- which tied the list of names to+ * the indices of an array that had no reason to grow, and so it never did.+ *+ * Nothing here is cached: every answer is either a static array filled by a+ * constructor, a value resolved once and remembered by the function that+ * owns it, or a runtime check that does its own remembering.  Haskell calls+ * this once, for a CAF.+ */+uint8_t *crypton_aes_cpu_init(void);++#ifdef WITH_ARMV8_CRYPTO+int crypton_aes_armv8_available(void);+int crypton_aes_armv8_pmull_available(void);+#endif+#ifdef WITH_ARMV8_SHA1+extern int crypton_sha1_armv8_available(void);+#endif+#ifdef WITH_ARMV8_SHA2+extern int crypton_sha256_armv8_available(void);+#endif+#ifdef WITH_ARMV8_SHA512+extern int crypton_sha512_armv8_available(void);+#endif++/* Keep in step with Crypto/System/CPU.hs, which is where these are named.+ * 2 is missing: it was RDRAND, which crypton no longer dispatches on.  The+ * number is left out rather than reused, so that the rest keep the values+ * they had. */+#define OPT_AESNI      0+#define OPT_PCLMUL     1+#define OPT_SSSE3      3+#define OPT_AVX        4+#define OPT_AVX2       5+#define OPT_SHANI      6+#define OPT_MOVBE      7+#define OPT_ADX        8+#define OPT_VAES       9+#define OPT_VAES512   10+#define OPT_NEON      11+#define OPT_ARMAES    12+#define OPT_ARMPMULL  13+#define OPT_ARMSHA1   14+#define OPT_ARMSHA2   15+#define OPT_ARMSHA512 16+#define OPT_PPCAES    17+#define OPT_PPCVPMSUM 18+#define OPT_IFMA      19++int crypton_cpu_option(unsigned int option)+{+#ifdef ARCH_X86+	uint32_t f = crypton_x86_simd_features();++	switch (option) {+	case OPT_AESNI:    return crypton_aes_cpu_init()[CPU_AESNI] != 0;+	case OPT_PCLMUL:   return crypton_aes_cpu_init()[CPU_PCLMUL] != 0;+	case OPT_SSSE3:    return (f & CRYPTON_X86_SSSE3)   != 0;+	case OPT_AVX:      return (f & CRYPTON_X86_AVX)     != 0;+	case OPT_AVX2:     return (f & CRYPTON_X86_AVX2)    != 0;+	case OPT_SHANI:    return (f & CRYPTON_X86_SHA_NI)  != 0;+	case OPT_MOVBE:    return (f & CRYPTON_X86_MOVBE)   != 0;+	case OPT_ADX:      return (f & CRYPTON_X86_ADX)     != 0;+	case OPT_VAES:     return (f & CRYPTON_X86_VAES)    != 0;+	case OPT_VAES512:  return (f & CRYPTON_X86_VAES512) != 0;+	case OPT_IFMA:     return (f & CRYPTON_X86_IFMA)    != 0;+	default:           return 0;+	}+#else+	switch (option) {+	/* NEON is not optional on AArch64, and crypton_armcap_P is set from+	 * the start for it.  Where there is no ARM assembly at all there is+	 * nothing to say. */+#ifdef CRYPTON_ARM_ASM+	case OPT_NEON:     return (crypton_armcap_P & CRYPTON_ARMCAP_NEON) != 0;+#endif+#ifdef WITH_ARMV8_CRYPTO+	/* The array rather than the check: it is what the branch table was+	 * actually built from, so it says what will run, not merely what the+	 * processor has. */+	case OPT_ARMAES:   return crypton_aes_cpu_init()[CPU_AESNI]  != 0;+	case OPT_ARMPMULL: return crypton_aes_cpu_init()[CPU_PCLMUL] != 0;+#endif+#ifdef WITH_ARMV8_SHA1+	case OPT_ARMSHA1:  return crypton_sha1_armv8_available()   != 0;+#endif+#ifdef WITH_ARMV8_SHA2+	case OPT_ARMSHA2:  return crypton_sha256_armv8_available() != 0;+#endif+#ifdef WITH_ARMV8_SHA512+	case OPT_ARMSHA512: return crypton_sha512_armv8_available() != 0;+#endif+#ifdef WITH_PPC8_CRYPTO+	/* The array rather than the check, for the same reason as the ARM pair+	 * above: it says what the branch table was built from. */+	case OPT_PPCAES:    return crypton_aes_cpu_init()[CPU_AESNI]  != 0;+	case OPT_PPCVPMSUM: return crypton_aes_cpu_init()[CPU_PCLMUL] != 0;+#endif+	default:           return 0;+	}+#endif+}
cbits/crypton_cpu.h view
@@ -49,6 +49,25 @@  * in, and the byte-swapping load it reads the message with */ #define CRYPTON_X86_AVX    16 #define CRYPTON_X86_MOVBE  32+/* MULX, ADCX and ADOX together: the two independent carry chains the+ * vendored s2n-bignum assembly wants.  They are general-purpose register+ * instructions, so unlike the vector ones above they ask nothing of the+ * operating system. */+#define CRYPTON_X86_ADX    64+/* The AES and carry-less multiply instructions in their 256-bit form, which+ * do two blocks where the 128-bit ones do one.  They are VEX-encoded and use+ * the vector registers AVX2 already needs the operating system to save, so+ * they ask nothing further of it -- but AVX2 itself is asked about, since+ * without it there is nowhere to put them. */+#define CRYPTON_X86_VAES   128+/* The same pair in their 512-bit form, four blocks to an instruction.  These+ * are EVEX-encoded and need the AVX-512 state as well, which is three more+ * bits of XCR0 than AVX2 wants: the mask registers and the two upper halves+ * of the vector registers. */+#define CRYPTON_X86_VAES512 256+/* AVX-512 IFMA, the 52-bit multiply-adds, with F and VL for the 256-bit form+ * RSA uses them in: EVEX-encoded, so they want the AVX-512 state saved too. */+#define CRYPTON_X86_IFMA   512 #ifdef ARCH_X86 uint32_t crypton_x86_simd_features(void); #endif@@ -81,6 +100,64 @@ #define CRYPTON_ARMCAP_SHA256 (1 << 4) extern unsigned int crypton_armcap_P; #endif++/*+ * Which of the optional ARMv8 instruction sets this processor has.+ *+ * Asked here rather than in each file that wants one.  Five files used to+ * carry the same three-way conditional -- Apple, Linux, and otherwise zero+ * -- and the "otherwise" is not a statement about the processor but about+ * which systems someone had thought of.  It is what left FreeBSD running+ * the table-driven AES and the C SHA on hardware that has the+ * instructions.  In one place the next system is added once.+ *+ * AArch32 answers here too.  It has the same AES, PMULL, SHA-1 and SHA-256+ * instructions, reports them in a different word of the auxiliary vector,+ * and has no SHA-512 or SHA-3 to report at all -- those two are AArch64's+ * alone, so the bits exist and are never set.+ */+#if defined(__aarch64__) || defined(__arm__)+#define CRYPTON_ARM_AES    (1u << 0)+#define CRYPTON_ARM_PMULL  (1u << 1)+#define CRYPTON_ARM_SHA1   (1u << 2)+#define CRYPTON_ARM_SHA2   (1u << 3)+#define CRYPTON_ARM_SHA512 (1u << 4)+#define CRYPTON_ARM_SHA3   (1u << 5)+unsigned int crypton_arm_features(void);+#endif++/*+ * Which of the optional PowerISA instruction sets this processor has.+ *+ * One bit so far: the vector AES and the vector carry-less multiply that+ * PowerISA 2.07 brought and POWER8 was the first to implement.  The system+ * reports it in the second capability word, as AArch32 does, and with its+ * own numbering again.+ */+#if defined(__powerpc64__) || defined(__PPC64__)+#define CRYPTON_PPC_VCRYPTO (1u << 0)+unsigned int crypton_ppc_features(void);+#endif++/*+ * The two slots of the array crypton_aes.c fills and crypton_aes_cpu_init+ * hands out.  Here rather than in that file because crypton_cpu.c reads+ * them too, to answer for Crypto.System.CPU.+ */+#define CPU_AESNI        0+#define CPU_PCLMUL       1+#define CPU_OPTION_COUNT 2++/*+ * One question at a time, numbered the way Crypto.System.CPU numbers its+ * ProcessorOption rather than the way any array here is indexed.  The+ * numbering is the module's to choose and this answers for it; reading an+ * array by the Haskell constructor's Enum index is what kept that type at+ * three names.  Returns 1 if the processor has it and crypton was built to+ * use it, 0 otherwise -- including for anything this does not know, so an+ * older library answers a newer caller rather than failing to link.+ */+int crypton_cpu_option(unsigned int option);  #ifdef USE_AESNI void crypton_aesni_initialize_hw(void (*init_table)(int, int));
cbits/crypton_ecc.c view
@@ -23,8 +23,12 @@  */ #include <stdlib.h> #include <crypton_bignum.h>+#include <crypton_bzero.h> #include <crypton_ecc.h> #include <crypton_powm.h>+#ifdef CRYPTON_S2N_BIGNUM+#include <crypton_ecc_s2n.h>+#endif  /* four bits of scalar per window, so a table of sixteen and no leftover  * bits: a byte holds exactly two windows */@@ -220,12 +224,15 @@  static void ctx_free(curve_ctx *c) {+	/* crypton_bzero rather than memset: this memory is freed on the next+	 * line, and a store to memory about to die is one an optimizer may+	 * drop.  Both buffers have held scalars. */ 	if (c->space != NULL) {-		memset(c->space, 0, c->words * sizeof(limb_t));+		crypton_bzero(c->space, c->words * sizeof(limb_t)); 		free(c->space); 	} 	if (c->bytes != NULL) {-		memset(c->bytes, 0, 2 * c->plen);+		crypton_bzero(c->bytes, 2 * c->plen); 		free(c->bytes); 	} 	c->space = NULL;@@ -284,10 +291,10 @@ 		ctx_free(c); 		return -1; 	}-	mont_r2(c->r2, mp, n, mont_t);+	c->f.n0 = mont_n0(mp[0]);+	mont_r2(c->r2, mp, c->f.n0, n, mont_t);  	c->f.n = n;-	c->f.n0 = mont_n0(mp[0]); 	c->f.p = mp; 	c->f.a = ma; 	c->f.b3 = mb3;@@ -411,6 +418,20 @@ 	curve_ctx c; 	uint32_t n, i, j; 	int ret = -1;++#ifdef CRYPTON_S2N_BIGNUM+	/* Two of the curves that reach here have hand-written assembly, six+	 * to ten times faster than what follows; see cbits/s2n/README.md.+	 * Anything else, including those two named with a different a or b,+	 * goes on down. */+	{+		int s2n_ret;++		if (crypton_s2n_ecc_mul(&s2n_ret, outx, outy, px, py, k, klen,+		                        a, b, p, plen))+			return s2n_ret;+	}+#endif  	if (klen == 0 || ctx_init(&c, a, b, p, plen) != 0) 		return -1;
+ cbits/crypton_ecc_s2n.c view
@@ -0,0 +1,204 @@+#include <string.h>++#include "crypton_ecc_s2n.h"+#include "crypton_ecc_s2n_curves.h"+#include "crypton_cpu.h"++/* P-384, Montgomery domain, six words a coordinate */+extern void p384_montjscalarmul(uint64_t *res, const uint64_t *s, const uint64_t *p);+extern void p384_montjscalarmul_alt(uint64_t *res, const uint64_t *s, const uint64_t *p);+extern void bignum_tomont_p384(uint64_t *z, const uint64_t *x);+extern void bignum_tomont_p384_alt(uint64_t *z, const uint64_t *x);+extern void bignum_deamont_p384(uint64_t *z, const uint64_t *x);+extern void bignum_deamont_p384_alt(uint64_t *z, const uint64_t *x);+extern void bignum_montmul_p384(uint64_t *z, const uint64_t *x, const uint64_t *y);+extern void bignum_montmul_p384_alt(uint64_t *z, const uint64_t *x, const uint64_t *y);+extern void bignum_montsqr_p384(uint64_t *z, const uint64_t *x);+extern void bignum_montsqr_p384_alt(uint64_t *z, const uint64_t *x);+extern void bignum_montinv_p384(uint64_t *z, const uint64_t *x);++/* P-521, ordinary values, nine words a coordinate */+extern void p521_jscalarmul(uint64_t *res, const uint64_t *s, const uint64_t *p);+extern void p521_jscalarmul_alt(uint64_t *res, const uint64_t *s, const uint64_t *p);+extern void bignum_mul_p521(uint64_t *z, const uint64_t *x, const uint64_t *y);+extern void bignum_mul_p521_alt(uint64_t *z, const uint64_t *x, const uint64_t *y);+extern void bignum_sqr_p521(uint64_t *z, const uint64_t *x);+extern void bignum_sqr_p521_alt(uint64_t *z, const uint64_t *x);+extern void bignum_inv_p521(uint64_t *z, const uint64_t *x);++/* One flavour or the other, all the way through: on x86-64 the plain form+ * wants MULX, ADCX and ADOX and _alt is the fallback, so mixing them would+ * fault on a machine without those. */+struct p384_asm {+	void (*tomont)(uint64_t *, const uint64_t *);+	void (*deamont)(uint64_t *, const uint64_t *);+	void (*montmul)(uint64_t *, const uint64_t *, const uint64_t *);+	void (*montsqr)(uint64_t *, const uint64_t *);+	void (*jscalarmul)(uint64_t *, const uint64_t *, const uint64_t *);+};+struct p521_asm {+	void (*mul)(uint64_t *, const uint64_t *, const uint64_t *);+	void (*sqr)(uint64_t *, const uint64_t *);+	void (*jscalarmul)(uint64_t *, const uint64_t *, const uint64_t *);+};++static const struct p384_asm p384_std = {+	bignum_tomont_p384, bignum_deamont_p384, bignum_montmul_p384,+	bignum_montsqr_p384, p384_montjscalarmul+};+static const struct p384_asm p384_alt = {+	bignum_tomont_p384_alt, bignum_deamont_p384_alt,+	bignum_montmul_p384_alt, bignum_montsqr_p384_alt,+	p384_montjscalarmul_alt+};+static const struct p521_asm p521_std = {+	bignum_mul_p521, bignum_sqr_p521, p521_jscalarmul+};+static const struct p521_asm p521_alt = {+	bignum_mul_p521_alt, bignum_sqr_p521_alt, p521_jscalarmul_alt+};++/* The same question as for P-256, answered the same way; see+ * cbits/p256/p256_s2n.c. */+static int use_alt(void)+{+#if defined(__aarch64__) || defined(__arm64__)+#ifdef __APPLE__+	return 1;+#else+	return 0;+#endif+#else+	return (crypton_x86_simd_features() & CRYPTON_X86_ADX) == 0;+#endif+}++#define MAXWORDS 9++static void be_to_le64(uint64_t *w, const uint8_t *b, uint32_t len)+{+	uint32_t i;++	for (i = 0; i < MAXWORDS; i++)+		w[i] = 0;+	for (i = 0; i < len; i++) {+		uint32_t pos = len - 1 - i;+		w[pos / 8] |= (uint64_t)b[i] << (8 * (pos % 8));+	}+}++static void le64_to_be(uint8_t *b, uint32_t len, const uint64_t *w)+{+	uint32_t i;++	for (i = 0; i < len; i++)+		b[len - 1 - i] = (uint8_t)(w[i / 8] >> (8 * (i % 8)));+}++static int is_zero(const uint64_t *w, int words)+{+	uint64_t acc = 0;+	int i;++	for (i = 0; i < words; i++)+		acc |= w[i];+	return acc == 0;+}++static int mul_p384(uint8_t *outx, uint8_t *outy, const uint8_t *px,+                    const uint8_t *py, const uint8_t *k, uint32_t klen)+{+	const struct p384_asm *f = use_alt() ? &p384_alt : &p384_std;+	uint64_t pt[18], res[18], sc[MAXWORDS], t[MAXWORDS];+	uint64_t zi[6], zi2[6], zi3[6], num[6];+	static const uint64_t one[6] = {1, 0, 0, 0, 0, 0};++	be_to_le64(t, px, P384_PLEN);+	f->tomont(pt, t);+	be_to_le64(t, py, P384_PLEN);+	f->tomont(pt + 6, t);+	f->tomont(pt + 12, one);+	be_to_le64(sc, k, klen);++	f->jscalarmul(res, sc, pt);+	if (is_zero(res + 12, 6))+		return 1;++	/* affine again: x = X/Z^2, y = Y/Z^3, with the inverse taken in the+	 * Montgomery domain so that it lands where the rest of these are */+	bignum_montinv_p384(zi, res + 12);+	f->montsqr(zi2, zi);+	f->montmul(zi3, zi2, zi);+	f->montmul(num, res, zi2);+	f->deamont(t, num);+	le64_to_be(outx, P384_PLEN, t);+	f->montmul(num, res + 6, zi3);+	f->deamont(t, num);+	le64_to_be(outy, P384_PLEN, t);+	return 0;+}++static int mul_p521(uint8_t *outx, uint8_t *outy, const uint8_t *px,+                    const uint8_t *py, const uint8_t *k, uint32_t klen)+{+	const struct p521_asm *f = use_alt() ? &p521_alt : &p521_std;+	uint64_t pt[27], res[27], sc[MAXWORDS], t[MAXWORDS];+	uint64_t zi[9], zi2[9], zi3[9];+	static const uint64_t one[9] = {1, 0, 0, 0, 0, 0, 0, 0, 0};++	be_to_le64(pt, px, P521_PLEN);+	be_to_le64(pt + 9, py, P521_PLEN);+	memcpy(pt + 18, one, sizeof(one));+	be_to_le64(sc, k, klen);++	f->jscalarmul(res, sc, pt);+	if (is_zero(res + 18, 9))+		return 1;++	bignum_inv_p521(zi, res + 18);+	f->sqr(zi2, zi);+	f->mul(zi3, zi2, zi);+	f->mul(t, res, zi2);+	le64_to_be(outx, P521_PLEN, t);+	f->mul(t, res + 9, zi3);+	le64_to_be(outy, P521_PLEN, t);+	return 0;+}++int crypton_s2n_ecc_mul(int *ret, uint8_t *outx, uint8_t *outy,+                        const uint8_t *px, const uint8_t *py,+                        const uint8_t *k, uint32_t klen,+                        const uint8_t *a, const uint8_t *b,+                        const uint8_t *p, uint32_t plen)+{+	int is384;++	if (plen == P384_PLEN && memcmp(p, P384_P, plen) == 0+	    && memcmp(a, P384_A, plen) == 0 && memcmp(b, P384_B, plen) == 0)+		is384 = 1;+	else if (plen == P521_PLEN && memcmp(p, P521_P, plen) == 0+	         && memcmp(a, P521_A, plen) == 0+	         && memcmp(b, P521_B, plen) == 0)+		is384 = 0;+	else+		return 0; /* some other curve; the C answers it */++	/* A scalar longer than the prime is not something the word arrays+	 * here hold, and it is not what any caller of these two curves+	 * sends, so leave it to the C rather than grow a second path. */+	if (klen == 0 || klen > plen)+		return 0;++	/* The C does not require the coordinates to be reduced -- it takes+	 * whatever fits in its limbs and lets the conversion to Montgomery+	 * form reduce it.  Rather than carry a reduction here to match, hand+	 * that case back: nothing sends one, and this way the two cannot+	 * disagree about it.  (A differential test against the C found this;+	 * the first version of this check returned -1 and was wrong.) */+	if (memcmp(px, p, plen) >= 0 || memcmp(py, p, plen) >= 0)+		return 0;++	*ret = is384 ? mul_p384(outx, outy, px, py, k, klen)+	             : mul_p521(outx, outy, px, py, k, klen);+	return 1;+}
+ cbits/crypton_ecc_s2n.h view
@@ -0,0 +1,27 @@+/*+ * P-384 and P-521 through the vendored s2n-bignum assembly, for the two+ * curves it knows among the ones crypton_ecc_mul is asked about.+ *+ * s2n-bignum has no affine wrapper for these two -- only a scalar+ * multiplication on Jacobian points, Montgomery-domain for P-384 and plain+ * for P-521 -- so the conversions in and out are built here out of its own+ * field operations.  See cbits/s2n/README.md.+ */+#ifndef CRYPTON_ECC_S2N_H+#define CRYPTON_ECC_S2N_H++#include <stdint.h>++/*+ * Returns 1 if this was a curve it knows and it answered, with *ret set to+ * what crypton_ecc_mul should return -- 0 and the point in outx and outy, 1+ * for the point at infinity, or -1 for arguments it will not take.  Returns+ * 0 if the curve is not one of its two and nothing was written.+ */+int crypton_s2n_ecc_mul(int *ret, uint8_t *outx, uint8_t *outy,+                        const uint8_t *px, const uint8_t *py,+                        const uint8_t *k, uint32_t klen,+                        const uint8_t *a, const uint8_t *b,+                        const uint8_t *p, uint32_t plen);++#endif
+ cbits/crypton_ecc_s2n_curves.h view
@@ -0,0 +1,76 @@+/*+ * p, a and b of the two curves the vendored s2n-bignum assembly knows, as+ * big-endian bytes, which is how crypton_ecc_mul is given a curve.  They are+ * here to be compared against, not computed with: a caller naming some other+ * curve with the same sizes has to go to the C.+ *+ * Generated from `openssl ecparam -name secp384r1 -param_enc explicit -text`+ * and the same for secp521r1, rather than transcribed.+ */+#ifndef CRYPTON_ECC_S2N_CURVES_H+#define CRYPTON_ECC_S2N_CURVES_H++#include <stdint.h>++#define P384_PLEN 48+static const uint8_t P384_P[48] = {+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xfe,+	0xff,0xff,0xff,0xff,0x00,0x00,0x00,0x00,+	0x00,0x00,0x00,0x00,0xff,0xff,0xff,0xff+};+static const uint8_t P384_A[48] = {+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xfe,+	0xff,0xff,0xff,0xff,0x00,0x00,0x00,0x00,+	0x00,0x00,0x00,0x00,0xff,0xff,0xff,0xfc+};+static const uint8_t P384_B[48] = {+	0xb3,0x31,0x2f,0xa7,0xe2,0x3e,0xe7,0xe4,+	0x98,0x8e,0x05,0x6b,0xe3,0xf8,0x2d,0x19,+	0x18,0x1d,0x9c,0x6e,0xfe,0x81,0x41,0x12,+	0x03,0x14,0x08,0x8f,0x50,0x13,0x87,0x5a,+	0xc6,0x56,0x39,0x8d,0x8a,0x2e,0xd1,0x9d,+	0x2a,0x85,0xc8,0xed,0xd3,0xec,0x2a,0xef+};++#define P521_PLEN 66+static const uint8_t P521_P[66] = {+	0x01,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff+};+static const uint8_t P521_A[66] = {+	0x01,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,+	0xff,0xfc+};+static const uint8_t P521_B[66] = {+	0x00,0x51,0x95,0x3e,0xb9,0x61,0x8e,0x1c,+	0x9a,0x1f,0x92,0x9a,0x21,0xa0,0xb6,0x85,+	0x40,0xee,0xa2,0xda,0x72,0x5b,0x99,0xb3,+	0x15,0xf3,0xb8,0xb4,0x89,0x91,0x8e,0xf1,+	0x09,0xe1,0x56,0x19,0x39,0x51,0xec,0x7e,+	0x93,0x7b,0x16,0x52,0xc0,0xbd,0x3b,0xb1,+	0xbf,0x07,0x35,0x73,0xdf,0x88,0x3d,0x2c,+	0x34,0xf1,0xef,0x45,0x1f,0xd4,0x6b,0x50,+	0x3f,0x00+};++#endif
cbits/crypton_f2m.c view
@@ -25,6 +25,8 @@ #include <stdlib.h> #include <string.h> #include <crypton_cpu.h>+#include "crypton_armv8_target.h"+#include <crypton_bzero.h> #include <crypton_f2m.h>  typedef uint64_t limb_t;@@ -89,7 +91,7 @@ #define PMULL_ATTR #define PMULL_ALWAYS 1 #else-#define PMULL_ATTR __attribute__((target("+crypto")))+#define PMULL_ATTR CRYPTON_TARGET_ARMV8_CRYPTO #define PMULL_ALWAYS 0 #endif @@ -543,8 +545,10 @@ 	ret = 0;  done:+	/* freed on the next line, so a plain memset here is a store the+	 * optimizer may drop */ 	if (space != NULL) {-		memset(space, 0, words * sizeof(limb_t));+		crypton_bzero(space, words * sizeof(limb_t)); 		free(space); 	} 	return ret;
cbits/crypton_md4.c view
@@ -48,16 +48,17 @@ #define K3 	0x6ED9EBA1 #define R(a,b,c,d,f,k,s,i) (a = rol32(a + f(b,c,d) + w[i] + k, s)) -static void md4_do_chunk(struct md4_ctx *ctx, uint32_t *buf)+/* The words are read out of the block rather than the block being pointed at+ * as though it were an array of them; see crypton_md5.c. */+static void md4_do_chunk(struct md4_ctx *ctx, const uint8_t *buf) { 	uint32_t a, b, c, d;-#ifdef ARCH_IS_BIG_ENDIAN 	uint32_t w[16];-	cpu_to_le32_array(w, (uint32_t *) buf, 16);-#else-	uint32_t *w = buf;-#endif+	int wi; +	for (wi = 0; wi < 16; wi++)+		w[wi] = load_le32(buf + 4 * wi);+ 	a = ctx->h[0]; b = ctx->h[1]; c = ctx->h[2]; d = ctx->h[3];  	R(a, b, c, d, f1, K1, 3, 0);@@ -125,24 +126,15 @@  	if (index && len >= to_fill) { 		memcpy(ctx->buf + index, data, to_fill);-		md4_do_chunk(ctx, (uint32_t *) ctx->buf);+		md4_do_chunk(ctx, ctx->buf); 		len -= to_fill; 		data += to_fill; 		index = 0; 	} -	if (need_alignment(data, 4)) {-		uint32_t tramp[16];-		ASSERT_ALIGNMENT(tramp, 4);-		for (; len >= 64; len -= 64, data += 64) {-			memcpy(tramp, data, 64);-			md4_do_chunk(ctx, tramp);-		}-	} else {-		/* process as much 64-block as possible */-		for (; len >= 64; len -= 64, data += 64)-			md4_do_chunk(ctx, (uint32_t *) data);-	}+	/* No trampoline: load_le32 does not ask for a boundary. */+	for (; len >= 64; len -= 64, data += 64)+		md4_do_chunk(ctx, data);  	/* append data into buf */ 	if (len)
cbits/crypton_md5.c view
@@ -45,15 +45,22 @@ #define f4(x, y, z)	(y ^ (x | ~z)) #define R(f, a, b, c, d, i, k, s) a += f(b, c, d) + w[i] + k; a = rol32(a, s); a += b -static void md5_do_chunk(struct md5_ctx *ctx, uint32_t *buf)+/* The sixteen words are read out of the block rather than the block being+ * pointed at as though it were an array of them.  A caller's pointer cast to+ * uint32_t * is a pointer the standard says may not exist unless the address+ * is aligned for it, and reading through it is undefined whether or not the+ * machine minds; UndefinedBehaviorSanitizer counted sixty-four of these.+ * load_le32 is a memcpy, which every compiler here turns into the one load+ * the cast used to be, and it takes the endianness with it -- so the two+ * arms this replaces are one. */+static void md5_do_chunk(struct md5_ctx *ctx, const uint8_t *buf) { 	uint32_t a, b, c, d;-#ifdef ARCH_IS_BIG_ENDIAN 	uint32_t w[16];-	cpu_to_le32_array(w, buf, 16);-#else-	uint32_t *w = buf;-#endif+	int wi;++	for (wi = 0; wi < 16; wi++)+		w[wi] = load_le32(buf + 4 * wi); 	a = ctx->h[0]; b = ctx->h[1]; c = ctx->h[2]; d = ctx->h[3];  	R(f1, a, b, c, d, 0, 0xd76aa478, 7);@@ -138,24 +145,16 @@  	if (index && len >= to_fill) { 		memcpy(ctx->buf + index, data, to_fill);-		md5_do_chunk(ctx, (uint32_t *) ctx->buf);+		md5_do_chunk(ctx, ctx->buf); 		len -= to_fill; 		data += to_fill; 		index = 0; 	} -	if (need_alignment(data, 4)) {-		uint32_t tramp[16];-		ASSERT_ALIGNMENT(tramp, 4);-		for (; len >= 64; len -= 64, data += 64) {-			memcpy(tramp, data, 64);-			md5_do_chunk(ctx, tramp);-		}-	} else {-		/* process as much 64-block as possible */-		for (; len >= 64; len -= 64, data += 64)-			md5_do_chunk(ctx, (uint32_t *) data);-	}+	/* No trampoline for a block that is not on a four-byte boundary: the+	 * words are read with load_le32 now, which does not ask. */+	for (; len >= 64; len -= 64, data += 64)+		md5_do_chunk(ctx, data);  	/* append data into buf */ 	if (len)
+ cbits/crypton_modinv.c view
@@ -0,0 +1,74 @@+/*+ * Inversion modulo an odd number through s2n-bignum, whose routine takes a+ * fixed number of division steps rather than an exponentiation: twenty to+ * thirty times less work than Fermat's little theorem at the sizes here.+ * Measured on an Apple M4, inverting modulo a curve order:+ *+ *              Fermat     this+ *     P-256    6.02 us    0.80+ *     P-384    31.3       1.20+ *     P-521    63.2       2.05+ *+ * It uses no instruction beyond the base architecture on either x86-64 or+ * AArch64, so unlike the rest of the vendored assembly there is nothing to+ * ask the processor first.+ */+#include <string.h>++#include "crypton_modinv.h"++#ifdef CRYPTON_S2N_BIGNUM++extern void bignum_modinv(uint64_t k, uint64_t *z, const uint64_t *a,+                          const uint64_t *b, uint64_t *t);++/* 4096 bits and no more, which covers every modulus that reaches here -- the+ * order of a curve, or a prime factor of an RSA modulus -- and keeps the+ * working space on the stack.  Anything larger is handed back. */+#define MODINV_MAXWORDS 64++int crypton_modinv_sec(uint8_t *z, const uint8_t *a, const uint8_t *m,+                       uint32_t len)+{+	uint64_t aw[MODINV_MAXWORDS], mw[MODINV_MAXWORDS];+	uint64_t zw[MODINV_MAXWORDS], t[3 * MODINV_MAXWORDS];+	uint32_t k = (len + 7) / 8;+	uint32_t i;++	/* An even modulus is the one case it answers without saying it+	 * cannot: it returns a number that is not an inverse rather than+	 * failing, so keep it away from here.  Every caller's modulus is odd. */+	if (len == 0 || k > MODINV_MAXWORDS || (m[len - 1] & 1) == 0)+		return 1;++	for (i = 0; i < k; i++) {+		aw[i] = 0;+		mw[i] = 0;+	}+	for (i = 0; i < len; i++) {+		uint32_t pos = len - 1 - i;++		aw[pos / 8] |= (uint64_t)a[i] << (8 * (pos % 8));+		mw[pos / 8] |= (uint64_t)m[i] << (8 * (pos % 8));+	}++	bignum_modinv(k, zw, aw, mw, t);++	for (i = 0; i < len; i++)+		z[len - 1 - i] = (uint8_t)(zw[i / 8] >> (8 * (i % 8)));+	return 0;+}++#else++int crypton_modinv_sec(uint8_t *z, const uint8_t *a, const uint8_t *m,+                       uint32_t len)+{+	(void)z;+	(void)a;+	(void)m;+	(void)len;+	return 1;+}++#endif
+ cbits/crypton_modinv.h view
@@ -0,0 +1,22 @@+#ifndef CRYPTON_MODINV_H+#define CRYPTON_MODINV_H++#include <stdint.h>++/* z = a^-1 mod m, all three big-endian byte strings of len bytes.+ *+ * Returns 0 with the answer in z, and 1 without touching z when it will not+ * do this one: the assembly it needs is not built, the modulus is even, or+ * the numbers are larger than it keeps room for.  A 1 is not an error, it is+ * "ask something else".+ *+ * The answer is not checked here.  When a has no inverse the routine+ * underneath returns something that is not one rather than saying so, so the+ * caller has to multiply out and look -- which is what Crypto.Number.+ * ModArithmetic.inverseSafe already did for the exponentiation this+ * replaces.+ */+int crypton_modinv_sec(uint8_t *z, const uint8_t *a, const uint8_t *m,+                       uint32_t len);++#endif
+ cbits/crypton_mont.h view
@@ -0,0 +1,13445 @@+/* Generated by cbits/crypton_mont.py; do not edit.+ *+ * Montgomery multiplication and squaring for AArch64 at 16, 24 and 32 limbs,+ * the halves of RSA-2048, -3072 and -4096, and squaring at 48, RSA-3072's+ * modulus.  Each answers what mont_mul and mont_sqr answer. */+#ifndef CRYPTON_MONT_H+#define CRYPTON_MONT_H++#include "crypton_bignum.h"++/* (t2:t1:t0) += x * y */+#define MONT_MAC(t0, t1, t2, x, y)                                      \+	do {                                                                \+		limb_t lo_, hi_;                                                \+		__asm__("mul\t%0, %5, %6\n\t"                                   \+		        "umulh\t%1, %5, %6\n\t"                                 \+		        "adds\t%2, %2, %0\n\t"                                  \+		        "adcs\t%3, %3, %1\n\t"                                  \+		        "adc\t%4, %4, xzr"                                       \+		        : "=&r"(lo_), "=&r"(hi_), "+r"(t0), "+r"(t1), "+r"(t2)  \+		        : "r"(x), "r"(y)                                        \+		        : "cc");                                                \+	} while (0)++/* (s2:s1:s0) = x * y */+#define MONT_MUL(s0, s1, s2, x, y)                                      \+	__asm__("mul\t%0, %3, %4\n\t"                                       \+	        "umulh\t%1, %3, %4\n\t"                                     \+	        "mov\t%2, xzr"                                               \+	        : "=&r"(s0), "=&r"(s1), "=r"(s2)                            \+	        : "r"(x), "r"(y))++/* (t2:t1:t0) += (s2:s1:s0) */+#define MONT_ADD(t0, t1, t2, s0, s1, s2)                                \+	__asm__("adds\t%0, %0, %3\n\t"                                      \+	        "adcs\t%1, %1, %4\n\t"                                      \+	        "adc\t%2, %2, %5"                                            \+	        : "+r"(t0), "+r"(t1), "+r"(t2)                              \+	        : "r"(s0), "r"(s1), "r"(s2)                                 \+	        : "cc")++/* (s2:s1:s0) *= 2 */+#define MONT_DBL(s0, s1, s2)                                            \+	do {                                                                \+		s2 = (s2 << 1) | (s1 >> 63);                                    \+		s1 = (s1 << 1) | (s0 >> 63);                                    \+		s0 <<= 1;                                                       \+	} while (0)++/* r = a * b * R^-1 mod m, for a and b under m; t is 2n limbs of scratch */+static void crypton_mont_mul16(limb_t *r, const limb_t *a, const limb_t *b,+                               const limb_t *m, limb_t n0, limb_t *t)+{+	limb_t *u = t, *z = t + 16;+	limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take;++	/* column 0 */+	MONT_MAC(t0, t1, t2, a[0], b[0]);+	u[0] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[0], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 1 */+	MONT_MAC(t0, t1, t2, a[0], b[1]);+	MONT_MUL(v0, v1, v2, a[1], b[0]);+	MONT_MUL(w0, w1, w2, u[0], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[1] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[1], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 2 */+	MONT_MAC(t0, t1, t2, a[0], b[2]);+	MONT_MUL(v0, v1, v2, a[1], b[1]);+	MONT_MUL(w0, w1, w2, a[2], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[2]);+	MONT_MAC(v0, v1, v2, u[1], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[2] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[2], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 3 */+	MONT_MAC(t0, t1, t2, a[0], b[3]);+	MONT_MUL(v0, v1, v2, a[1], b[2]);+	MONT_MUL(w0, w1, w2, a[2], b[1]);+	MONT_MAC(t0, t1, t2, a[3], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[3]);+	MONT_MAC(w0, w1, w2, u[1], m[2]);+	MONT_MAC(t0, t1, t2, u[2], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[3] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[3], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 4 */+	MONT_MAC(t0, t1, t2, a[0], b[4]);+	MONT_MUL(v0, v1, v2, a[1], b[3]);+	MONT_MUL(w0, w1, w2, a[2], b[2]);+	MONT_MAC(t0, t1, t2, a[3], b[1]);+	MONT_MAC(v0, v1, v2, a[4], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[4]);+	MONT_MAC(t0, t1, t2, u[1], m[3]);+	MONT_MAC(v0, v1, v2, u[2], m[2]);+	MONT_MAC(w0, w1, w2, u[3], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[4] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[4], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 5 */+	MONT_MAC(t0, t1, t2, a[0], b[5]);+	MONT_MUL(v0, v1, v2, a[1], b[4]);+	MONT_MUL(w0, w1, w2, a[2], b[3]);+	MONT_MAC(t0, t1, t2, a[3], b[2]);+	MONT_MAC(v0, v1, v2, a[4], b[1]);+	MONT_MAC(w0, w1, w2, a[5], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[5]);+	MONT_MAC(v0, v1, v2, u[1], m[4]);+	MONT_MAC(w0, w1, w2, u[2], m[3]);+	MONT_MAC(t0, t1, t2, u[3], m[2]);+	MONT_MAC(v0, v1, v2, u[4], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[5] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[5], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 6 */+	MONT_MAC(t0, t1, t2, a[0], b[6]);+	MONT_MUL(v0, v1, v2, a[1], b[5]);+	MONT_MUL(w0, w1, w2, a[2], b[4]);+	MONT_MAC(t0, t1, t2, a[3], b[3]);+	MONT_MAC(v0, v1, v2, a[4], b[2]);+	MONT_MAC(w0, w1, w2, a[5], b[1]);+	MONT_MAC(t0, t1, t2, a[6], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[6]);+	MONT_MAC(w0, w1, w2, u[1], m[5]);+	MONT_MAC(t0, t1, t2, u[2], m[4]);+	MONT_MAC(v0, v1, v2, u[3], m[3]);+	MONT_MAC(w0, w1, w2, u[4], m[2]);+	MONT_MAC(t0, t1, t2, u[5], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[6] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[6], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 7 */+	MONT_MAC(t0, t1, t2, a[0], b[7]);+	MONT_MUL(v0, v1, v2, a[1], b[6]);+	MONT_MUL(w0, w1, w2, a[2], b[5]);+	MONT_MAC(t0, t1, t2, a[3], b[4]);+	MONT_MAC(v0, v1, v2, a[4], b[3]);+	MONT_MAC(w0, w1, w2, a[5], b[2]);+	MONT_MAC(t0, t1, t2, a[6], b[1]);+	MONT_MAC(v0, v1, v2, a[7], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[7]);+	MONT_MAC(t0, t1, t2, u[1], m[6]);+	MONT_MAC(v0, v1, v2, u[2], m[5]);+	MONT_MAC(w0, w1, w2, u[3], m[4]);+	MONT_MAC(t0, t1, t2, u[4], m[3]);+	MONT_MAC(v0, v1, v2, u[5], m[2]);+	MONT_MAC(w0, w1, w2, u[6], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[7] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[7], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 8 */+	MONT_MAC(t0, t1, t2, a[0], b[8]);+	MONT_MUL(v0, v1, v2, a[1], b[7]);+	MONT_MUL(w0, w1, w2, a[2], b[6]);+	MONT_MAC(t0, t1, t2, a[3], b[5]);+	MONT_MAC(v0, v1, v2, a[4], b[4]);+	MONT_MAC(w0, w1, w2, a[5], b[3]);+	MONT_MAC(t0, t1, t2, a[6], b[2]);+	MONT_MAC(v0, v1, v2, a[7], b[1]);+	MONT_MAC(w0, w1, w2, a[8], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[8]);+	MONT_MAC(v0, v1, v2, u[1], m[7]);+	MONT_MAC(w0, w1, w2, u[2], m[6]);+	MONT_MAC(t0, t1, t2, u[3], m[5]);+	MONT_MAC(v0, v1, v2, u[4], m[4]);+	MONT_MAC(w0, w1, w2, u[5], m[3]);+	MONT_MAC(t0, t1, t2, u[6], m[2]);+	MONT_MAC(v0, v1, v2, u[7], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[8] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[8], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 9 */+	MONT_MAC(t0, t1, t2, a[0], b[9]);+	MONT_MUL(v0, v1, v2, a[1], b[8]);+	MONT_MUL(w0, w1, w2, a[2], b[7]);+	MONT_MAC(t0, t1, t2, a[3], b[6]);+	MONT_MAC(v0, v1, v2, a[4], b[5]);+	MONT_MAC(w0, w1, w2, a[5], b[4]);+	MONT_MAC(t0, t1, t2, a[6], b[3]);+	MONT_MAC(v0, v1, v2, a[7], b[2]);+	MONT_MAC(w0, w1, w2, a[8], b[1]);+	MONT_MAC(t0, t1, t2, a[9], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[9]);+	MONT_MAC(w0, w1, w2, u[1], m[8]);+	MONT_MAC(t0, t1, t2, u[2], m[7]);+	MONT_MAC(v0, v1, v2, u[3], m[6]);+	MONT_MAC(w0, w1, w2, u[4], m[5]);+	MONT_MAC(t0, t1, t2, u[5], m[4]);+	MONT_MAC(v0, v1, v2, u[6], m[3]);+	MONT_MAC(w0, w1, w2, u[7], m[2]);+	MONT_MAC(t0, t1, t2, u[8], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[9] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[9], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 10 */+	MONT_MAC(t0, t1, t2, a[0], b[10]);+	MONT_MUL(v0, v1, v2, a[1], b[9]);+	MONT_MUL(w0, w1, w2, a[2], b[8]);+	MONT_MAC(t0, t1, t2, a[3], b[7]);+	MONT_MAC(v0, v1, v2, a[4], b[6]);+	MONT_MAC(w0, w1, w2, a[5], b[5]);+	MONT_MAC(t0, t1, t2, a[6], b[4]);+	MONT_MAC(v0, v1, v2, a[7], b[3]);+	MONT_MAC(w0, w1, w2, a[8], b[2]);+	MONT_MAC(t0, t1, t2, a[9], b[1]);+	MONT_MAC(v0, v1, v2, a[10], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[10]);+	MONT_MAC(t0, t1, t2, u[1], m[9]);+	MONT_MAC(v0, v1, v2, u[2], m[8]);+	MONT_MAC(w0, w1, w2, u[3], m[7]);+	MONT_MAC(t0, t1, t2, u[4], m[6]);+	MONT_MAC(v0, v1, v2, u[5], m[5]);+	MONT_MAC(w0, w1, w2, u[6], m[4]);+	MONT_MAC(t0, t1, t2, u[7], m[3]);+	MONT_MAC(v0, v1, v2, u[8], m[2]);+	MONT_MAC(w0, w1, w2, u[9], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[10] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[10], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 11 */+	MONT_MAC(t0, t1, t2, a[0], b[11]);+	MONT_MUL(v0, v1, v2, a[1], b[10]);+	MONT_MUL(w0, w1, w2, a[2], b[9]);+	MONT_MAC(t0, t1, t2, a[3], b[8]);+	MONT_MAC(v0, v1, v2, a[4], b[7]);+	MONT_MAC(w0, w1, w2, a[5], b[6]);+	MONT_MAC(t0, t1, t2, a[6], b[5]);+	MONT_MAC(v0, v1, v2, a[7], b[4]);+	MONT_MAC(w0, w1, w2, a[8], b[3]);+	MONT_MAC(t0, t1, t2, a[9], b[2]);+	MONT_MAC(v0, v1, v2, a[10], b[1]);+	MONT_MAC(w0, w1, w2, a[11], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[11]);+	MONT_MAC(v0, v1, v2, u[1], m[10]);+	MONT_MAC(w0, w1, w2, u[2], m[9]);+	MONT_MAC(t0, t1, t2, u[3], m[8]);+	MONT_MAC(v0, v1, v2, u[4], m[7]);+	MONT_MAC(w0, w1, w2, u[5], m[6]);+	MONT_MAC(t0, t1, t2, u[6], m[5]);+	MONT_MAC(v0, v1, v2, u[7], m[4]);+	MONT_MAC(w0, w1, w2, u[8], m[3]);+	MONT_MAC(t0, t1, t2, u[9], m[2]);+	MONT_MAC(v0, v1, v2, u[10], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[11] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[11], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 12 */+	MONT_MAC(t0, t1, t2, a[0], b[12]);+	MONT_MUL(v0, v1, v2, a[1], b[11]);+	MONT_MUL(w0, w1, w2, a[2], b[10]);+	MONT_MAC(t0, t1, t2, a[3], b[9]);+	MONT_MAC(v0, v1, v2, a[4], b[8]);+	MONT_MAC(w0, w1, w2, a[5], b[7]);+	MONT_MAC(t0, t1, t2, a[6], b[6]);+	MONT_MAC(v0, v1, v2, a[7], b[5]);+	MONT_MAC(w0, w1, w2, a[8], b[4]);+	MONT_MAC(t0, t1, t2, a[9], b[3]);+	MONT_MAC(v0, v1, v2, a[10], b[2]);+	MONT_MAC(w0, w1, w2, a[11], b[1]);+	MONT_MAC(t0, t1, t2, a[12], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[12]);+	MONT_MAC(w0, w1, w2, u[1], m[11]);+	MONT_MAC(t0, t1, t2, u[2], m[10]);+	MONT_MAC(v0, v1, v2, u[3], m[9]);+	MONT_MAC(w0, w1, w2, u[4], m[8]);+	MONT_MAC(t0, t1, t2, u[5], m[7]);+	MONT_MAC(v0, v1, v2, u[6], m[6]);+	MONT_MAC(w0, w1, w2, u[7], m[5]);+	MONT_MAC(t0, t1, t2, u[8], m[4]);+	MONT_MAC(v0, v1, v2, u[9], m[3]);+	MONT_MAC(w0, w1, w2, u[10], m[2]);+	MONT_MAC(t0, t1, t2, u[11], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[12] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[12], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 13 */+	MONT_MAC(t0, t1, t2, a[0], b[13]);+	MONT_MUL(v0, v1, v2, a[1], b[12]);+	MONT_MUL(w0, w1, w2, a[2], b[11]);+	MONT_MAC(t0, t1, t2, a[3], b[10]);+	MONT_MAC(v0, v1, v2, a[4], b[9]);+	MONT_MAC(w0, w1, w2, a[5], b[8]);+	MONT_MAC(t0, t1, t2, a[6], b[7]);+	MONT_MAC(v0, v1, v2, a[7], b[6]);+	MONT_MAC(w0, w1, w2, a[8], b[5]);+	MONT_MAC(t0, t1, t2, a[9], b[4]);+	MONT_MAC(v0, v1, v2, a[10], b[3]);+	MONT_MAC(w0, w1, w2, a[11], b[2]);+	MONT_MAC(t0, t1, t2, a[12], b[1]);+	MONT_MAC(v0, v1, v2, a[13], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[13]);+	MONT_MAC(t0, t1, t2, u[1], m[12]);+	MONT_MAC(v0, v1, v2, u[2], m[11]);+	MONT_MAC(w0, w1, w2, u[3], m[10]);+	MONT_MAC(t0, t1, t2, u[4], m[9]);+	MONT_MAC(v0, v1, v2, u[5], m[8]);+	MONT_MAC(w0, w1, w2, u[6], m[7]);+	MONT_MAC(t0, t1, t2, u[7], m[6]);+	MONT_MAC(v0, v1, v2, u[8], m[5]);+	MONT_MAC(w0, w1, w2, u[9], m[4]);+	MONT_MAC(t0, t1, t2, u[10], m[3]);+	MONT_MAC(v0, v1, v2, u[11], m[2]);+	MONT_MAC(w0, w1, w2, u[12], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[13] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[13], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 14 */+	MONT_MAC(t0, t1, t2, a[0], b[14]);+	MONT_MUL(v0, v1, v2, a[1], b[13]);+	MONT_MUL(w0, w1, w2, a[2], b[12]);+	MONT_MAC(t0, t1, t2, a[3], b[11]);+	MONT_MAC(v0, v1, v2, a[4], b[10]);+	MONT_MAC(w0, w1, w2, a[5], b[9]);+	MONT_MAC(t0, t1, t2, a[6], b[8]);+	MONT_MAC(v0, v1, v2, a[7], b[7]);+	MONT_MAC(w0, w1, w2, a[8], b[6]);+	MONT_MAC(t0, t1, t2, a[9], b[5]);+	MONT_MAC(v0, v1, v2, a[10], b[4]);+	MONT_MAC(w0, w1, w2, a[11], b[3]);+	MONT_MAC(t0, t1, t2, a[12], b[2]);+	MONT_MAC(v0, v1, v2, a[13], b[1]);+	MONT_MAC(w0, w1, w2, a[14], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[14]);+	MONT_MAC(v0, v1, v2, u[1], m[13]);+	MONT_MAC(w0, w1, w2, u[2], m[12]);+	MONT_MAC(t0, t1, t2, u[3], m[11]);+	MONT_MAC(v0, v1, v2, u[4], m[10]);+	MONT_MAC(w0, w1, w2, u[5], m[9]);+	MONT_MAC(t0, t1, t2, u[6], m[8]);+	MONT_MAC(v0, v1, v2, u[7], m[7]);+	MONT_MAC(w0, w1, w2, u[8], m[6]);+	MONT_MAC(t0, t1, t2, u[9], m[5]);+	MONT_MAC(v0, v1, v2, u[10], m[4]);+	MONT_MAC(w0, w1, w2, u[11], m[3]);+	MONT_MAC(t0, t1, t2, u[12], m[2]);+	MONT_MAC(v0, v1, v2, u[13], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[14] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[14], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 15 */+	MONT_MAC(t0, t1, t2, a[0], b[15]);+	MONT_MUL(v0, v1, v2, a[1], b[14]);+	MONT_MUL(w0, w1, w2, a[2], b[13]);+	MONT_MAC(t0, t1, t2, a[3], b[12]);+	MONT_MAC(v0, v1, v2, a[4], b[11]);+	MONT_MAC(w0, w1, w2, a[5], b[10]);+	MONT_MAC(t0, t1, t2, a[6], b[9]);+	MONT_MAC(v0, v1, v2, a[7], b[8]);+	MONT_MAC(w0, w1, w2, a[8], b[7]);+	MONT_MAC(t0, t1, t2, a[9], b[6]);+	MONT_MAC(v0, v1, v2, a[10], b[5]);+	MONT_MAC(w0, w1, w2, a[11], b[4]);+	MONT_MAC(t0, t1, t2, a[12], b[3]);+	MONT_MAC(v0, v1, v2, a[13], b[2]);+	MONT_MAC(w0, w1, w2, a[14], b[1]);+	MONT_MAC(t0, t1, t2, a[15], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[15]);+	MONT_MAC(w0, w1, w2, u[1], m[14]);+	MONT_MAC(t0, t1, t2, u[2], m[13]);+	MONT_MAC(v0, v1, v2, u[3], m[12]);+	MONT_MAC(w0, w1, w2, u[4], m[11]);+	MONT_MAC(t0, t1, t2, u[5], m[10]);+	MONT_MAC(v0, v1, v2, u[6], m[9]);+	MONT_MAC(w0, w1, w2, u[7], m[8]);+	MONT_MAC(t0, t1, t2, u[8], m[7]);+	MONT_MAC(v0, v1, v2, u[9], m[6]);+	MONT_MAC(w0, w1, w2, u[10], m[5]);+	MONT_MAC(t0, t1, t2, u[11], m[4]);+	MONT_MAC(v0, v1, v2, u[12], m[3]);+	MONT_MAC(w0, w1, w2, u[13], m[2]);+	MONT_MAC(t0, t1, t2, u[14], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[15] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[15], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 16 */+	MONT_MAC(t0, t1, t2, a[1], b[15]);+	MONT_MUL(v0, v1, v2, a[2], b[14]);+	MONT_MUL(w0, w1, w2, a[3], b[13]);+	MONT_MAC(t0, t1, t2, a[4], b[12]);+	MONT_MAC(v0, v1, v2, a[5], b[11]);+	MONT_MAC(w0, w1, w2, a[6], b[10]);+	MONT_MAC(t0, t1, t2, a[7], b[9]);+	MONT_MAC(v0, v1, v2, a[8], b[8]);+	MONT_MAC(w0, w1, w2, a[9], b[7]);+	MONT_MAC(t0, t1, t2, a[10], b[6]);+	MONT_MAC(v0, v1, v2, a[11], b[5]);+	MONT_MAC(w0, w1, w2, a[12], b[4]);+	MONT_MAC(t0, t1, t2, a[13], b[3]);+	MONT_MAC(v0, v1, v2, a[14], b[2]);+	MONT_MAC(w0, w1, w2, a[15], b[1]);+	MONT_MAC(t0, t1, t2, u[1], m[15]);+	MONT_MAC(v0, v1, v2, u[2], m[14]);+	MONT_MAC(w0, w1, w2, u[3], m[13]);+	MONT_MAC(t0, t1, t2, u[4], m[12]);+	MONT_MAC(v0, v1, v2, u[5], m[11]);+	MONT_MAC(w0, w1, w2, u[6], m[10]);+	MONT_MAC(t0, t1, t2, u[7], m[9]);+	MONT_MAC(v0, v1, v2, u[8], m[8]);+	MONT_MAC(w0, w1, w2, u[9], m[7]);+	MONT_MAC(t0, t1, t2, u[10], m[6]);+	MONT_MAC(v0, v1, v2, u[11], m[5]);+	MONT_MAC(w0, w1, w2, u[12], m[4]);+	MONT_MAC(t0, t1, t2, u[13], m[3]);+	MONT_MAC(v0, v1, v2, u[14], m[2]);+	MONT_MAC(w0, w1, w2, u[15], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[0] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 17 */+	MONT_MAC(t0, t1, t2, a[2], b[15]);+	MONT_MUL(v0, v1, v2, a[3], b[14]);+	MONT_MUL(w0, w1, w2, a[4], b[13]);+	MONT_MAC(t0, t1, t2, a[5], b[12]);+	MONT_MAC(v0, v1, v2, a[6], b[11]);+	MONT_MAC(w0, w1, w2, a[7], b[10]);+	MONT_MAC(t0, t1, t2, a[8], b[9]);+	MONT_MAC(v0, v1, v2, a[9], b[8]);+	MONT_MAC(w0, w1, w2, a[10], b[7]);+	MONT_MAC(t0, t1, t2, a[11], b[6]);+	MONT_MAC(v0, v1, v2, a[12], b[5]);+	MONT_MAC(w0, w1, w2, a[13], b[4]);+	MONT_MAC(t0, t1, t2, a[14], b[3]);+	MONT_MAC(v0, v1, v2, a[15], b[2]);+	MONT_MAC(w0, w1, w2, u[2], m[15]);+	MONT_MAC(t0, t1, t2, u[3], m[14]);+	MONT_MAC(v0, v1, v2, u[4], m[13]);+	MONT_MAC(w0, w1, w2, u[5], m[12]);+	MONT_MAC(t0, t1, t2, u[6], m[11]);+	MONT_MAC(v0, v1, v2, u[7], m[10]);+	MONT_MAC(w0, w1, w2, u[8], m[9]);+	MONT_MAC(t0, t1, t2, u[9], m[8]);+	MONT_MAC(v0, v1, v2, u[10], m[7]);+	MONT_MAC(w0, w1, w2, u[11], m[6]);+	MONT_MAC(t0, t1, t2, u[12], m[5]);+	MONT_MAC(v0, v1, v2, u[13], m[4]);+	MONT_MAC(w0, w1, w2, u[14], m[3]);+	MONT_MAC(t0, t1, t2, u[15], m[2]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[1] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 18 */+	MONT_MAC(t0, t1, t2, a[3], b[15]);+	MONT_MUL(v0, v1, v2, a[4], b[14]);+	MONT_MUL(w0, w1, w2, a[5], b[13]);+	MONT_MAC(t0, t1, t2, a[6], b[12]);+	MONT_MAC(v0, v1, v2, a[7], b[11]);+	MONT_MAC(w0, w1, w2, a[8], b[10]);+	MONT_MAC(t0, t1, t2, a[9], b[9]);+	MONT_MAC(v0, v1, v2, a[10], b[8]);+	MONT_MAC(w0, w1, w2, a[11], b[7]);+	MONT_MAC(t0, t1, t2, a[12], b[6]);+	MONT_MAC(v0, v1, v2, a[13], b[5]);+	MONT_MAC(w0, w1, w2, a[14], b[4]);+	MONT_MAC(t0, t1, t2, a[15], b[3]);+	MONT_MAC(v0, v1, v2, u[3], m[15]);+	MONT_MAC(w0, w1, w2, u[4], m[14]);+	MONT_MAC(t0, t1, t2, u[5], m[13]);+	MONT_MAC(v0, v1, v2, u[6], m[12]);+	MONT_MAC(w0, w1, w2, u[7], m[11]);+	MONT_MAC(t0, t1, t2, u[8], m[10]);+	MONT_MAC(v0, v1, v2, u[9], m[9]);+	MONT_MAC(w0, w1, w2, u[10], m[8]);+	MONT_MAC(t0, t1, t2, u[11], m[7]);+	MONT_MAC(v0, v1, v2, u[12], m[6]);+	MONT_MAC(w0, w1, w2, u[13], m[5]);+	MONT_MAC(t0, t1, t2, u[14], m[4]);+	MONT_MAC(v0, v1, v2, u[15], m[3]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[2] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 19 */+	MONT_MAC(t0, t1, t2, a[4], b[15]);+	MONT_MUL(v0, v1, v2, a[5], b[14]);+	MONT_MUL(w0, w1, w2, a[6], b[13]);+	MONT_MAC(t0, t1, t2, a[7], b[12]);+	MONT_MAC(v0, v1, v2, a[8], b[11]);+	MONT_MAC(w0, w1, w2, a[9], b[10]);+	MONT_MAC(t0, t1, t2, a[10], b[9]);+	MONT_MAC(v0, v1, v2, a[11], b[8]);+	MONT_MAC(w0, w1, w2, a[12], b[7]);+	MONT_MAC(t0, t1, t2, a[13], b[6]);+	MONT_MAC(v0, v1, v2, a[14], b[5]);+	MONT_MAC(w0, w1, w2, a[15], b[4]);+	MONT_MAC(t0, t1, t2, u[4], m[15]);+	MONT_MAC(v0, v1, v2, u[5], m[14]);+	MONT_MAC(w0, w1, w2, u[6], m[13]);+	MONT_MAC(t0, t1, t2, u[7], m[12]);+	MONT_MAC(v0, v1, v2, u[8], m[11]);+	MONT_MAC(w0, w1, w2, u[9], m[10]);+	MONT_MAC(t0, t1, t2, u[10], m[9]);+	MONT_MAC(v0, v1, v2, u[11], m[8]);+	MONT_MAC(w0, w1, w2, u[12], m[7]);+	MONT_MAC(t0, t1, t2, u[13], m[6]);+	MONT_MAC(v0, v1, v2, u[14], m[5]);+	MONT_MAC(w0, w1, w2, u[15], m[4]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[3] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 20 */+	MONT_MAC(t0, t1, t2, a[5], b[15]);+	MONT_MUL(v0, v1, v2, a[6], b[14]);+	MONT_MUL(w0, w1, w2, a[7], b[13]);+	MONT_MAC(t0, t1, t2, a[8], b[12]);+	MONT_MAC(v0, v1, v2, a[9], b[11]);+	MONT_MAC(w0, w1, w2, a[10], b[10]);+	MONT_MAC(t0, t1, t2, a[11], b[9]);+	MONT_MAC(v0, v1, v2, a[12], b[8]);+	MONT_MAC(w0, w1, w2, a[13], b[7]);+	MONT_MAC(t0, t1, t2, a[14], b[6]);+	MONT_MAC(v0, v1, v2, a[15], b[5]);+	MONT_MAC(w0, w1, w2, u[5], m[15]);+	MONT_MAC(t0, t1, t2, u[6], m[14]);+	MONT_MAC(v0, v1, v2, u[7], m[13]);+	MONT_MAC(w0, w1, w2, u[8], m[12]);+	MONT_MAC(t0, t1, t2, u[9], m[11]);+	MONT_MAC(v0, v1, v2, u[10], m[10]);+	MONT_MAC(w0, w1, w2, u[11], m[9]);+	MONT_MAC(t0, t1, t2, u[12], m[8]);+	MONT_MAC(v0, v1, v2, u[13], m[7]);+	MONT_MAC(w0, w1, w2, u[14], m[6]);+	MONT_MAC(t0, t1, t2, u[15], m[5]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[4] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 21 */+	MONT_MAC(t0, t1, t2, a[6], b[15]);+	MONT_MUL(v0, v1, v2, a[7], b[14]);+	MONT_MUL(w0, w1, w2, a[8], b[13]);+	MONT_MAC(t0, t1, t2, a[9], b[12]);+	MONT_MAC(v0, v1, v2, a[10], b[11]);+	MONT_MAC(w0, w1, w2, a[11], b[10]);+	MONT_MAC(t0, t1, t2, a[12], b[9]);+	MONT_MAC(v0, v1, v2, a[13], b[8]);+	MONT_MAC(w0, w1, w2, a[14], b[7]);+	MONT_MAC(t0, t1, t2, a[15], b[6]);+	MONT_MAC(v0, v1, v2, u[6], m[15]);+	MONT_MAC(w0, w1, w2, u[7], m[14]);+	MONT_MAC(t0, t1, t2, u[8], m[13]);+	MONT_MAC(v0, v1, v2, u[9], m[12]);+	MONT_MAC(w0, w1, w2, u[10], m[11]);+	MONT_MAC(t0, t1, t2, u[11], m[10]);+	MONT_MAC(v0, v1, v2, u[12], m[9]);+	MONT_MAC(w0, w1, w2, u[13], m[8]);+	MONT_MAC(t0, t1, t2, u[14], m[7]);+	MONT_MAC(v0, v1, v2, u[15], m[6]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[5] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 22 */+	MONT_MAC(t0, t1, t2, a[7], b[15]);+	MONT_MUL(v0, v1, v2, a[8], b[14]);+	MONT_MUL(w0, w1, w2, a[9], b[13]);+	MONT_MAC(t0, t1, t2, a[10], b[12]);+	MONT_MAC(v0, v1, v2, a[11], b[11]);+	MONT_MAC(w0, w1, w2, a[12], b[10]);+	MONT_MAC(t0, t1, t2, a[13], b[9]);+	MONT_MAC(v0, v1, v2, a[14], b[8]);+	MONT_MAC(w0, w1, w2, a[15], b[7]);+	MONT_MAC(t0, t1, t2, u[7], m[15]);+	MONT_MAC(v0, v1, v2, u[8], m[14]);+	MONT_MAC(w0, w1, w2, u[9], m[13]);+	MONT_MAC(t0, t1, t2, u[10], m[12]);+	MONT_MAC(v0, v1, v2, u[11], m[11]);+	MONT_MAC(w0, w1, w2, u[12], m[10]);+	MONT_MAC(t0, t1, t2, u[13], m[9]);+	MONT_MAC(v0, v1, v2, u[14], m[8]);+	MONT_MAC(w0, w1, w2, u[15], m[7]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[6] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 23 */+	MONT_MAC(t0, t1, t2, a[8], b[15]);+	MONT_MUL(v0, v1, v2, a[9], b[14]);+	MONT_MUL(w0, w1, w2, a[10], b[13]);+	MONT_MAC(t0, t1, t2, a[11], b[12]);+	MONT_MAC(v0, v1, v2, a[12], b[11]);+	MONT_MAC(w0, w1, w2, a[13], b[10]);+	MONT_MAC(t0, t1, t2, a[14], b[9]);+	MONT_MAC(v0, v1, v2, a[15], b[8]);+	MONT_MAC(w0, w1, w2, u[8], m[15]);+	MONT_MAC(t0, t1, t2, u[9], m[14]);+	MONT_MAC(v0, v1, v2, u[10], m[13]);+	MONT_MAC(w0, w1, w2, u[11], m[12]);+	MONT_MAC(t0, t1, t2, u[12], m[11]);+	MONT_MAC(v0, v1, v2, u[13], m[10]);+	MONT_MAC(w0, w1, w2, u[14], m[9]);+	MONT_MAC(t0, t1, t2, u[15], m[8]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[7] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 24 */+	MONT_MAC(t0, t1, t2, a[9], b[15]);+	MONT_MUL(v0, v1, v2, a[10], b[14]);+	MONT_MUL(w0, w1, w2, a[11], b[13]);+	MONT_MAC(t0, t1, t2, a[12], b[12]);+	MONT_MAC(v0, v1, v2, a[13], b[11]);+	MONT_MAC(w0, w1, w2, a[14], b[10]);+	MONT_MAC(t0, t1, t2, a[15], b[9]);+	MONT_MAC(v0, v1, v2, u[9], m[15]);+	MONT_MAC(w0, w1, w2, u[10], m[14]);+	MONT_MAC(t0, t1, t2, u[11], m[13]);+	MONT_MAC(v0, v1, v2, u[12], m[12]);+	MONT_MAC(w0, w1, w2, u[13], m[11]);+	MONT_MAC(t0, t1, t2, u[14], m[10]);+	MONT_MAC(v0, v1, v2, u[15], m[9]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[8] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 25 */+	MONT_MAC(t0, t1, t2, a[10], b[15]);+	MONT_MUL(v0, v1, v2, a[11], b[14]);+	MONT_MUL(w0, w1, w2, a[12], b[13]);+	MONT_MAC(t0, t1, t2, a[13], b[12]);+	MONT_MAC(v0, v1, v2, a[14], b[11]);+	MONT_MAC(w0, w1, w2, a[15], b[10]);+	MONT_MAC(t0, t1, t2, u[10], m[15]);+	MONT_MAC(v0, v1, v2, u[11], m[14]);+	MONT_MAC(w0, w1, w2, u[12], m[13]);+	MONT_MAC(t0, t1, t2, u[13], m[12]);+	MONT_MAC(v0, v1, v2, u[14], m[11]);+	MONT_MAC(w0, w1, w2, u[15], m[10]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[9] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 26 */+	MONT_MAC(t0, t1, t2, a[11], b[15]);+	MONT_MUL(v0, v1, v2, a[12], b[14]);+	MONT_MUL(w0, w1, w2, a[13], b[13]);+	MONT_MAC(t0, t1, t2, a[14], b[12]);+	MONT_MAC(v0, v1, v2, a[15], b[11]);+	MONT_MAC(w0, w1, w2, u[11], m[15]);+	MONT_MAC(t0, t1, t2, u[12], m[14]);+	MONT_MAC(v0, v1, v2, u[13], m[13]);+	MONT_MAC(w0, w1, w2, u[14], m[12]);+	MONT_MAC(t0, t1, t2, u[15], m[11]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[10] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 27 */+	MONT_MAC(t0, t1, t2, a[12], b[15]);+	MONT_MUL(v0, v1, v2, a[13], b[14]);+	MONT_MUL(w0, w1, w2, a[14], b[13]);+	MONT_MAC(t0, t1, t2, a[15], b[12]);+	MONT_MAC(v0, v1, v2, u[12], m[15]);+	MONT_MAC(w0, w1, w2, u[13], m[14]);+	MONT_MAC(t0, t1, t2, u[14], m[13]);+	MONT_MAC(v0, v1, v2, u[15], m[12]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[11] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 28 */+	MONT_MAC(t0, t1, t2, a[13], b[15]);+	MONT_MUL(v0, v1, v2, a[14], b[14]);+	MONT_MUL(w0, w1, w2, a[15], b[13]);+	MONT_MAC(t0, t1, t2, u[13], m[15]);+	MONT_MAC(v0, v1, v2, u[14], m[14]);+	MONT_MAC(w0, w1, w2, u[15], m[13]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[12] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 29 */+	MONT_MAC(t0, t1, t2, a[14], b[15]);+	MONT_MUL(v0, v1, v2, a[15], b[14]);+	MONT_MUL(w0, w1, w2, u[14], m[15]);+	MONT_MAC(t0, t1, t2, u[15], m[14]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[13] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 30 */+	MONT_MAC(t0, t1, t2, a[15], b[15]);+	MONT_MUL(v0, v1, v2, u[15], m[15]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	z[14] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	z[15] = t0;++	/* under 2m, with t1 the bit above n limbs: at most one subtraction */+	borrow = sub_n(r, z, m, 16);+	take = t1 | (borrow ^ 1);+	select_n(r, r, z, take & 1, 16);+}++/* r = a * a * R^-1 mod m, for a under m; t is 2n limbs of scratch */+static void crypton_mont_sqr16(limb_t *r, const limb_t *a,+                               const limb_t *m, limb_t n0, limb_t *t)+{+	limb_t *u = t, *z = t + 16;+	limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take;+	limb_t s0, s1, s2, c0, c1, c2;++	/* column 0 */+	MONT_MAC(t0, t1, t2, a[0], a[0]);+	u[0] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[0], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 1 */+	MONT_MUL(s0, s1, s2, a[0], a[1]);+	MONT_MAC(t0, t1, t2, u[0], m[1]);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[1] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[1], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 2 */+	MONT_MUL(s0, s1, s2, a[0], a[2]);+	MONT_MAC(t0, t1, t2, a[1], a[1]);+	MONT_MUL(v0, v1, v2, u[0], m[2]);+	MONT_MUL(w0, w1, w2, u[1], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[2] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[2], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 3 */+	MONT_MUL(s0, s1, s2, a[0], a[3]);+	MONT_MUL(c0, c1, c2, a[1], a[2]);+	MONT_MAC(t0, t1, t2, u[0], m[3]);+	MONT_MUL(v0, v1, v2, u[1], m[2]);+	MONT_MUL(w0, w1, w2, u[2], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[3] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[3], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 4 */+	MONT_MUL(s0, s1, s2, a[0], a[4]);+	MONT_MUL(c0, c1, c2, a[1], a[3]);+	MONT_MAC(t0, t1, t2, a[2], a[2]);+	MONT_MUL(v0, v1, v2, u[0], m[4]);+	MONT_MUL(w0, w1, w2, u[1], m[3]);+	MONT_MAC(t0, t1, t2, u[2], m[2]);+	MONT_MAC(v0, v1, v2, u[3], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[4] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[4], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 5 */+	MONT_MUL(s0, s1, s2, a[0], a[5]);+	MONT_MUL(c0, c1, c2, a[1], a[4]);+	MONT_MAC(s0, s1, s2, a[2], a[3]);+	MONT_MAC(t0, t1, t2, u[0], m[5]);+	MONT_MUL(v0, v1, v2, u[1], m[4]);+	MONT_MUL(w0, w1, w2, u[2], m[3]);+	MONT_MAC(t0, t1, t2, u[3], m[2]);+	MONT_MAC(v0, v1, v2, u[4], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[5] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[5], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 6 */+	MONT_MUL(s0, s1, s2, a[0], a[6]);+	MONT_MUL(c0, c1, c2, a[1], a[5]);+	MONT_MAC(s0, s1, s2, a[2], a[4]);+	MONT_MAC(t0, t1, t2, a[3], a[3]);+	MONT_MUL(v0, v1, v2, u[0], m[6]);+	MONT_MUL(w0, w1, w2, u[1], m[5]);+	MONT_MAC(t0, t1, t2, u[2], m[4]);+	MONT_MAC(v0, v1, v2, u[3], m[3]);+	MONT_MAC(w0, w1, w2, u[4], m[2]);+	MONT_MAC(t0, t1, t2, u[5], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[6] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[6], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 7 */+	MONT_MUL(s0, s1, s2, a[0], a[7]);+	MONT_MUL(c0, c1, c2, a[1], a[6]);+	MONT_MAC(s0, s1, s2, a[2], a[5]);+	MONT_MAC(c0, c1, c2, a[3], a[4]);+	MONT_MAC(t0, t1, t2, u[0], m[7]);+	MONT_MUL(v0, v1, v2, u[1], m[6]);+	MONT_MUL(w0, w1, w2, u[2], m[5]);+	MONT_MAC(t0, t1, t2, u[3], m[4]);+	MONT_MAC(v0, v1, v2, u[4], m[3]);+	MONT_MAC(w0, w1, w2, u[5], m[2]);+	MONT_MAC(t0, t1, t2, u[6], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[7] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[7], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 8 */+	MONT_MUL(s0, s1, s2, a[0], a[8]);+	MONT_MUL(c0, c1, c2, a[1], a[7]);+	MONT_MAC(s0, s1, s2, a[2], a[6]);+	MONT_MAC(c0, c1, c2, a[3], a[5]);+	MONT_MAC(t0, t1, t2, a[4], a[4]);+	MONT_MUL(v0, v1, v2, u[0], m[8]);+	MONT_MUL(w0, w1, w2, u[1], m[7]);+	MONT_MAC(t0, t1, t2, u[2], m[6]);+	MONT_MAC(v0, v1, v2, u[3], m[5]);+	MONT_MAC(w0, w1, w2, u[4], m[4]);+	MONT_MAC(t0, t1, t2, u[5], m[3]);+	MONT_MAC(v0, v1, v2, u[6], m[2]);+	MONT_MAC(w0, w1, w2, u[7], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[8] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[8], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 9 */+	MONT_MUL(s0, s1, s2, a[0], a[9]);+	MONT_MUL(c0, c1, c2, a[1], a[8]);+	MONT_MAC(s0, s1, s2, a[2], a[7]);+	MONT_MAC(c0, c1, c2, a[3], a[6]);+	MONT_MAC(s0, s1, s2, a[4], a[5]);+	MONT_MAC(t0, t1, t2, u[0], m[9]);+	MONT_MUL(v0, v1, v2, u[1], m[8]);+	MONT_MUL(w0, w1, w2, u[2], m[7]);+	MONT_MAC(t0, t1, t2, u[3], m[6]);+	MONT_MAC(v0, v1, v2, u[4], m[5]);+	MONT_MAC(w0, w1, w2, u[5], m[4]);+	MONT_MAC(t0, t1, t2, u[6], m[3]);+	MONT_MAC(v0, v1, v2, u[7], m[2]);+	MONT_MAC(w0, w1, w2, u[8], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[9] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[9], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 10 */+	MONT_MUL(s0, s1, s2, a[0], a[10]);+	MONT_MUL(c0, c1, c2, a[1], a[9]);+	MONT_MAC(s0, s1, s2, a[2], a[8]);+	MONT_MAC(c0, c1, c2, a[3], a[7]);+	MONT_MAC(s0, s1, s2, a[4], a[6]);+	MONT_MAC(t0, t1, t2, a[5], a[5]);+	MONT_MUL(v0, v1, v2, u[0], m[10]);+	MONT_MUL(w0, w1, w2, u[1], m[9]);+	MONT_MAC(t0, t1, t2, u[2], m[8]);+	MONT_MAC(v0, v1, v2, u[3], m[7]);+	MONT_MAC(w0, w1, w2, u[4], m[6]);+	MONT_MAC(t0, t1, t2, u[5], m[5]);+	MONT_MAC(v0, v1, v2, u[6], m[4]);+	MONT_MAC(w0, w1, w2, u[7], m[3]);+	MONT_MAC(t0, t1, t2, u[8], m[2]);+	MONT_MAC(v0, v1, v2, u[9], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[10] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[10], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 11 */+	MONT_MUL(s0, s1, s2, a[0], a[11]);+	MONT_MUL(c0, c1, c2, a[1], a[10]);+	MONT_MAC(s0, s1, s2, a[2], a[9]);+	MONT_MAC(c0, c1, c2, a[3], a[8]);+	MONT_MAC(s0, s1, s2, a[4], a[7]);+	MONT_MAC(c0, c1, c2, a[5], a[6]);+	MONT_MAC(t0, t1, t2, u[0], m[11]);+	MONT_MUL(v0, v1, v2, u[1], m[10]);+	MONT_MUL(w0, w1, w2, u[2], m[9]);+	MONT_MAC(t0, t1, t2, u[3], m[8]);+	MONT_MAC(v0, v1, v2, u[4], m[7]);+	MONT_MAC(w0, w1, w2, u[5], m[6]);+	MONT_MAC(t0, t1, t2, u[6], m[5]);+	MONT_MAC(v0, v1, v2, u[7], m[4]);+	MONT_MAC(w0, w1, w2, u[8], m[3]);+	MONT_MAC(t0, t1, t2, u[9], m[2]);+	MONT_MAC(v0, v1, v2, u[10], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[11] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[11], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 12 */+	MONT_MUL(s0, s1, s2, a[0], a[12]);+	MONT_MUL(c0, c1, c2, a[1], a[11]);+	MONT_MAC(s0, s1, s2, a[2], a[10]);+	MONT_MAC(c0, c1, c2, a[3], a[9]);+	MONT_MAC(s0, s1, s2, a[4], a[8]);+	MONT_MAC(c0, c1, c2, a[5], a[7]);+	MONT_MAC(t0, t1, t2, a[6], a[6]);+	MONT_MUL(v0, v1, v2, u[0], m[12]);+	MONT_MUL(w0, w1, w2, u[1], m[11]);+	MONT_MAC(t0, t1, t2, u[2], m[10]);+	MONT_MAC(v0, v1, v2, u[3], m[9]);+	MONT_MAC(w0, w1, w2, u[4], m[8]);+	MONT_MAC(t0, t1, t2, u[5], m[7]);+	MONT_MAC(v0, v1, v2, u[6], m[6]);+	MONT_MAC(w0, w1, w2, u[7], m[5]);+	MONT_MAC(t0, t1, t2, u[8], m[4]);+	MONT_MAC(v0, v1, v2, u[9], m[3]);+	MONT_MAC(w0, w1, w2, u[10], m[2]);+	MONT_MAC(t0, t1, t2, u[11], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[12] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[12], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 13 */+	MONT_MUL(s0, s1, s2, a[0], a[13]);+	MONT_MUL(c0, c1, c2, a[1], a[12]);+	MONT_MAC(s0, s1, s2, a[2], a[11]);+	MONT_MAC(c0, c1, c2, a[3], a[10]);+	MONT_MAC(s0, s1, s2, a[4], a[9]);+	MONT_MAC(c0, c1, c2, a[5], a[8]);+	MONT_MAC(s0, s1, s2, a[6], a[7]);+	MONT_MAC(t0, t1, t2, u[0], m[13]);+	MONT_MUL(v0, v1, v2, u[1], m[12]);+	MONT_MUL(w0, w1, w2, u[2], m[11]);+	MONT_MAC(t0, t1, t2, u[3], m[10]);+	MONT_MAC(v0, v1, v2, u[4], m[9]);+	MONT_MAC(w0, w1, w2, u[5], m[8]);+	MONT_MAC(t0, t1, t2, u[6], m[7]);+	MONT_MAC(v0, v1, v2, u[7], m[6]);+	MONT_MAC(w0, w1, w2, u[8], m[5]);+	MONT_MAC(t0, t1, t2, u[9], m[4]);+	MONT_MAC(v0, v1, v2, u[10], m[3]);+	MONT_MAC(w0, w1, w2, u[11], m[2]);+	MONT_MAC(t0, t1, t2, u[12], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[13] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[13], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 14 */+	MONT_MUL(s0, s1, s2, a[0], a[14]);+	MONT_MUL(c0, c1, c2, a[1], a[13]);+	MONT_MAC(s0, s1, s2, a[2], a[12]);+	MONT_MAC(c0, c1, c2, a[3], a[11]);+	MONT_MAC(s0, s1, s2, a[4], a[10]);+	MONT_MAC(c0, c1, c2, a[5], a[9]);+	MONT_MAC(s0, s1, s2, a[6], a[8]);+	MONT_MAC(t0, t1, t2, a[7], a[7]);+	MONT_MUL(v0, v1, v2, u[0], m[14]);+	MONT_MUL(w0, w1, w2, u[1], m[13]);+	MONT_MAC(t0, t1, t2, u[2], m[12]);+	MONT_MAC(v0, v1, v2, u[3], m[11]);+	MONT_MAC(w0, w1, w2, u[4], m[10]);+	MONT_MAC(t0, t1, t2, u[5], m[9]);+	MONT_MAC(v0, v1, v2, u[6], m[8]);+	MONT_MAC(w0, w1, w2, u[7], m[7]);+	MONT_MAC(t0, t1, t2, u[8], m[6]);+	MONT_MAC(v0, v1, v2, u[9], m[5]);+	MONT_MAC(w0, w1, w2, u[10], m[4]);+	MONT_MAC(t0, t1, t2, u[11], m[3]);+	MONT_MAC(v0, v1, v2, u[12], m[2]);+	MONT_MAC(w0, w1, w2, u[13], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[14] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[14], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 15 */+	MONT_MUL(s0, s1, s2, a[0], a[15]);+	MONT_MUL(c0, c1, c2, a[1], a[14]);+	MONT_MAC(s0, s1, s2, a[2], a[13]);+	MONT_MAC(c0, c1, c2, a[3], a[12]);+	MONT_MAC(s0, s1, s2, a[4], a[11]);+	MONT_MAC(c0, c1, c2, a[5], a[10]);+	MONT_MAC(s0, s1, s2, a[6], a[9]);+	MONT_MAC(c0, c1, c2, a[7], a[8]);+	MONT_MAC(t0, t1, t2, u[0], m[15]);+	MONT_MUL(v0, v1, v2, u[1], m[14]);+	MONT_MUL(w0, w1, w2, u[2], m[13]);+	MONT_MAC(t0, t1, t2, u[3], m[12]);+	MONT_MAC(v0, v1, v2, u[4], m[11]);+	MONT_MAC(w0, w1, w2, u[5], m[10]);+	MONT_MAC(t0, t1, t2, u[6], m[9]);+	MONT_MAC(v0, v1, v2, u[7], m[8]);+	MONT_MAC(w0, w1, w2, u[8], m[7]);+	MONT_MAC(t0, t1, t2, u[9], m[6]);+	MONT_MAC(v0, v1, v2, u[10], m[5]);+	MONT_MAC(w0, w1, w2, u[11], m[4]);+	MONT_MAC(t0, t1, t2, u[12], m[3]);+	MONT_MAC(v0, v1, v2, u[13], m[2]);+	MONT_MAC(w0, w1, w2, u[14], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[15] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[15], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 16 */+	MONT_MUL(s0, s1, s2, a[1], a[15]);+	MONT_MUL(c0, c1, c2, a[2], a[14]);+	MONT_MAC(s0, s1, s2, a[3], a[13]);+	MONT_MAC(c0, c1, c2, a[4], a[12]);+	MONT_MAC(s0, s1, s2, a[5], a[11]);+	MONT_MAC(c0, c1, c2, a[6], a[10]);+	MONT_MAC(s0, s1, s2, a[7], a[9]);+	MONT_MAC(t0, t1, t2, a[8], a[8]);+	MONT_MUL(v0, v1, v2, u[1], m[15]);+	MONT_MUL(w0, w1, w2, u[2], m[14]);+	MONT_MAC(t0, t1, t2, u[3], m[13]);+	MONT_MAC(v0, v1, v2, u[4], m[12]);+	MONT_MAC(w0, w1, w2, u[5], m[11]);+	MONT_MAC(t0, t1, t2, u[6], m[10]);+	MONT_MAC(v0, v1, v2, u[7], m[9]);+	MONT_MAC(w0, w1, w2, u[8], m[8]);+	MONT_MAC(t0, t1, t2, u[9], m[7]);+	MONT_MAC(v0, v1, v2, u[10], m[6]);+	MONT_MAC(w0, w1, w2, u[11], m[5]);+	MONT_MAC(t0, t1, t2, u[12], m[4]);+	MONT_MAC(v0, v1, v2, u[13], m[3]);+	MONT_MAC(w0, w1, w2, u[14], m[2]);+	MONT_MAC(t0, t1, t2, u[15], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[0] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 17 */+	MONT_MUL(s0, s1, s2, a[2], a[15]);+	MONT_MUL(c0, c1, c2, a[3], a[14]);+	MONT_MAC(s0, s1, s2, a[4], a[13]);+	MONT_MAC(c0, c1, c2, a[5], a[12]);+	MONT_MAC(s0, s1, s2, a[6], a[11]);+	MONT_MAC(c0, c1, c2, a[7], a[10]);+	MONT_MAC(s0, s1, s2, a[8], a[9]);+	MONT_MAC(t0, t1, t2, u[2], m[15]);+	MONT_MUL(v0, v1, v2, u[3], m[14]);+	MONT_MUL(w0, w1, w2, u[4], m[13]);+	MONT_MAC(t0, t1, t2, u[5], m[12]);+	MONT_MAC(v0, v1, v2, u[6], m[11]);+	MONT_MAC(w0, w1, w2, u[7], m[10]);+	MONT_MAC(t0, t1, t2, u[8], m[9]);+	MONT_MAC(v0, v1, v2, u[9], m[8]);+	MONT_MAC(w0, w1, w2, u[10], m[7]);+	MONT_MAC(t0, t1, t2, u[11], m[6]);+	MONT_MAC(v0, v1, v2, u[12], m[5]);+	MONT_MAC(w0, w1, w2, u[13], m[4]);+	MONT_MAC(t0, t1, t2, u[14], m[3]);+	MONT_MAC(v0, v1, v2, u[15], m[2]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[1] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 18 */+	MONT_MUL(s0, s1, s2, a[3], a[15]);+	MONT_MUL(c0, c1, c2, a[4], a[14]);+	MONT_MAC(s0, s1, s2, a[5], a[13]);+	MONT_MAC(c0, c1, c2, a[6], a[12]);+	MONT_MAC(s0, s1, s2, a[7], a[11]);+	MONT_MAC(c0, c1, c2, a[8], a[10]);+	MONT_MAC(t0, t1, t2, a[9], a[9]);+	MONT_MUL(v0, v1, v2, u[3], m[15]);+	MONT_MUL(w0, w1, w2, u[4], m[14]);+	MONT_MAC(t0, t1, t2, u[5], m[13]);+	MONT_MAC(v0, v1, v2, u[6], m[12]);+	MONT_MAC(w0, w1, w2, u[7], m[11]);+	MONT_MAC(t0, t1, t2, u[8], m[10]);+	MONT_MAC(v0, v1, v2, u[9], m[9]);+	MONT_MAC(w0, w1, w2, u[10], m[8]);+	MONT_MAC(t0, t1, t2, u[11], m[7]);+	MONT_MAC(v0, v1, v2, u[12], m[6]);+	MONT_MAC(w0, w1, w2, u[13], m[5]);+	MONT_MAC(t0, t1, t2, u[14], m[4]);+	MONT_MAC(v0, v1, v2, u[15], m[3]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[2] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 19 */+	MONT_MUL(s0, s1, s2, a[4], a[15]);+	MONT_MUL(c0, c1, c2, a[5], a[14]);+	MONT_MAC(s0, s1, s2, a[6], a[13]);+	MONT_MAC(c0, c1, c2, a[7], a[12]);+	MONT_MAC(s0, s1, s2, a[8], a[11]);+	MONT_MAC(c0, c1, c2, a[9], a[10]);+	MONT_MAC(t0, t1, t2, u[4], m[15]);+	MONT_MUL(v0, v1, v2, u[5], m[14]);+	MONT_MUL(w0, w1, w2, u[6], m[13]);+	MONT_MAC(t0, t1, t2, u[7], m[12]);+	MONT_MAC(v0, v1, v2, u[8], m[11]);+	MONT_MAC(w0, w1, w2, u[9], m[10]);+	MONT_MAC(t0, t1, t2, u[10], m[9]);+	MONT_MAC(v0, v1, v2, u[11], m[8]);+	MONT_MAC(w0, w1, w2, u[12], m[7]);+	MONT_MAC(t0, t1, t2, u[13], m[6]);+	MONT_MAC(v0, v1, v2, u[14], m[5]);+	MONT_MAC(w0, w1, w2, u[15], m[4]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[3] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 20 */+	MONT_MUL(s0, s1, s2, a[5], a[15]);+	MONT_MUL(c0, c1, c2, a[6], a[14]);+	MONT_MAC(s0, s1, s2, a[7], a[13]);+	MONT_MAC(c0, c1, c2, a[8], a[12]);+	MONT_MAC(s0, s1, s2, a[9], a[11]);+	MONT_MAC(t0, t1, t2, a[10], a[10]);+	MONT_MUL(v0, v1, v2, u[5], m[15]);+	MONT_MUL(w0, w1, w2, u[6], m[14]);+	MONT_MAC(t0, t1, t2, u[7], m[13]);+	MONT_MAC(v0, v1, v2, u[8], m[12]);+	MONT_MAC(w0, w1, w2, u[9], m[11]);+	MONT_MAC(t0, t1, t2, u[10], m[10]);+	MONT_MAC(v0, v1, v2, u[11], m[9]);+	MONT_MAC(w0, w1, w2, u[12], m[8]);+	MONT_MAC(t0, t1, t2, u[13], m[7]);+	MONT_MAC(v0, v1, v2, u[14], m[6]);+	MONT_MAC(w0, w1, w2, u[15], m[5]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[4] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 21 */+	MONT_MUL(s0, s1, s2, a[6], a[15]);+	MONT_MUL(c0, c1, c2, a[7], a[14]);+	MONT_MAC(s0, s1, s2, a[8], a[13]);+	MONT_MAC(c0, c1, c2, a[9], a[12]);+	MONT_MAC(s0, s1, s2, a[10], a[11]);+	MONT_MAC(t0, t1, t2, u[6], m[15]);+	MONT_MUL(v0, v1, v2, u[7], m[14]);+	MONT_MUL(w0, w1, w2, u[8], m[13]);+	MONT_MAC(t0, t1, t2, u[9], m[12]);+	MONT_MAC(v0, v1, v2, u[10], m[11]);+	MONT_MAC(w0, w1, w2, u[11], m[10]);+	MONT_MAC(t0, t1, t2, u[12], m[9]);+	MONT_MAC(v0, v1, v2, u[13], m[8]);+	MONT_MAC(w0, w1, w2, u[14], m[7]);+	MONT_MAC(t0, t1, t2, u[15], m[6]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[5] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 22 */+	MONT_MUL(s0, s1, s2, a[7], a[15]);+	MONT_MUL(c0, c1, c2, a[8], a[14]);+	MONT_MAC(s0, s1, s2, a[9], a[13]);+	MONT_MAC(c0, c1, c2, a[10], a[12]);+	MONT_MAC(t0, t1, t2, a[11], a[11]);+	MONT_MUL(v0, v1, v2, u[7], m[15]);+	MONT_MUL(w0, w1, w2, u[8], m[14]);+	MONT_MAC(t0, t1, t2, u[9], m[13]);+	MONT_MAC(v0, v1, v2, u[10], m[12]);+	MONT_MAC(w0, w1, w2, u[11], m[11]);+	MONT_MAC(t0, t1, t2, u[12], m[10]);+	MONT_MAC(v0, v1, v2, u[13], m[9]);+	MONT_MAC(w0, w1, w2, u[14], m[8]);+	MONT_MAC(t0, t1, t2, u[15], m[7]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[6] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 23 */+	MONT_MUL(s0, s1, s2, a[8], a[15]);+	MONT_MUL(c0, c1, c2, a[9], a[14]);+	MONT_MAC(s0, s1, s2, a[10], a[13]);+	MONT_MAC(c0, c1, c2, a[11], a[12]);+	MONT_MAC(t0, t1, t2, u[8], m[15]);+	MONT_MUL(v0, v1, v2, u[9], m[14]);+	MONT_MUL(w0, w1, w2, u[10], m[13]);+	MONT_MAC(t0, t1, t2, u[11], m[12]);+	MONT_MAC(v0, v1, v2, u[12], m[11]);+	MONT_MAC(w0, w1, w2, u[13], m[10]);+	MONT_MAC(t0, t1, t2, u[14], m[9]);+	MONT_MAC(v0, v1, v2, u[15], m[8]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[7] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 24 */+	MONT_MUL(s0, s1, s2, a[9], a[15]);+	MONT_MUL(c0, c1, c2, a[10], a[14]);+	MONT_MAC(s0, s1, s2, a[11], a[13]);+	MONT_MAC(t0, t1, t2, a[12], a[12]);+	MONT_MUL(v0, v1, v2, u[9], m[15]);+	MONT_MUL(w0, w1, w2, u[10], m[14]);+	MONT_MAC(t0, t1, t2, u[11], m[13]);+	MONT_MAC(v0, v1, v2, u[12], m[12]);+	MONT_MAC(w0, w1, w2, u[13], m[11]);+	MONT_MAC(t0, t1, t2, u[14], m[10]);+	MONT_MAC(v0, v1, v2, u[15], m[9]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[8] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 25 */+	MONT_MUL(s0, s1, s2, a[10], a[15]);+	MONT_MUL(c0, c1, c2, a[11], a[14]);+	MONT_MAC(s0, s1, s2, a[12], a[13]);+	MONT_MAC(t0, t1, t2, u[10], m[15]);+	MONT_MUL(v0, v1, v2, u[11], m[14]);+	MONT_MUL(w0, w1, w2, u[12], m[13]);+	MONT_MAC(t0, t1, t2, u[13], m[12]);+	MONT_MAC(v0, v1, v2, u[14], m[11]);+	MONT_MAC(w0, w1, w2, u[15], m[10]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[9] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 26 */+	MONT_MUL(s0, s1, s2, a[11], a[15]);+	MONT_MUL(c0, c1, c2, a[12], a[14]);+	MONT_MAC(t0, t1, t2, a[13], a[13]);+	MONT_MUL(v0, v1, v2, u[11], m[15]);+	MONT_MUL(w0, w1, w2, u[12], m[14]);+	MONT_MAC(t0, t1, t2, u[13], m[13]);+	MONT_MAC(v0, v1, v2, u[14], m[12]);+	MONT_MAC(w0, w1, w2, u[15], m[11]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[10] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 27 */+	MONT_MUL(s0, s1, s2, a[12], a[15]);+	MONT_MUL(c0, c1, c2, a[13], a[14]);+	MONT_MAC(t0, t1, t2, u[12], m[15]);+	MONT_MUL(v0, v1, v2, u[13], m[14]);+	MONT_MUL(w0, w1, w2, u[14], m[13]);+	MONT_MAC(t0, t1, t2, u[15], m[12]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[11] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 28 */+	MONT_MUL(s0, s1, s2, a[13], a[15]);+	MONT_MAC(t0, t1, t2, a[14], a[14]);+	MONT_MUL(v0, v1, v2, u[13], m[15]);+	MONT_MUL(w0, w1, w2, u[14], m[14]);+	MONT_MAC(t0, t1, t2, u[15], m[13]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[12] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 29 */+	MONT_MUL(s0, s1, s2, a[14], a[15]);+	MONT_MAC(t0, t1, t2, u[14], m[15]);+	MONT_MUL(v0, v1, v2, u[15], m[14]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[13] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 30 */+	MONT_MAC(t0, t1, t2, a[15], a[15]);+	MONT_MUL(v0, v1, v2, u[15], m[15]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	z[14] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	z[15] = t0;++	/* under 2m, with t1 the bit above n limbs: at most one subtraction */+	borrow = sub_n(r, z, m, 16);+	take = t1 | (borrow ^ 1);+	select_n(r, r, z, take & 1, 16);+}++/* r = a * b * R^-1 mod m, for a and b under m; t is 2n limbs of scratch */+static void crypton_mont_mul24(limb_t *r, const limb_t *a, const limb_t *b,+                               const limb_t *m, limb_t n0, limb_t *t)+{+	limb_t *u = t, *z = t + 24;+	limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take;++	/* column 0 */+	MONT_MAC(t0, t1, t2, a[0], b[0]);+	u[0] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[0], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 1 */+	MONT_MAC(t0, t1, t2, a[0], b[1]);+	MONT_MUL(v0, v1, v2, a[1], b[0]);+	MONT_MUL(w0, w1, w2, u[0], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[1] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[1], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 2 */+	MONT_MAC(t0, t1, t2, a[0], b[2]);+	MONT_MUL(v0, v1, v2, a[1], b[1]);+	MONT_MUL(w0, w1, w2, a[2], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[2]);+	MONT_MAC(v0, v1, v2, u[1], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[2] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[2], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 3 */+	MONT_MAC(t0, t1, t2, a[0], b[3]);+	MONT_MUL(v0, v1, v2, a[1], b[2]);+	MONT_MUL(w0, w1, w2, a[2], b[1]);+	MONT_MAC(t0, t1, t2, a[3], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[3]);+	MONT_MAC(w0, w1, w2, u[1], m[2]);+	MONT_MAC(t0, t1, t2, u[2], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[3] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[3], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 4 */+	MONT_MAC(t0, t1, t2, a[0], b[4]);+	MONT_MUL(v0, v1, v2, a[1], b[3]);+	MONT_MUL(w0, w1, w2, a[2], b[2]);+	MONT_MAC(t0, t1, t2, a[3], b[1]);+	MONT_MAC(v0, v1, v2, a[4], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[4]);+	MONT_MAC(t0, t1, t2, u[1], m[3]);+	MONT_MAC(v0, v1, v2, u[2], m[2]);+	MONT_MAC(w0, w1, w2, u[3], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[4] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[4], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 5 */+	MONT_MAC(t0, t1, t2, a[0], b[5]);+	MONT_MUL(v0, v1, v2, a[1], b[4]);+	MONT_MUL(w0, w1, w2, a[2], b[3]);+	MONT_MAC(t0, t1, t2, a[3], b[2]);+	MONT_MAC(v0, v1, v2, a[4], b[1]);+	MONT_MAC(w0, w1, w2, a[5], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[5]);+	MONT_MAC(v0, v1, v2, u[1], m[4]);+	MONT_MAC(w0, w1, w2, u[2], m[3]);+	MONT_MAC(t0, t1, t2, u[3], m[2]);+	MONT_MAC(v0, v1, v2, u[4], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[5] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[5], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 6 */+	MONT_MAC(t0, t1, t2, a[0], b[6]);+	MONT_MUL(v0, v1, v2, a[1], b[5]);+	MONT_MUL(w0, w1, w2, a[2], b[4]);+	MONT_MAC(t0, t1, t2, a[3], b[3]);+	MONT_MAC(v0, v1, v2, a[4], b[2]);+	MONT_MAC(w0, w1, w2, a[5], b[1]);+	MONT_MAC(t0, t1, t2, a[6], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[6]);+	MONT_MAC(w0, w1, w2, u[1], m[5]);+	MONT_MAC(t0, t1, t2, u[2], m[4]);+	MONT_MAC(v0, v1, v2, u[3], m[3]);+	MONT_MAC(w0, w1, w2, u[4], m[2]);+	MONT_MAC(t0, t1, t2, u[5], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[6] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[6], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 7 */+	MONT_MAC(t0, t1, t2, a[0], b[7]);+	MONT_MUL(v0, v1, v2, a[1], b[6]);+	MONT_MUL(w0, w1, w2, a[2], b[5]);+	MONT_MAC(t0, t1, t2, a[3], b[4]);+	MONT_MAC(v0, v1, v2, a[4], b[3]);+	MONT_MAC(w0, w1, w2, a[5], b[2]);+	MONT_MAC(t0, t1, t2, a[6], b[1]);+	MONT_MAC(v0, v1, v2, a[7], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[7]);+	MONT_MAC(t0, t1, t2, u[1], m[6]);+	MONT_MAC(v0, v1, v2, u[2], m[5]);+	MONT_MAC(w0, w1, w2, u[3], m[4]);+	MONT_MAC(t0, t1, t2, u[4], m[3]);+	MONT_MAC(v0, v1, v2, u[5], m[2]);+	MONT_MAC(w0, w1, w2, u[6], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[7] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[7], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 8 */+	MONT_MAC(t0, t1, t2, a[0], b[8]);+	MONT_MUL(v0, v1, v2, a[1], b[7]);+	MONT_MUL(w0, w1, w2, a[2], b[6]);+	MONT_MAC(t0, t1, t2, a[3], b[5]);+	MONT_MAC(v0, v1, v2, a[4], b[4]);+	MONT_MAC(w0, w1, w2, a[5], b[3]);+	MONT_MAC(t0, t1, t2, a[6], b[2]);+	MONT_MAC(v0, v1, v2, a[7], b[1]);+	MONT_MAC(w0, w1, w2, a[8], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[8]);+	MONT_MAC(v0, v1, v2, u[1], m[7]);+	MONT_MAC(w0, w1, w2, u[2], m[6]);+	MONT_MAC(t0, t1, t2, u[3], m[5]);+	MONT_MAC(v0, v1, v2, u[4], m[4]);+	MONT_MAC(w0, w1, w2, u[5], m[3]);+	MONT_MAC(t0, t1, t2, u[6], m[2]);+	MONT_MAC(v0, v1, v2, u[7], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[8] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[8], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 9 */+	MONT_MAC(t0, t1, t2, a[0], b[9]);+	MONT_MUL(v0, v1, v2, a[1], b[8]);+	MONT_MUL(w0, w1, w2, a[2], b[7]);+	MONT_MAC(t0, t1, t2, a[3], b[6]);+	MONT_MAC(v0, v1, v2, a[4], b[5]);+	MONT_MAC(w0, w1, w2, a[5], b[4]);+	MONT_MAC(t0, t1, t2, a[6], b[3]);+	MONT_MAC(v0, v1, v2, a[7], b[2]);+	MONT_MAC(w0, w1, w2, a[8], b[1]);+	MONT_MAC(t0, t1, t2, a[9], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[9]);+	MONT_MAC(w0, w1, w2, u[1], m[8]);+	MONT_MAC(t0, t1, t2, u[2], m[7]);+	MONT_MAC(v0, v1, v2, u[3], m[6]);+	MONT_MAC(w0, w1, w2, u[4], m[5]);+	MONT_MAC(t0, t1, t2, u[5], m[4]);+	MONT_MAC(v0, v1, v2, u[6], m[3]);+	MONT_MAC(w0, w1, w2, u[7], m[2]);+	MONT_MAC(t0, t1, t2, u[8], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[9] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[9], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 10 */+	MONT_MAC(t0, t1, t2, a[0], b[10]);+	MONT_MUL(v0, v1, v2, a[1], b[9]);+	MONT_MUL(w0, w1, w2, a[2], b[8]);+	MONT_MAC(t0, t1, t2, a[3], b[7]);+	MONT_MAC(v0, v1, v2, a[4], b[6]);+	MONT_MAC(w0, w1, w2, a[5], b[5]);+	MONT_MAC(t0, t1, t2, a[6], b[4]);+	MONT_MAC(v0, v1, v2, a[7], b[3]);+	MONT_MAC(w0, w1, w2, a[8], b[2]);+	MONT_MAC(t0, t1, t2, a[9], b[1]);+	MONT_MAC(v0, v1, v2, a[10], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[10]);+	MONT_MAC(t0, t1, t2, u[1], m[9]);+	MONT_MAC(v0, v1, v2, u[2], m[8]);+	MONT_MAC(w0, w1, w2, u[3], m[7]);+	MONT_MAC(t0, t1, t2, u[4], m[6]);+	MONT_MAC(v0, v1, v2, u[5], m[5]);+	MONT_MAC(w0, w1, w2, u[6], m[4]);+	MONT_MAC(t0, t1, t2, u[7], m[3]);+	MONT_MAC(v0, v1, v2, u[8], m[2]);+	MONT_MAC(w0, w1, w2, u[9], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[10] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[10], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 11 */+	MONT_MAC(t0, t1, t2, a[0], b[11]);+	MONT_MUL(v0, v1, v2, a[1], b[10]);+	MONT_MUL(w0, w1, w2, a[2], b[9]);+	MONT_MAC(t0, t1, t2, a[3], b[8]);+	MONT_MAC(v0, v1, v2, a[4], b[7]);+	MONT_MAC(w0, w1, w2, a[5], b[6]);+	MONT_MAC(t0, t1, t2, a[6], b[5]);+	MONT_MAC(v0, v1, v2, a[7], b[4]);+	MONT_MAC(w0, w1, w2, a[8], b[3]);+	MONT_MAC(t0, t1, t2, a[9], b[2]);+	MONT_MAC(v0, v1, v2, a[10], b[1]);+	MONT_MAC(w0, w1, w2, a[11], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[11]);+	MONT_MAC(v0, v1, v2, u[1], m[10]);+	MONT_MAC(w0, w1, w2, u[2], m[9]);+	MONT_MAC(t0, t1, t2, u[3], m[8]);+	MONT_MAC(v0, v1, v2, u[4], m[7]);+	MONT_MAC(w0, w1, w2, u[5], m[6]);+	MONT_MAC(t0, t1, t2, u[6], m[5]);+	MONT_MAC(v0, v1, v2, u[7], m[4]);+	MONT_MAC(w0, w1, w2, u[8], m[3]);+	MONT_MAC(t0, t1, t2, u[9], m[2]);+	MONT_MAC(v0, v1, v2, u[10], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[11] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[11], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 12 */+	MONT_MAC(t0, t1, t2, a[0], b[12]);+	MONT_MUL(v0, v1, v2, a[1], b[11]);+	MONT_MUL(w0, w1, w2, a[2], b[10]);+	MONT_MAC(t0, t1, t2, a[3], b[9]);+	MONT_MAC(v0, v1, v2, a[4], b[8]);+	MONT_MAC(w0, w1, w2, a[5], b[7]);+	MONT_MAC(t0, t1, t2, a[6], b[6]);+	MONT_MAC(v0, v1, v2, a[7], b[5]);+	MONT_MAC(w0, w1, w2, a[8], b[4]);+	MONT_MAC(t0, t1, t2, a[9], b[3]);+	MONT_MAC(v0, v1, v2, a[10], b[2]);+	MONT_MAC(w0, w1, w2, a[11], b[1]);+	MONT_MAC(t0, t1, t2, a[12], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[12]);+	MONT_MAC(w0, w1, w2, u[1], m[11]);+	MONT_MAC(t0, t1, t2, u[2], m[10]);+	MONT_MAC(v0, v1, v2, u[3], m[9]);+	MONT_MAC(w0, w1, w2, u[4], m[8]);+	MONT_MAC(t0, t1, t2, u[5], m[7]);+	MONT_MAC(v0, v1, v2, u[6], m[6]);+	MONT_MAC(w0, w1, w2, u[7], m[5]);+	MONT_MAC(t0, t1, t2, u[8], m[4]);+	MONT_MAC(v0, v1, v2, u[9], m[3]);+	MONT_MAC(w0, w1, w2, u[10], m[2]);+	MONT_MAC(t0, t1, t2, u[11], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[12] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[12], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 13 */+	MONT_MAC(t0, t1, t2, a[0], b[13]);+	MONT_MUL(v0, v1, v2, a[1], b[12]);+	MONT_MUL(w0, w1, w2, a[2], b[11]);+	MONT_MAC(t0, t1, t2, a[3], b[10]);+	MONT_MAC(v0, v1, v2, a[4], b[9]);+	MONT_MAC(w0, w1, w2, a[5], b[8]);+	MONT_MAC(t0, t1, t2, a[6], b[7]);+	MONT_MAC(v0, v1, v2, a[7], b[6]);+	MONT_MAC(w0, w1, w2, a[8], b[5]);+	MONT_MAC(t0, t1, t2, a[9], b[4]);+	MONT_MAC(v0, v1, v2, a[10], b[3]);+	MONT_MAC(w0, w1, w2, a[11], b[2]);+	MONT_MAC(t0, t1, t2, a[12], b[1]);+	MONT_MAC(v0, v1, v2, a[13], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[13]);+	MONT_MAC(t0, t1, t2, u[1], m[12]);+	MONT_MAC(v0, v1, v2, u[2], m[11]);+	MONT_MAC(w0, w1, w2, u[3], m[10]);+	MONT_MAC(t0, t1, t2, u[4], m[9]);+	MONT_MAC(v0, v1, v2, u[5], m[8]);+	MONT_MAC(w0, w1, w2, u[6], m[7]);+	MONT_MAC(t0, t1, t2, u[7], m[6]);+	MONT_MAC(v0, v1, v2, u[8], m[5]);+	MONT_MAC(w0, w1, w2, u[9], m[4]);+	MONT_MAC(t0, t1, t2, u[10], m[3]);+	MONT_MAC(v0, v1, v2, u[11], m[2]);+	MONT_MAC(w0, w1, w2, u[12], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[13] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[13], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 14 */+	MONT_MAC(t0, t1, t2, a[0], b[14]);+	MONT_MUL(v0, v1, v2, a[1], b[13]);+	MONT_MUL(w0, w1, w2, a[2], b[12]);+	MONT_MAC(t0, t1, t2, a[3], b[11]);+	MONT_MAC(v0, v1, v2, a[4], b[10]);+	MONT_MAC(w0, w1, w2, a[5], b[9]);+	MONT_MAC(t0, t1, t2, a[6], b[8]);+	MONT_MAC(v0, v1, v2, a[7], b[7]);+	MONT_MAC(w0, w1, w2, a[8], b[6]);+	MONT_MAC(t0, t1, t2, a[9], b[5]);+	MONT_MAC(v0, v1, v2, a[10], b[4]);+	MONT_MAC(w0, w1, w2, a[11], b[3]);+	MONT_MAC(t0, t1, t2, a[12], b[2]);+	MONT_MAC(v0, v1, v2, a[13], b[1]);+	MONT_MAC(w0, w1, w2, a[14], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[14]);+	MONT_MAC(v0, v1, v2, u[1], m[13]);+	MONT_MAC(w0, w1, w2, u[2], m[12]);+	MONT_MAC(t0, t1, t2, u[3], m[11]);+	MONT_MAC(v0, v1, v2, u[4], m[10]);+	MONT_MAC(w0, w1, w2, u[5], m[9]);+	MONT_MAC(t0, t1, t2, u[6], m[8]);+	MONT_MAC(v0, v1, v2, u[7], m[7]);+	MONT_MAC(w0, w1, w2, u[8], m[6]);+	MONT_MAC(t0, t1, t2, u[9], m[5]);+	MONT_MAC(v0, v1, v2, u[10], m[4]);+	MONT_MAC(w0, w1, w2, u[11], m[3]);+	MONT_MAC(t0, t1, t2, u[12], m[2]);+	MONT_MAC(v0, v1, v2, u[13], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[14] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[14], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 15 */+	MONT_MAC(t0, t1, t2, a[0], b[15]);+	MONT_MUL(v0, v1, v2, a[1], b[14]);+	MONT_MUL(w0, w1, w2, a[2], b[13]);+	MONT_MAC(t0, t1, t2, a[3], b[12]);+	MONT_MAC(v0, v1, v2, a[4], b[11]);+	MONT_MAC(w0, w1, w2, a[5], b[10]);+	MONT_MAC(t0, t1, t2, a[6], b[9]);+	MONT_MAC(v0, v1, v2, a[7], b[8]);+	MONT_MAC(w0, w1, w2, a[8], b[7]);+	MONT_MAC(t0, t1, t2, a[9], b[6]);+	MONT_MAC(v0, v1, v2, a[10], b[5]);+	MONT_MAC(w0, w1, w2, a[11], b[4]);+	MONT_MAC(t0, t1, t2, a[12], b[3]);+	MONT_MAC(v0, v1, v2, a[13], b[2]);+	MONT_MAC(w0, w1, w2, a[14], b[1]);+	MONT_MAC(t0, t1, t2, a[15], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[15]);+	MONT_MAC(w0, w1, w2, u[1], m[14]);+	MONT_MAC(t0, t1, t2, u[2], m[13]);+	MONT_MAC(v0, v1, v2, u[3], m[12]);+	MONT_MAC(w0, w1, w2, u[4], m[11]);+	MONT_MAC(t0, t1, t2, u[5], m[10]);+	MONT_MAC(v0, v1, v2, u[6], m[9]);+	MONT_MAC(w0, w1, w2, u[7], m[8]);+	MONT_MAC(t0, t1, t2, u[8], m[7]);+	MONT_MAC(v0, v1, v2, u[9], m[6]);+	MONT_MAC(w0, w1, w2, u[10], m[5]);+	MONT_MAC(t0, t1, t2, u[11], m[4]);+	MONT_MAC(v0, v1, v2, u[12], m[3]);+	MONT_MAC(w0, w1, w2, u[13], m[2]);+	MONT_MAC(t0, t1, t2, u[14], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[15] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[15], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 16 */+	MONT_MAC(t0, t1, t2, a[0], b[16]);+	MONT_MUL(v0, v1, v2, a[1], b[15]);+	MONT_MUL(w0, w1, w2, a[2], b[14]);+	MONT_MAC(t0, t1, t2, a[3], b[13]);+	MONT_MAC(v0, v1, v2, a[4], b[12]);+	MONT_MAC(w0, w1, w2, a[5], b[11]);+	MONT_MAC(t0, t1, t2, a[6], b[10]);+	MONT_MAC(v0, v1, v2, a[7], b[9]);+	MONT_MAC(w0, w1, w2, a[8], b[8]);+	MONT_MAC(t0, t1, t2, a[9], b[7]);+	MONT_MAC(v0, v1, v2, a[10], b[6]);+	MONT_MAC(w0, w1, w2, a[11], b[5]);+	MONT_MAC(t0, t1, t2, a[12], b[4]);+	MONT_MAC(v0, v1, v2, a[13], b[3]);+	MONT_MAC(w0, w1, w2, a[14], b[2]);+	MONT_MAC(t0, t1, t2, a[15], b[1]);+	MONT_MAC(v0, v1, v2, a[16], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[16]);+	MONT_MAC(t0, t1, t2, u[1], m[15]);+	MONT_MAC(v0, v1, v2, u[2], m[14]);+	MONT_MAC(w0, w1, w2, u[3], m[13]);+	MONT_MAC(t0, t1, t2, u[4], m[12]);+	MONT_MAC(v0, v1, v2, u[5], m[11]);+	MONT_MAC(w0, w1, w2, u[6], m[10]);+	MONT_MAC(t0, t1, t2, u[7], m[9]);+	MONT_MAC(v0, v1, v2, u[8], m[8]);+	MONT_MAC(w0, w1, w2, u[9], m[7]);+	MONT_MAC(t0, t1, t2, u[10], m[6]);+	MONT_MAC(v0, v1, v2, u[11], m[5]);+	MONT_MAC(w0, w1, w2, u[12], m[4]);+	MONT_MAC(t0, t1, t2, u[13], m[3]);+	MONT_MAC(v0, v1, v2, u[14], m[2]);+	MONT_MAC(w0, w1, w2, u[15], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[16] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[16], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 17 */+	MONT_MAC(t0, t1, t2, a[0], b[17]);+	MONT_MUL(v0, v1, v2, a[1], b[16]);+	MONT_MUL(w0, w1, w2, a[2], b[15]);+	MONT_MAC(t0, t1, t2, a[3], b[14]);+	MONT_MAC(v0, v1, v2, a[4], b[13]);+	MONT_MAC(w0, w1, w2, a[5], b[12]);+	MONT_MAC(t0, t1, t2, a[6], b[11]);+	MONT_MAC(v0, v1, v2, a[7], b[10]);+	MONT_MAC(w0, w1, w2, a[8], b[9]);+	MONT_MAC(t0, t1, t2, a[9], b[8]);+	MONT_MAC(v0, v1, v2, a[10], b[7]);+	MONT_MAC(w0, w1, w2, a[11], b[6]);+	MONT_MAC(t0, t1, t2, a[12], b[5]);+	MONT_MAC(v0, v1, v2, a[13], b[4]);+	MONT_MAC(w0, w1, w2, a[14], b[3]);+	MONT_MAC(t0, t1, t2, a[15], b[2]);+	MONT_MAC(v0, v1, v2, a[16], b[1]);+	MONT_MAC(w0, w1, w2, a[17], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[17]);+	MONT_MAC(v0, v1, v2, u[1], m[16]);+	MONT_MAC(w0, w1, w2, u[2], m[15]);+	MONT_MAC(t0, t1, t2, u[3], m[14]);+	MONT_MAC(v0, v1, v2, u[4], m[13]);+	MONT_MAC(w0, w1, w2, u[5], m[12]);+	MONT_MAC(t0, t1, t2, u[6], m[11]);+	MONT_MAC(v0, v1, v2, u[7], m[10]);+	MONT_MAC(w0, w1, w2, u[8], m[9]);+	MONT_MAC(t0, t1, t2, u[9], m[8]);+	MONT_MAC(v0, v1, v2, u[10], m[7]);+	MONT_MAC(w0, w1, w2, u[11], m[6]);+	MONT_MAC(t0, t1, t2, u[12], m[5]);+	MONT_MAC(v0, v1, v2, u[13], m[4]);+	MONT_MAC(w0, w1, w2, u[14], m[3]);+	MONT_MAC(t0, t1, t2, u[15], m[2]);+	MONT_MAC(v0, v1, v2, u[16], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[17] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[17], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 18 */+	MONT_MAC(t0, t1, t2, a[0], b[18]);+	MONT_MUL(v0, v1, v2, a[1], b[17]);+	MONT_MUL(w0, w1, w2, a[2], b[16]);+	MONT_MAC(t0, t1, t2, a[3], b[15]);+	MONT_MAC(v0, v1, v2, a[4], b[14]);+	MONT_MAC(w0, w1, w2, a[5], b[13]);+	MONT_MAC(t0, t1, t2, a[6], b[12]);+	MONT_MAC(v0, v1, v2, a[7], b[11]);+	MONT_MAC(w0, w1, w2, a[8], b[10]);+	MONT_MAC(t0, t1, t2, a[9], b[9]);+	MONT_MAC(v0, v1, v2, a[10], b[8]);+	MONT_MAC(w0, w1, w2, a[11], b[7]);+	MONT_MAC(t0, t1, t2, a[12], b[6]);+	MONT_MAC(v0, v1, v2, a[13], b[5]);+	MONT_MAC(w0, w1, w2, a[14], b[4]);+	MONT_MAC(t0, t1, t2, a[15], b[3]);+	MONT_MAC(v0, v1, v2, a[16], b[2]);+	MONT_MAC(w0, w1, w2, a[17], b[1]);+	MONT_MAC(t0, t1, t2, a[18], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[18]);+	MONT_MAC(w0, w1, w2, u[1], m[17]);+	MONT_MAC(t0, t1, t2, u[2], m[16]);+	MONT_MAC(v0, v1, v2, u[3], m[15]);+	MONT_MAC(w0, w1, w2, u[4], m[14]);+	MONT_MAC(t0, t1, t2, u[5], m[13]);+	MONT_MAC(v0, v1, v2, u[6], m[12]);+	MONT_MAC(w0, w1, w2, u[7], m[11]);+	MONT_MAC(t0, t1, t2, u[8], m[10]);+	MONT_MAC(v0, v1, v2, u[9], m[9]);+	MONT_MAC(w0, w1, w2, u[10], m[8]);+	MONT_MAC(t0, t1, t2, u[11], m[7]);+	MONT_MAC(v0, v1, v2, u[12], m[6]);+	MONT_MAC(w0, w1, w2, u[13], m[5]);+	MONT_MAC(t0, t1, t2, u[14], m[4]);+	MONT_MAC(v0, v1, v2, u[15], m[3]);+	MONT_MAC(w0, w1, w2, u[16], m[2]);+	MONT_MAC(t0, t1, t2, u[17], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[18] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[18], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 19 */+	MONT_MAC(t0, t1, t2, a[0], b[19]);+	MONT_MUL(v0, v1, v2, a[1], b[18]);+	MONT_MUL(w0, w1, w2, a[2], b[17]);+	MONT_MAC(t0, t1, t2, a[3], b[16]);+	MONT_MAC(v0, v1, v2, a[4], b[15]);+	MONT_MAC(w0, w1, w2, a[5], b[14]);+	MONT_MAC(t0, t1, t2, a[6], b[13]);+	MONT_MAC(v0, v1, v2, a[7], b[12]);+	MONT_MAC(w0, w1, w2, a[8], b[11]);+	MONT_MAC(t0, t1, t2, a[9], b[10]);+	MONT_MAC(v0, v1, v2, a[10], b[9]);+	MONT_MAC(w0, w1, w2, a[11], b[8]);+	MONT_MAC(t0, t1, t2, a[12], b[7]);+	MONT_MAC(v0, v1, v2, a[13], b[6]);+	MONT_MAC(w0, w1, w2, a[14], b[5]);+	MONT_MAC(t0, t1, t2, a[15], b[4]);+	MONT_MAC(v0, v1, v2, a[16], b[3]);+	MONT_MAC(w0, w1, w2, a[17], b[2]);+	MONT_MAC(t0, t1, t2, a[18], b[1]);+	MONT_MAC(v0, v1, v2, a[19], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[19]);+	MONT_MAC(t0, t1, t2, u[1], m[18]);+	MONT_MAC(v0, v1, v2, u[2], m[17]);+	MONT_MAC(w0, w1, w2, u[3], m[16]);+	MONT_MAC(t0, t1, t2, u[4], m[15]);+	MONT_MAC(v0, v1, v2, u[5], m[14]);+	MONT_MAC(w0, w1, w2, u[6], m[13]);+	MONT_MAC(t0, t1, t2, u[7], m[12]);+	MONT_MAC(v0, v1, v2, u[8], m[11]);+	MONT_MAC(w0, w1, w2, u[9], m[10]);+	MONT_MAC(t0, t1, t2, u[10], m[9]);+	MONT_MAC(v0, v1, v2, u[11], m[8]);+	MONT_MAC(w0, w1, w2, u[12], m[7]);+	MONT_MAC(t0, t1, t2, u[13], m[6]);+	MONT_MAC(v0, v1, v2, u[14], m[5]);+	MONT_MAC(w0, w1, w2, u[15], m[4]);+	MONT_MAC(t0, t1, t2, u[16], m[3]);+	MONT_MAC(v0, v1, v2, u[17], m[2]);+	MONT_MAC(w0, w1, w2, u[18], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[19] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[19], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 20 */+	MONT_MAC(t0, t1, t2, a[0], b[20]);+	MONT_MUL(v0, v1, v2, a[1], b[19]);+	MONT_MUL(w0, w1, w2, a[2], b[18]);+	MONT_MAC(t0, t1, t2, a[3], b[17]);+	MONT_MAC(v0, v1, v2, a[4], b[16]);+	MONT_MAC(w0, w1, w2, a[5], b[15]);+	MONT_MAC(t0, t1, t2, a[6], b[14]);+	MONT_MAC(v0, v1, v2, a[7], b[13]);+	MONT_MAC(w0, w1, w2, a[8], b[12]);+	MONT_MAC(t0, t1, t2, a[9], b[11]);+	MONT_MAC(v0, v1, v2, a[10], b[10]);+	MONT_MAC(w0, w1, w2, a[11], b[9]);+	MONT_MAC(t0, t1, t2, a[12], b[8]);+	MONT_MAC(v0, v1, v2, a[13], b[7]);+	MONT_MAC(w0, w1, w2, a[14], b[6]);+	MONT_MAC(t0, t1, t2, a[15], b[5]);+	MONT_MAC(v0, v1, v2, a[16], b[4]);+	MONT_MAC(w0, w1, w2, a[17], b[3]);+	MONT_MAC(t0, t1, t2, a[18], b[2]);+	MONT_MAC(v0, v1, v2, a[19], b[1]);+	MONT_MAC(w0, w1, w2, a[20], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[20]);+	MONT_MAC(v0, v1, v2, u[1], m[19]);+	MONT_MAC(w0, w1, w2, u[2], m[18]);+	MONT_MAC(t0, t1, t2, u[3], m[17]);+	MONT_MAC(v0, v1, v2, u[4], m[16]);+	MONT_MAC(w0, w1, w2, u[5], m[15]);+	MONT_MAC(t0, t1, t2, u[6], m[14]);+	MONT_MAC(v0, v1, v2, u[7], m[13]);+	MONT_MAC(w0, w1, w2, u[8], m[12]);+	MONT_MAC(t0, t1, t2, u[9], m[11]);+	MONT_MAC(v0, v1, v2, u[10], m[10]);+	MONT_MAC(w0, w1, w2, u[11], m[9]);+	MONT_MAC(t0, t1, t2, u[12], m[8]);+	MONT_MAC(v0, v1, v2, u[13], m[7]);+	MONT_MAC(w0, w1, w2, u[14], m[6]);+	MONT_MAC(t0, t1, t2, u[15], m[5]);+	MONT_MAC(v0, v1, v2, u[16], m[4]);+	MONT_MAC(w0, w1, w2, u[17], m[3]);+	MONT_MAC(t0, t1, t2, u[18], m[2]);+	MONT_MAC(v0, v1, v2, u[19], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[20] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[20], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 21 */+	MONT_MAC(t0, t1, t2, a[0], b[21]);+	MONT_MUL(v0, v1, v2, a[1], b[20]);+	MONT_MUL(w0, w1, w2, a[2], b[19]);+	MONT_MAC(t0, t1, t2, a[3], b[18]);+	MONT_MAC(v0, v1, v2, a[4], b[17]);+	MONT_MAC(w0, w1, w2, a[5], b[16]);+	MONT_MAC(t0, t1, t2, a[6], b[15]);+	MONT_MAC(v0, v1, v2, a[7], b[14]);+	MONT_MAC(w0, w1, w2, a[8], b[13]);+	MONT_MAC(t0, t1, t2, a[9], b[12]);+	MONT_MAC(v0, v1, v2, a[10], b[11]);+	MONT_MAC(w0, w1, w2, a[11], b[10]);+	MONT_MAC(t0, t1, t2, a[12], b[9]);+	MONT_MAC(v0, v1, v2, a[13], b[8]);+	MONT_MAC(w0, w1, w2, a[14], b[7]);+	MONT_MAC(t0, t1, t2, a[15], b[6]);+	MONT_MAC(v0, v1, v2, a[16], b[5]);+	MONT_MAC(w0, w1, w2, a[17], b[4]);+	MONT_MAC(t0, t1, t2, a[18], b[3]);+	MONT_MAC(v0, v1, v2, a[19], b[2]);+	MONT_MAC(w0, w1, w2, a[20], b[1]);+	MONT_MAC(t0, t1, t2, a[21], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[21]);+	MONT_MAC(w0, w1, w2, u[1], m[20]);+	MONT_MAC(t0, t1, t2, u[2], m[19]);+	MONT_MAC(v0, v1, v2, u[3], m[18]);+	MONT_MAC(w0, w1, w2, u[4], m[17]);+	MONT_MAC(t0, t1, t2, u[5], m[16]);+	MONT_MAC(v0, v1, v2, u[6], m[15]);+	MONT_MAC(w0, w1, w2, u[7], m[14]);+	MONT_MAC(t0, t1, t2, u[8], m[13]);+	MONT_MAC(v0, v1, v2, u[9], m[12]);+	MONT_MAC(w0, w1, w2, u[10], m[11]);+	MONT_MAC(t0, t1, t2, u[11], m[10]);+	MONT_MAC(v0, v1, v2, u[12], m[9]);+	MONT_MAC(w0, w1, w2, u[13], m[8]);+	MONT_MAC(t0, t1, t2, u[14], m[7]);+	MONT_MAC(v0, v1, v2, u[15], m[6]);+	MONT_MAC(w0, w1, w2, u[16], m[5]);+	MONT_MAC(t0, t1, t2, u[17], m[4]);+	MONT_MAC(v0, v1, v2, u[18], m[3]);+	MONT_MAC(w0, w1, w2, u[19], m[2]);+	MONT_MAC(t0, t1, t2, u[20], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[21] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[21], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 22 */+	MONT_MAC(t0, t1, t2, a[0], b[22]);+	MONT_MUL(v0, v1, v2, a[1], b[21]);+	MONT_MUL(w0, w1, w2, a[2], b[20]);+	MONT_MAC(t0, t1, t2, a[3], b[19]);+	MONT_MAC(v0, v1, v2, a[4], b[18]);+	MONT_MAC(w0, w1, w2, a[5], b[17]);+	MONT_MAC(t0, t1, t2, a[6], b[16]);+	MONT_MAC(v0, v1, v2, a[7], b[15]);+	MONT_MAC(w0, w1, w2, a[8], b[14]);+	MONT_MAC(t0, t1, t2, a[9], b[13]);+	MONT_MAC(v0, v1, v2, a[10], b[12]);+	MONT_MAC(w0, w1, w2, a[11], b[11]);+	MONT_MAC(t0, t1, t2, a[12], b[10]);+	MONT_MAC(v0, v1, v2, a[13], b[9]);+	MONT_MAC(w0, w1, w2, a[14], b[8]);+	MONT_MAC(t0, t1, t2, a[15], b[7]);+	MONT_MAC(v0, v1, v2, a[16], b[6]);+	MONT_MAC(w0, w1, w2, a[17], b[5]);+	MONT_MAC(t0, t1, t2, a[18], b[4]);+	MONT_MAC(v0, v1, v2, a[19], b[3]);+	MONT_MAC(w0, w1, w2, a[20], b[2]);+	MONT_MAC(t0, t1, t2, a[21], b[1]);+	MONT_MAC(v0, v1, v2, a[22], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[22]);+	MONT_MAC(t0, t1, t2, u[1], m[21]);+	MONT_MAC(v0, v1, v2, u[2], m[20]);+	MONT_MAC(w0, w1, w2, u[3], m[19]);+	MONT_MAC(t0, t1, t2, u[4], m[18]);+	MONT_MAC(v0, v1, v2, u[5], m[17]);+	MONT_MAC(w0, w1, w2, u[6], m[16]);+	MONT_MAC(t0, t1, t2, u[7], m[15]);+	MONT_MAC(v0, v1, v2, u[8], m[14]);+	MONT_MAC(w0, w1, w2, u[9], m[13]);+	MONT_MAC(t0, t1, t2, u[10], m[12]);+	MONT_MAC(v0, v1, v2, u[11], m[11]);+	MONT_MAC(w0, w1, w2, u[12], m[10]);+	MONT_MAC(t0, t1, t2, u[13], m[9]);+	MONT_MAC(v0, v1, v2, u[14], m[8]);+	MONT_MAC(w0, w1, w2, u[15], m[7]);+	MONT_MAC(t0, t1, t2, u[16], m[6]);+	MONT_MAC(v0, v1, v2, u[17], m[5]);+	MONT_MAC(w0, w1, w2, u[18], m[4]);+	MONT_MAC(t0, t1, t2, u[19], m[3]);+	MONT_MAC(v0, v1, v2, u[20], m[2]);+	MONT_MAC(w0, w1, w2, u[21], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[22] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[22], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 23 */+	MONT_MAC(t0, t1, t2, a[0], b[23]);+	MONT_MUL(v0, v1, v2, a[1], b[22]);+	MONT_MUL(w0, w1, w2, a[2], b[21]);+	MONT_MAC(t0, t1, t2, a[3], b[20]);+	MONT_MAC(v0, v1, v2, a[4], b[19]);+	MONT_MAC(w0, w1, w2, a[5], b[18]);+	MONT_MAC(t0, t1, t2, a[6], b[17]);+	MONT_MAC(v0, v1, v2, a[7], b[16]);+	MONT_MAC(w0, w1, w2, a[8], b[15]);+	MONT_MAC(t0, t1, t2, a[9], b[14]);+	MONT_MAC(v0, v1, v2, a[10], b[13]);+	MONT_MAC(w0, w1, w2, a[11], b[12]);+	MONT_MAC(t0, t1, t2, a[12], b[11]);+	MONT_MAC(v0, v1, v2, a[13], b[10]);+	MONT_MAC(w0, w1, w2, a[14], b[9]);+	MONT_MAC(t0, t1, t2, a[15], b[8]);+	MONT_MAC(v0, v1, v2, a[16], b[7]);+	MONT_MAC(w0, w1, w2, a[17], b[6]);+	MONT_MAC(t0, t1, t2, a[18], b[5]);+	MONT_MAC(v0, v1, v2, a[19], b[4]);+	MONT_MAC(w0, w1, w2, a[20], b[3]);+	MONT_MAC(t0, t1, t2, a[21], b[2]);+	MONT_MAC(v0, v1, v2, a[22], b[1]);+	MONT_MAC(w0, w1, w2, a[23], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[23]);+	MONT_MAC(v0, v1, v2, u[1], m[22]);+	MONT_MAC(w0, w1, w2, u[2], m[21]);+	MONT_MAC(t0, t1, t2, u[3], m[20]);+	MONT_MAC(v0, v1, v2, u[4], m[19]);+	MONT_MAC(w0, w1, w2, u[5], m[18]);+	MONT_MAC(t0, t1, t2, u[6], m[17]);+	MONT_MAC(v0, v1, v2, u[7], m[16]);+	MONT_MAC(w0, w1, w2, u[8], m[15]);+	MONT_MAC(t0, t1, t2, u[9], m[14]);+	MONT_MAC(v0, v1, v2, u[10], m[13]);+	MONT_MAC(w0, w1, w2, u[11], m[12]);+	MONT_MAC(t0, t1, t2, u[12], m[11]);+	MONT_MAC(v0, v1, v2, u[13], m[10]);+	MONT_MAC(w0, w1, w2, u[14], m[9]);+	MONT_MAC(t0, t1, t2, u[15], m[8]);+	MONT_MAC(v0, v1, v2, u[16], m[7]);+	MONT_MAC(w0, w1, w2, u[17], m[6]);+	MONT_MAC(t0, t1, t2, u[18], m[5]);+	MONT_MAC(v0, v1, v2, u[19], m[4]);+	MONT_MAC(w0, w1, w2, u[20], m[3]);+	MONT_MAC(t0, t1, t2, u[21], m[2]);+	MONT_MAC(v0, v1, v2, u[22], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[23] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[23], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 24 */+	MONT_MAC(t0, t1, t2, a[1], b[23]);+	MONT_MUL(v0, v1, v2, a[2], b[22]);+	MONT_MUL(w0, w1, w2, a[3], b[21]);+	MONT_MAC(t0, t1, t2, a[4], b[20]);+	MONT_MAC(v0, v1, v2, a[5], b[19]);+	MONT_MAC(w0, w1, w2, a[6], b[18]);+	MONT_MAC(t0, t1, t2, a[7], b[17]);+	MONT_MAC(v0, v1, v2, a[8], b[16]);+	MONT_MAC(w0, w1, w2, a[9], b[15]);+	MONT_MAC(t0, t1, t2, a[10], b[14]);+	MONT_MAC(v0, v1, v2, a[11], b[13]);+	MONT_MAC(w0, w1, w2, a[12], b[12]);+	MONT_MAC(t0, t1, t2, a[13], b[11]);+	MONT_MAC(v0, v1, v2, a[14], b[10]);+	MONT_MAC(w0, w1, w2, a[15], b[9]);+	MONT_MAC(t0, t1, t2, a[16], b[8]);+	MONT_MAC(v0, v1, v2, a[17], b[7]);+	MONT_MAC(w0, w1, w2, a[18], b[6]);+	MONT_MAC(t0, t1, t2, a[19], b[5]);+	MONT_MAC(v0, v1, v2, a[20], b[4]);+	MONT_MAC(w0, w1, w2, a[21], b[3]);+	MONT_MAC(t0, t1, t2, a[22], b[2]);+	MONT_MAC(v0, v1, v2, a[23], b[1]);+	MONT_MAC(w0, w1, w2, u[1], m[23]);+	MONT_MAC(t0, t1, t2, u[2], m[22]);+	MONT_MAC(v0, v1, v2, u[3], m[21]);+	MONT_MAC(w0, w1, w2, u[4], m[20]);+	MONT_MAC(t0, t1, t2, u[5], m[19]);+	MONT_MAC(v0, v1, v2, u[6], m[18]);+	MONT_MAC(w0, w1, w2, u[7], m[17]);+	MONT_MAC(t0, t1, t2, u[8], m[16]);+	MONT_MAC(v0, v1, v2, u[9], m[15]);+	MONT_MAC(w0, w1, w2, u[10], m[14]);+	MONT_MAC(t0, t1, t2, u[11], m[13]);+	MONT_MAC(v0, v1, v2, u[12], m[12]);+	MONT_MAC(w0, w1, w2, u[13], m[11]);+	MONT_MAC(t0, t1, t2, u[14], m[10]);+	MONT_MAC(v0, v1, v2, u[15], m[9]);+	MONT_MAC(w0, w1, w2, u[16], m[8]);+	MONT_MAC(t0, t1, t2, u[17], m[7]);+	MONT_MAC(v0, v1, v2, u[18], m[6]);+	MONT_MAC(w0, w1, w2, u[19], m[5]);+	MONT_MAC(t0, t1, t2, u[20], m[4]);+	MONT_MAC(v0, v1, v2, u[21], m[3]);+	MONT_MAC(w0, w1, w2, u[22], m[2]);+	MONT_MAC(t0, t1, t2, u[23], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[0] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 25 */+	MONT_MAC(t0, t1, t2, a[2], b[23]);+	MONT_MUL(v0, v1, v2, a[3], b[22]);+	MONT_MUL(w0, w1, w2, a[4], b[21]);+	MONT_MAC(t0, t1, t2, a[5], b[20]);+	MONT_MAC(v0, v1, v2, a[6], b[19]);+	MONT_MAC(w0, w1, w2, a[7], b[18]);+	MONT_MAC(t0, t1, t2, a[8], b[17]);+	MONT_MAC(v0, v1, v2, a[9], b[16]);+	MONT_MAC(w0, w1, w2, a[10], b[15]);+	MONT_MAC(t0, t1, t2, a[11], b[14]);+	MONT_MAC(v0, v1, v2, a[12], b[13]);+	MONT_MAC(w0, w1, w2, a[13], b[12]);+	MONT_MAC(t0, t1, t2, a[14], b[11]);+	MONT_MAC(v0, v1, v2, a[15], b[10]);+	MONT_MAC(w0, w1, w2, a[16], b[9]);+	MONT_MAC(t0, t1, t2, a[17], b[8]);+	MONT_MAC(v0, v1, v2, a[18], b[7]);+	MONT_MAC(w0, w1, w2, a[19], b[6]);+	MONT_MAC(t0, t1, t2, a[20], b[5]);+	MONT_MAC(v0, v1, v2, a[21], b[4]);+	MONT_MAC(w0, w1, w2, a[22], b[3]);+	MONT_MAC(t0, t1, t2, a[23], b[2]);+	MONT_MAC(v0, v1, v2, u[2], m[23]);+	MONT_MAC(w0, w1, w2, u[3], m[22]);+	MONT_MAC(t0, t1, t2, u[4], m[21]);+	MONT_MAC(v0, v1, v2, u[5], m[20]);+	MONT_MAC(w0, w1, w2, u[6], m[19]);+	MONT_MAC(t0, t1, t2, u[7], m[18]);+	MONT_MAC(v0, v1, v2, u[8], m[17]);+	MONT_MAC(w0, w1, w2, u[9], m[16]);+	MONT_MAC(t0, t1, t2, u[10], m[15]);+	MONT_MAC(v0, v1, v2, u[11], m[14]);+	MONT_MAC(w0, w1, w2, u[12], m[13]);+	MONT_MAC(t0, t1, t2, u[13], m[12]);+	MONT_MAC(v0, v1, v2, u[14], m[11]);+	MONT_MAC(w0, w1, w2, u[15], m[10]);+	MONT_MAC(t0, t1, t2, u[16], m[9]);+	MONT_MAC(v0, v1, v2, u[17], m[8]);+	MONT_MAC(w0, w1, w2, u[18], m[7]);+	MONT_MAC(t0, t1, t2, u[19], m[6]);+	MONT_MAC(v0, v1, v2, u[20], m[5]);+	MONT_MAC(w0, w1, w2, u[21], m[4]);+	MONT_MAC(t0, t1, t2, u[22], m[3]);+	MONT_MAC(v0, v1, v2, u[23], m[2]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[1] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 26 */+	MONT_MAC(t0, t1, t2, a[3], b[23]);+	MONT_MUL(v0, v1, v2, a[4], b[22]);+	MONT_MUL(w0, w1, w2, a[5], b[21]);+	MONT_MAC(t0, t1, t2, a[6], b[20]);+	MONT_MAC(v0, v1, v2, a[7], b[19]);+	MONT_MAC(w0, w1, w2, a[8], b[18]);+	MONT_MAC(t0, t1, t2, a[9], b[17]);+	MONT_MAC(v0, v1, v2, a[10], b[16]);+	MONT_MAC(w0, w1, w2, a[11], b[15]);+	MONT_MAC(t0, t1, t2, a[12], b[14]);+	MONT_MAC(v0, v1, v2, a[13], b[13]);+	MONT_MAC(w0, w1, w2, a[14], b[12]);+	MONT_MAC(t0, t1, t2, a[15], b[11]);+	MONT_MAC(v0, v1, v2, a[16], b[10]);+	MONT_MAC(w0, w1, w2, a[17], b[9]);+	MONT_MAC(t0, t1, t2, a[18], b[8]);+	MONT_MAC(v0, v1, v2, a[19], b[7]);+	MONT_MAC(w0, w1, w2, a[20], b[6]);+	MONT_MAC(t0, t1, t2, a[21], b[5]);+	MONT_MAC(v0, v1, v2, a[22], b[4]);+	MONT_MAC(w0, w1, w2, a[23], b[3]);+	MONT_MAC(t0, t1, t2, u[3], m[23]);+	MONT_MAC(v0, v1, v2, u[4], m[22]);+	MONT_MAC(w0, w1, w2, u[5], m[21]);+	MONT_MAC(t0, t1, t2, u[6], m[20]);+	MONT_MAC(v0, v1, v2, u[7], m[19]);+	MONT_MAC(w0, w1, w2, u[8], m[18]);+	MONT_MAC(t0, t1, t2, u[9], m[17]);+	MONT_MAC(v0, v1, v2, u[10], m[16]);+	MONT_MAC(w0, w1, w2, u[11], m[15]);+	MONT_MAC(t0, t1, t2, u[12], m[14]);+	MONT_MAC(v0, v1, v2, u[13], m[13]);+	MONT_MAC(w0, w1, w2, u[14], m[12]);+	MONT_MAC(t0, t1, t2, u[15], m[11]);+	MONT_MAC(v0, v1, v2, u[16], m[10]);+	MONT_MAC(w0, w1, w2, u[17], m[9]);+	MONT_MAC(t0, t1, t2, u[18], m[8]);+	MONT_MAC(v0, v1, v2, u[19], m[7]);+	MONT_MAC(w0, w1, w2, u[20], m[6]);+	MONT_MAC(t0, t1, t2, u[21], m[5]);+	MONT_MAC(v0, v1, v2, u[22], m[4]);+	MONT_MAC(w0, w1, w2, u[23], m[3]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[2] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 27 */+	MONT_MAC(t0, t1, t2, a[4], b[23]);+	MONT_MUL(v0, v1, v2, a[5], b[22]);+	MONT_MUL(w0, w1, w2, a[6], b[21]);+	MONT_MAC(t0, t1, t2, a[7], b[20]);+	MONT_MAC(v0, v1, v2, a[8], b[19]);+	MONT_MAC(w0, w1, w2, a[9], b[18]);+	MONT_MAC(t0, t1, t2, a[10], b[17]);+	MONT_MAC(v0, v1, v2, a[11], b[16]);+	MONT_MAC(w0, w1, w2, a[12], b[15]);+	MONT_MAC(t0, t1, t2, a[13], b[14]);+	MONT_MAC(v0, v1, v2, a[14], b[13]);+	MONT_MAC(w0, w1, w2, a[15], b[12]);+	MONT_MAC(t0, t1, t2, a[16], b[11]);+	MONT_MAC(v0, v1, v2, a[17], b[10]);+	MONT_MAC(w0, w1, w2, a[18], b[9]);+	MONT_MAC(t0, t1, t2, a[19], b[8]);+	MONT_MAC(v0, v1, v2, a[20], b[7]);+	MONT_MAC(w0, w1, w2, a[21], b[6]);+	MONT_MAC(t0, t1, t2, a[22], b[5]);+	MONT_MAC(v0, v1, v2, a[23], b[4]);+	MONT_MAC(w0, w1, w2, u[4], m[23]);+	MONT_MAC(t0, t1, t2, u[5], m[22]);+	MONT_MAC(v0, v1, v2, u[6], m[21]);+	MONT_MAC(w0, w1, w2, u[7], m[20]);+	MONT_MAC(t0, t1, t2, u[8], m[19]);+	MONT_MAC(v0, v1, v2, u[9], m[18]);+	MONT_MAC(w0, w1, w2, u[10], m[17]);+	MONT_MAC(t0, t1, t2, u[11], m[16]);+	MONT_MAC(v0, v1, v2, u[12], m[15]);+	MONT_MAC(w0, w1, w2, u[13], m[14]);+	MONT_MAC(t0, t1, t2, u[14], m[13]);+	MONT_MAC(v0, v1, v2, u[15], m[12]);+	MONT_MAC(w0, w1, w2, u[16], m[11]);+	MONT_MAC(t0, t1, t2, u[17], m[10]);+	MONT_MAC(v0, v1, v2, u[18], m[9]);+	MONT_MAC(w0, w1, w2, u[19], m[8]);+	MONT_MAC(t0, t1, t2, u[20], m[7]);+	MONT_MAC(v0, v1, v2, u[21], m[6]);+	MONT_MAC(w0, w1, w2, u[22], m[5]);+	MONT_MAC(t0, t1, t2, u[23], m[4]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[3] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 28 */+	MONT_MAC(t0, t1, t2, a[5], b[23]);+	MONT_MUL(v0, v1, v2, a[6], b[22]);+	MONT_MUL(w0, w1, w2, a[7], b[21]);+	MONT_MAC(t0, t1, t2, a[8], b[20]);+	MONT_MAC(v0, v1, v2, a[9], b[19]);+	MONT_MAC(w0, w1, w2, a[10], b[18]);+	MONT_MAC(t0, t1, t2, a[11], b[17]);+	MONT_MAC(v0, v1, v2, a[12], b[16]);+	MONT_MAC(w0, w1, w2, a[13], b[15]);+	MONT_MAC(t0, t1, t2, a[14], b[14]);+	MONT_MAC(v0, v1, v2, a[15], b[13]);+	MONT_MAC(w0, w1, w2, a[16], b[12]);+	MONT_MAC(t0, t1, t2, a[17], b[11]);+	MONT_MAC(v0, v1, v2, a[18], b[10]);+	MONT_MAC(w0, w1, w2, a[19], b[9]);+	MONT_MAC(t0, t1, t2, a[20], b[8]);+	MONT_MAC(v0, v1, v2, a[21], b[7]);+	MONT_MAC(w0, w1, w2, a[22], b[6]);+	MONT_MAC(t0, t1, t2, a[23], b[5]);+	MONT_MAC(v0, v1, v2, u[5], m[23]);+	MONT_MAC(w0, w1, w2, u[6], m[22]);+	MONT_MAC(t0, t1, t2, u[7], m[21]);+	MONT_MAC(v0, v1, v2, u[8], m[20]);+	MONT_MAC(w0, w1, w2, u[9], m[19]);+	MONT_MAC(t0, t1, t2, u[10], m[18]);+	MONT_MAC(v0, v1, v2, u[11], m[17]);+	MONT_MAC(w0, w1, w2, u[12], m[16]);+	MONT_MAC(t0, t1, t2, u[13], m[15]);+	MONT_MAC(v0, v1, v2, u[14], m[14]);+	MONT_MAC(w0, w1, w2, u[15], m[13]);+	MONT_MAC(t0, t1, t2, u[16], m[12]);+	MONT_MAC(v0, v1, v2, u[17], m[11]);+	MONT_MAC(w0, w1, w2, u[18], m[10]);+	MONT_MAC(t0, t1, t2, u[19], m[9]);+	MONT_MAC(v0, v1, v2, u[20], m[8]);+	MONT_MAC(w0, w1, w2, u[21], m[7]);+	MONT_MAC(t0, t1, t2, u[22], m[6]);+	MONT_MAC(v0, v1, v2, u[23], m[5]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[4] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 29 */+	MONT_MAC(t0, t1, t2, a[6], b[23]);+	MONT_MUL(v0, v1, v2, a[7], b[22]);+	MONT_MUL(w0, w1, w2, a[8], b[21]);+	MONT_MAC(t0, t1, t2, a[9], b[20]);+	MONT_MAC(v0, v1, v2, a[10], b[19]);+	MONT_MAC(w0, w1, w2, a[11], b[18]);+	MONT_MAC(t0, t1, t2, a[12], b[17]);+	MONT_MAC(v0, v1, v2, a[13], b[16]);+	MONT_MAC(w0, w1, w2, a[14], b[15]);+	MONT_MAC(t0, t1, t2, a[15], b[14]);+	MONT_MAC(v0, v1, v2, a[16], b[13]);+	MONT_MAC(w0, w1, w2, a[17], b[12]);+	MONT_MAC(t0, t1, t2, a[18], b[11]);+	MONT_MAC(v0, v1, v2, a[19], b[10]);+	MONT_MAC(w0, w1, w2, a[20], b[9]);+	MONT_MAC(t0, t1, t2, a[21], b[8]);+	MONT_MAC(v0, v1, v2, a[22], b[7]);+	MONT_MAC(w0, w1, w2, a[23], b[6]);+	MONT_MAC(t0, t1, t2, u[6], m[23]);+	MONT_MAC(v0, v1, v2, u[7], m[22]);+	MONT_MAC(w0, w1, w2, u[8], m[21]);+	MONT_MAC(t0, t1, t2, u[9], m[20]);+	MONT_MAC(v0, v1, v2, u[10], m[19]);+	MONT_MAC(w0, w1, w2, u[11], m[18]);+	MONT_MAC(t0, t1, t2, u[12], m[17]);+	MONT_MAC(v0, v1, v2, u[13], m[16]);+	MONT_MAC(w0, w1, w2, u[14], m[15]);+	MONT_MAC(t0, t1, t2, u[15], m[14]);+	MONT_MAC(v0, v1, v2, u[16], m[13]);+	MONT_MAC(w0, w1, w2, u[17], m[12]);+	MONT_MAC(t0, t1, t2, u[18], m[11]);+	MONT_MAC(v0, v1, v2, u[19], m[10]);+	MONT_MAC(w0, w1, w2, u[20], m[9]);+	MONT_MAC(t0, t1, t2, u[21], m[8]);+	MONT_MAC(v0, v1, v2, u[22], m[7]);+	MONT_MAC(w0, w1, w2, u[23], m[6]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[5] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 30 */+	MONT_MAC(t0, t1, t2, a[7], b[23]);+	MONT_MUL(v0, v1, v2, a[8], b[22]);+	MONT_MUL(w0, w1, w2, a[9], b[21]);+	MONT_MAC(t0, t1, t2, a[10], b[20]);+	MONT_MAC(v0, v1, v2, a[11], b[19]);+	MONT_MAC(w0, w1, w2, a[12], b[18]);+	MONT_MAC(t0, t1, t2, a[13], b[17]);+	MONT_MAC(v0, v1, v2, a[14], b[16]);+	MONT_MAC(w0, w1, w2, a[15], b[15]);+	MONT_MAC(t0, t1, t2, a[16], b[14]);+	MONT_MAC(v0, v1, v2, a[17], b[13]);+	MONT_MAC(w0, w1, w2, a[18], b[12]);+	MONT_MAC(t0, t1, t2, a[19], b[11]);+	MONT_MAC(v0, v1, v2, a[20], b[10]);+	MONT_MAC(w0, w1, w2, a[21], b[9]);+	MONT_MAC(t0, t1, t2, a[22], b[8]);+	MONT_MAC(v0, v1, v2, a[23], b[7]);+	MONT_MAC(w0, w1, w2, u[7], m[23]);+	MONT_MAC(t0, t1, t2, u[8], m[22]);+	MONT_MAC(v0, v1, v2, u[9], m[21]);+	MONT_MAC(w0, w1, w2, u[10], m[20]);+	MONT_MAC(t0, t1, t2, u[11], m[19]);+	MONT_MAC(v0, v1, v2, u[12], m[18]);+	MONT_MAC(w0, w1, w2, u[13], m[17]);+	MONT_MAC(t0, t1, t2, u[14], m[16]);+	MONT_MAC(v0, v1, v2, u[15], m[15]);+	MONT_MAC(w0, w1, w2, u[16], m[14]);+	MONT_MAC(t0, t1, t2, u[17], m[13]);+	MONT_MAC(v0, v1, v2, u[18], m[12]);+	MONT_MAC(w0, w1, w2, u[19], m[11]);+	MONT_MAC(t0, t1, t2, u[20], m[10]);+	MONT_MAC(v0, v1, v2, u[21], m[9]);+	MONT_MAC(w0, w1, w2, u[22], m[8]);+	MONT_MAC(t0, t1, t2, u[23], m[7]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[6] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 31 */+	MONT_MAC(t0, t1, t2, a[8], b[23]);+	MONT_MUL(v0, v1, v2, a[9], b[22]);+	MONT_MUL(w0, w1, w2, a[10], b[21]);+	MONT_MAC(t0, t1, t2, a[11], b[20]);+	MONT_MAC(v0, v1, v2, a[12], b[19]);+	MONT_MAC(w0, w1, w2, a[13], b[18]);+	MONT_MAC(t0, t1, t2, a[14], b[17]);+	MONT_MAC(v0, v1, v2, a[15], b[16]);+	MONT_MAC(w0, w1, w2, a[16], b[15]);+	MONT_MAC(t0, t1, t2, a[17], b[14]);+	MONT_MAC(v0, v1, v2, a[18], b[13]);+	MONT_MAC(w0, w1, w2, a[19], b[12]);+	MONT_MAC(t0, t1, t2, a[20], b[11]);+	MONT_MAC(v0, v1, v2, a[21], b[10]);+	MONT_MAC(w0, w1, w2, a[22], b[9]);+	MONT_MAC(t0, t1, t2, a[23], b[8]);+	MONT_MAC(v0, v1, v2, u[8], m[23]);+	MONT_MAC(w0, w1, w2, u[9], m[22]);+	MONT_MAC(t0, t1, t2, u[10], m[21]);+	MONT_MAC(v0, v1, v2, u[11], m[20]);+	MONT_MAC(w0, w1, w2, u[12], m[19]);+	MONT_MAC(t0, t1, t2, u[13], m[18]);+	MONT_MAC(v0, v1, v2, u[14], m[17]);+	MONT_MAC(w0, w1, w2, u[15], m[16]);+	MONT_MAC(t0, t1, t2, u[16], m[15]);+	MONT_MAC(v0, v1, v2, u[17], m[14]);+	MONT_MAC(w0, w1, w2, u[18], m[13]);+	MONT_MAC(t0, t1, t2, u[19], m[12]);+	MONT_MAC(v0, v1, v2, u[20], m[11]);+	MONT_MAC(w0, w1, w2, u[21], m[10]);+	MONT_MAC(t0, t1, t2, u[22], m[9]);+	MONT_MAC(v0, v1, v2, u[23], m[8]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[7] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 32 */+	MONT_MAC(t0, t1, t2, a[9], b[23]);+	MONT_MUL(v0, v1, v2, a[10], b[22]);+	MONT_MUL(w0, w1, w2, a[11], b[21]);+	MONT_MAC(t0, t1, t2, a[12], b[20]);+	MONT_MAC(v0, v1, v2, a[13], b[19]);+	MONT_MAC(w0, w1, w2, a[14], b[18]);+	MONT_MAC(t0, t1, t2, a[15], b[17]);+	MONT_MAC(v0, v1, v2, a[16], b[16]);+	MONT_MAC(w0, w1, w2, a[17], b[15]);+	MONT_MAC(t0, t1, t2, a[18], b[14]);+	MONT_MAC(v0, v1, v2, a[19], b[13]);+	MONT_MAC(w0, w1, w2, a[20], b[12]);+	MONT_MAC(t0, t1, t2, a[21], b[11]);+	MONT_MAC(v0, v1, v2, a[22], b[10]);+	MONT_MAC(w0, w1, w2, a[23], b[9]);+	MONT_MAC(t0, t1, t2, u[9], m[23]);+	MONT_MAC(v0, v1, v2, u[10], m[22]);+	MONT_MAC(w0, w1, w2, u[11], m[21]);+	MONT_MAC(t0, t1, t2, u[12], m[20]);+	MONT_MAC(v0, v1, v2, u[13], m[19]);+	MONT_MAC(w0, w1, w2, u[14], m[18]);+	MONT_MAC(t0, t1, t2, u[15], m[17]);+	MONT_MAC(v0, v1, v2, u[16], m[16]);+	MONT_MAC(w0, w1, w2, u[17], m[15]);+	MONT_MAC(t0, t1, t2, u[18], m[14]);+	MONT_MAC(v0, v1, v2, u[19], m[13]);+	MONT_MAC(w0, w1, w2, u[20], m[12]);+	MONT_MAC(t0, t1, t2, u[21], m[11]);+	MONT_MAC(v0, v1, v2, u[22], m[10]);+	MONT_MAC(w0, w1, w2, u[23], m[9]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[8] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 33 */+	MONT_MAC(t0, t1, t2, a[10], b[23]);+	MONT_MUL(v0, v1, v2, a[11], b[22]);+	MONT_MUL(w0, w1, w2, a[12], b[21]);+	MONT_MAC(t0, t1, t2, a[13], b[20]);+	MONT_MAC(v0, v1, v2, a[14], b[19]);+	MONT_MAC(w0, w1, w2, a[15], b[18]);+	MONT_MAC(t0, t1, t2, a[16], b[17]);+	MONT_MAC(v0, v1, v2, a[17], b[16]);+	MONT_MAC(w0, w1, w2, a[18], b[15]);+	MONT_MAC(t0, t1, t2, a[19], b[14]);+	MONT_MAC(v0, v1, v2, a[20], b[13]);+	MONT_MAC(w0, w1, w2, a[21], b[12]);+	MONT_MAC(t0, t1, t2, a[22], b[11]);+	MONT_MAC(v0, v1, v2, a[23], b[10]);+	MONT_MAC(w0, w1, w2, u[10], m[23]);+	MONT_MAC(t0, t1, t2, u[11], m[22]);+	MONT_MAC(v0, v1, v2, u[12], m[21]);+	MONT_MAC(w0, w1, w2, u[13], m[20]);+	MONT_MAC(t0, t1, t2, u[14], m[19]);+	MONT_MAC(v0, v1, v2, u[15], m[18]);+	MONT_MAC(w0, w1, w2, u[16], m[17]);+	MONT_MAC(t0, t1, t2, u[17], m[16]);+	MONT_MAC(v0, v1, v2, u[18], m[15]);+	MONT_MAC(w0, w1, w2, u[19], m[14]);+	MONT_MAC(t0, t1, t2, u[20], m[13]);+	MONT_MAC(v0, v1, v2, u[21], m[12]);+	MONT_MAC(w0, w1, w2, u[22], m[11]);+	MONT_MAC(t0, t1, t2, u[23], m[10]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[9] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 34 */+	MONT_MAC(t0, t1, t2, a[11], b[23]);+	MONT_MUL(v0, v1, v2, a[12], b[22]);+	MONT_MUL(w0, w1, w2, a[13], b[21]);+	MONT_MAC(t0, t1, t2, a[14], b[20]);+	MONT_MAC(v0, v1, v2, a[15], b[19]);+	MONT_MAC(w0, w1, w2, a[16], b[18]);+	MONT_MAC(t0, t1, t2, a[17], b[17]);+	MONT_MAC(v0, v1, v2, a[18], b[16]);+	MONT_MAC(w0, w1, w2, a[19], b[15]);+	MONT_MAC(t0, t1, t2, a[20], b[14]);+	MONT_MAC(v0, v1, v2, a[21], b[13]);+	MONT_MAC(w0, w1, w2, a[22], b[12]);+	MONT_MAC(t0, t1, t2, a[23], b[11]);+	MONT_MAC(v0, v1, v2, u[11], m[23]);+	MONT_MAC(w0, w1, w2, u[12], m[22]);+	MONT_MAC(t0, t1, t2, u[13], m[21]);+	MONT_MAC(v0, v1, v2, u[14], m[20]);+	MONT_MAC(w0, w1, w2, u[15], m[19]);+	MONT_MAC(t0, t1, t2, u[16], m[18]);+	MONT_MAC(v0, v1, v2, u[17], m[17]);+	MONT_MAC(w0, w1, w2, u[18], m[16]);+	MONT_MAC(t0, t1, t2, u[19], m[15]);+	MONT_MAC(v0, v1, v2, u[20], m[14]);+	MONT_MAC(w0, w1, w2, u[21], m[13]);+	MONT_MAC(t0, t1, t2, u[22], m[12]);+	MONT_MAC(v0, v1, v2, u[23], m[11]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[10] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 35 */+	MONT_MAC(t0, t1, t2, a[12], b[23]);+	MONT_MUL(v0, v1, v2, a[13], b[22]);+	MONT_MUL(w0, w1, w2, a[14], b[21]);+	MONT_MAC(t0, t1, t2, a[15], b[20]);+	MONT_MAC(v0, v1, v2, a[16], b[19]);+	MONT_MAC(w0, w1, w2, a[17], b[18]);+	MONT_MAC(t0, t1, t2, a[18], b[17]);+	MONT_MAC(v0, v1, v2, a[19], b[16]);+	MONT_MAC(w0, w1, w2, a[20], b[15]);+	MONT_MAC(t0, t1, t2, a[21], b[14]);+	MONT_MAC(v0, v1, v2, a[22], b[13]);+	MONT_MAC(w0, w1, w2, a[23], b[12]);+	MONT_MAC(t0, t1, t2, u[12], m[23]);+	MONT_MAC(v0, v1, v2, u[13], m[22]);+	MONT_MAC(w0, w1, w2, u[14], m[21]);+	MONT_MAC(t0, t1, t2, u[15], m[20]);+	MONT_MAC(v0, v1, v2, u[16], m[19]);+	MONT_MAC(w0, w1, w2, u[17], m[18]);+	MONT_MAC(t0, t1, t2, u[18], m[17]);+	MONT_MAC(v0, v1, v2, u[19], m[16]);+	MONT_MAC(w0, w1, w2, u[20], m[15]);+	MONT_MAC(t0, t1, t2, u[21], m[14]);+	MONT_MAC(v0, v1, v2, u[22], m[13]);+	MONT_MAC(w0, w1, w2, u[23], m[12]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[11] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 36 */+	MONT_MAC(t0, t1, t2, a[13], b[23]);+	MONT_MUL(v0, v1, v2, a[14], b[22]);+	MONT_MUL(w0, w1, w2, a[15], b[21]);+	MONT_MAC(t0, t1, t2, a[16], b[20]);+	MONT_MAC(v0, v1, v2, a[17], b[19]);+	MONT_MAC(w0, w1, w2, a[18], b[18]);+	MONT_MAC(t0, t1, t2, a[19], b[17]);+	MONT_MAC(v0, v1, v2, a[20], b[16]);+	MONT_MAC(w0, w1, w2, a[21], b[15]);+	MONT_MAC(t0, t1, t2, a[22], b[14]);+	MONT_MAC(v0, v1, v2, a[23], b[13]);+	MONT_MAC(w0, w1, w2, u[13], m[23]);+	MONT_MAC(t0, t1, t2, u[14], m[22]);+	MONT_MAC(v0, v1, v2, u[15], m[21]);+	MONT_MAC(w0, w1, w2, u[16], m[20]);+	MONT_MAC(t0, t1, t2, u[17], m[19]);+	MONT_MAC(v0, v1, v2, u[18], m[18]);+	MONT_MAC(w0, w1, w2, u[19], m[17]);+	MONT_MAC(t0, t1, t2, u[20], m[16]);+	MONT_MAC(v0, v1, v2, u[21], m[15]);+	MONT_MAC(w0, w1, w2, u[22], m[14]);+	MONT_MAC(t0, t1, t2, u[23], m[13]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[12] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 37 */+	MONT_MAC(t0, t1, t2, a[14], b[23]);+	MONT_MUL(v0, v1, v2, a[15], b[22]);+	MONT_MUL(w0, w1, w2, a[16], b[21]);+	MONT_MAC(t0, t1, t2, a[17], b[20]);+	MONT_MAC(v0, v1, v2, a[18], b[19]);+	MONT_MAC(w0, w1, w2, a[19], b[18]);+	MONT_MAC(t0, t1, t2, a[20], b[17]);+	MONT_MAC(v0, v1, v2, a[21], b[16]);+	MONT_MAC(w0, w1, w2, a[22], b[15]);+	MONT_MAC(t0, t1, t2, a[23], b[14]);+	MONT_MAC(v0, v1, v2, u[14], m[23]);+	MONT_MAC(w0, w1, w2, u[15], m[22]);+	MONT_MAC(t0, t1, t2, u[16], m[21]);+	MONT_MAC(v0, v1, v2, u[17], m[20]);+	MONT_MAC(w0, w1, w2, u[18], m[19]);+	MONT_MAC(t0, t1, t2, u[19], m[18]);+	MONT_MAC(v0, v1, v2, u[20], m[17]);+	MONT_MAC(w0, w1, w2, u[21], m[16]);+	MONT_MAC(t0, t1, t2, u[22], m[15]);+	MONT_MAC(v0, v1, v2, u[23], m[14]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[13] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 38 */+	MONT_MAC(t0, t1, t2, a[15], b[23]);+	MONT_MUL(v0, v1, v2, a[16], b[22]);+	MONT_MUL(w0, w1, w2, a[17], b[21]);+	MONT_MAC(t0, t1, t2, a[18], b[20]);+	MONT_MAC(v0, v1, v2, a[19], b[19]);+	MONT_MAC(w0, w1, w2, a[20], b[18]);+	MONT_MAC(t0, t1, t2, a[21], b[17]);+	MONT_MAC(v0, v1, v2, a[22], b[16]);+	MONT_MAC(w0, w1, w2, a[23], b[15]);+	MONT_MAC(t0, t1, t2, u[15], m[23]);+	MONT_MAC(v0, v1, v2, u[16], m[22]);+	MONT_MAC(w0, w1, w2, u[17], m[21]);+	MONT_MAC(t0, t1, t2, u[18], m[20]);+	MONT_MAC(v0, v1, v2, u[19], m[19]);+	MONT_MAC(w0, w1, w2, u[20], m[18]);+	MONT_MAC(t0, t1, t2, u[21], m[17]);+	MONT_MAC(v0, v1, v2, u[22], m[16]);+	MONT_MAC(w0, w1, w2, u[23], m[15]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[14] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 39 */+	MONT_MAC(t0, t1, t2, a[16], b[23]);+	MONT_MUL(v0, v1, v2, a[17], b[22]);+	MONT_MUL(w0, w1, w2, a[18], b[21]);+	MONT_MAC(t0, t1, t2, a[19], b[20]);+	MONT_MAC(v0, v1, v2, a[20], b[19]);+	MONT_MAC(w0, w1, w2, a[21], b[18]);+	MONT_MAC(t0, t1, t2, a[22], b[17]);+	MONT_MAC(v0, v1, v2, a[23], b[16]);+	MONT_MAC(w0, w1, w2, u[16], m[23]);+	MONT_MAC(t0, t1, t2, u[17], m[22]);+	MONT_MAC(v0, v1, v2, u[18], m[21]);+	MONT_MAC(w0, w1, w2, u[19], m[20]);+	MONT_MAC(t0, t1, t2, u[20], m[19]);+	MONT_MAC(v0, v1, v2, u[21], m[18]);+	MONT_MAC(w0, w1, w2, u[22], m[17]);+	MONT_MAC(t0, t1, t2, u[23], m[16]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[15] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 40 */+	MONT_MAC(t0, t1, t2, a[17], b[23]);+	MONT_MUL(v0, v1, v2, a[18], b[22]);+	MONT_MUL(w0, w1, w2, a[19], b[21]);+	MONT_MAC(t0, t1, t2, a[20], b[20]);+	MONT_MAC(v0, v1, v2, a[21], b[19]);+	MONT_MAC(w0, w1, w2, a[22], b[18]);+	MONT_MAC(t0, t1, t2, a[23], b[17]);+	MONT_MAC(v0, v1, v2, u[17], m[23]);+	MONT_MAC(w0, w1, w2, u[18], m[22]);+	MONT_MAC(t0, t1, t2, u[19], m[21]);+	MONT_MAC(v0, v1, v2, u[20], m[20]);+	MONT_MAC(w0, w1, w2, u[21], m[19]);+	MONT_MAC(t0, t1, t2, u[22], m[18]);+	MONT_MAC(v0, v1, v2, u[23], m[17]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[16] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 41 */+	MONT_MAC(t0, t1, t2, a[18], b[23]);+	MONT_MUL(v0, v1, v2, a[19], b[22]);+	MONT_MUL(w0, w1, w2, a[20], b[21]);+	MONT_MAC(t0, t1, t2, a[21], b[20]);+	MONT_MAC(v0, v1, v2, a[22], b[19]);+	MONT_MAC(w0, w1, w2, a[23], b[18]);+	MONT_MAC(t0, t1, t2, u[18], m[23]);+	MONT_MAC(v0, v1, v2, u[19], m[22]);+	MONT_MAC(w0, w1, w2, u[20], m[21]);+	MONT_MAC(t0, t1, t2, u[21], m[20]);+	MONT_MAC(v0, v1, v2, u[22], m[19]);+	MONT_MAC(w0, w1, w2, u[23], m[18]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[17] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 42 */+	MONT_MAC(t0, t1, t2, a[19], b[23]);+	MONT_MUL(v0, v1, v2, a[20], b[22]);+	MONT_MUL(w0, w1, w2, a[21], b[21]);+	MONT_MAC(t0, t1, t2, a[22], b[20]);+	MONT_MAC(v0, v1, v2, a[23], b[19]);+	MONT_MAC(w0, w1, w2, u[19], m[23]);+	MONT_MAC(t0, t1, t2, u[20], m[22]);+	MONT_MAC(v0, v1, v2, u[21], m[21]);+	MONT_MAC(w0, w1, w2, u[22], m[20]);+	MONT_MAC(t0, t1, t2, u[23], m[19]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[18] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 43 */+	MONT_MAC(t0, t1, t2, a[20], b[23]);+	MONT_MUL(v0, v1, v2, a[21], b[22]);+	MONT_MUL(w0, w1, w2, a[22], b[21]);+	MONT_MAC(t0, t1, t2, a[23], b[20]);+	MONT_MAC(v0, v1, v2, u[20], m[23]);+	MONT_MAC(w0, w1, w2, u[21], m[22]);+	MONT_MAC(t0, t1, t2, u[22], m[21]);+	MONT_MAC(v0, v1, v2, u[23], m[20]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[19] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 44 */+	MONT_MAC(t0, t1, t2, a[21], b[23]);+	MONT_MUL(v0, v1, v2, a[22], b[22]);+	MONT_MUL(w0, w1, w2, a[23], b[21]);+	MONT_MAC(t0, t1, t2, u[21], m[23]);+	MONT_MAC(v0, v1, v2, u[22], m[22]);+	MONT_MAC(w0, w1, w2, u[23], m[21]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[20] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 45 */+	MONT_MAC(t0, t1, t2, a[22], b[23]);+	MONT_MUL(v0, v1, v2, a[23], b[22]);+	MONT_MUL(w0, w1, w2, u[22], m[23]);+	MONT_MAC(t0, t1, t2, u[23], m[22]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[21] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 46 */+	MONT_MAC(t0, t1, t2, a[23], b[23]);+	MONT_MUL(v0, v1, v2, u[23], m[23]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	z[22] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	z[23] = t0;++	/* under 2m, with t1 the bit above n limbs: at most one subtraction */+	borrow = sub_n(r, z, m, 24);+	take = t1 | (borrow ^ 1);+	select_n(r, r, z, take & 1, 24);+}++/* r = a * a * R^-1 mod m, for a under m; t is 2n limbs of scratch */+static void crypton_mont_sqr24(limb_t *r, const limb_t *a,+                               const limb_t *m, limb_t n0, limb_t *t)+{+	limb_t *u = t, *z = t + 24;+	limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take;+	limb_t s0, s1, s2, c0, c1, c2;++	/* column 0 */+	MONT_MAC(t0, t1, t2, a[0], a[0]);+	u[0] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[0], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 1 */+	MONT_MUL(s0, s1, s2, a[0], a[1]);+	MONT_MAC(t0, t1, t2, u[0], m[1]);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[1] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[1], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 2 */+	MONT_MUL(s0, s1, s2, a[0], a[2]);+	MONT_MAC(t0, t1, t2, a[1], a[1]);+	MONT_MUL(v0, v1, v2, u[0], m[2]);+	MONT_MUL(w0, w1, w2, u[1], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[2] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[2], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 3 */+	MONT_MUL(s0, s1, s2, a[0], a[3]);+	MONT_MUL(c0, c1, c2, a[1], a[2]);+	MONT_MAC(t0, t1, t2, u[0], m[3]);+	MONT_MUL(v0, v1, v2, u[1], m[2]);+	MONT_MUL(w0, w1, w2, u[2], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[3] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[3], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 4 */+	MONT_MUL(s0, s1, s2, a[0], a[4]);+	MONT_MUL(c0, c1, c2, a[1], a[3]);+	MONT_MAC(t0, t1, t2, a[2], a[2]);+	MONT_MUL(v0, v1, v2, u[0], m[4]);+	MONT_MUL(w0, w1, w2, u[1], m[3]);+	MONT_MAC(t0, t1, t2, u[2], m[2]);+	MONT_MAC(v0, v1, v2, u[3], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[4] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[4], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 5 */+	MONT_MUL(s0, s1, s2, a[0], a[5]);+	MONT_MUL(c0, c1, c2, a[1], a[4]);+	MONT_MAC(s0, s1, s2, a[2], a[3]);+	MONT_MAC(t0, t1, t2, u[0], m[5]);+	MONT_MUL(v0, v1, v2, u[1], m[4]);+	MONT_MUL(w0, w1, w2, u[2], m[3]);+	MONT_MAC(t0, t1, t2, u[3], m[2]);+	MONT_MAC(v0, v1, v2, u[4], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[5] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[5], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 6 */+	MONT_MUL(s0, s1, s2, a[0], a[6]);+	MONT_MUL(c0, c1, c2, a[1], a[5]);+	MONT_MAC(s0, s1, s2, a[2], a[4]);+	MONT_MAC(t0, t1, t2, a[3], a[3]);+	MONT_MUL(v0, v1, v2, u[0], m[6]);+	MONT_MUL(w0, w1, w2, u[1], m[5]);+	MONT_MAC(t0, t1, t2, u[2], m[4]);+	MONT_MAC(v0, v1, v2, u[3], m[3]);+	MONT_MAC(w0, w1, w2, u[4], m[2]);+	MONT_MAC(t0, t1, t2, u[5], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[6] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[6], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 7 */+	MONT_MUL(s0, s1, s2, a[0], a[7]);+	MONT_MUL(c0, c1, c2, a[1], a[6]);+	MONT_MAC(s0, s1, s2, a[2], a[5]);+	MONT_MAC(c0, c1, c2, a[3], a[4]);+	MONT_MAC(t0, t1, t2, u[0], m[7]);+	MONT_MUL(v0, v1, v2, u[1], m[6]);+	MONT_MUL(w0, w1, w2, u[2], m[5]);+	MONT_MAC(t0, t1, t2, u[3], m[4]);+	MONT_MAC(v0, v1, v2, u[4], m[3]);+	MONT_MAC(w0, w1, w2, u[5], m[2]);+	MONT_MAC(t0, t1, t2, u[6], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[7] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[7], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 8 */+	MONT_MUL(s0, s1, s2, a[0], a[8]);+	MONT_MUL(c0, c1, c2, a[1], a[7]);+	MONT_MAC(s0, s1, s2, a[2], a[6]);+	MONT_MAC(c0, c1, c2, a[3], a[5]);+	MONT_MAC(t0, t1, t2, a[4], a[4]);+	MONT_MUL(v0, v1, v2, u[0], m[8]);+	MONT_MUL(w0, w1, w2, u[1], m[7]);+	MONT_MAC(t0, t1, t2, u[2], m[6]);+	MONT_MAC(v0, v1, v2, u[3], m[5]);+	MONT_MAC(w0, w1, w2, u[4], m[4]);+	MONT_MAC(t0, t1, t2, u[5], m[3]);+	MONT_MAC(v0, v1, v2, u[6], m[2]);+	MONT_MAC(w0, w1, w2, u[7], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[8] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[8], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 9 */+	MONT_MUL(s0, s1, s2, a[0], a[9]);+	MONT_MUL(c0, c1, c2, a[1], a[8]);+	MONT_MAC(s0, s1, s2, a[2], a[7]);+	MONT_MAC(c0, c1, c2, a[3], a[6]);+	MONT_MAC(s0, s1, s2, a[4], a[5]);+	MONT_MAC(t0, t1, t2, u[0], m[9]);+	MONT_MUL(v0, v1, v2, u[1], m[8]);+	MONT_MUL(w0, w1, w2, u[2], m[7]);+	MONT_MAC(t0, t1, t2, u[3], m[6]);+	MONT_MAC(v0, v1, v2, u[4], m[5]);+	MONT_MAC(w0, w1, w2, u[5], m[4]);+	MONT_MAC(t0, t1, t2, u[6], m[3]);+	MONT_MAC(v0, v1, v2, u[7], m[2]);+	MONT_MAC(w0, w1, w2, u[8], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[9] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[9], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 10 */+	MONT_MUL(s0, s1, s2, a[0], a[10]);+	MONT_MUL(c0, c1, c2, a[1], a[9]);+	MONT_MAC(s0, s1, s2, a[2], a[8]);+	MONT_MAC(c0, c1, c2, a[3], a[7]);+	MONT_MAC(s0, s1, s2, a[4], a[6]);+	MONT_MAC(t0, t1, t2, a[5], a[5]);+	MONT_MUL(v0, v1, v2, u[0], m[10]);+	MONT_MUL(w0, w1, w2, u[1], m[9]);+	MONT_MAC(t0, t1, t2, u[2], m[8]);+	MONT_MAC(v0, v1, v2, u[3], m[7]);+	MONT_MAC(w0, w1, w2, u[4], m[6]);+	MONT_MAC(t0, t1, t2, u[5], m[5]);+	MONT_MAC(v0, v1, v2, u[6], m[4]);+	MONT_MAC(w0, w1, w2, u[7], m[3]);+	MONT_MAC(t0, t1, t2, u[8], m[2]);+	MONT_MAC(v0, v1, v2, u[9], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[10] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[10], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 11 */+	MONT_MUL(s0, s1, s2, a[0], a[11]);+	MONT_MUL(c0, c1, c2, a[1], a[10]);+	MONT_MAC(s0, s1, s2, a[2], a[9]);+	MONT_MAC(c0, c1, c2, a[3], a[8]);+	MONT_MAC(s0, s1, s2, a[4], a[7]);+	MONT_MAC(c0, c1, c2, a[5], a[6]);+	MONT_MAC(t0, t1, t2, u[0], m[11]);+	MONT_MUL(v0, v1, v2, u[1], m[10]);+	MONT_MUL(w0, w1, w2, u[2], m[9]);+	MONT_MAC(t0, t1, t2, u[3], m[8]);+	MONT_MAC(v0, v1, v2, u[4], m[7]);+	MONT_MAC(w0, w1, w2, u[5], m[6]);+	MONT_MAC(t0, t1, t2, u[6], m[5]);+	MONT_MAC(v0, v1, v2, u[7], m[4]);+	MONT_MAC(w0, w1, w2, u[8], m[3]);+	MONT_MAC(t0, t1, t2, u[9], m[2]);+	MONT_MAC(v0, v1, v2, u[10], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[11] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[11], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 12 */+	MONT_MUL(s0, s1, s2, a[0], a[12]);+	MONT_MUL(c0, c1, c2, a[1], a[11]);+	MONT_MAC(s0, s1, s2, a[2], a[10]);+	MONT_MAC(c0, c1, c2, a[3], a[9]);+	MONT_MAC(s0, s1, s2, a[4], a[8]);+	MONT_MAC(c0, c1, c2, a[5], a[7]);+	MONT_MAC(t0, t1, t2, a[6], a[6]);+	MONT_MUL(v0, v1, v2, u[0], m[12]);+	MONT_MUL(w0, w1, w2, u[1], m[11]);+	MONT_MAC(t0, t1, t2, u[2], m[10]);+	MONT_MAC(v0, v1, v2, u[3], m[9]);+	MONT_MAC(w0, w1, w2, u[4], m[8]);+	MONT_MAC(t0, t1, t2, u[5], m[7]);+	MONT_MAC(v0, v1, v2, u[6], m[6]);+	MONT_MAC(w0, w1, w2, u[7], m[5]);+	MONT_MAC(t0, t1, t2, u[8], m[4]);+	MONT_MAC(v0, v1, v2, u[9], m[3]);+	MONT_MAC(w0, w1, w2, u[10], m[2]);+	MONT_MAC(t0, t1, t2, u[11], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[12] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[12], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 13 */+	MONT_MUL(s0, s1, s2, a[0], a[13]);+	MONT_MUL(c0, c1, c2, a[1], a[12]);+	MONT_MAC(s0, s1, s2, a[2], a[11]);+	MONT_MAC(c0, c1, c2, a[3], a[10]);+	MONT_MAC(s0, s1, s2, a[4], a[9]);+	MONT_MAC(c0, c1, c2, a[5], a[8]);+	MONT_MAC(s0, s1, s2, a[6], a[7]);+	MONT_MAC(t0, t1, t2, u[0], m[13]);+	MONT_MUL(v0, v1, v2, u[1], m[12]);+	MONT_MUL(w0, w1, w2, u[2], m[11]);+	MONT_MAC(t0, t1, t2, u[3], m[10]);+	MONT_MAC(v0, v1, v2, u[4], m[9]);+	MONT_MAC(w0, w1, w2, u[5], m[8]);+	MONT_MAC(t0, t1, t2, u[6], m[7]);+	MONT_MAC(v0, v1, v2, u[7], m[6]);+	MONT_MAC(w0, w1, w2, u[8], m[5]);+	MONT_MAC(t0, t1, t2, u[9], m[4]);+	MONT_MAC(v0, v1, v2, u[10], m[3]);+	MONT_MAC(w0, w1, w2, u[11], m[2]);+	MONT_MAC(t0, t1, t2, u[12], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[13] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[13], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 14 */+	MONT_MUL(s0, s1, s2, a[0], a[14]);+	MONT_MUL(c0, c1, c2, a[1], a[13]);+	MONT_MAC(s0, s1, s2, a[2], a[12]);+	MONT_MAC(c0, c1, c2, a[3], a[11]);+	MONT_MAC(s0, s1, s2, a[4], a[10]);+	MONT_MAC(c0, c1, c2, a[5], a[9]);+	MONT_MAC(s0, s1, s2, a[6], a[8]);+	MONT_MAC(t0, t1, t2, a[7], a[7]);+	MONT_MUL(v0, v1, v2, u[0], m[14]);+	MONT_MUL(w0, w1, w2, u[1], m[13]);+	MONT_MAC(t0, t1, t2, u[2], m[12]);+	MONT_MAC(v0, v1, v2, u[3], m[11]);+	MONT_MAC(w0, w1, w2, u[4], m[10]);+	MONT_MAC(t0, t1, t2, u[5], m[9]);+	MONT_MAC(v0, v1, v2, u[6], m[8]);+	MONT_MAC(w0, w1, w2, u[7], m[7]);+	MONT_MAC(t0, t1, t2, u[8], m[6]);+	MONT_MAC(v0, v1, v2, u[9], m[5]);+	MONT_MAC(w0, w1, w2, u[10], m[4]);+	MONT_MAC(t0, t1, t2, u[11], m[3]);+	MONT_MAC(v0, v1, v2, u[12], m[2]);+	MONT_MAC(w0, w1, w2, u[13], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[14] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[14], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 15 */+	MONT_MUL(s0, s1, s2, a[0], a[15]);+	MONT_MUL(c0, c1, c2, a[1], a[14]);+	MONT_MAC(s0, s1, s2, a[2], a[13]);+	MONT_MAC(c0, c1, c2, a[3], a[12]);+	MONT_MAC(s0, s1, s2, a[4], a[11]);+	MONT_MAC(c0, c1, c2, a[5], a[10]);+	MONT_MAC(s0, s1, s2, a[6], a[9]);+	MONT_MAC(c0, c1, c2, a[7], a[8]);+	MONT_MAC(t0, t1, t2, u[0], m[15]);+	MONT_MUL(v0, v1, v2, u[1], m[14]);+	MONT_MUL(w0, w1, w2, u[2], m[13]);+	MONT_MAC(t0, t1, t2, u[3], m[12]);+	MONT_MAC(v0, v1, v2, u[4], m[11]);+	MONT_MAC(w0, w1, w2, u[5], m[10]);+	MONT_MAC(t0, t1, t2, u[6], m[9]);+	MONT_MAC(v0, v1, v2, u[7], m[8]);+	MONT_MAC(w0, w1, w2, u[8], m[7]);+	MONT_MAC(t0, t1, t2, u[9], m[6]);+	MONT_MAC(v0, v1, v2, u[10], m[5]);+	MONT_MAC(w0, w1, w2, u[11], m[4]);+	MONT_MAC(t0, t1, t2, u[12], m[3]);+	MONT_MAC(v0, v1, v2, u[13], m[2]);+	MONT_MAC(w0, w1, w2, u[14], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[15] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[15], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 16 */+	MONT_MUL(s0, s1, s2, a[0], a[16]);+	MONT_MUL(c0, c1, c2, a[1], a[15]);+	MONT_MAC(s0, s1, s2, a[2], a[14]);+	MONT_MAC(c0, c1, c2, a[3], a[13]);+	MONT_MAC(s0, s1, s2, a[4], a[12]);+	MONT_MAC(c0, c1, c2, a[5], a[11]);+	MONT_MAC(s0, s1, s2, a[6], a[10]);+	MONT_MAC(c0, c1, c2, a[7], a[9]);+	MONT_MAC(t0, t1, t2, a[8], a[8]);+	MONT_MUL(v0, v1, v2, u[0], m[16]);+	MONT_MUL(w0, w1, w2, u[1], m[15]);+	MONT_MAC(t0, t1, t2, u[2], m[14]);+	MONT_MAC(v0, v1, v2, u[3], m[13]);+	MONT_MAC(w0, w1, w2, u[4], m[12]);+	MONT_MAC(t0, t1, t2, u[5], m[11]);+	MONT_MAC(v0, v1, v2, u[6], m[10]);+	MONT_MAC(w0, w1, w2, u[7], m[9]);+	MONT_MAC(t0, t1, t2, u[8], m[8]);+	MONT_MAC(v0, v1, v2, u[9], m[7]);+	MONT_MAC(w0, w1, w2, u[10], m[6]);+	MONT_MAC(t0, t1, t2, u[11], m[5]);+	MONT_MAC(v0, v1, v2, u[12], m[4]);+	MONT_MAC(w0, w1, w2, u[13], m[3]);+	MONT_MAC(t0, t1, t2, u[14], m[2]);+	MONT_MAC(v0, v1, v2, u[15], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[16] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[16], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 17 */+	MONT_MUL(s0, s1, s2, a[0], a[17]);+	MONT_MUL(c0, c1, c2, a[1], a[16]);+	MONT_MAC(s0, s1, s2, a[2], a[15]);+	MONT_MAC(c0, c1, c2, a[3], a[14]);+	MONT_MAC(s0, s1, s2, a[4], a[13]);+	MONT_MAC(c0, c1, c2, a[5], a[12]);+	MONT_MAC(s0, s1, s2, a[6], a[11]);+	MONT_MAC(c0, c1, c2, a[7], a[10]);+	MONT_MAC(s0, s1, s2, a[8], a[9]);+	MONT_MAC(t0, t1, t2, u[0], m[17]);+	MONT_MUL(v0, v1, v2, u[1], m[16]);+	MONT_MUL(w0, w1, w2, u[2], m[15]);+	MONT_MAC(t0, t1, t2, u[3], m[14]);+	MONT_MAC(v0, v1, v2, u[4], m[13]);+	MONT_MAC(w0, w1, w2, u[5], m[12]);+	MONT_MAC(t0, t1, t2, u[6], m[11]);+	MONT_MAC(v0, v1, v2, u[7], m[10]);+	MONT_MAC(w0, w1, w2, u[8], m[9]);+	MONT_MAC(t0, t1, t2, u[9], m[8]);+	MONT_MAC(v0, v1, v2, u[10], m[7]);+	MONT_MAC(w0, w1, w2, u[11], m[6]);+	MONT_MAC(t0, t1, t2, u[12], m[5]);+	MONT_MAC(v0, v1, v2, u[13], m[4]);+	MONT_MAC(w0, w1, w2, u[14], m[3]);+	MONT_MAC(t0, t1, t2, u[15], m[2]);+	MONT_MAC(v0, v1, v2, u[16], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[17] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[17], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 18 */+	MONT_MUL(s0, s1, s2, a[0], a[18]);+	MONT_MUL(c0, c1, c2, a[1], a[17]);+	MONT_MAC(s0, s1, s2, a[2], a[16]);+	MONT_MAC(c0, c1, c2, a[3], a[15]);+	MONT_MAC(s0, s1, s2, a[4], a[14]);+	MONT_MAC(c0, c1, c2, a[5], a[13]);+	MONT_MAC(s0, s1, s2, a[6], a[12]);+	MONT_MAC(c0, c1, c2, a[7], a[11]);+	MONT_MAC(s0, s1, s2, a[8], a[10]);+	MONT_MAC(t0, t1, t2, a[9], a[9]);+	MONT_MUL(v0, v1, v2, u[0], m[18]);+	MONT_MUL(w0, w1, w2, u[1], m[17]);+	MONT_MAC(t0, t1, t2, u[2], m[16]);+	MONT_MAC(v0, v1, v2, u[3], m[15]);+	MONT_MAC(w0, w1, w2, u[4], m[14]);+	MONT_MAC(t0, t1, t2, u[5], m[13]);+	MONT_MAC(v0, v1, v2, u[6], m[12]);+	MONT_MAC(w0, w1, w2, u[7], m[11]);+	MONT_MAC(t0, t1, t2, u[8], m[10]);+	MONT_MAC(v0, v1, v2, u[9], m[9]);+	MONT_MAC(w0, w1, w2, u[10], m[8]);+	MONT_MAC(t0, t1, t2, u[11], m[7]);+	MONT_MAC(v0, v1, v2, u[12], m[6]);+	MONT_MAC(w0, w1, w2, u[13], m[5]);+	MONT_MAC(t0, t1, t2, u[14], m[4]);+	MONT_MAC(v0, v1, v2, u[15], m[3]);+	MONT_MAC(w0, w1, w2, u[16], m[2]);+	MONT_MAC(t0, t1, t2, u[17], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[18] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[18], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 19 */+	MONT_MUL(s0, s1, s2, a[0], a[19]);+	MONT_MUL(c0, c1, c2, a[1], a[18]);+	MONT_MAC(s0, s1, s2, a[2], a[17]);+	MONT_MAC(c0, c1, c2, a[3], a[16]);+	MONT_MAC(s0, s1, s2, a[4], a[15]);+	MONT_MAC(c0, c1, c2, a[5], a[14]);+	MONT_MAC(s0, s1, s2, a[6], a[13]);+	MONT_MAC(c0, c1, c2, a[7], a[12]);+	MONT_MAC(s0, s1, s2, a[8], a[11]);+	MONT_MAC(c0, c1, c2, a[9], a[10]);+	MONT_MAC(t0, t1, t2, u[0], m[19]);+	MONT_MUL(v0, v1, v2, u[1], m[18]);+	MONT_MUL(w0, w1, w2, u[2], m[17]);+	MONT_MAC(t0, t1, t2, u[3], m[16]);+	MONT_MAC(v0, v1, v2, u[4], m[15]);+	MONT_MAC(w0, w1, w2, u[5], m[14]);+	MONT_MAC(t0, t1, t2, u[6], m[13]);+	MONT_MAC(v0, v1, v2, u[7], m[12]);+	MONT_MAC(w0, w1, w2, u[8], m[11]);+	MONT_MAC(t0, t1, t2, u[9], m[10]);+	MONT_MAC(v0, v1, v2, u[10], m[9]);+	MONT_MAC(w0, w1, w2, u[11], m[8]);+	MONT_MAC(t0, t1, t2, u[12], m[7]);+	MONT_MAC(v0, v1, v2, u[13], m[6]);+	MONT_MAC(w0, w1, w2, u[14], m[5]);+	MONT_MAC(t0, t1, t2, u[15], m[4]);+	MONT_MAC(v0, v1, v2, u[16], m[3]);+	MONT_MAC(w0, w1, w2, u[17], m[2]);+	MONT_MAC(t0, t1, t2, u[18], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[19] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[19], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 20 */+	MONT_MUL(s0, s1, s2, a[0], a[20]);+	MONT_MUL(c0, c1, c2, a[1], a[19]);+	MONT_MAC(s0, s1, s2, a[2], a[18]);+	MONT_MAC(c0, c1, c2, a[3], a[17]);+	MONT_MAC(s0, s1, s2, a[4], a[16]);+	MONT_MAC(c0, c1, c2, a[5], a[15]);+	MONT_MAC(s0, s1, s2, a[6], a[14]);+	MONT_MAC(c0, c1, c2, a[7], a[13]);+	MONT_MAC(s0, s1, s2, a[8], a[12]);+	MONT_MAC(c0, c1, c2, a[9], a[11]);+	MONT_MAC(t0, t1, t2, a[10], a[10]);+	MONT_MUL(v0, v1, v2, u[0], m[20]);+	MONT_MUL(w0, w1, w2, u[1], m[19]);+	MONT_MAC(t0, t1, t2, u[2], m[18]);+	MONT_MAC(v0, v1, v2, u[3], m[17]);+	MONT_MAC(w0, w1, w2, u[4], m[16]);+	MONT_MAC(t0, t1, t2, u[5], m[15]);+	MONT_MAC(v0, v1, v2, u[6], m[14]);+	MONT_MAC(w0, w1, w2, u[7], m[13]);+	MONT_MAC(t0, t1, t2, u[8], m[12]);+	MONT_MAC(v0, v1, v2, u[9], m[11]);+	MONT_MAC(w0, w1, w2, u[10], m[10]);+	MONT_MAC(t0, t1, t2, u[11], m[9]);+	MONT_MAC(v0, v1, v2, u[12], m[8]);+	MONT_MAC(w0, w1, w2, u[13], m[7]);+	MONT_MAC(t0, t1, t2, u[14], m[6]);+	MONT_MAC(v0, v1, v2, u[15], m[5]);+	MONT_MAC(w0, w1, w2, u[16], m[4]);+	MONT_MAC(t0, t1, t2, u[17], m[3]);+	MONT_MAC(v0, v1, v2, u[18], m[2]);+	MONT_MAC(w0, w1, w2, u[19], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[20] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[20], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 21 */+	MONT_MUL(s0, s1, s2, a[0], a[21]);+	MONT_MUL(c0, c1, c2, a[1], a[20]);+	MONT_MAC(s0, s1, s2, a[2], a[19]);+	MONT_MAC(c0, c1, c2, a[3], a[18]);+	MONT_MAC(s0, s1, s2, a[4], a[17]);+	MONT_MAC(c0, c1, c2, a[5], a[16]);+	MONT_MAC(s0, s1, s2, a[6], a[15]);+	MONT_MAC(c0, c1, c2, a[7], a[14]);+	MONT_MAC(s0, s1, s2, a[8], a[13]);+	MONT_MAC(c0, c1, c2, a[9], a[12]);+	MONT_MAC(s0, s1, s2, a[10], a[11]);+	MONT_MAC(t0, t1, t2, u[0], m[21]);+	MONT_MUL(v0, v1, v2, u[1], m[20]);+	MONT_MUL(w0, w1, w2, u[2], m[19]);+	MONT_MAC(t0, t1, t2, u[3], m[18]);+	MONT_MAC(v0, v1, v2, u[4], m[17]);+	MONT_MAC(w0, w1, w2, u[5], m[16]);+	MONT_MAC(t0, t1, t2, u[6], m[15]);+	MONT_MAC(v0, v1, v2, u[7], m[14]);+	MONT_MAC(w0, w1, w2, u[8], m[13]);+	MONT_MAC(t0, t1, t2, u[9], m[12]);+	MONT_MAC(v0, v1, v2, u[10], m[11]);+	MONT_MAC(w0, w1, w2, u[11], m[10]);+	MONT_MAC(t0, t1, t2, u[12], m[9]);+	MONT_MAC(v0, v1, v2, u[13], m[8]);+	MONT_MAC(w0, w1, w2, u[14], m[7]);+	MONT_MAC(t0, t1, t2, u[15], m[6]);+	MONT_MAC(v0, v1, v2, u[16], m[5]);+	MONT_MAC(w0, w1, w2, u[17], m[4]);+	MONT_MAC(t0, t1, t2, u[18], m[3]);+	MONT_MAC(v0, v1, v2, u[19], m[2]);+	MONT_MAC(w0, w1, w2, u[20], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[21] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[21], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 22 */+	MONT_MUL(s0, s1, s2, a[0], a[22]);+	MONT_MUL(c0, c1, c2, a[1], a[21]);+	MONT_MAC(s0, s1, s2, a[2], a[20]);+	MONT_MAC(c0, c1, c2, a[3], a[19]);+	MONT_MAC(s0, s1, s2, a[4], a[18]);+	MONT_MAC(c0, c1, c2, a[5], a[17]);+	MONT_MAC(s0, s1, s2, a[6], a[16]);+	MONT_MAC(c0, c1, c2, a[7], a[15]);+	MONT_MAC(s0, s1, s2, a[8], a[14]);+	MONT_MAC(c0, c1, c2, a[9], a[13]);+	MONT_MAC(s0, s1, s2, a[10], a[12]);+	MONT_MAC(t0, t1, t2, a[11], a[11]);+	MONT_MUL(v0, v1, v2, u[0], m[22]);+	MONT_MUL(w0, w1, w2, u[1], m[21]);+	MONT_MAC(t0, t1, t2, u[2], m[20]);+	MONT_MAC(v0, v1, v2, u[3], m[19]);+	MONT_MAC(w0, w1, w2, u[4], m[18]);+	MONT_MAC(t0, t1, t2, u[5], m[17]);+	MONT_MAC(v0, v1, v2, u[6], m[16]);+	MONT_MAC(w0, w1, w2, u[7], m[15]);+	MONT_MAC(t0, t1, t2, u[8], m[14]);+	MONT_MAC(v0, v1, v2, u[9], m[13]);+	MONT_MAC(w0, w1, w2, u[10], m[12]);+	MONT_MAC(t0, t1, t2, u[11], m[11]);+	MONT_MAC(v0, v1, v2, u[12], m[10]);+	MONT_MAC(w0, w1, w2, u[13], m[9]);+	MONT_MAC(t0, t1, t2, u[14], m[8]);+	MONT_MAC(v0, v1, v2, u[15], m[7]);+	MONT_MAC(w0, w1, w2, u[16], m[6]);+	MONT_MAC(t0, t1, t2, u[17], m[5]);+	MONT_MAC(v0, v1, v2, u[18], m[4]);+	MONT_MAC(w0, w1, w2, u[19], m[3]);+	MONT_MAC(t0, t1, t2, u[20], m[2]);+	MONT_MAC(v0, v1, v2, u[21], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[22] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[22], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 23 */+	MONT_MUL(s0, s1, s2, a[0], a[23]);+	MONT_MUL(c0, c1, c2, a[1], a[22]);+	MONT_MAC(s0, s1, s2, a[2], a[21]);+	MONT_MAC(c0, c1, c2, a[3], a[20]);+	MONT_MAC(s0, s1, s2, a[4], a[19]);+	MONT_MAC(c0, c1, c2, a[5], a[18]);+	MONT_MAC(s0, s1, s2, a[6], a[17]);+	MONT_MAC(c0, c1, c2, a[7], a[16]);+	MONT_MAC(s0, s1, s2, a[8], a[15]);+	MONT_MAC(c0, c1, c2, a[9], a[14]);+	MONT_MAC(s0, s1, s2, a[10], a[13]);+	MONT_MAC(c0, c1, c2, a[11], a[12]);+	MONT_MAC(t0, t1, t2, u[0], m[23]);+	MONT_MUL(v0, v1, v2, u[1], m[22]);+	MONT_MUL(w0, w1, w2, u[2], m[21]);+	MONT_MAC(t0, t1, t2, u[3], m[20]);+	MONT_MAC(v0, v1, v2, u[4], m[19]);+	MONT_MAC(w0, w1, w2, u[5], m[18]);+	MONT_MAC(t0, t1, t2, u[6], m[17]);+	MONT_MAC(v0, v1, v2, u[7], m[16]);+	MONT_MAC(w0, w1, w2, u[8], m[15]);+	MONT_MAC(t0, t1, t2, u[9], m[14]);+	MONT_MAC(v0, v1, v2, u[10], m[13]);+	MONT_MAC(w0, w1, w2, u[11], m[12]);+	MONT_MAC(t0, t1, t2, u[12], m[11]);+	MONT_MAC(v0, v1, v2, u[13], m[10]);+	MONT_MAC(w0, w1, w2, u[14], m[9]);+	MONT_MAC(t0, t1, t2, u[15], m[8]);+	MONT_MAC(v0, v1, v2, u[16], m[7]);+	MONT_MAC(w0, w1, w2, u[17], m[6]);+	MONT_MAC(t0, t1, t2, u[18], m[5]);+	MONT_MAC(v0, v1, v2, u[19], m[4]);+	MONT_MAC(w0, w1, w2, u[20], m[3]);+	MONT_MAC(t0, t1, t2, u[21], m[2]);+	MONT_MAC(v0, v1, v2, u[22], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[23] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[23], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 24 */+	MONT_MUL(s0, s1, s2, a[1], a[23]);+	MONT_MUL(c0, c1, c2, a[2], a[22]);+	MONT_MAC(s0, s1, s2, a[3], a[21]);+	MONT_MAC(c0, c1, c2, a[4], a[20]);+	MONT_MAC(s0, s1, s2, a[5], a[19]);+	MONT_MAC(c0, c1, c2, a[6], a[18]);+	MONT_MAC(s0, s1, s2, a[7], a[17]);+	MONT_MAC(c0, c1, c2, a[8], a[16]);+	MONT_MAC(s0, s1, s2, a[9], a[15]);+	MONT_MAC(c0, c1, c2, a[10], a[14]);+	MONT_MAC(s0, s1, s2, a[11], a[13]);+	MONT_MAC(t0, t1, t2, a[12], a[12]);+	MONT_MUL(v0, v1, v2, u[1], m[23]);+	MONT_MUL(w0, w1, w2, u[2], m[22]);+	MONT_MAC(t0, t1, t2, u[3], m[21]);+	MONT_MAC(v0, v1, v2, u[4], m[20]);+	MONT_MAC(w0, w1, w2, u[5], m[19]);+	MONT_MAC(t0, t1, t2, u[6], m[18]);+	MONT_MAC(v0, v1, v2, u[7], m[17]);+	MONT_MAC(w0, w1, w2, u[8], m[16]);+	MONT_MAC(t0, t1, t2, u[9], m[15]);+	MONT_MAC(v0, v1, v2, u[10], m[14]);+	MONT_MAC(w0, w1, w2, u[11], m[13]);+	MONT_MAC(t0, t1, t2, u[12], m[12]);+	MONT_MAC(v0, v1, v2, u[13], m[11]);+	MONT_MAC(w0, w1, w2, u[14], m[10]);+	MONT_MAC(t0, t1, t2, u[15], m[9]);+	MONT_MAC(v0, v1, v2, u[16], m[8]);+	MONT_MAC(w0, w1, w2, u[17], m[7]);+	MONT_MAC(t0, t1, t2, u[18], m[6]);+	MONT_MAC(v0, v1, v2, u[19], m[5]);+	MONT_MAC(w0, w1, w2, u[20], m[4]);+	MONT_MAC(t0, t1, t2, u[21], m[3]);+	MONT_MAC(v0, v1, v2, u[22], m[2]);+	MONT_MAC(w0, w1, w2, u[23], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[0] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 25 */+	MONT_MUL(s0, s1, s2, a[2], a[23]);+	MONT_MUL(c0, c1, c2, a[3], a[22]);+	MONT_MAC(s0, s1, s2, a[4], a[21]);+	MONT_MAC(c0, c1, c2, a[5], a[20]);+	MONT_MAC(s0, s1, s2, a[6], a[19]);+	MONT_MAC(c0, c1, c2, a[7], a[18]);+	MONT_MAC(s0, s1, s2, a[8], a[17]);+	MONT_MAC(c0, c1, c2, a[9], a[16]);+	MONT_MAC(s0, s1, s2, a[10], a[15]);+	MONT_MAC(c0, c1, c2, a[11], a[14]);+	MONT_MAC(s0, s1, s2, a[12], a[13]);+	MONT_MAC(t0, t1, t2, u[2], m[23]);+	MONT_MUL(v0, v1, v2, u[3], m[22]);+	MONT_MUL(w0, w1, w2, u[4], m[21]);+	MONT_MAC(t0, t1, t2, u[5], m[20]);+	MONT_MAC(v0, v1, v2, u[6], m[19]);+	MONT_MAC(w0, w1, w2, u[7], m[18]);+	MONT_MAC(t0, t1, t2, u[8], m[17]);+	MONT_MAC(v0, v1, v2, u[9], m[16]);+	MONT_MAC(w0, w1, w2, u[10], m[15]);+	MONT_MAC(t0, t1, t2, u[11], m[14]);+	MONT_MAC(v0, v1, v2, u[12], m[13]);+	MONT_MAC(w0, w1, w2, u[13], m[12]);+	MONT_MAC(t0, t1, t2, u[14], m[11]);+	MONT_MAC(v0, v1, v2, u[15], m[10]);+	MONT_MAC(w0, w1, w2, u[16], m[9]);+	MONT_MAC(t0, t1, t2, u[17], m[8]);+	MONT_MAC(v0, v1, v2, u[18], m[7]);+	MONT_MAC(w0, w1, w2, u[19], m[6]);+	MONT_MAC(t0, t1, t2, u[20], m[5]);+	MONT_MAC(v0, v1, v2, u[21], m[4]);+	MONT_MAC(w0, w1, w2, u[22], m[3]);+	MONT_MAC(t0, t1, t2, u[23], m[2]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[1] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 26 */+	MONT_MUL(s0, s1, s2, a[3], a[23]);+	MONT_MUL(c0, c1, c2, a[4], a[22]);+	MONT_MAC(s0, s1, s2, a[5], a[21]);+	MONT_MAC(c0, c1, c2, a[6], a[20]);+	MONT_MAC(s0, s1, s2, a[7], a[19]);+	MONT_MAC(c0, c1, c2, a[8], a[18]);+	MONT_MAC(s0, s1, s2, a[9], a[17]);+	MONT_MAC(c0, c1, c2, a[10], a[16]);+	MONT_MAC(s0, s1, s2, a[11], a[15]);+	MONT_MAC(c0, c1, c2, a[12], a[14]);+	MONT_MAC(t0, t1, t2, a[13], a[13]);+	MONT_MUL(v0, v1, v2, u[3], m[23]);+	MONT_MUL(w0, w1, w2, u[4], m[22]);+	MONT_MAC(t0, t1, t2, u[5], m[21]);+	MONT_MAC(v0, v1, v2, u[6], m[20]);+	MONT_MAC(w0, w1, w2, u[7], m[19]);+	MONT_MAC(t0, t1, t2, u[8], m[18]);+	MONT_MAC(v0, v1, v2, u[9], m[17]);+	MONT_MAC(w0, w1, w2, u[10], m[16]);+	MONT_MAC(t0, t1, t2, u[11], m[15]);+	MONT_MAC(v0, v1, v2, u[12], m[14]);+	MONT_MAC(w0, w1, w2, u[13], m[13]);+	MONT_MAC(t0, t1, t2, u[14], m[12]);+	MONT_MAC(v0, v1, v2, u[15], m[11]);+	MONT_MAC(w0, w1, w2, u[16], m[10]);+	MONT_MAC(t0, t1, t2, u[17], m[9]);+	MONT_MAC(v0, v1, v2, u[18], m[8]);+	MONT_MAC(w0, w1, w2, u[19], m[7]);+	MONT_MAC(t0, t1, t2, u[20], m[6]);+	MONT_MAC(v0, v1, v2, u[21], m[5]);+	MONT_MAC(w0, w1, w2, u[22], m[4]);+	MONT_MAC(t0, t1, t2, u[23], m[3]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[2] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 27 */+	MONT_MUL(s0, s1, s2, a[4], a[23]);+	MONT_MUL(c0, c1, c2, a[5], a[22]);+	MONT_MAC(s0, s1, s2, a[6], a[21]);+	MONT_MAC(c0, c1, c2, a[7], a[20]);+	MONT_MAC(s0, s1, s2, a[8], a[19]);+	MONT_MAC(c0, c1, c2, a[9], a[18]);+	MONT_MAC(s0, s1, s2, a[10], a[17]);+	MONT_MAC(c0, c1, c2, a[11], a[16]);+	MONT_MAC(s0, s1, s2, a[12], a[15]);+	MONT_MAC(c0, c1, c2, a[13], a[14]);+	MONT_MAC(t0, t1, t2, u[4], m[23]);+	MONT_MUL(v0, v1, v2, u[5], m[22]);+	MONT_MUL(w0, w1, w2, u[6], m[21]);+	MONT_MAC(t0, t1, t2, u[7], m[20]);+	MONT_MAC(v0, v1, v2, u[8], m[19]);+	MONT_MAC(w0, w1, w2, u[9], m[18]);+	MONT_MAC(t0, t1, t2, u[10], m[17]);+	MONT_MAC(v0, v1, v2, u[11], m[16]);+	MONT_MAC(w0, w1, w2, u[12], m[15]);+	MONT_MAC(t0, t1, t2, u[13], m[14]);+	MONT_MAC(v0, v1, v2, u[14], m[13]);+	MONT_MAC(w0, w1, w2, u[15], m[12]);+	MONT_MAC(t0, t1, t2, u[16], m[11]);+	MONT_MAC(v0, v1, v2, u[17], m[10]);+	MONT_MAC(w0, w1, w2, u[18], m[9]);+	MONT_MAC(t0, t1, t2, u[19], m[8]);+	MONT_MAC(v0, v1, v2, u[20], m[7]);+	MONT_MAC(w0, w1, w2, u[21], m[6]);+	MONT_MAC(t0, t1, t2, u[22], m[5]);+	MONT_MAC(v0, v1, v2, u[23], m[4]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[3] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 28 */+	MONT_MUL(s0, s1, s2, a[5], a[23]);+	MONT_MUL(c0, c1, c2, a[6], a[22]);+	MONT_MAC(s0, s1, s2, a[7], a[21]);+	MONT_MAC(c0, c1, c2, a[8], a[20]);+	MONT_MAC(s0, s1, s2, a[9], a[19]);+	MONT_MAC(c0, c1, c2, a[10], a[18]);+	MONT_MAC(s0, s1, s2, a[11], a[17]);+	MONT_MAC(c0, c1, c2, a[12], a[16]);+	MONT_MAC(s0, s1, s2, a[13], a[15]);+	MONT_MAC(t0, t1, t2, a[14], a[14]);+	MONT_MUL(v0, v1, v2, u[5], m[23]);+	MONT_MUL(w0, w1, w2, u[6], m[22]);+	MONT_MAC(t0, t1, t2, u[7], m[21]);+	MONT_MAC(v0, v1, v2, u[8], m[20]);+	MONT_MAC(w0, w1, w2, u[9], m[19]);+	MONT_MAC(t0, t1, t2, u[10], m[18]);+	MONT_MAC(v0, v1, v2, u[11], m[17]);+	MONT_MAC(w0, w1, w2, u[12], m[16]);+	MONT_MAC(t0, t1, t2, u[13], m[15]);+	MONT_MAC(v0, v1, v2, u[14], m[14]);+	MONT_MAC(w0, w1, w2, u[15], m[13]);+	MONT_MAC(t0, t1, t2, u[16], m[12]);+	MONT_MAC(v0, v1, v2, u[17], m[11]);+	MONT_MAC(w0, w1, w2, u[18], m[10]);+	MONT_MAC(t0, t1, t2, u[19], m[9]);+	MONT_MAC(v0, v1, v2, u[20], m[8]);+	MONT_MAC(w0, w1, w2, u[21], m[7]);+	MONT_MAC(t0, t1, t2, u[22], m[6]);+	MONT_MAC(v0, v1, v2, u[23], m[5]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[4] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 29 */+	MONT_MUL(s0, s1, s2, a[6], a[23]);+	MONT_MUL(c0, c1, c2, a[7], a[22]);+	MONT_MAC(s0, s1, s2, a[8], a[21]);+	MONT_MAC(c0, c1, c2, a[9], a[20]);+	MONT_MAC(s0, s1, s2, a[10], a[19]);+	MONT_MAC(c0, c1, c2, a[11], a[18]);+	MONT_MAC(s0, s1, s2, a[12], a[17]);+	MONT_MAC(c0, c1, c2, a[13], a[16]);+	MONT_MAC(s0, s1, s2, a[14], a[15]);+	MONT_MAC(t0, t1, t2, u[6], m[23]);+	MONT_MUL(v0, v1, v2, u[7], m[22]);+	MONT_MUL(w0, w1, w2, u[8], m[21]);+	MONT_MAC(t0, t1, t2, u[9], m[20]);+	MONT_MAC(v0, v1, v2, u[10], m[19]);+	MONT_MAC(w0, w1, w2, u[11], m[18]);+	MONT_MAC(t0, t1, t2, u[12], m[17]);+	MONT_MAC(v0, v1, v2, u[13], m[16]);+	MONT_MAC(w0, w1, w2, u[14], m[15]);+	MONT_MAC(t0, t1, t2, u[15], m[14]);+	MONT_MAC(v0, v1, v2, u[16], m[13]);+	MONT_MAC(w0, w1, w2, u[17], m[12]);+	MONT_MAC(t0, t1, t2, u[18], m[11]);+	MONT_MAC(v0, v1, v2, u[19], m[10]);+	MONT_MAC(w0, w1, w2, u[20], m[9]);+	MONT_MAC(t0, t1, t2, u[21], m[8]);+	MONT_MAC(v0, v1, v2, u[22], m[7]);+	MONT_MAC(w0, w1, w2, u[23], m[6]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[5] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 30 */+	MONT_MUL(s0, s1, s2, a[7], a[23]);+	MONT_MUL(c0, c1, c2, a[8], a[22]);+	MONT_MAC(s0, s1, s2, a[9], a[21]);+	MONT_MAC(c0, c1, c2, a[10], a[20]);+	MONT_MAC(s0, s1, s2, a[11], a[19]);+	MONT_MAC(c0, c1, c2, a[12], a[18]);+	MONT_MAC(s0, s1, s2, a[13], a[17]);+	MONT_MAC(c0, c1, c2, a[14], a[16]);+	MONT_MAC(t0, t1, t2, a[15], a[15]);+	MONT_MUL(v0, v1, v2, u[7], m[23]);+	MONT_MUL(w0, w1, w2, u[8], m[22]);+	MONT_MAC(t0, t1, t2, u[9], m[21]);+	MONT_MAC(v0, v1, v2, u[10], m[20]);+	MONT_MAC(w0, w1, w2, u[11], m[19]);+	MONT_MAC(t0, t1, t2, u[12], m[18]);+	MONT_MAC(v0, v1, v2, u[13], m[17]);+	MONT_MAC(w0, w1, w2, u[14], m[16]);+	MONT_MAC(t0, t1, t2, u[15], m[15]);+	MONT_MAC(v0, v1, v2, u[16], m[14]);+	MONT_MAC(w0, w1, w2, u[17], m[13]);+	MONT_MAC(t0, t1, t2, u[18], m[12]);+	MONT_MAC(v0, v1, v2, u[19], m[11]);+	MONT_MAC(w0, w1, w2, u[20], m[10]);+	MONT_MAC(t0, t1, t2, u[21], m[9]);+	MONT_MAC(v0, v1, v2, u[22], m[8]);+	MONT_MAC(w0, w1, w2, u[23], m[7]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[6] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 31 */+	MONT_MUL(s0, s1, s2, a[8], a[23]);+	MONT_MUL(c0, c1, c2, a[9], a[22]);+	MONT_MAC(s0, s1, s2, a[10], a[21]);+	MONT_MAC(c0, c1, c2, a[11], a[20]);+	MONT_MAC(s0, s1, s2, a[12], a[19]);+	MONT_MAC(c0, c1, c2, a[13], a[18]);+	MONT_MAC(s0, s1, s2, a[14], a[17]);+	MONT_MAC(c0, c1, c2, a[15], a[16]);+	MONT_MAC(t0, t1, t2, u[8], m[23]);+	MONT_MUL(v0, v1, v2, u[9], m[22]);+	MONT_MUL(w0, w1, w2, u[10], m[21]);+	MONT_MAC(t0, t1, t2, u[11], m[20]);+	MONT_MAC(v0, v1, v2, u[12], m[19]);+	MONT_MAC(w0, w1, w2, u[13], m[18]);+	MONT_MAC(t0, t1, t2, u[14], m[17]);+	MONT_MAC(v0, v1, v2, u[15], m[16]);+	MONT_MAC(w0, w1, w2, u[16], m[15]);+	MONT_MAC(t0, t1, t2, u[17], m[14]);+	MONT_MAC(v0, v1, v2, u[18], m[13]);+	MONT_MAC(w0, w1, w2, u[19], m[12]);+	MONT_MAC(t0, t1, t2, u[20], m[11]);+	MONT_MAC(v0, v1, v2, u[21], m[10]);+	MONT_MAC(w0, w1, w2, u[22], m[9]);+	MONT_MAC(t0, t1, t2, u[23], m[8]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[7] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 32 */+	MONT_MUL(s0, s1, s2, a[9], a[23]);+	MONT_MUL(c0, c1, c2, a[10], a[22]);+	MONT_MAC(s0, s1, s2, a[11], a[21]);+	MONT_MAC(c0, c1, c2, a[12], a[20]);+	MONT_MAC(s0, s1, s2, a[13], a[19]);+	MONT_MAC(c0, c1, c2, a[14], a[18]);+	MONT_MAC(s0, s1, s2, a[15], a[17]);+	MONT_MAC(t0, t1, t2, a[16], a[16]);+	MONT_MUL(v0, v1, v2, u[9], m[23]);+	MONT_MUL(w0, w1, w2, u[10], m[22]);+	MONT_MAC(t0, t1, t2, u[11], m[21]);+	MONT_MAC(v0, v1, v2, u[12], m[20]);+	MONT_MAC(w0, w1, w2, u[13], m[19]);+	MONT_MAC(t0, t1, t2, u[14], m[18]);+	MONT_MAC(v0, v1, v2, u[15], m[17]);+	MONT_MAC(w0, w1, w2, u[16], m[16]);+	MONT_MAC(t0, t1, t2, u[17], m[15]);+	MONT_MAC(v0, v1, v2, u[18], m[14]);+	MONT_MAC(w0, w1, w2, u[19], m[13]);+	MONT_MAC(t0, t1, t2, u[20], m[12]);+	MONT_MAC(v0, v1, v2, u[21], m[11]);+	MONT_MAC(w0, w1, w2, u[22], m[10]);+	MONT_MAC(t0, t1, t2, u[23], m[9]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[8] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 33 */+	MONT_MUL(s0, s1, s2, a[10], a[23]);+	MONT_MUL(c0, c1, c2, a[11], a[22]);+	MONT_MAC(s0, s1, s2, a[12], a[21]);+	MONT_MAC(c0, c1, c2, a[13], a[20]);+	MONT_MAC(s0, s1, s2, a[14], a[19]);+	MONT_MAC(c0, c1, c2, a[15], a[18]);+	MONT_MAC(s0, s1, s2, a[16], a[17]);+	MONT_MAC(t0, t1, t2, u[10], m[23]);+	MONT_MUL(v0, v1, v2, u[11], m[22]);+	MONT_MUL(w0, w1, w2, u[12], m[21]);+	MONT_MAC(t0, t1, t2, u[13], m[20]);+	MONT_MAC(v0, v1, v2, u[14], m[19]);+	MONT_MAC(w0, w1, w2, u[15], m[18]);+	MONT_MAC(t0, t1, t2, u[16], m[17]);+	MONT_MAC(v0, v1, v2, u[17], m[16]);+	MONT_MAC(w0, w1, w2, u[18], m[15]);+	MONT_MAC(t0, t1, t2, u[19], m[14]);+	MONT_MAC(v0, v1, v2, u[20], m[13]);+	MONT_MAC(w0, w1, w2, u[21], m[12]);+	MONT_MAC(t0, t1, t2, u[22], m[11]);+	MONT_MAC(v0, v1, v2, u[23], m[10]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[9] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 34 */+	MONT_MUL(s0, s1, s2, a[11], a[23]);+	MONT_MUL(c0, c1, c2, a[12], a[22]);+	MONT_MAC(s0, s1, s2, a[13], a[21]);+	MONT_MAC(c0, c1, c2, a[14], a[20]);+	MONT_MAC(s0, s1, s2, a[15], a[19]);+	MONT_MAC(c0, c1, c2, a[16], a[18]);+	MONT_MAC(t0, t1, t2, a[17], a[17]);+	MONT_MUL(v0, v1, v2, u[11], m[23]);+	MONT_MUL(w0, w1, w2, u[12], m[22]);+	MONT_MAC(t0, t1, t2, u[13], m[21]);+	MONT_MAC(v0, v1, v2, u[14], m[20]);+	MONT_MAC(w0, w1, w2, u[15], m[19]);+	MONT_MAC(t0, t1, t2, u[16], m[18]);+	MONT_MAC(v0, v1, v2, u[17], m[17]);+	MONT_MAC(w0, w1, w2, u[18], m[16]);+	MONT_MAC(t0, t1, t2, u[19], m[15]);+	MONT_MAC(v0, v1, v2, u[20], m[14]);+	MONT_MAC(w0, w1, w2, u[21], m[13]);+	MONT_MAC(t0, t1, t2, u[22], m[12]);+	MONT_MAC(v0, v1, v2, u[23], m[11]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[10] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 35 */+	MONT_MUL(s0, s1, s2, a[12], a[23]);+	MONT_MUL(c0, c1, c2, a[13], a[22]);+	MONT_MAC(s0, s1, s2, a[14], a[21]);+	MONT_MAC(c0, c1, c2, a[15], a[20]);+	MONT_MAC(s0, s1, s2, a[16], a[19]);+	MONT_MAC(c0, c1, c2, a[17], a[18]);+	MONT_MAC(t0, t1, t2, u[12], m[23]);+	MONT_MUL(v0, v1, v2, u[13], m[22]);+	MONT_MUL(w0, w1, w2, u[14], m[21]);+	MONT_MAC(t0, t1, t2, u[15], m[20]);+	MONT_MAC(v0, v1, v2, u[16], m[19]);+	MONT_MAC(w0, w1, w2, u[17], m[18]);+	MONT_MAC(t0, t1, t2, u[18], m[17]);+	MONT_MAC(v0, v1, v2, u[19], m[16]);+	MONT_MAC(w0, w1, w2, u[20], m[15]);+	MONT_MAC(t0, t1, t2, u[21], m[14]);+	MONT_MAC(v0, v1, v2, u[22], m[13]);+	MONT_MAC(w0, w1, w2, u[23], m[12]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[11] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 36 */+	MONT_MUL(s0, s1, s2, a[13], a[23]);+	MONT_MUL(c0, c1, c2, a[14], a[22]);+	MONT_MAC(s0, s1, s2, a[15], a[21]);+	MONT_MAC(c0, c1, c2, a[16], a[20]);+	MONT_MAC(s0, s1, s2, a[17], a[19]);+	MONT_MAC(t0, t1, t2, a[18], a[18]);+	MONT_MUL(v0, v1, v2, u[13], m[23]);+	MONT_MUL(w0, w1, w2, u[14], m[22]);+	MONT_MAC(t0, t1, t2, u[15], m[21]);+	MONT_MAC(v0, v1, v2, u[16], m[20]);+	MONT_MAC(w0, w1, w2, u[17], m[19]);+	MONT_MAC(t0, t1, t2, u[18], m[18]);+	MONT_MAC(v0, v1, v2, u[19], m[17]);+	MONT_MAC(w0, w1, w2, u[20], m[16]);+	MONT_MAC(t0, t1, t2, u[21], m[15]);+	MONT_MAC(v0, v1, v2, u[22], m[14]);+	MONT_MAC(w0, w1, w2, u[23], m[13]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[12] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 37 */+	MONT_MUL(s0, s1, s2, a[14], a[23]);+	MONT_MUL(c0, c1, c2, a[15], a[22]);+	MONT_MAC(s0, s1, s2, a[16], a[21]);+	MONT_MAC(c0, c1, c2, a[17], a[20]);+	MONT_MAC(s0, s1, s2, a[18], a[19]);+	MONT_MAC(t0, t1, t2, u[14], m[23]);+	MONT_MUL(v0, v1, v2, u[15], m[22]);+	MONT_MUL(w0, w1, w2, u[16], m[21]);+	MONT_MAC(t0, t1, t2, u[17], m[20]);+	MONT_MAC(v0, v1, v2, u[18], m[19]);+	MONT_MAC(w0, w1, w2, u[19], m[18]);+	MONT_MAC(t0, t1, t2, u[20], m[17]);+	MONT_MAC(v0, v1, v2, u[21], m[16]);+	MONT_MAC(w0, w1, w2, u[22], m[15]);+	MONT_MAC(t0, t1, t2, u[23], m[14]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[13] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 38 */+	MONT_MUL(s0, s1, s2, a[15], a[23]);+	MONT_MUL(c0, c1, c2, a[16], a[22]);+	MONT_MAC(s0, s1, s2, a[17], a[21]);+	MONT_MAC(c0, c1, c2, a[18], a[20]);+	MONT_MAC(t0, t1, t2, a[19], a[19]);+	MONT_MUL(v0, v1, v2, u[15], m[23]);+	MONT_MUL(w0, w1, w2, u[16], m[22]);+	MONT_MAC(t0, t1, t2, u[17], m[21]);+	MONT_MAC(v0, v1, v2, u[18], m[20]);+	MONT_MAC(w0, w1, w2, u[19], m[19]);+	MONT_MAC(t0, t1, t2, u[20], m[18]);+	MONT_MAC(v0, v1, v2, u[21], m[17]);+	MONT_MAC(w0, w1, w2, u[22], m[16]);+	MONT_MAC(t0, t1, t2, u[23], m[15]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[14] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 39 */+	MONT_MUL(s0, s1, s2, a[16], a[23]);+	MONT_MUL(c0, c1, c2, a[17], a[22]);+	MONT_MAC(s0, s1, s2, a[18], a[21]);+	MONT_MAC(c0, c1, c2, a[19], a[20]);+	MONT_MAC(t0, t1, t2, u[16], m[23]);+	MONT_MUL(v0, v1, v2, u[17], m[22]);+	MONT_MUL(w0, w1, w2, u[18], m[21]);+	MONT_MAC(t0, t1, t2, u[19], m[20]);+	MONT_MAC(v0, v1, v2, u[20], m[19]);+	MONT_MAC(w0, w1, w2, u[21], m[18]);+	MONT_MAC(t0, t1, t2, u[22], m[17]);+	MONT_MAC(v0, v1, v2, u[23], m[16]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[15] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 40 */+	MONT_MUL(s0, s1, s2, a[17], a[23]);+	MONT_MUL(c0, c1, c2, a[18], a[22]);+	MONT_MAC(s0, s1, s2, a[19], a[21]);+	MONT_MAC(t0, t1, t2, a[20], a[20]);+	MONT_MUL(v0, v1, v2, u[17], m[23]);+	MONT_MUL(w0, w1, w2, u[18], m[22]);+	MONT_MAC(t0, t1, t2, u[19], m[21]);+	MONT_MAC(v0, v1, v2, u[20], m[20]);+	MONT_MAC(w0, w1, w2, u[21], m[19]);+	MONT_MAC(t0, t1, t2, u[22], m[18]);+	MONT_MAC(v0, v1, v2, u[23], m[17]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[16] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 41 */+	MONT_MUL(s0, s1, s2, a[18], a[23]);+	MONT_MUL(c0, c1, c2, a[19], a[22]);+	MONT_MAC(s0, s1, s2, a[20], a[21]);+	MONT_MAC(t0, t1, t2, u[18], m[23]);+	MONT_MUL(v0, v1, v2, u[19], m[22]);+	MONT_MUL(w0, w1, w2, u[20], m[21]);+	MONT_MAC(t0, t1, t2, u[21], m[20]);+	MONT_MAC(v0, v1, v2, u[22], m[19]);+	MONT_MAC(w0, w1, w2, u[23], m[18]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[17] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 42 */+	MONT_MUL(s0, s1, s2, a[19], a[23]);+	MONT_MUL(c0, c1, c2, a[20], a[22]);+	MONT_MAC(t0, t1, t2, a[21], a[21]);+	MONT_MUL(v0, v1, v2, u[19], m[23]);+	MONT_MUL(w0, w1, w2, u[20], m[22]);+	MONT_MAC(t0, t1, t2, u[21], m[21]);+	MONT_MAC(v0, v1, v2, u[22], m[20]);+	MONT_MAC(w0, w1, w2, u[23], m[19]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[18] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 43 */+	MONT_MUL(s0, s1, s2, a[20], a[23]);+	MONT_MUL(c0, c1, c2, a[21], a[22]);+	MONT_MAC(t0, t1, t2, u[20], m[23]);+	MONT_MUL(v0, v1, v2, u[21], m[22]);+	MONT_MUL(w0, w1, w2, u[22], m[21]);+	MONT_MAC(t0, t1, t2, u[23], m[20]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[19] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 44 */+	MONT_MUL(s0, s1, s2, a[21], a[23]);+	MONT_MAC(t0, t1, t2, a[22], a[22]);+	MONT_MUL(v0, v1, v2, u[21], m[23]);+	MONT_MUL(w0, w1, w2, u[22], m[22]);+	MONT_MAC(t0, t1, t2, u[23], m[21]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[20] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 45 */+	MONT_MUL(s0, s1, s2, a[22], a[23]);+	MONT_MAC(t0, t1, t2, u[22], m[23]);+	MONT_MUL(v0, v1, v2, u[23], m[22]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[21] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 46 */+	MONT_MAC(t0, t1, t2, a[23], a[23]);+	MONT_MUL(v0, v1, v2, u[23], m[23]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	z[22] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	z[23] = t0;++	/* under 2m, with t1 the bit above n limbs: at most one subtraction */+	borrow = sub_n(r, z, m, 24);+	take = t1 | (borrow ^ 1);+	select_n(r, r, z, take & 1, 24);+}++/* r = a * b * R^-1 mod m, for a and b under m; t is 2n limbs of scratch */+static void crypton_mont_mul32(limb_t *r, const limb_t *a, const limb_t *b,+                               const limb_t *m, limb_t n0, limb_t *t)+{+	limb_t *u = t, *z = t + 32;+	limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take;++	/* column 0 */+	MONT_MAC(t0, t1, t2, a[0], b[0]);+	u[0] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[0], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 1 */+	MONT_MAC(t0, t1, t2, a[0], b[1]);+	MONT_MUL(v0, v1, v2, a[1], b[0]);+	MONT_MUL(w0, w1, w2, u[0], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[1] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[1], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 2 */+	MONT_MAC(t0, t1, t2, a[0], b[2]);+	MONT_MUL(v0, v1, v2, a[1], b[1]);+	MONT_MUL(w0, w1, w2, a[2], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[2]);+	MONT_MAC(v0, v1, v2, u[1], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[2] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[2], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 3 */+	MONT_MAC(t0, t1, t2, a[0], b[3]);+	MONT_MUL(v0, v1, v2, a[1], b[2]);+	MONT_MUL(w0, w1, w2, a[2], b[1]);+	MONT_MAC(t0, t1, t2, a[3], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[3]);+	MONT_MAC(w0, w1, w2, u[1], m[2]);+	MONT_MAC(t0, t1, t2, u[2], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[3] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[3], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 4 */+	MONT_MAC(t0, t1, t2, a[0], b[4]);+	MONT_MUL(v0, v1, v2, a[1], b[3]);+	MONT_MUL(w0, w1, w2, a[2], b[2]);+	MONT_MAC(t0, t1, t2, a[3], b[1]);+	MONT_MAC(v0, v1, v2, a[4], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[4]);+	MONT_MAC(t0, t1, t2, u[1], m[3]);+	MONT_MAC(v0, v1, v2, u[2], m[2]);+	MONT_MAC(w0, w1, w2, u[3], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[4] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[4], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 5 */+	MONT_MAC(t0, t1, t2, a[0], b[5]);+	MONT_MUL(v0, v1, v2, a[1], b[4]);+	MONT_MUL(w0, w1, w2, a[2], b[3]);+	MONT_MAC(t0, t1, t2, a[3], b[2]);+	MONT_MAC(v0, v1, v2, a[4], b[1]);+	MONT_MAC(w0, w1, w2, a[5], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[5]);+	MONT_MAC(v0, v1, v2, u[1], m[4]);+	MONT_MAC(w0, w1, w2, u[2], m[3]);+	MONT_MAC(t0, t1, t2, u[3], m[2]);+	MONT_MAC(v0, v1, v2, u[4], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[5] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[5], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 6 */+	MONT_MAC(t0, t1, t2, a[0], b[6]);+	MONT_MUL(v0, v1, v2, a[1], b[5]);+	MONT_MUL(w0, w1, w2, a[2], b[4]);+	MONT_MAC(t0, t1, t2, a[3], b[3]);+	MONT_MAC(v0, v1, v2, a[4], b[2]);+	MONT_MAC(w0, w1, w2, a[5], b[1]);+	MONT_MAC(t0, t1, t2, a[6], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[6]);+	MONT_MAC(w0, w1, w2, u[1], m[5]);+	MONT_MAC(t0, t1, t2, u[2], m[4]);+	MONT_MAC(v0, v1, v2, u[3], m[3]);+	MONT_MAC(w0, w1, w2, u[4], m[2]);+	MONT_MAC(t0, t1, t2, u[5], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[6] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[6], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 7 */+	MONT_MAC(t0, t1, t2, a[0], b[7]);+	MONT_MUL(v0, v1, v2, a[1], b[6]);+	MONT_MUL(w0, w1, w2, a[2], b[5]);+	MONT_MAC(t0, t1, t2, a[3], b[4]);+	MONT_MAC(v0, v1, v2, a[4], b[3]);+	MONT_MAC(w0, w1, w2, a[5], b[2]);+	MONT_MAC(t0, t1, t2, a[6], b[1]);+	MONT_MAC(v0, v1, v2, a[7], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[7]);+	MONT_MAC(t0, t1, t2, u[1], m[6]);+	MONT_MAC(v0, v1, v2, u[2], m[5]);+	MONT_MAC(w0, w1, w2, u[3], m[4]);+	MONT_MAC(t0, t1, t2, u[4], m[3]);+	MONT_MAC(v0, v1, v2, u[5], m[2]);+	MONT_MAC(w0, w1, w2, u[6], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[7] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[7], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 8 */+	MONT_MAC(t0, t1, t2, a[0], b[8]);+	MONT_MUL(v0, v1, v2, a[1], b[7]);+	MONT_MUL(w0, w1, w2, a[2], b[6]);+	MONT_MAC(t0, t1, t2, a[3], b[5]);+	MONT_MAC(v0, v1, v2, a[4], b[4]);+	MONT_MAC(w0, w1, w2, a[5], b[3]);+	MONT_MAC(t0, t1, t2, a[6], b[2]);+	MONT_MAC(v0, v1, v2, a[7], b[1]);+	MONT_MAC(w0, w1, w2, a[8], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[8]);+	MONT_MAC(v0, v1, v2, u[1], m[7]);+	MONT_MAC(w0, w1, w2, u[2], m[6]);+	MONT_MAC(t0, t1, t2, u[3], m[5]);+	MONT_MAC(v0, v1, v2, u[4], m[4]);+	MONT_MAC(w0, w1, w2, u[5], m[3]);+	MONT_MAC(t0, t1, t2, u[6], m[2]);+	MONT_MAC(v0, v1, v2, u[7], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[8] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[8], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 9 */+	MONT_MAC(t0, t1, t2, a[0], b[9]);+	MONT_MUL(v0, v1, v2, a[1], b[8]);+	MONT_MUL(w0, w1, w2, a[2], b[7]);+	MONT_MAC(t0, t1, t2, a[3], b[6]);+	MONT_MAC(v0, v1, v2, a[4], b[5]);+	MONT_MAC(w0, w1, w2, a[5], b[4]);+	MONT_MAC(t0, t1, t2, a[6], b[3]);+	MONT_MAC(v0, v1, v2, a[7], b[2]);+	MONT_MAC(w0, w1, w2, a[8], b[1]);+	MONT_MAC(t0, t1, t2, a[9], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[9]);+	MONT_MAC(w0, w1, w2, u[1], m[8]);+	MONT_MAC(t0, t1, t2, u[2], m[7]);+	MONT_MAC(v0, v1, v2, u[3], m[6]);+	MONT_MAC(w0, w1, w2, u[4], m[5]);+	MONT_MAC(t0, t1, t2, u[5], m[4]);+	MONT_MAC(v0, v1, v2, u[6], m[3]);+	MONT_MAC(w0, w1, w2, u[7], m[2]);+	MONT_MAC(t0, t1, t2, u[8], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[9] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[9], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 10 */+	MONT_MAC(t0, t1, t2, a[0], b[10]);+	MONT_MUL(v0, v1, v2, a[1], b[9]);+	MONT_MUL(w0, w1, w2, a[2], b[8]);+	MONT_MAC(t0, t1, t2, a[3], b[7]);+	MONT_MAC(v0, v1, v2, a[4], b[6]);+	MONT_MAC(w0, w1, w2, a[5], b[5]);+	MONT_MAC(t0, t1, t2, a[6], b[4]);+	MONT_MAC(v0, v1, v2, a[7], b[3]);+	MONT_MAC(w0, w1, w2, a[8], b[2]);+	MONT_MAC(t0, t1, t2, a[9], b[1]);+	MONT_MAC(v0, v1, v2, a[10], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[10]);+	MONT_MAC(t0, t1, t2, u[1], m[9]);+	MONT_MAC(v0, v1, v2, u[2], m[8]);+	MONT_MAC(w0, w1, w2, u[3], m[7]);+	MONT_MAC(t0, t1, t2, u[4], m[6]);+	MONT_MAC(v0, v1, v2, u[5], m[5]);+	MONT_MAC(w0, w1, w2, u[6], m[4]);+	MONT_MAC(t0, t1, t2, u[7], m[3]);+	MONT_MAC(v0, v1, v2, u[8], m[2]);+	MONT_MAC(w0, w1, w2, u[9], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[10] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[10], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 11 */+	MONT_MAC(t0, t1, t2, a[0], b[11]);+	MONT_MUL(v0, v1, v2, a[1], b[10]);+	MONT_MUL(w0, w1, w2, a[2], b[9]);+	MONT_MAC(t0, t1, t2, a[3], b[8]);+	MONT_MAC(v0, v1, v2, a[4], b[7]);+	MONT_MAC(w0, w1, w2, a[5], b[6]);+	MONT_MAC(t0, t1, t2, a[6], b[5]);+	MONT_MAC(v0, v1, v2, a[7], b[4]);+	MONT_MAC(w0, w1, w2, a[8], b[3]);+	MONT_MAC(t0, t1, t2, a[9], b[2]);+	MONT_MAC(v0, v1, v2, a[10], b[1]);+	MONT_MAC(w0, w1, w2, a[11], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[11]);+	MONT_MAC(v0, v1, v2, u[1], m[10]);+	MONT_MAC(w0, w1, w2, u[2], m[9]);+	MONT_MAC(t0, t1, t2, u[3], m[8]);+	MONT_MAC(v0, v1, v2, u[4], m[7]);+	MONT_MAC(w0, w1, w2, u[5], m[6]);+	MONT_MAC(t0, t1, t2, u[6], m[5]);+	MONT_MAC(v0, v1, v2, u[7], m[4]);+	MONT_MAC(w0, w1, w2, u[8], m[3]);+	MONT_MAC(t0, t1, t2, u[9], m[2]);+	MONT_MAC(v0, v1, v2, u[10], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[11] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[11], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 12 */+	MONT_MAC(t0, t1, t2, a[0], b[12]);+	MONT_MUL(v0, v1, v2, a[1], b[11]);+	MONT_MUL(w0, w1, w2, a[2], b[10]);+	MONT_MAC(t0, t1, t2, a[3], b[9]);+	MONT_MAC(v0, v1, v2, a[4], b[8]);+	MONT_MAC(w0, w1, w2, a[5], b[7]);+	MONT_MAC(t0, t1, t2, a[6], b[6]);+	MONT_MAC(v0, v1, v2, a[7], b[5]);+	MONT_MAC(w0, w1, w2, a[8], b[4]);+	MONT_MAC(t0, t1, t2, a[9], b[3]);+	MONT_MAC(v0, v1, v2, a[10], b[2]);+	MONT_MAC(w0, w1, w2, a[11], b[1]);+	MONT_MAC(t0, t1, t2, a[12], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[12]);+	MONT_MAC(w0, w1, w2, u[1], m[11]);+	MONT_MAC(t0, t1, t2, u[2], m[10]);+	MONT_MAC(v0, v1, v2, u[3], m[9]);+	MONT_MAC(w0, w1, w2, u[4], m[8]);+	MONT_MAC(t0, t1, t2, u[5], m[7]);+	MONT_MAC(v0, v1, v2, u[6], m[6]);+	MONT_MAC(w0, w1, w2, u[7], m[5]);+	MONT_MAC(t0, t1, t2, u[8], m[4]);+	MONT_MAC(v0, v1, v2, u[9], m[3]);+	MONT_MAC(w0, w1, w2, u[10], m[2]);+	MONT_MAC(t0, t1, t2, u[11], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[12] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[12], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 13 */+	MONT_MAC(t0, t1, t2, a[0], b[13]);+	MONT_MUL(v0, v1, v2, a[1], b[12]);+	MONT_MUL(w0, w1, w2, a[2], b[11]);+	MONT_MAC(t0, t1, t2, a[3], b[10]);+	MONT_MAC(v0, v1, v2, a[4], b[9]);+	MONT_MAC(w0, w1, w2, a[5], b[8]);+	MONT_MAC(t0, t1, t2, a[6], b[7]);+	MONT_MAC(v0, v1, v2, a[7], b[6]);+	MONT_MAC(w0, w1, w2, a[8], b[5]);+	MONT_MAC(t0, t1, t2, a[9], b[4]);+	MONT_MAC(v0, v1, v2, a[10], b[3]);+	MONT_MAC(w0, w1, w2, a[11], b[2]);+	MONT_MAC(t0, t1, t2, a[12], b[1]);+	MONT_MAC(v0, v1, v2, a[13], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[13]);+	MONT_MAC(t0, t1, t2, u[1], m[12]);+	MONT_MAC(v0, v1, v2, u[2], m[11]);+	MONT_MAC(w0, w1, w2, u[3], m[10]);+	MONT_MAC(t0, t1, t2, u[4], m[9]);+	MONT_MAC(v0, v1, v2, u[5], m[8]);+	MONT_MAC(w0, w1, w2, u[6], m[7]);+	MONT_MAC(t0, t1, t2, u[7], m[6]);+	MONT_MAC(v0, v1, v2, u[8], m[5]);+	MONT_MAC(w0, w1, w2, u[9], m[4]);+	MONT_MAC(t0, t1, t2, u[10], m[3]);+	MONT_MAC(v0, v1, v2, u[11], m[2]);+	MONT_MAC(w0, w1, w2, u[12], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[13] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[13], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 14 */+	MONT_MAC(t0, t1, t2, a[0], b[14]);+	MONT_MUL(v0, v1, v2, a[1], b[13]);+	MONT_MUL(w0, w1, w2, a[2], b[12]);+	MONT_MAC(t0, t1, t2, a[3], b[11]);+	MONT_MAC(v0, v1, v2, a[4], b[10]);+	MONT_MAC(w0, w1, w2, a[5], b[9]);+	MONT_MAC(t0, t1, t2, a[6], b[8]);+	MONT_MAC(v0, v1, v2, a[7], b[7]);+	MONT_MAC(w0, w1, w2, a[8], b[6]);+	MONT_MAC(t0, t1, t2, a[9], b[5]);+	MONT_MAC(v0, v1, v2, a[10], b[4]);+	MONT_MAC(w0, w1, w2, a[11], b[3]);+	MONT_MAC(t0, t1, t2, a[12], b[2]);+	MONT_MAC(v0, v1, v2, a[13], b[1]);+	MONT_MAC(w0, w1, w2, a[14], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[14]);+	MONT_MAC(v0, v1, v2, u[1], m[13]);+	MONT_MAC(w0, w1, w2, u[2], m[12]);+	MONT_MAC(t0, t1, t2, u[3], m[11]);+	MONT_MAC(v0, v1, v2, u[4], m[10]);+	MONT_MAC(w0, w1, w2, u[5], m[9]);+	MONT_MAC(t0, t1, t2, u[6], m[8]);+	MONT_MAC(v0, v1, v2, u[7], m[7]);+	MONT_MAC(w0, w1, w2, u[8], m[6]);+	MONT_MAC(t0, t1, t2, u[9], m[5]);+	MONT_MAC(v0, v1, v2, u[10], m[4]);+	MONT_MAC(w0, w1, w2, u[11], m[3]);+	MONT_MAC(t0, t1, t2, u[12], m[2]);+	MONT_MAC(v0, v1, v2, u[13], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[14] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[14], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 15 */+	MONT_MAC(t0, t1, t2, a[0], b[15]);+	MONT_MUL(v0, v1, v2, a[1], b[14]);+	MONT_MUL(w0, w1, w2, a[2], b[13]);+	MONT_MAC(t0, t1, t2, a[3], b[12]);+	MONT_MAC(v0, v1, v2, a[4], b[11]);+	MONT_MAC(w0, w1, w2, a[5], b[10]);+	MONT_MAC(t0, t1, t2, a[6], b[9]);+	MONT_MAC(v0, v1, v2, a[7], b[8]);+	MONT_MAC(w0, w1, w2, a[8], b[7]);+	MONT_MAC(t0, t1, t2, a[9], b[6]);+	MONT_MAC(v0, v1, v2, a[10], b[5]);+	MONT_MAC(w0, w1, w2, a[11], b[4]);+	MONT_MAC(t0, t1, t2, a[12], b[3]);+	MONT_MAC(v0, v1, v2, a[13], b[2]);+	MONT_MAC(w0, w1, w2, a[14], b[1]);+	MONT_MAC(t0, t1, t2, a[15], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[15]);+	MONT_MAC(w0, w1, w2, u[1], m[14]);+	MONT_MAC(t0, t1, t2, u[2], m[13]);+	MONT_MAC(v0, v1, v2, u[3], m[12]);+	MONT_MAC(w0, w1, w2, u[4], m[11]);+	MONT_MAC(t0, t1, t2, u[5], m[10]);+	MONT_MAC(v0, v1, v2, u[6], m[9]);+	MONT_MAC(w0, w1, w2, u[7], m[8]);+	MONT_MAC(t0, t1, t2, u[8], m[7]);+	MONT_MAC(v0, v1, v2, u[9], m[6]);+	MONT_MAC(w0, w1, w2, u[10], m[5]);+	MONT_MAC(t0, t1, t2, u[11], m[4]);+	MONT_MAC(v0, v1, v2, u[12], m[3]);+	MONT_MAC(w0, w1, w2, u[13], m[2]);+	MONT_MAC(t0, t1, t2, u[14], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[15] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[15], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 16 */+	MONT_MAC(t0, t1, t2, a[0], b[16]);+	MONT_MUL(v0, v1, v2, a[1], b[15]);+	MONT_MUL(w0, w1, w2, a[2], b[14]);+	MONT_MAC(t0, t1, t2, a[3], b[13]);+	MONT_MAC(v0, v1, v2, a[4], b[12]);+	MONT_MAC(w0, w1, w2, a[5], b[11]);+	MONT_MAC(t0, t1, t2, a[6], b[10]);+	MONT_MAC(v0, v1, v2, a[7], b[9]);+	MONT_MAC(w0, w1, w2, a[8], b[8]);+	MONT_MAC(t0, t1, t2, a[9], b[7]);+	MONT_MAC(v0, v1, v2, a[10], b[6]);+	MONT_MAC(w0, w1, w2, a[11], b[5]);+	MONT_MAC(t0, t1, t2, a[12], b[4]);+	MONT_MAC(v0, v1, v2, a[13], b[3]);+	MONT_MAC(w0, w1, w2, a[14], b[2]);+	MONT_MAC(t0, t1, t2, a[15], b[1]);+	MONT_MAC(v0, v1, v2, a[16], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[16]);+	MONT_MAC(t0, t1, t2, u[1], m[15]);+	MONT_MAC(v0, v1, v2, u[2], m[14]);+	MONT_MAC(w0, w1, w2, u[3], m[13]);+	MONT_MAC(t0, t1, t2, u[4], m[12]);+	MONT_MAC(v0, v1, v2, u[5], m[11]);+	MONT_MAC(w0, w1, w2, u[6], m[10]);+	MONT_MAC(t0, t1, t2, u[7], m[9]);+	MONT_MAC(v0, v1, v2, u[8], m[8]);+	MONT_MAC(w0, w1, w2, u[9], m[7]);+	MONT_MAC(t0, t1, t2, u[10], m[6]);+	MONT_MAC(v0, v1, v2, u[11], m[5]);+	MONT_MAC(w0, w1, w2, u[12], m[4]);+	MONT_MAC(t0, t1, t2, u[13], m[3]);+	MONT_MAC(v0, v1, v2, u[14], m[2]);+	MONT_MAC(w0, w1, w2, u[15], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[16] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[16], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 17 */+	MONT_MAC(t0, t1, t2, a[0], b[17]);+	MONT_MUL(v0, v1, v2, a[1], b[16]);+	MONT_MUL(w0, w1, w2, a[2], b[15]);+	MONT_MAC(t0, t1, t2, a[3], b[14]);+	MONT_MAC(v0, v1, v2, a[4], b[13]);+	MONT_MAC(w0, w1, w2, a[5], b[12]);+	MONT_MAC(t0, t1, t2, a[6], b[11]);+	MONT_MAC(v0, v1, v2, a[7], b[10]);+	MONT_MAC(w0, w1, w2, a[8], b[9]);+	MONT_MAC(t0, t1, t2, a[9], b[8]);+	MONT_MAC(v0, v1, v2, a[10], b[7]);+	MONT_MAC(w0, w1, w2, a[11], b[6]);+	MONT_MAC(t0, t1, t2, a[12], b[5]);+	MONT_MAC(v0, v1, v2, a[13], b[4]);+	MONT_MAC(w0, w1, w2, a[14], b[3]);+	MONT_MAC(t0, t1, t2, a[15], b[2]);+	MONT_MAC(v0, v1, v2, a[16], b[1]);+	MONT_MAC(w0, w1, w2, a[17], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[17]);+	MONT_MAC(v0, v1, v2, u[1], m[16]);+	MONT_MAC(w0, w1, w2, u[2], m[15]);+	MONT_MAC(t0, t1, t2, u[3], m[14]);+	MONT_MAC(v0, v1, v2, u[4], m[13]);+	MONT_MAC(w0, w1, w2, u[5], m[12]);+	MONT_MAC(t0, t1, t2, u[6], m[11]);+	MONT_MAC(v0, v1, v2, u[7], m[10]);+	MONT_MAC(w0, w1, w2, u[8], m[9]);+	MONT_MAC(t0, t1, t2, u[9], m[8]);+	MONT_MAC(v0, v1, v2, u[10], m[7]);+	MONT_MAC(w0, w1, w2, u[11], m[6]);+	MONT_MAC(t0, t1, t2, u[12], m[5]);+	MONT_MAC(v0, v1, v2, u[13], m[4]);+	MONT_MAC(w0, w1, w2, u[14], m[3]);+	MONT_MAC(t0, t1, t2, u[15], m[2]);+	MONT_MAC(v0, v1, v2, u[16], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[17] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[17], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 18 */+	MONT_MAC(t0, t1, t2, a[0], b[18]);+	MONT_MUL(v0, v1, v2, a[1], b[17]);+	MONT_MUL(w0, w1, w2, a[2], b[16]);+	MONT_MAC(t0, t1, t2, a[3], b[15]);+	MONT_MAC(v0, v1, v2, a[4], b[14]);+	MONT_MAC(w0, w1, w2, a[5], b[13]);+	MONT_MAC(t0, t1, t2, a[6], b[12]);+	MONT_MAC(v0, v1, v2, a[7], b[11]);+	MONT_MAC(w0, w1, w2, a[8], b[10]);+	MONT_MAC(t0, t1, t2, a[9], b[9]);+	MONT_MAC(v0, v1, v2, a[10], b[8]);+	MONT_MAC(w0, w1, w2, a[11], b[7]);+	MONT_MAC(t0, t1, t2, a[12], b[6]);+	MONT_MAC(v0, v1, v2, a[13], b[5]);+	MONT_MAC(w0, w1, w2, a[14], b[4]);+	MONT_MAC(t0, t1, t2, a[15], b[3]);+	MONT_MAC(v0, v1, v2, a[16], b[2]);+	MONT_MAC(w0, w1, w2, a[17], b[1]);+	MONT_MAC(t0, t1, t2, a[18], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[18]);+	MONT_MAC(w0, w1, w2, u[1], m[17]);+	MONT_MAC(t0, t1, t2, u[2], m[16]);+	MONT_MAC(v0, v1, v2, u[3], m[15]);+	MONT_MAC(w0, w1, w2, u[4], m[14]);+	MONT_MAC(t0, t1, t2, u[5], m[13]);+	MONT_MAC(v0, v1, v2, u[6], m[12]);+	MONT_MAC(w0, w1, w2, u[7], m[11]);+	MONT_MAC(t0, t1, t2, u[8], m[10]);+	MONT_MAC(v0, v1, v2, u[9], m[9]);+	MONT_MAC(w0, w1, w2, u[10], m[8]);+	MONT_MAC(t0, t1, t2, u[11], m[7]);+	MONT_MAC(v0, v1, v2, u[12], m[6]);+	MONT_MAC(w0, w1, w2, u[13], m[5]);+	MONT_MAC(t0, t1, t2, u[14], m[4]);+	MONT_MAC(v0, v1, v2, u[15], m[3]);+	MONT_MAC(w0, w1, w2, u[16], m[2]);+	MONT_MAC(t0, t1, t2, u[17], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[18] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[18], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 19 */+	MONT_MAC(t0, t1, t2, a[0], b[19]);+	MONT_MUL(v0, v1, v2, a[1], b[18]);+	MONT_MUL(w0, w1, w2, a[2], b[17]);+	MONT_MAC(t0, t1, t2, a[3], b[16]);+	MONT_MAC(v0, v1, v2, a[4], b[15]);+	MONT_MAC(w0, w1, w2, a[5], b[14]);+	MONT_MAC(t0, t1, t2, a[6], b[13]);+	MONT_MAC(v0, v1, v2, a[7], b[12]);+	MONT_MAC(w0, w1, w2, a[8], b[11]);+	MONT_MAC(t0, t1, t2, a[9], b[10]);+	MONT_MAC(v0, v1, v2, a[10], b[9]);+	MONT_MAC(w0, w1, w2, a[11], b[8]);+	MONT_MAC(t0, t1, t2, a[12], b[7]);+	MONT_MAC(v0, v1, v2, a[13], b[6]);+	MONT_MAC(w0, w1, w2, a[14], b[5]);+	MONT_MAC(t0, t1, t2, a[15], b[4]);+	MONT_MAC(v0, v1, v2, a[16], b[3]);+	MONT_MAC(w0, w1, w2, a[17], b[2]);+	MONT_MAC(t0, t1, t2, a[18], b[1]);+	MONT_MAC(v0, v1, v2, a[19], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[19]);+	MONT_MAC(t0, t1, t2, u[1], m[18]);+	MONT_MAC(v0, v1, v2, u[2], m[17]);+	MONT_MAC(w0, w1, w2, u[3], m[16]);+	MONT_MAC(t0, t1, t2, u[4], m[15]);+	MONT_MAC(v0, v1, v2, u[5], m[14]);+	MONT_MAC(w0, w1, w2, u[6], m[13]);+	MONT_MAC(t0, t1, t2, u[7], m[12]);+	MONT_MAC(v0, v1, v2, u[8], m[11]);+	MONT_MAC(w0, w1, w2, u[9], m[10]);+	MONT_MAC(t0, t1, t2, u[10], m[9]);+	MONT_MAC(v0, v1, v2, u[11], m[8]);+	MONT_MAC(w0, w1, w2, u[12], m[7]);+	MONT_MAC(t0, t1, t2, u[13], m[6]);+	MONT_MAC(v0, v1, v2, u[14], m[5]);+	MONT_MAC(w0, w1, w2, u[15], m[4]);+	MONT_MAC(t0, t1, t2, u[16], m[3]);+	MONT_MAC(v0, v1, v2, u[17], m[2]);+	MONT_MAC(w0, w1, w2, u[18], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[19] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[19], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 20 */+	MONT_MAC(t0, t1, t2, a[0], b[20]);+	MONT_MUL(v0, v1, v2, a[1], b[19]);+	MONT_MUL(w0, w1, w2, a[2], b[18]);+	MONT_MAC(t0, t1, t2, a[3], b[17]);+	MONT_MAC(v0, v1, v2, a[4], b[16]);+	MONT_MAC(w0, w1, w2, a[5], b[15]);+	MONT_MAC(t0, t1, t2, a[6], b[14]);+	MONT_MAC(v0, v1, v2, a[7], b[13]);+	MONT_MAC(w0, w1, w2, a[8], b[12]);+	MONT_MAC(t0, t1, t2, a[9], b[11]);+	MONT_MAC(v0, v1, v2, a[10], b[10]);+	MONT_MAC(w0, w1, w2, a[11], b[9]);+	MONT_MAC(t0, t1, t2, a[12], b[8]);+	MONT_MAC(v0, v1, v2, a[13], b[7]);+	MONT_MAC(w0, w1, w2, a[14], b[6]);+	MONT_MAC(t0, t1, t2, a[15], b[5]);+	MONT_MAC(v0, v1, v2, a[16], b[4]);+	MONT_MAC(w0, w1, w2, a[17], b[3]);+	MONT_MAC(t0, t1, t2, a[18], b[2]);+	MONT_MAC(v0, v1, v2, a[19], b[1]);+	MONT_MAC(w0, w1, w2, a[20], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[20]);+	MONT_MAC(v0, v1, v2, u[1], m[19]);+	MONT_MAC(w0, w1, w2, u[2], m[18]);+	MONT_MAC(t0, t1, t2, u[3], m[17]);+	MONT_MAC(v0, v1, v2, u[4], m[16]);+	MONT_MAC(w0, w1, w2, u[5], m[15]);+	MONT_MAC(t0, t1, t2, u[6], m[14]);+	MONT_MAC(v0, v1, v2, u[7], m[13]);+	MONT_MAC(w0, w1, w2, u[8], m[12]);+	MONT_MAC(t0, t1, t2, u[9], m[11]);+	MONT_MAC(v0, v1, v2, u[10], m[10]);+	MONT_MAC(w0, w1, w2, u[11], m[9]);+	MONT_MAC(t0, t1, t2, u[12], m[8]);+	MONT_MAC(v0, v1, v2, u[13], m[7]);+	MONT_MAC(w0, w1, w2, u[14], m[6]);+	MONT_MAC(t0, t1, t2, u[15], m[5]);+	MONT_MAC(v0, v1, v2, u[16], m[4]);+	MONT_MAC(w0, w1, w2, u[17], m[3]);+	MONT_MAC(t0, t1, t2, u[18], m[2]);+	MONT_MAC(v0, v1, v2, u[19], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[20] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[20], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 21 */+	MONT_MAC(t0, t1, t2, a[0], b[21]);+	MONT_MUL(v0, v1, v2, a[1], b[20]);+	MONT_MUL(w0, w1, w2, a[2], b[19]);+	MONT_MAC(t0, t1, t2, a[3], b[18]);+	MONT_MAC(v0, v1, v2, a[4], b[17]);+	MONT_MAC(w0, w1, w2, a[5], b[16]);+	MONT_MAC(t0, t1, t2, a[6], b[15]);+	MONT_MAC(v0, v1, v2, a[7], b[14]);+	MONT_MAC(w0, w1, w2, a[8], b[13]);+	MONT_MAC(t0, t1, t2, a[9], b[12]);+	MONT_MAC(v0, v1, v2, a[10], b[11]);+	MONT_MAC(w0, w1, w2, a[11], b[10]);+	MONT_MAC(t0, t1, t2, a[12], b[9]);+	MONT_MAC(v0, v1, v2, a[13], b[8]);+	MONT_MAC(w0, w1, w2, a[14], b[7]);+	MONT_MAC(t0, t1, t2, a[15], b[6]);+	MONT_MAC(v0, v1, v2, a[16], b[5]);+	MONT_MAC(w0, w1, w2, a[17], b[4]);+	MONT_MAC(t0, t1, t2, a[18], b[3]);+	MONT_MAC(v0, v1, v2, a[19], b[2]);+	MONT_MAC(w0, w1, w2, a[20], b[1]);+	MONT_MAC(t0, t1, t2, a[21], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[21]);+	MONT_MAC(w0, w1, w2, u[1], m[20]);+	MONT_MAC(t0, t1, t2, u[2], m[19]);+	MONT_MAC(v0, v1, v2, u[3], m[18]);+	MONT_MAC(w0, w1, w2, u[4], m[17]);+	MONT_MAC(t0, t1, t2, u[5], m[16]);+	MONT_MAC(v0, v1, v2, u[6], m[15]);+	MONT_MAC(w0, w1, w2, u[7], m[14]);+	MONT_MAC(t0, t1, t2, u[8], m[13]);+	MONT_MAC(v0, v1, v2, u[9], m[12]);+	MONT_MAC(w0, w1, w2, u[10], m[11]);+	MONT_MAC(t0, t1, t2, u[11], m[10]);+	MONT_MAC(v0, v1, v2, u[12], m[9]);+	MONT_MAC(w0, w1, w2, u[13], m[8]);+	MONT_MAC(t0, t1, t2, u[14], m[7]);+	MONT_MAC(v0, v1, v2, u[15], m[6]);+	MONT_MAC(w0, w1, w2, u[16], m[5]);+	MONT_MAC(t0, t1, t2, u[17], m[4]);+	MONT_MAC(v0, v1, v2, u[18], m[3]);+	MONT_MAC(w0, w1, w2, u[19], m[2]);+	MONT_MAC(t0, t1, t2, u[20], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[21] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[21], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 22 */+	MONT_MAC(t0, t1, t2, a[0], b[22]);+	MONT_MUL(v0, v1, v2, a[1], b[21]);+	MONT_MUL(w0, w1, w2, a[2], b[20]);+	MONT_MAC(t0, t1, t2, a[3], b[19]);+	MONT_MAC(v0, v1, v2, a[4], b[18]);+	MONT_MAC(w0, w1, w2, a[5], b[17]);+	MONT_MAC(t0, t1, t2, a[6], b[16]);+	MONT_MAC(v0, v1, v2, a[7], b[15]);+	MONT_MAC(w0, w1, w2, a[8], b[14]);+	MONT_MAC(t0, t1, t2, a[9], b[13]);+	MONT_MAC(v0, v1, v2, a[10], b[12]);+	MONT_MAC(w0, w1, w2, a[11], b[11]);+	MONT_MAC(t0, t1, t2, a[12], b[10]);+	MONT_MAC(v0, v1, v2, a[13], b[9]);+	MONT_MAC(w0, w1, w2, a[14], b[8]);+	MONT_MAC(t0, t1, t2, a[15], b[7]);+	MONT_MAC(v0, v1, v2, a[16], b[6]);+	MONT_MAC(w0, w1, w2, a[17], b[5]);+	MONT_MAC(t0, t1, t2, a[18], b[4]);+	MONT_MAC(v0, v1, v2, a[19], b[3]);+	MONT_MAC(w0, w1, w2, a[20], b[2]);+	MONT_MAC(t0, t1, t2, a[21], b[1]);+	MONT_MAC(v0, v1, v2, a[22], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[22]);+	MONT_MAC(t0, t1, t2, u[1], m[21]);+	MONT_MAC(v0, v1, v2, u[2], m[20]);+	MONT_MAC(w0, w1, w2, u[3], m[19]);+	MONT_MAC(t0, t1, t2, u[4], m[18]);+	MONT_MAC(v0, v1, v2, u[5], m[17]);+	MONT_MAC(w0, w1, w2, u[6], m[16]);+	MONT_MAC(t0, t1, t2, u[7], m[15]);+	MONT_MAC(v0, v1, v2, u[8], m[14]);+	MONT_MAC(w0, w1, w2, u[9], m[13]);+	MONT_MAC(t0, t1, t2, u[10], m[12]);+	MONT_MAC(v0, v1, v2, u[11], m[11]);+	MONT_MAC(w0, w1, w2, u[12], m[10]);+	MONT_MAC(t0, t1, t2, u[13], m[9]);+	MONT_MAC(v0, v1, v2, u[14], m[8]);+	MONT_MAC(w0, w1, w2, u[15], m[7]);+	MONT_MAC(t0, t1, t2, u[16], m[6]);+	MONT_MAC(v0, v1, v2, u[17], m[5]);+	MONT_MAC(w0, w1, w2, u[18], m[4]);+	MONT_MAC(t0, t1, t2, u[19], m[3]);+	MONT_MAC(v0, v1, v2, u[20], m[2]);+	MONT_MAC(w0, w1, w2, u[21], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[22] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[22], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 23 */+	MONT_MAC(t0, t1, t2, a[0], b[23]);+	MONT_MUL(v0, v1, v2, a[1], b[22]);+	MONT_MUL(w0, w1, w2, a[2], b[21]);+	MONT_MAC(t0, t1, t2, a[3], b[20]);+	MONT_MAC(v0, v1, v2, a[4], b[19]);+	MONT_MAC(w0, w1, w2, a[5], b[18]);+	MONT_MAC(t0, t1, t2, a[6], b[17]);+	MONT_MAC(v0, v1, v2, a[7], b[16]);+	MONT_MAC(w0, w1, w2, a[8], b[15]);+	MONT_MAC(t0, t1, t2, a[9], b[14]);+	MONT_MAC(v0, v1, v2, a[10], b[13]);+	MONT_MAC(w0, w1, w2, a[11], b[12]);+	MONT_MAC(t0, t1, t2, a[12], b[11]);+	MONT_MAC(v0, v1, v2, a[13], b[10]);+	MONT_MAC(w0, w1, w2, a[14], b[9]);+	MONT_MAC(t0, t1, t2, a[15], b[8]);+	MONT_MAC(v0, v1, v2, a[16], b[7]);+	MONT_MAC(w0, w1, w2, a[17], b[6]);+	MONT_MAC(t0, t1, t2, a[18], b[5]);+	MONT_MAC(v0, v1, v2, a[19], b[4]);+	MONT_MAC(w0, w1, w2, a[20], b[3]);+	MONT_MAC(t0, t1, t2, a[21], b[2]);+	MONT_MAC(v0, v1, v2, a[22], b[1]);+	MONT_MAC(w0, w1, w2, a[23], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[23]);+	MONT_MAC(v0, v1, v2, u[1], m[22]);+	MONT_MAC(w0, w1, w2, u[2], m[21]);+	MONT_MAC(t0, t1, t2, u[3], m[20]);+	MONT_MAC(v0, v1, v2, u[4], m[19]);+	MONT_MAC(w0, w1, w2, u[5], m[18]);+	MONT_MAC(t0, t1, t2, u[6], m[17]);+	MONT_MAC(v0, v1, v2, u[7], m[16]);+	MONT_MAC(w0, w1, w2, u[8], m[15]);+	MONT_MAC(t0, t1, t2, u[9], m[14]);+	MONT_MAC(v0, v1, v2, u[10], m[13]);+	MONT_MAC(w0, w1, w2, u[11], m[12]);+	MONT_MAC(t0, t1, t2, u[12], m[11]);+	MONT_MAC(v0, v1, v2, u[13], m[10]);+	MONT_MAC(w0, w1, w2, u[14], m[9]);+	MONT_MAC(t0, t1, t2, u[15], m[8]);+	MONT_MAC(v0, v1, v2, u[16], m[7]);+	MONT_MAC(w0, w1, w2, u[17], m[6]);+	MONT_MAC(t0, t1, t2, u[18], m[5]);+	MONT_MAC(v0, v1, v2, u[19], m[4]);+	MONT_MAC(w0, w1, w2, u[20], m[3]);+	MONT_MAC(t0, t1, t2, u[21], m[2]);+	MONT_MAC(v0, v1, v2, u[22], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[23] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[23], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 24 */+	MONT_MAC(t0, t1, t2, a[0], b[24]);+	MONT_MUL(v0, v1, v2, a[1], b[23]);+	MONT_MUL(w0, w1, w2, a[2], b[22]);+	MONT_MAC(t0, t1, t2, a[3], b[21]);+	MONT_MAC(v0, v1, v2, a[4], b[20]);+	MONT_MAC(w0, w1, w2, a[5], b[19]);+	MONT_MAC(t0, t1, t2, a[6], b[18]);+	MONT_MAC(v0, v1, v2, a[7], b[17]);+	MONT_MAC(w0, w1, w2, a[8], b[16]);+	MONT_MAC(t0, t1, t2, a[9], b[15]);+	MONT_MAC(v0, v1, v2, a[10], b[14]);+	MONT_MAC(w0, w1, w2, a[11], b[13]);+	MONT_MAC(t0, t1, t2, a[12], b[12]);+	MONT_MAC(v0, v1, v2, a[13], b[11]);+	MONT_MAC(w0, w1, w2, a[14], b[10]);+	MONT_MAC(t0, t1, t2, a[15], b[9]);+	MONT_MAC(v0, v1, v2, a[16], b[8]);+	MONT_MAC(w0, w1, w2, a[17], b[7]);+	MONT_MAC(t0, t1, t2, a[18], b[6]);+	MONT_MAC(v0, v1, v2, a[19], b[5]);+	MONT_MAC(w0, w1, w2, a[20], b[4]);+	MONT_MAC(t0, t1, t2, a[21], b[3]);+	MONT_MAC(v0, v1, v2, a[22], b[2]);+	MONT_MAC(w0, w1, w2, a[23], b[1]);+	MONT_MAC(t0, t1, t2, a[24], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[24]);+	MONT_MAC(w0, w1, w2, u[1], m[23]);+	MONT_MAC(t0, t1, t2, u[2], m[22]);+	MONT_MAC(v0, v1, v2, u[3], m[21]);+	MONT_MAC(w0, w1, w2, u[4], m[20]);+	MONT_MAC(t0, t1, t2, u[5], m[19]);+	MONT_MAC(v0, v1, v2, u[6], m[18]);+	MONT_MAC(w0, w1, w2, u[7], m[17]);+	MONT_MAC(t0, t1, t2, u[8], m[16]);+	MONT_MAC(v0, v1, v2, u[9], m[15]);+	MONT_MAC(w0, w1, w2, u[10], m[14]);+	MONT_MAC(t0, t1, t2, u[11], m[13]);+	MONT_MAC(v0, v1, v2, u[12], m[12]);+	MONT_MAC(w0, w1, w2, u[13], m[11]);+	MONT_MAC(t0, t1, t2, u[14], m[10]);+	MONT_MAC(v0, v1, v2, u[15], m[9]);+	MONT_MAC(w0, w1, w2, u[16], m[8]);+	MONT_MAC(t0, t1, t2, u[17], m[7]);+	MONT_MAC(v0, v1, v2, u[18], m[6]);+	MONT_MAC(w0, w1, w2, u[19], m[5]);+	MONT_MAC(t0, t1, t2, u[20], m[4]);+	MONT_MAC(v0, v1, v2, u[21], m[3]);+	MONT_MAC(w0, w1, w2, u[22], m[2]);+	MONT_MAC(t0, t1, t2, u[23], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[24] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[24], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 25 */+	MONT_MAC(t0, t1, t2, a[0], b[25]);+	MONT_MUL(v0, v1, v2, a[1], b[24]);+	MONT_MUL(w0, w1, w2, a[2], b[23]);+	MONT_MAC(t0, t1, t2, a[3], b[22]);+	MONT_MAC(v0, v1, v2, a[4], b[21]);+	MONT_MAC(w0, w1, w2, a[5], b[20]);+	MONT_MAC(t0, t1, t2, a[6], b[19]);+	MONT_MAC(v0, v1, v2, a[7], b[18]);+	MONT_MAC(w0, w1, w2, a[8], b[17]);+	MONT_MAC(t0, t1, t2, a[9], b[16]);+	MONT_MAC(v0, v1, v2, a[10], b[15]);+	MONT_MAC(w0, w1, w2, a[11], b[14]);+	MONT_MAC(t0, t1, t2, a[12], b[13]);+	MONT_MAC(v0, v1, v2, a[13], b[12]);+	MONT_MAC(w0, w1, w2, a[14], b[11]);+	MONT_MAC(t0, t1, t2, a[15], b[10]);+	MONT_MAC(v0, v1, v2, a[16], b[9]);+	MONT_MAC(w0, w1, w2, a[17], b[8]);+	MONT_MAC(t0, t1, t2, a[18], b[7]);+	MONT_MAC(v0, v1, v2, a[19], b[6]);+	MONT_MAC(w0, w1, w2, a[20], b[5]);+	MONT_MAC(t0, t1, t2, a[21], b[4]);+	MONT_MAC(v0, v1, v2, a[22], b[3]);+	MONT_MAC(w0, w1, w2, a[23], b[2]);+	MONT_MAC(t0, t1, t2, a[24], b[1]);+	MONT_MAC(v0, v1, v2, a[25], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[25]);+	MONT_MAC(t0, t1, t2, u[1], m[24]);+	MONT_MAC(v0, v1, v2, u[2], m[23]);+	MONT_MAC(w0, w1, w2, u[3], m[22]);+	MONT_MAC(t0, t1, t2, u[4], m[21]);+	MONT_MAC(v0, v1, v2, u[5], m[20]);+	MONT_MAC(w0, w1, w2, u[6], m[19]);+	MONT_MAC(t0, t1, t2, u[7], m[18]);+	MONT_MAC(v0, v1, v2, u[8], m[17]);+	MONT_MAC(w0, w1, w2, u[9], m[16]);+	MONT_MAC(t0, t1, t2, u[10], m[15]);+	MONT_MAC(v0, v1, v2, u[11], m[14]);+	MONT_MAC(w0, w1, w2, u[12], m[13]);+	MONT_MAC(t0, t1, t2, u[13], m[12]);+	MONT_MAC(v0, v1, v2, u[14], m[11]);+	MONT_MAC(w0, w1, w2, u[15], m[10]);+	MONT_MAC(t0, t1, t2, u[16], m[9]);+	MONT_MAC(v0, v1, v2, u[17], m[8]);+	MONT_MAC(w0, w1, w2, u[18], m[7]);+	MONT_MAC(t0, t1, t2, u[19], m[6]);+	MONT_MAC(v0, v1, v2, u[20], m[5]);+	MONT_MAC(w0, w1, w2, u[21], m[4]);+	MONT_MAC(t0, t1, t2, u[22], m[3]);+	MONT_MAC(v0, v1, v2, u[23], m[2]);+	MONT_MAC(w0, w1, w2, u[24], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[25] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[25], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 26 */+	MONT_MAC(t0, t1, t2, a[0], b[26]);+	MONT_MUL(v0, v1, v2, a[1], b[25]);+	MONT_MUL(w0, w1, w2, a[2], b[24]);+	MONT_MAC(t0, t1, t2, a[3], b[23]);+	MONT_MAC(v0, v1, v2, a[4], b[22]);+	MONT_MAC(w0, w1, w2, a[5], b[21]);+	MONT_MAC(t0, t1, t2, a[6], b[20]);+	MONT_MAC(v0, v1, v2, a[7], b[19]);+	MONT_MAC(w0, w1, w2, a[8], b[18]);+	MONT_MAC(t0, t1, t2, a[9], b[17]);+	MONT_MAC(v0, v1, v2, a[10], b[16]);+	MONT_MAC(w0, w1, w2, a[11], b[15]);+	MONT_MAC(t0, t1, t2, a[12], b[14]);+	MONT_MAC(v0, v1, v2, a[13], b[13]);+	MONT_MAC(w0, w1, w2, a[14], b[12]);+	MONT_MAC(t0, t1, t2, a[15], b[11]);+	MONT_MAC(v0, v1, v2, a[16], b[10]);+	MONT_MAC(w0, w1, w2, a[17], b[9]);+	MONT_MAC(t0, t1, t2, a[18], b[8]);+	MONT_MAC(v0, v1, v2, a[19], b[7]);+	MONT_MAC(w0, w1, w2, a[20], b[6]);+	MONT_MAC(t0, t1, t2, a[21], b[5]);+	MONT_MAC(v0, v1, v2, a[22], b[4]);+	MONT_MAC(w0, w1, w2, a[23], b[3]);+	MONT_MAC(t0, t1, t2, a[24], b[2]);+	MONT_MAC(v0, v1, v2, a[25], b[1]);+	MONT_MAC(w0, w1, w2, a[26], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[26]);+	MONT_MAC(v0, v1, v2, u[1], m[25]);+	MONT_MAC(w0, w1, w2, u[2], m[24]);+	MONT_MAC(t0, t1, t2, u[3], m[23]);+	MONT_MAC(v0, v1, v2, u[4], m[22]);+	MONT_MAC(w0, w1, w2, u[5], m[21]);+	MONT_MAC(t0, t1, t2, u[6], m[20]);+	MONT_MAC(v0, v1, v2, u[7], m[19]);+	MONT_MAC(w0, w1, w2, u[8], m[18]);+	MONT_MAC(t0, t1, t2, u[9], m[17]);+	MONT_MAC(v0, v1, v2, u[10], m[16]);+	MONT_MAC(w0, w1, w2, u[11], m[15]);+	MONT_MAC(t0, t1, t2, u[12], m[14]);+	MONT_MAC(v0, v1, v2, u[13], m[13]);+	MONT_MAC(w0, w1, w2, u[14], m[12]);+	MONT_MAC(t0, t1, t2, u[15], m[11]);+	MONT_MAC(v0, v1, v2, u[16], m[10]);+	MONT_MAC(w0, w1, w2, u[17], m[9]);+	MONT_MAC(t0, t1, t2, u[18], m[8]);+	MONT_MAC(v0, v1, v2, u[19], m[7]);+	MONT_MAC(w0, w1, w2, u[20], m[6]);+	MONT_MAC(t0, t1, t2, u[21], m[5]);+	MONT_MAC(v0, v1, v2, u[22], m[4]);+	MONT_MAC(w0, w1, w2, u[23], m[3]);+	MONT_MAC(t0, t1, t2, u[24], m[2]);+	MONT_MAC(v0, v1, v2, u[25], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[26] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[26], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 27 */+	MONT_MAC(t0, t1, t2, a[0], b[27]);+	MONT_MUL(v0, v1, v2, a[1], b[26]);+	MONT_MUL(w0, w1, w2, a[2], b[25]);+	MONT_MAC(t0, t1, t2, a[3], b[24]);+	MONT_MAC(v0, v1, v2, a[4], b[23]);+	MONT_MAC(w0, w1, w2, a[5], b[22]);+	MONT_MAC(t0, t1, t2, a[6], b[21]);+	MONT_MAC(v0, v1, v2, a[7], b[20]);+	MONT_MAC(w0, w1, w2, a[8], b[19]);+	MONT_MAC(t0, t1, t2, a[9], b[18]);+	MONT_MAC(v0, v1, v2, a[10], b[17]);+	MONT_MAC(w0, w1, w2, a[11], b[16]);+	MONT_MAC(t0, t1, t2, a[12], b[15]);+	MONT_MAC(v0, v1, v2, a[13], b[14]);+	MONT_MAC(w0, w1, w2, a[14], b[13]);+	MONT_MAC(t0, t1, t2, a[15], b[12]);+	MONT_MAC(v0, v1, v2, a[16], b[11]);+	MONT_MAC(w0, w1, w2, a[17], b[10]);+	MONT_MAC(t0, t1, t2, a[18], b[9]);+	MONT_MAC(v0, v1, v2, a[19], b[8]);+	MONT_MAC(w0, w1, w2, a[20], b[7]);+	MONT_MAC(t0, t1, t2, a[21], b[6]);+	MONT_MAC(v0, v1, v2, a[22], b[5]);+	MONT_MAC(w0, w1, w2, a[23], b[4]);+	MONT_MAC(t0, t1, t2, a[24], b[3]);+	MONT_MAC(v0, v1, v2, a[25], b[2]);+	MONT_MAC(w0, w1, w2, a[26], b[1]);+	MONT_MAC(t0, t1, t2, a[27], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[27]);+	MONT_MAC(w0, w1, w2, u[1], m[26]);+	MONT_MAC(t0, t1, t2, u[2], m[25]);+	MONT_MAC(v0, v1, v2, u[3], m[24]);+	MONT_MAC(w0, w1, w2, u[4], m[23]);+	MONT_MAC(t0, t1, t2, u[5], m[22]);+	MONT_MAC(v0, v1, v2, u[6], m[21]);+	MONT_MAC(w0, w1, w2, u[7], m[20]);+	MONT_MAC(t0, t1, t2, u[8], m[19]);+	MONT_MAC(v0, v1, v2, u[9], m[18]);+	MONT_MAC(w0, w1, w2, u[10], m[17]);+	MONT_MAC(t0, t1, t2, u[11], m[16]);+	MONT_MAC(v0, v1, v2, u[12], m[15]);+	MONT_MAC(w0, w1, w2, u[13], m[14]);+	MONT_MAC(t0, t1, t2, u[14], m[13]);+	MONT_MAC(v0, v1, v2, u[15], m[12]);+	MONT_MAC(w0, w1, w2, u[16], m[11]);+	MONT_MAC(t0, t1, t2, u[17], m[10]);+	MONT_MAC(v0, v1, v2, u[18], m[9]);+	MONT_MAC(w0, w1, w2, u[19], m[8]);+	MONT_MAC(t0, t1, t2, u[20], m[7]);+	MONT_MAC(v0, v1, v2, u[21], m[6]);+	MONT_MAC(w0, w1, w2, u[22], m[5]);+	MONT_MAC(t0, t1, t2, u[23], m[4]);+	MONT_MAC(v0, v1, v2, u[24], m[3]);+	MONT_MAC(w0, w1, w2, u[25], m[2]);+	MONT_MAC(t0, t1, t2, u[26], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[27] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[27], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 28 */+	MONT_MAC(t0, t1, t2, a[0], b[28]);+	MONT_MUL(v0, v1, v2, a[1], b[27]);+	MONT_MUL(w0, w1, w2, a[2], b[26]);+	MONT_MAC(t0, t1, t2, a[3], b[25]);+	MONT_MAC(v0, v1, v2, a[4], b[24]);+	MONT_MAC(w0, w1, w2, a[5], b[23]);+	MONT_MAC(t0, t1, t2, a[6], b[22]);+	MONT_MAC(v0, v1, v2, a[7], b[21]);+	MONT_MAC(w0, w1, w2, a[8], b[20]);+	MONT_MAC(t0, t1, t2, a[9], b[19]);+	MONT_MAC(v0, v1, v2, a[10], b[18]);+	MONT_MAC(w0, w1, w2, a[11], b[17]);+	MONT_MAC(t0, t1, t2, a[12], b[16]);+	MONT_MAC(v0, v1, v2, a[13], b[15]);+	MONT_MAC(w0, w1, w2, a[14], b[14]);+	MONT_MAC(t0, t1, t2, a[15], b[13]);+	MONT_MAC(v0, v1, v2, a[16], b[12]);+	MONT_MAC(w0, w1, w2, a[17], b[11]);+	MONT_MAC(t0, t1, t2, a[18], b[10]);+	MONT_MAC(v0, v1, v2, a[19], b[9]);+	MONT_MAC(w0, w1, w2, a[20], b[8]);+	MONT_MAC(t0, t1, t2, a[21], b[7]);+	MONT_MAC(v0, v1, v2, a[22], b[6]);+	MONT_MAC(w0, w1, w2, a[23], b[5]);+	MONT_MAC(t0, t1, t2, a[24], b[4]);+	MONT_MAC(v0, v1, v2, a[25], b[3]);+	MONT_MAC(w0, w1, w2, a[26], b[2]);+	MONT_MAC(t0, t1, t2, a[27], b[1]);+	MONT_MAC(v0, v1, v2, a[28], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[28]);+	MONT_MAC(t0, t1, t2, u[1], m[27]);+	MONT_MAC(v0, v1, v2, u[2], m[26]);+	MONT_MAC(w0, w1, w2, u[3], m[25]);+	MONT_MAC(t0, t1, t2, u[4], m[24]);+	MONT_MAC(v0, v1, v2, u[5], m[23]);+	MONT_MAC(w0, w1, w2, u[6], m[22]);+	MONT_MAC(t0, t1, t2, u[7], m[21]);+	MONT_MAC(v0, v1, v2, u[8], m[20]);+	MONT_MAC(w0, w1, w2, u[9], m[19]);+	MONT_MAC(t0, t1, t2, u[10], m[18]);+	MONT_MAC(v0, v1, v2, u[11], m[17]);+	MONT_MAC(w0, w1, w2, u[12], m[16]);+	MONT_MAC(t0, t1, t2, u[13], m[15]);+	MONT_MAC(v0, v1, v2, u[14], m[14]);+	MONT_MAC(w0, w1, w2, u[15], m[13]);+	MONT_MAC(t0, t1, t2, u[16], m[12]);+	MONT_MAC(v0, v1, v2, u[17], m[11]);+	MONT_MAC(w0, w1, w2, u[18], m[10]);+	MONT_MAC(t0, t1, t2, u[19], m[9]);+	MONT_MAC(v0, v1, v2, u[20], m[8]);+	MONT_MAC(w0, w1, w2, u[21], m[7]);+	MONT_MAC(t0, t1, t2, u[22], m[6]);+	MONT_MAC(v0, v1, v2, u[23], m[5]);+	MONT_MAC(w0, w1, w2, u[24], m[4]);+	MONT_MAC(t0, t1, t2, u[25], m[3]);+	MONT_MAC(v0, v1, v2, u[26], m[2]);+	MONT_MAC(w0, w1, w2, u[27], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[28] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[28], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 29 */+	MONT_MAC(t0, t1, t2, a[0], b[29]);+	MONT_MUL(v0, v1, v2, a[1], b[28]);+	MONT_MUL(w0, w1, w2, a[2], b[27]);+	MONT_MAC(t0, t1, t2, a[3], b[26]);+	MONT_MAC(v0, v1, v2, a[4], b[25]);+	MONT_MAC(w0, w1, w2, a[5], b[24]);+	MONT_MAC(t0, t1, t2, a[6], b[23]);+	MONT_MAC(v0, v1, v2, a[7], b[22]);+	MONT_MAC(w0, w1, w2, a[8], b[21]);+	MONT_MAC(t0, t1, t2, a[9], b[20]);+	MONT_MAC(v0, v1, v2, a[10], b[19]);+	MONT_MAC(w0, w1, w2, a[11], b[18]);+	MONT_MAC(t0, t1, t2, a[12], b[17]);+	MONT_MAC(v0, v1, v2, a[13], b[16]);+	MONT_MAC(w0, w1, w2, a[14], b[15]);+	MONT_MAC(t0, t1, t2, a[15], b[14]);+	MONT_MAC(v0, v1, v2, a[16], b[13]);+	MONT_MAC(w0, w1, w2, a[17], b[12]);+	MONT_MAC(t0, t1, t2, a[18], b[11]);+	MONT_MAC(v0, v1, v2, a[19], b[10]);+	MONT_MAC(w0, w1, w2, a[20], b[9]);+	MONT_MAC(t0, t1, t2, a[21], b[8]);+	MONT_MAC(v0, v1, v2, a[22], b[7]);+	MONT_MAC(w0, w1, w2, a[23], b[6]);+	MONT_MAC(t0, t1, t2, a[24], b[5]);+	MONT_MAC(v0, v1, v2, a[25], b[4]);+	MONT_MAC(w0, w1, w2, a[26], b[3]);+	MONT_MAC(t0, t1, t2, a[27], b[2]);+	MONT_MAC(v0, v1, v2, a[28], b[1]);+	MONT_MAC(w0, w1, w2, a[29], b[0]);+	MONT_MAC(t0, t1, t2, u[0], m[29]);+	MONT_MAC(v0, v1, v2, u[1], m[28]);+	MONT_MAC(w0, w1, w2, u[2], m[27]);+	MONT_MAC(t0, t1, t2, u[3], m[26]);+	MONT_MAC(v0, v1, v2, u[4], m[25]);+	MONT_MAC(w0, w1, w2, u[5], m[24]);+	MONT_MAC(t0, t1, t2, u[6], m[23]);+	MONT_MAC(v0, v1, v2, u[7], m[22]);+	MONT_MAC(w0, w1, w2, u[8], m[21]);+	MONT_MAC(t0, t1, t2, u[9], m[20]);+	MONT_MAC(v0, v1, v2, u[10], m[19]);+	MONT_MAC(w0, w1, w2, u[11], m[18]);+	MONT_MAC(t0, t1, t2, u[12], m[17]);+	MONT_MAC(v0, v1, v2, u[13], m[16]);+	MONT_MAC(w0, w1, w2, u[14], m[15]);+	MONT_MAC(t0, t1, t2, u[15], m[14]);+	MONT_MAC(v0, v1, v2, u[16], m[13]);+	MONT_MAC(w0, w1, w2, u[17], m[12]);+	MONT_MAC(t0, t1, t2, u[18], m[11]);+	MONT_MAC(v0, v1, v2, u[19], m[10]);+	MONT_MAC(w0, w1, w2, u[20], m[9]);+	MONT_MAC(t0, t1, t2, u[21], m[8]);+	MONT_MAC(v0, v1, v2, u[22], m[7]);+	MONT_MAC(w0, w1, w2, u[23], m[6]);+	MONT_MAC(t0, t1, t2, u[24], m[5]);+	MONT_MAC(v0, v1, v2, u[25], m[4]);+	MONT_MAC(w0, w1, w2, u[26], m[3]);+	MONT_MAC(t0, t1, t2, u[27], m[2]);+	MONT_MAC(v0, v1, v2, u[28], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[29] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[29], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 30 */+	MONT_MAC(t0, t1, t2, a[0], b[30]);+	MONT_MUL(v0, v1, v2, a[1], b[29]);+	MONT_MUL(w0, w1, w2, a[2], b[28]);+	MONT_MAC(t0, t1, t2, a[3], b[27]);+	MONT_MAC(v0, v1, v2, a[4], b[26]);+	MONT_MAC(w0, w1, w2, a[5], b[25]);+	MONT_MAC(t0, t1, t2, a[6], b[24]);+	MONT_MAC(v0, v1, v2, a[7], b[23]);+	MONT_MAC(w0, w1, w2, a[8], b[22]);+	MONT_MAC(t0, t1, t2, a[9], b[21]);+	MONT_MAC(v0, v1, v2, a[10], b[20]);+	MONT_MAC(w0, w1, w2, a[11], b[19]);+	MONT_MAC(t0, t1, t2, a[12], b[18]);+	MONT_MAC(v0, v1, v2, a[13], b[17]);+	MONT_MAC(w0, w1, w2, a[14], b[16]);+	MONT_MAC(t0, t1, t2, a[15], b[15]);+	MONT_MAC(v0, v1, v2, a[16], b[14]);+	MONT_MAC(w0, w1, w2, a[17], b[13]);+	MONT_MAC(t0, t1, t2, a[18], b[12]);+	MONT_MAC(v0, v1, v2, a[19], b[11]);+	MONT_MAC(w0, w1, w2, a[20], b[10]);+	MONT_MAC(t0, t1, t2, a[21], b[9]);+	MONT_MAC(v0, v1, v2, a[22], b[8]);+	MONT_MAC(w0, w1, w2, a[23], b[7]);+	MONT_MAC(t0, t1, t2, a[24], b[6]);+	MONT_MAC(v0, v1, v2, a[25], b[5]);+	MONT_MAC(w0, w1, w2, a[26], b[4]);+	MONT_MAC(t0, t1, t2, a[27], b[3]);+	MONT_MAC(v0, v1, v2, a[28], b[2]);+	MONT_MAC(w0, w1, w2, a[29], b[1]);+	MONT_MAC(t0, t1, t2, a[30], b[0]);+	MONT_MAC(v0, v1, v2, u[0], m[30]);+	MONT_MAC(w0, w1, w2, u[1], m[29]);+	MONT_MAC(t0, t1, t2, u[2], m[28]);+	MONT_MAC(v0, v1, v2, u[3], m[27]);+	MONT_MAC(w0, w1, w2, u[4], m[26]);+	MONT_MAC(t0, t1, t2, u[5], m[25]);+	MONT_MAC(v0, v1, v2, u[6], m[24]);+	MONT_MAC(w0, w1, w2, u[7], m[23]);+	MONT_MAC(t0, t1, t2, u[8], m[22]);+	MONT_MAC(v0, v1, v2, u[9], m[21]);+	MONT_MAC(w0, w1, w2, u[10], m[20]);+	MONT_MAC(t0, t1, t2, u[11], m[19]);+	MONT_MAC(v0, v1, v2, u[12], m[18]);+	MONT_MAC(w0, w1, w2, u[13], m[17]);+	MONT_MAC(t0, t1, t2, u[14], m[16]);+	MONT_MAC(v0, v1, v2, u[15], m[15]);+	MONT_MAC(w0, w1, w2, u[16], m[14]);+	MONT_MAC(t0, t1, t2, u[17], m[13]);+	MONT_MAC(v0, v1, v2, u[18], m[12]);+	MONT_MAC(w0, w1, w2, u[19], m[11]);+	MONT_MAC(t0, t1, t2, u[20], m[10]);+	MONT_MAC(v0, v1, v2, u[21], m[9]);+	MONT_MAC(w0, w1, w2, u[22], m[8]);+	MONT_MAC(t0, t1, t2, u[23], m[7]);+	MONT_MAC(v0, v1, v2, u[24], m[6]);+	MONT_MAC(w0, w1, w2, u[25], m[5]);+	MONT_MAC(t0, t1, t2, u[26], m[4]);+	MONT_MAC(v0, v1, v2, u[27], m[3]);+	MONT_MAC(w0, w1, w2, u[28], m[2]);+	MONT_MAC(t0, t1, t2, u[29], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[30] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[30], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 31 */+	MONT_MAC(t0, t1, t2, a[0], b[31]);+	MONT_MUL(v0, v1, v2, a[1], b[30]);+	MONT_MUL(w0, w1, w2, a[2], b[29]);+	MONT_MAC(t0, t1, t2, a[3], b[28]);+	MONT_MAC(v0, v1, v2, a[4], b[27]);+	MONT_MAC(w0, w1, w2, a[5], b[26]);+	MONT_MAC(t0, t1, t2, a[6], b[25]);+	MONT_MAC(v0, v1, v2, a[7], b[24]);+	MONT_MAC(w0, w1, w2, a[8], b[23]);+	MONT_MAC(t0, t1, t2, a[9], b[22]);+	MONT_MAC(v0, v1, v2, a[10], b[21]);+	MONT_MAC(w0, w1, w2, a[11], b[20]);+	MONT_MAC(t0, t1, t2, a[12], b[19]);+	MONT_MAC(v0, v1, v2, a[13], b[18]);+	MONT_MAC(w0, w1, w2, a[14], b[17]);+	MONT_MAC(t0, t1, t2, a[15], b[16]);+	MONT_MAC(v0, v1, v2, a[16], b[15]);+	MONT_MAC(w0, w1, w2, a[17], b[14]);+	MONT_MAC(t0, t1, t2, a[18], b[13]);+	MONT_MAC(v0, v1, v2, a[19], b[12]);+	MONT_MAC(w0, w1, w2, a[20], b[11]);+	MONT_MAC(t0, t1, t2, a[21], b[10]);+	MONT_MAC(v0, v1, v2, a[22], b[9]);+	MONT_MAC(w0, w1, w2, a[23], b[8]);+	MONT_MAC(t0, t1, t2, a[24], b[7]);+	MONT_MAC(v0, v1, v2, a[25], b[6]);+	MONT_MAC(w0, w1, w2, a[26], b[5]);+	MONT_MAC(t0, t1, t2, a[27], b[4]);+	MONT_MAC(v0, v1, v2, a[28], b[3]);+	MONT_MAC(w0, w1, w2, a[29], b[2]);+	MONT_MAC(t0, t1, t2, a[30], b[1]);+	MONT_MAC(v0, v1, v2, a[31], b[0]);+	MONT_MAC(w0, w1, w2, u[0], m[31]);+	MONT_MAC(t0, t1, t2, u[1], m[30]);+	MONT_MAC(v0, v1, v2, u[2], m[29]);+	MONT_MAC(w0, w1, w2, u[3], m[28]);+	MONT_MAC(t0, t1, t2, u[4], m[27]);+	MONT_MAC(v0, v1, v2, u[5], m[26]);+	MONT_MAC(w0, w1, w2, u[6], m[25]);+	MONT_MAC(t0, t1, t2, u[7], m[24]);+	MONT_MAC(v0, v1, v2, u[8], m[23]);+	MONT_MAC(w0, w1, w2, u[9], m[22]);+	MONT_MAC(t0, t1, t2, u[10], m[21]);+	MONT_MAC(v0, v1, v2, u[11], m[20]);+	MONT_MAC(w0, w1, w2, u[12], m[19]);+	MONT_MAC(t0, t1, t2, u[13], m[18]);+	MONT_MAC(v0, v1, v2, u[14], m[17]);+	MONT_MAC(w0, w1, w2, u[15], m[16]);+	MONT_MAC(t0, t1, t2, u[16], m[15]);+	MONT_MAC(v0, v1, v2, u[17], m[14]);+	MONT_MAC(w0, w1, w2, u[18], m[13]);+	MONT_MAC(t0, t1, t2, u[19], m[12]);+	MONT_MAC(v0, v1, v2, u[20], m[11]);+	MONT_MAC(w0, w1, w2, u[21], m[10]);+	MONT_MAC(t0, t1, t2, u[22], m[9]);+	MONT_MAC(v0, v1, v2, u[23], m[8]);+	MONT_MAC(w0, w1, w2, u[24], m[7]);+	MONT_MAC(t0, t1, t2, u[25], m[6]);+	MONT_MAC(v0, v1, v2, u[26], m[5]);+	MONT_MAC(w0, w1, w2, u[27], m[4]);+	MONT_MAC(t0, t1, t2, u[28], m[3]);+	MONT_MAC(v0, v1, v2, u[29], m[2]);+	MONT_MAC(w0, w1, w2, u[30], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	u[31] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[31], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 32 */+	MONT_MAC(t0, t1, t2, a[1], b[31]);+	MONT_MUL(v0, v1, v2, a[2], b[30]);+	MONT_MUL(w0, w1, w2, a[3], b[29]);+	MONT_MAC(t0, t1, t2, a[4], b[28]);+	MONT_MAC(v0, v1, v2, a[5], b[27]);+	MONT_MAC(w0, w1, w2, a[6], b[26]);+	MONT_MAC(t0, t1, t2, a[7], b[25]);+	MONT_MAC(v0, v1, v2, a[8], b[24]);+	MONT_MAC(w0, w1, w2, a[9], b[23]);+	MONT_MAC(t0, t1, t2, a[10], b[22]);+	MONT_MAC(v0, v1, v2, a[11], b[21]);+	MONT_MAC(w0, w1, w2, a[12], b[20]);+	MONT_MAC(t0, t1, t2, a[13], b[19]);+	MONT_MAC(v0, v1, v2, a[14], b[18]);+	MONT_MAC(w0, w1, w2, a[15], b[17]);+	MONT_MAC(t0, t1, t2, a[16], b[16]);+	MONT_MAC(v0, v1, v2, a[17], b[15]);+	MONT_MAC(w0, w1, w2, a[18], b[14]);+	MONT_MAC(t0, t1, t2, a[19], b[13]);+	MONT_MAC(v0, v1, v2, a[20], b[12]);+	MONT_MAC(w0, w1, w2, a[21], b[11]);+	MONT_MAC(t0, t1, t2, a[22], b[10]);+	MONT_MAC(v0, v1, v2, a[23], b[9]);+	MONT_MAC(w0, w1, w2, a[24], b[8]);+	MONT_MAC(t0, t1, t2, a[25], b[7]);+	MONT_MAC(v0, v1, v2, a[26], b[6]);+	MONT_MAC(w0, w1, w2, a[27], b[5]);+	MONT_MAC(t0, t1, t2, a[28], b[4]);+	MONT_MAC(v0, v1, v2, a[29], b[3]);+	MONT_MAC(w0, w1, w2, a[30], b[2]);+	MONT_MAC(t0, t1, t2, a[31], b[1]);+	MONT_MAC(v0, v1, v2, u[1], m[31]);+	MONT_MAC(w0, w1, w2, u[2], m[30]);+	MONT_MAC(t0, t1, t2, u[3], m[29]);+	MONT_MAC(v0, v1, v2, u[4], m[28]);+	MONT_MAC(w0, w1, w2, u[5], m[27]);+	MONT_MAC(t0, t1, t2, u[6], m[26]);+	MONT_MAC(v0, v1, v2, u[7], m[25]);+	MONT_MAC(w0, w1, w2, u[8], m[24]);+	MONT_MAC(t0, t1, t2, u[9], m[23]);+	MONT_MAC(v0, v1, v2, u[10], m[22]);+	MONT_MAC(w0, w1, w2, u[11], m[21]);+	MONT_MAC(t0, t1, t2, u[12], m[20]);+	MONT_MAC(v0, v1, v2, u[13], m[19]);+	MONT_MAC(w0, w1, w2, u[14], m[18]);+	MONT_MAC(t0, t1, t2, u[15], m[17]);+	MONT_MAC(v0, v1, v2, u[16], m[16]);+	MONT_MAC(w0, w1, w2, u[17], m[15]);+	MONT_MAC(t0, t1, t2, u[18], m[14]);+	MONT_MAC(v0, v1, v2, u[19], m[13]);+	MONT_MAC(w0, w1, w2, u[20], m[12]);+	MONT_MAC(t0, t1, t2, u[21], m[11]);+	MONT_MAC(v0, v1, v2, u[22], m[10]);+	MONT_MAC(w0, w1, w2, u[23], m[9]);+	MONT_MAC(t0, t1, t2, u[24], m[8]);+	MONT_MAC(v0, v1, v2, u[25], m[7]);+	MONT_MAC(w0, w1, w2, u[26], m[6]);+	MONT_MAC(t0, t1, t2, u[27], m[5]);+	MONT_MAC(v0, v1, v2, u[28], m[4]);+	MONT_MAC(w0, w1, w2, u[29], m[3]);+	MONT_MAC(t0, t1, t2, u[30], m[2]);+	MONT_MAC(v0, v1, v2, u[31], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[0] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 33 */+	MONT_MAC(t0, t1, t2, a[2], b[31]);+	MONT_MUL(v0, v1, v2, a[3], b[30]);+	MONT_MUL(w0, w1, w2, a[4], b[29]);+	MONT_MAC(t0, t1, t2, a[5], b[28]);+	MONT_MAC(v0, v1, v2, a[6], b[27]);+	MONT_MAC(w0, w1, w2, a[7], b[26]);+	MONT_MAC(t0, t1, t2, a[8], b[25]);+	MONT_MAC(v0, v1, v2, a[9], b[24]);+	MONT_MAC(w0, w1, w2, a[10], b[23]);+	MONT_MAC(t0, t1, t2, a[11], b[22]);+	MONT_MAC(v0, v1, v2, a[12], b[21]);+	MONT_MAC(w0, w1, w2, a[13], b[20]);+	MONT_MAC(t0, t1, t2, a[14], b[19]);+	MONT_MAC(v0, v1, v2, a[15], b[18]);+	MONT_MAC(w0, w1, w2, a[16], b[17]);+	MONT_MAC(t0, t1, t2, a[17], b[16]);+	MONT_MAC(v0, v1, v2, a[18], b[15]);+	MONT_MAC(w0, w1, w2, a[19], b[14]);+	MONT_MAC(t0, t1, t2, a[20], b[13]);+	MONT_MAC(v0, v1, v2, a[21], b[12]);+	MONT_MAC(w0, w1, w2, a[22], b[11]);+	MONT_MAC(t0, t1, t2, a[23], b[10]);+	MONT_MAC(v0, v1, v2, a[24], b[9]);+	MONT_MAC(w0, w1, w2, a[25], b[8]);+	MONT_MAC(t0, t1, t2, a[26], b[7]);+	MONT_MAC(v0, v1, v2, a[27], b[6]);+	MONT_MAC(w0, w1, w2, a[28], b[5]);+	MONT_MAC(t0, t1, t2, a[29], b[4]);+	MONT_MAC(v0, v1, v2, a[30], b[3]);+	MONT_MAC(w0, w1, w2, a[31], b[2]);+	MONT_MAC(t0, t1, t2, u[2], m[31]);+	MONT_MAC(v0, v1, v2, u[3], m[30]);+	MONT_MAC(w0, w1, w2, u[4], m[29]);+	MONT_MAC(t0, t1, t2, u[5], m[28]);+	MONT_MAC(v0, v1, v2, u[6], m[27]);+	MONT_MAC(w0, w1, w2, u[7], m[26]);+	MONT_MAC(t0, t1, t2, u[8], m[25]);+	MONT_MAC(v0, v1, v2, u[9], m[24]);+	MONT_MAC(w0, w1, w2, u[10], m[23]);+	MONT_MAC(t0, t1, t2, u[11], m[22]);+	MONT_MAC(v0, v1, v2, u[12], m[21]);+	MONT_MAC(w0, w1, w2, u[13], m[20]);+	MONT_MAC(t0, t1, t2, u[14], m[19]);+	MONT_MAC(v0, v1, v2, u[15], m[18]);+	MONT_MAC(w0, w1, w2, u[16], m[17]);+	MONT_MAC(t0, t1, t2, u[17], m[16]);+	MONT_MAC(v0, v1, v2, u[18], m[15]);+	MONT_MAC(w0, w1, w2, u[19], m[14]);+	MONT_MAC(t0, t1, t2, u[20], m[13]);+	MONT_MAC(v0, v1, v2, u[21], m[12]);+	MONT_MAC(w0, w1, w2, u[22], m[11]);+	MONT_MAC(t0, t1, t2, u[23], m[10]);+	MONT_MAC(v0, v1, v2, u[24], m[9]);+	MONT_MAC(w0, w1, w2, u[25], m[8]);+	MONT_MAC(t0, t1, t2, u[26], m[7]);+	MONT_MAC(v0, v1, v2, u[27], m[6]);+	MONT_MAC(w0, w1, w2, u[28], m[5]);+	MONT_MAC(t0, t1, t2, u[29], m[4]);+	MONT_MAC(v0, v1, v2, u[30], m[3]);+	MONT_MAC(w0, w1, w2, u[31], m[2]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[1] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 34 */+	MONT_MAC(t0, t1, t2, a[3], b[31]);+	MONT_MUL(v0, v1, v2, a[4], b[30]);+	MONT_MUL(w0, w1, w2, a[5], b[29]);+	MONT_MAC(t0, t1, t2, a[6], b[28]);+	MONT_MAC(v0, v1, v2, a[7], b[27]);+	MONT_MAC(w0, w1, w2, a[8], b[26]);+	MONT_MAC(t0, t1, t2, a[9], b[25]);+	MONT_MAC(v0, v1, v2, a[10], b[24]);+	MONT_MAC(w0, w1, w2, a[11], b[23]);+	MONT_MAC(t0, t1, t2, a[12], b[22]);+	MONT_MAC(v0, v1, v2, a[13], b[21]);+	MONT_MAC(w0, w1, w2, a[14], b[20]);+	MONT_MAC(t0, t1, t2, a[15], b[19]);+	MONT_MAC(v0, v1, v2, a[16], b[18]);+	MONT_MAC(w0, w1, w2, a[17], b[17]);+	MONT_MAC(t0, t1, t2, a[18], b[16]);+	MONT_MAC(v0, v1, v2, a[19], b[15]);+	MONT_MAC(w0, w1, w2, a[20], b[14]);+	MONT_MAC(t0, t1, t2, a[21], b[13]);+	MONT_MAC(v0, v1, v2, a[22], b[12]);+	MONT_MAC(w0, w1, w2, a[23], b[11]);+	MONT_MAC(t0, t1, t2, a[24], b[10]);+	MONT_MAC(v0, v1, v2, a[25], b[9]);+	MONT_MAC(w0, w1, w2, a[26], b[8]);+	MONT_MAC(t0, t1, t2, a[27], b[7]);+	MONT_MAC(v0, v1, v2, a[28], b[6]);+	MONT_MAC(w0, w1, w2, a[29], b[5]);+	MONT_MAC(t0, t1, t2, a[30], b[4]);+	MONT_MAC(v0, v1, v2, a[31], b[3]);+	MONT_MAC(w0, w1, w2, u[3], m[31]);+	MONT_MAC(t0, t1, t2, u[4], m[30]);+	MONT_MAC(v0, v1, v2, u[5], m[29]);+	MONT_MAC(w0, w1, w2, u[6], m[28]);+	MONT_MAC(t0, t1, t2, u[7], m[27]);+	MONT_MAC(v0, v1, v2, u[8], m[26]);+	MONT_MAC(w0, w1, w2, u[9], m[25]);+	MONT_MAC(t0, t1, t2, u[10], m[24]);+	MONT_MAC(v0, v1, v2, u[11], m[23]);+	MONT_MAC(w0, w1, w2, u[12], m[22]);+	MONT_MAC(t0, t1, t2, u[13], m[21]);+	MONT_MAC(v0, v1, v2, u[14], m[20]);+	MONT_MAC(w0, w1, w2, u[15], m[19]);+	MONT_MAC(t0, t1, t2, u[16], m[18]);+	MONT_MAC(v0, v1, v2, u[17], m[17]);+	MONT_MAC(w0, w1, w2, u[18], m[16]);+	MONT_MAC(t0, t1, t2, u[19], m[15]);+	MONT_MAC(v0, v1, v2, u[20], m[14]);+	MONT_MAC(w0, w1, w2, u[21], m[13]);+	MONT_MAC(t0, t1, t2, u[22], m[12]);+	MONT_MAC(v0, v1, v2, u[23], m[11]);+	MONT_MAC(w0, w1, w2, u[24], m[10]);+	MONT_MAC(t0, t1, t2, u[25], m[9]);+	MONT_MAC(v0, v1, v2, u[26], m[8]);+	MONT_MAC(w0, w1, w2, u[27], m[7]);+	MONT_MAC(t0, t1, t2, u[28], m[6]);+	MONT_MAC(v0, v1, v2, u[29], m[5]);+	MONT_MAC(w0, w1, w2, u[30], m[4]);+	MONT_MAC(t0, t1, t2, u[31], m[3]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[2] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 35 */+	MONT_MAC(t0, t1, t2, a[4], b[31]);+	MONT_MUL(v0, v1, v2, a[5], b[30]);+	MONT_MUL(w0, w1, w2, a[6], b[29]);+	MONT_MAC(t0, t1, t2, a[7], b[28]);+	MONT_MAC(v0, v1, v2, a[8], b[27]);+	MONT_MAC(w0, w1, w2, a[9], b[26]);+	MONT_MAC(t0, t1, t2, a[10], b[25]);+	MONT_MAC(v0, v1, v2, a[11], b[24]);+	MONT_MAC(w0, w1, w2, a[12], b[23]);+	MONT_MAC(t0, t1, t2, a[13], b[22]);+	MONT_MAC(v0, v1, v2, a[14], b[21]);+	MONT_MAC(w0, w1, w2, a[15], b[20]);+	MONT_MAC(t0, t1, t2, a[16], b[19]);+	MONT_MAC(v0, v1, v2, a[17], b[18]);+	MONT_MAC(w0, w1, w2, a[18], b[17]);+	MONT_MAC(t0, t1, t2, a[19], b[16]);+	MONT_MAC(v0, v1, v2, a[20], b[15]);+	MONT_MAC(w0, w1, w2, a[21], b[14]);+	MONT_MAC(t0, t1, t2, a[22], b[13]);+	MONT_MAC(v0, v1, v2, a[23], b[12]);+	MONT_MAC(w0, w1, w2, a[24], b[11]);+	MONT_MAC(t0, t1, t2, a[25], b[10]);+	MONT_MAC(v0, v1, v2, a[26], b[9]);+	MONT_MAC(w0, w1, w2, a[27], b[8]);+	MONT_MAC(t0, t1, t2, a[28], b[7]);+	MONT_MAC(v0, v1, v2, a[29], b[6]);+	MONT_MAC(w0, w1, w2, a[30], b[5]);+	MONT_MAC(t0, t1, t2, a[31], b[4]);+	MONT_MAC(v0, v1, v2, u[4], m[31]);+	MONT_MAC(w0, w1, w2, u[5], m[30]);+	MONT_MAC(t0, t1, t2, u[6], m[29]);+	MONT_MAC(v0, v1, v2, u[7], m[28]);+	MONT_MAC(w0, w1, w2, u[8], m[27]);+	MONT_MAC(t0, t1, t2, u[9], m[26]);+	MONT_MAC(v0, v1, v2, u[10], m[25]);+	MONT_MAC(w0, w1, w2, u[11], m[24]);+	MONT_MAC(t0, t1, t2, u[12], m[23]);+	MONT_MAC(v0, v1, v2, u[13], m[22]);+	MONT_MAC(w0, w1, w2, u[14], m[21]);+	MONT_MAC(t0, t1, t2, u[15], m[20]);+	MONT_MAC(v0, v1, v2, u[16], m[19]);+	MONT_MAC(w0, w1, w2, u[17], m[18]);+	MONT_MAC(t0, t1, t2, u[18], m[17]);+	MONT_MAC(v0, v1, v2, u[19], m[16]);+	MONT_MAC(w0, w1, w2, u[20], m[15]);+	MONT_MAC(t0, t1, t2, u[21], m[14]);+	MONT_MAC(v0, v1, v2, u[22], m[13]);+	MONT_MAC(w0, w1, w2, u[23], m[12]);+	MONT_MAC(t0, t1, t2, u[24], m[11]);+	MONT_MAC(v0, v1, v2, u[25], m[10]);+	MONT_MAC(w0, w1, w2, u[26], m[9]);+	MONT_MAC(t0, t1, t2, u[27], m[8]);+	MONT_MAC(v0, v1, v2, u[28], m[7]);+	MONT_MAC(w0, w1, w2, u[29], m[6]);+	MONT_MAC(t0, t1, t2, u[30], m[5]);+	MONT_MAC(v0, v1, v2, u[31], m[4]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[3] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 36 */+	MONT_MAC(t0, t1, t2, a[5], b[31]);+	MONT_MUL(v0, v1, v2, a[6], b[30]);+	MONT_MUL(w0, w1, w2, a[7], b[29]);+	MONT_MAC(t0, t1, t2, a[8], b[28]);+	MONT_MAC(v0, v1, v2, a[9], b[27]);+	MONT_MAC(w0, w1, w2, a[10], b[26]);+	MONT_MAC(t0, t1, t2, a[11], b[25]);+	MONT_MAC(v0, v1, v2, a[12], b[24]);+	MONT_MAC(w0, w1, w2, a[13], b[23]);+	MONT_MAC(t0, t1, t2, a[14], b[22]);+	MONT_MAC(v0, v1, v2, a[15], b[21]);+	MONT_MAC(w0, w1, w2, a[16], b[20]);+	MONT_MAC(t0, t1, t2, a[17], b[19]);+	MONT_MAC(v0, v1, v2, a[18], b[18]);+	MONT_MAC(w0, w1, w2, a[19], b[17]);+	MONT_MAC(t0, t1, t2, a[20], b[16]);+	MONT_MAC(v0, v1, v2, a[21], b[15]);+	MONT_MAC(w0, w1, w2, a[22], b[14]);+	MONT_MAC(t0, t1, t2, a[23], b[13]);+	MONT_MAC(v0, v1, v2, a[24], b[12]);+	MONT_MAC(w0, w1, w2, a[25], b[11]);+	MONT_MAC(t0, t1, t2, a[26], b[10]);+	MONT_MAC(v0, v1, v2, a[27], b[9]);+	MONT_MAC(w0, w1, w2, a[28], b[8]);+	MONT_MAC(t0, t1, t2, a[29], b[7]);+	MONT_MAC(v0, v1, v2, a[30], b[6]);+	MONT_MAC(w0, w1, w2, a[31], b[5]);+	MONT_MAC(t0, t1, t2, u[5], m[31]);+	MONT_MAC(v0, v1, v2, u[6], m[30]);+	MONT_MAC(w0, w1, w2, u[7], m[29]);+	MONT_MAC(t0, t1, t2, u[8], m[28]);+	MONT_MAC(v0, v1, v2, u[9], m[27]);+	MONT_MAC(w0, w1, w2, u[10], m[26]);+	MONT_MAC(t0, t1, t2, u[11], m[25]);+	MONT_MAC(v0, v1, v2, u[12], m[24]);+	MONT_MAC(w0, w1, w2, u[13], m[23]);+	MONT_MAC(t0, t1, t2, u[14], m[22]);+	MONT_MAC(v0, v1, v2, u[15], m[21]);+	MONT_MAC(w0, w1, w2, u[16], m[20]);+	MONT_MAC(t0, t1, t2, u[17], m[19]);+	MONT_MAC(v0, v1, v2, u[18], m[18]);+	MONT_MAC(w0, w1, w2, u[19], m[17]);+	MONT_MAC(t0, t1, t2, u[20], m[16]);+	MONT_MAC(v0, v1, v2, u[21], m[15]);+	MONT_MAC(w0, w1, w2, u[22], m[14]);+	MONT_MAC(t0, t1, t2, u[23], m[13]);+	MONT_MAC(v0, v1, v2, u[24], m[12]);+	MONT_MAC(w0, w1, w2, u[25], m[11]);+	MONT_MAC(t0, t1, t2, u[26], m[10]);+	MONT_MAC(v0, v1, v2, u[27], m[9]);+	MONT_MAC(w0, w1, w2, u[28], m[8]);+	MONT_MAC(t0, t1, t2, u[29], m[7]);+	MONT_MAC(v0, v1, v2, u[30], m[6]);+	MONT_MAC(w0, w1, w2, u[31], m[5]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[4] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 37 */+	MONT_MAC(t0, t1, t2, a[6], b[31]);+	MONT_MUL(v0, v1, v2, a[7], b[30]);+	MONT_MUL(w0, w1, w2, a[8], b[29]);+	MONT_MAC(t0, t1, t2, a[9], b[28]);+	MONT_MAC(v0, v1, v2, a[10], b[27]);+	MONT_MAC(w0, w1, w2, a[11], b[26]);+	MONT_MAC(t0, t1, t2, a[12], b[25]);+	MONT_MAC(v0, v1, v2, a[13], b[24]);+	MONT_MAC(w0, w1, w2, a[14], b[23]);+	MONT_MAC(t0, t1, t2, a[15], b[22]);+	MONT_MAC(v0, v1, v2, a[16], b[21]);+	MONT_MAC(w0, w1, w2, a[17], b[20]);+	MONT_MAC(t0, t1, t2, a[18], b[19]);+	MONT_MAC(v0, v1, v2, a[19], b[18]);+	MONT_MAC(w0, w1, w2, a[20], b[17]);+	MONT_MAC(t0, t1, t2, a[21], b[16]);+	MONT_MAC(v0, v1, v2, a[22], b[15]);+	MONT_MAC(w0, w1, w2, a[23], b[14]);+	MONT_MAC(t0, t1, t2, a[24], b[13]);+	MONT_MAC(v0, v1, v2, a[25], b[12]);+	MONT_MAC(w0, w1, w2, a[26], b[11]);+	MONT_MAC(t0, t1, t2, a[27], b[10]);+	MONT_MAC(v0, v1, v2, a[28], b[9]);+	MONT_MAC(w0, w1, w2, a[29], b[8]);+	MONT_MAC(t0, t1, t2, a[30], b[7]);+	MONT_MAC(v0, v1, v2, a[31], b[6]);+	MONT_MAC(w0, w1, w2, u[6], m[31]);+	MONT_MAC(t0, t1, t2, u[7], m[30]);+	MONT_MAC(v0, v1, v2, u[8], m[29]);+	MONT_MAC(w0, w1, w2, u[9], m[28]);+	MONT_MAC(t0, t1, t2, u[10], m[27]);+	MONT_MAC(v0, v1, v2, u[11], m[26]);+	MONT_MAC(w0, w1, w2, u[12], m[25]);+	MONT_MAC(t0, t1, t2, u[13], m[24]);+	MONT_MAC(v0, v1, v2, u[14], m[23]);+	MONT_MAC(w0, w1, w2, u[15], m[22]);+	MONT_MAC(t0, t1, t2, u[16], m[21]);+	MONT_MAC(v0, v1, v2, u[17], m[20]);+	MONT_MAC(w0, w1, w2, u[18], m[19]);+	MONT_MAC(t0, t1, t2, u[19], m[18]);+	MONT_MAC(v0, v1, v2, u[20], m[17]);+	MONT_MAC(w0, w1, w2, u[21], m[16]);+	MONT_MAC(t0, t1, t2, u[22], m[15]);+	MONT_MAC(v0, v1, v2, u[23], m[14]);+	MONT_MAC(w0, w1, w2, u[24], m[13]);+	MONT_MAC(t0, t1, t2, u[25], m[12]);+	MONT_MAC(v0, v1, v2, u[26], m[11]);+	MONT_MAC(w0, w1, w2, u[27], m[10]);+	MONT_MAC(t0, t1, t2, u[28], m[9]);+	MONT_MAC(v0, v1, v2, u[29], m[8]);+	MONT_MAC(w0, w1, w2, u[30], m[7]);+	MONT_MAC(t0, t1, t2, u[31], m[6]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[5] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 38 */+	MONT_MAC(t0, t1, t2, a[7], b[31]);+	MONT_MUL(v0, v1, v2, a[8], b[30]);+	MONT_MUL(w0, w1, w2, a[9], b[29]);+	MONT_MAC(t0, t1, t2, a[10], b[28]);+	MONT_MAC(v0, v1, v2, a[11], b[27]);+	MONT_MAC(w0, w1, w2, a[12], b[26]);+	MONT_MAC(t0, t1, t2, a[13], b[25]);+	MONT_MAC(v0, v1, v2, a[14], b[24]);+	MONT_MAC(w0, w1, w2, a[15], b[23]);+	MONT_MAC(t0, t1, t2, a[16], b[22]);+	MONT_MAC(v0, v1, v2, a[17], b[21]);+	MONT_MAC(w0, w1, w2, a[18], b[20]);+	MONT_MAC(t0, t1, t2, a[19], b[19]);+	MONT_MAC(v0, v1, v2, a[20], b[18]);+	MONT_MAC(w0, w1, w2, a[21], b[17]);+	MONT_MAC(t0, t1, t2, a[22], b[16]);+	MONT_MAC(v0, v1, v2, a[23], b[15]);+	MONT_MAC(w0, w1, w2, a[24], b[14]);+	MONT_MAC(t0, t1, t2, a[25], b[13]);+	MONT_MAC(v0, v1, v2, a[26], b[12]);+	MONT_MAC(w0, w1, w2, a[27], b[11]);+	MONT_MAC(t0, t1, t2, a[28], b[10]);+	MONT_MAC(v0, v1, v2, a[29], b[9]);+	MONT_MAC(w0, w1, w2, a[30], b[8]);+	MONT_MAC(t0, t1, t2, a[31], b[7]);+	MONT_MAC(v0, v1, v2, u[7], m[31]);+	MONT_MAC(w0, w1, w2, u[8], m[30]);+	MONT_MAC(t0, t1, t2, u[9], m[29]);+	MONT_MAC(v0, v1, v2, u[10], m[28]);+	MONT_MAC(w0, w1, w2, u[11], m[27]);+	MONT_MAC(t0, t1, t2, u[12], m[26]);+	MONT_MAC(v0, v1, v2, u[13], m[25]);+	MONT_MAC(w0, w1, w2, u[14], m[24]);+	MONT_MAC(t0, t1, t2, u[15], m[23]);+	MONT_MAC(v0, v1, v2, u[16], m[22]);+	MONT_MAC(w0, w1, w2, u[17], m[21]);+	MONT_MAC(t0, t1, t2, u[18], m[20]);+	MONT_MAC(v0, v1, v2, u[19], m[19]);+	MONT_MAC(w0, w1, w2, u[20], m[18]);+	MONT_MAC(t0, t1, t2, u[21], m[17]);+	MONT_MAC(v0, v1, v2, u[22], m[16]);+	MONT_MAC(w0, w1, w2, u[23], m[15]);+	MONT_MAC(t0, t1, t2, u[24], m[14]);+	MONT_MAC(v0, v1, v2, u[25], m[13]);+	MONT_MAC(w0, w1, w2, u[26], m[12]);+	MONT_MAC(t0, t1, t2, u[27], m[11]);+	MONT_MAC(v0, v1, v2, u[28], m[10]);+	MONT_MAC(w0, w1, w2, u[29], m[9]);+	MONT_MAC(t0, t1, t2, u[30], m[8]);+	MONT_MAC(v0, v1, v2, u[31], m[7]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[6] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 39 */+	MONT_MAC(t0, t1, t2, a[8], b[31]);+	MONT_MUL(v0, v1, v2, a[9], b[30]);+	MONT_MUL(w0, w1, w2, a[10], b[29]);+	MONT_MAC(t0, t1, t2, a[11], b[28]);+	MONT_MAC(v0, v1, v2, a[12], b[27]);+	MONT_MAC(w0, w1, w2, a[13], b[26]);+	MONT_MAC(t0, t1, t2, a[14], b[25]);+	MONT_MAC(v0, v1, v2, a[15], b[24]);+	MONT_MAC(w0, w1, w2, a[16], b[23]);+	MONT_MAC(t0, t1, t2, a[17], b[22]);+	MONT_MAC(v0, v1, v2, a[18], b[21]);+	MONT_MAC(w0, w1, w2, a[19], b[20]);+	MONT_MAC(t0, t1, t2, a[20], b[19]);+	MONT_MAC(v0, v1, v2, a[21], b[18]);+	MONT_MAC(w0, w1, w2, a[22], b[17]);+	MONT_MAC(t0, t1, t2, a[23], b[16]);+	MONT_MAC(v0, v1, v2, a[24], b[15]);+	MONT_MAC(w0, w1, w2, a[25], b[14]);+	MONT_MAC(t0, t1, t2, a[26], b[13]);+	MONT_MAC(v0, v1, v2, a[27], b[12]);+	MONT_MAC(w0, w1, w2, a[28], b[11]);+	MONT_MAC(t0, t1, t2, a[29], b[10]);+	MONT_MAC(v0, v1, v2, a[30], b[9]);+	MONT_MAC(w0, w1, w2, a[31], b[8]);+	MONT_MAC(t0, t1, t2, u[8], m[31]);+	MONT_MAC(v0, v1, v2, u[9], m[30]);+	MONT_MAC(w0, w1, w2, u[10], m[29]);+	MONT_MAC(t0, t1, t2, u[11], m[28]);+	MONT_MAC(v0, v1, v2, u[12], m[27]);+	MONT_MAC(w0, w1, w2, u[13], m[26]);+	MONT_MAC(t0, t1, t2, u[14], m[25]);+	MONT_MAC(v0, v1, v2, u[15], m[24]);+	MONT_MAC(w0, w1, w2, u[16], m[23]);+	MONT_MAC(t0, t1, t2, u[17], m[22]);+	MONT_MAC(v0, v1, v2, u[18], m[21]);+	MONT_MAC(w0, w1, w2, u[19], m[20]);+	MONT_MAC(t0, t1, t2, u[20], m[19]);+	MONT_MAC(v0, v1, v2, u[21], m[18]);+	MONT_MAC(w0, w1, w2, u[22], m[17]);+	MONT_MAC(t0, t1, t2, u[23], m[16]);+	MONT_MAC(v0, v1, v2, u[24], m[15]);+	MONT_MAC(w0, w1, w2, u[25], m[14]);+	MONT_MAC(t0, t1, t2, u[26], m[13]);+	MONT_MAC(v0, v1, v2, u[27], m[12]);+	MONT_MAC(w0, w1, w2, u[28], m[11]);+	MONT_MAC(t0, t1, t2, u[29], m[10]);+	MONT_MAC(v0, v1, v2, u[30], m[9]);+	MONT_MAC(w0, w1, w2, u[31], m[8]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[7] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 40 */+	MONT_MAC(t0, t1, t2, a[9], b[31]);+	MONT_MUL(v0, v1, v2, a[10], b[30]);+	MONT_MUL(w0, w1, w2, a[11], b[29]);+	MONT_MAC(t0, t1, t2, a[12], b[28]);+	MONT_MAC(v0, v1, v2, a[13], b[27]);+	MONT_MAC(w0, w1, w2, a[14], b[26]);+	MONT_MAC(t0, t1, t2, a[15], b[25]);+	MONT_MAC(v0, v1, v2, a[16], b[24]);+	MONT_MAC(w0, w1, w2, a[17], b[23]);+	MONT_MAC(t0, t1, t2, a[18], b[22]);+	MONT_MAC(v0, v1, v2, a[19], b[21]);+	MONT_MAC(w0, w1, w2, a[20], b[20]);+	MONT_MAC(t0, t1, t2, a[21], b[19]);+	MONT_MAC(v0, v1, v2, a[22], b[18]);+	MONT_MAC(w0, w1, w2, a[23], b[17]);+	MONT_MAC(t0, t1, t2, a[24], b[16]);+	MONT_MAC(v0, v1, v2, a[25], b[15]);+	MONT_MAC(w0, w1, w2, a[26], b[14]);+	MONT_MAC(t0, t1, t2, a[27], b[13]);+	MONT_MAC(v0, v1, v2, a[28], b[12]);+	MONT_MAC(w0, w1, w2, a[29], b[11]);+	MONT_MAC(t0, t1, t2, a[30], b[10]);+	MONT_MAC(v0, v1, v2, a[31], b[9]);+	MONT_MAC(w0, w1, w2, u[9], m[31]);+	MONT_MAC(t0, t1, t2, u[10], m[30]);+	MONT_MAC(v0, v1, v2, u[11], m[29]);+	MONT_MAC(w0, w1, w2, u[12], m[28]);+	MONT_MAC(t0, t1, t2, u[13], m[27]);+	MONT_MAC(v0, v1, v2, u[14], m[26]);+	MONT_MAC(w0, w1, w2, u[15], m[25]);+	MONT_MAC(t0, t1, t2, u[16], m[24]);+	MONT_MAC(v0, v1, v2, u[17], m[23]);+	MONT_MAC(w0, w1, w2, u[18], m[22]);+	MONT_MAC(t0, t1, t2, u[19], m[21]);+	MONT_MAC(v0, v1, v2, u[20], m[20]);+	MONT_MAC(w0, w1, w2, u[21], m[19]);+	MONT_MAC(t0, t1, t2, u[22], m[18]);+	MONT_MAC(v0, v1, v2, u[23], m[17]);+	MONT_MAC(w0, w1, w2, u[24], m[16]);+	MONT_MAC(t0, t1, t2, u[25], m[15]);+	MONT_MAC(v0, v1, v2, u[26], m[14]);+	MONT_MAC(w0, w1, w2, u[27], m[13]);+	MONT_MAC(t0, t1, t2, u[28], m[12]);+	MONT_MAC(v0, v1, v2, u[29], m[11]);+	MONT_MAC(w0, w1, w2, u[30], m[10]);+	MONT_MAC(t0, t1, t2, u[31], m[9]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[8] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 41 */+	MONT_MAC(t0, t1, t2, a[10], b[31]);+	MONT_MUL(v0, v1, v2, a[11], b[30]);+	MONT_MUL(w0, w1, w2, a[12], b[29]);+	MONT_MAC(t0, t1, t2, a[13], b[28]);+	MONT_MAC(v0, v1, v2, a[14], b[27]);+	MONT_MAC(w0, w1, w2, a[15], b[26]);+	MONT_MAC(t0, t1, t2, a[16], b[25]);+	MONT_MAC(v0, v1, v2, a[17], b[24]);+	MONT_MAC(w0, w1, w2, a[18], b[23]);+	MONT_MAC(t0, t1, t2, a[19], b[22]);+	MONT_MAC(v0, v1, v2, a[20], b[21]);+	MONT_MAC(w0, w1, w2, a[21], b[20]);+	MONT_MAC(t0, t1, t2, a[22], b[19]);+	MONT_MAC(v0, v1, v2, a[23], b[18]);+	MONT_MAC(w0, w1, w2, a[24], b[17]);+	MONT_MAC(t0, t1, t2, a[25], b[16]);+	MONT_MAC(v0, v1, v2, a[26], b[15]);+	MONT_MAC(w0, w1, w2, a[27], b[14]);+	MONT_MAC(t0, t1, t2, a[28], b[13]);+	MONT_MAC(v0, v1, v2, a[29], b[12]);+	MONT_MAC(w0, w1, w2, a[30], b[11]);+	MONT_MAC(t0, t1, t2, a[31], b[10]);+	MONT_MAC(v0, v1, v2, u[10], m[31]);+	MONT_MAC(w0, w1, w2, u[11], m[30]);+	MONT_MAC(t0, t1, t2, u[12], m[29]);+	MONT_MAC(v0, v1, v2, u[13], m[28]);+	MONT_MAC(w0, w1, w2, u[14], m[27]);+	MONT_MAC(t0, t1, t2, u[15], m[26]);+	MONT_MAC(v0, v1, v2, u[16], m[25]);+	MONT_MAC(w0, w1, w2, u[17], m[24]);+	MONT_MAC(t0, t1, t2, u[18], m[23]);+	MONT_MAC(v0, v1, v2, u[19], m[22]);+	MONT_MAC(w0, w1, w2, u[20], m[21]);+	MONT_MAC(t0, t1, t2, u[21], m[20]);+	MONT_MAC(v0, v1, v2, u[22], m[19]);+	MONT_MAC(w0, w1, w2, u[23], m[18]);+	MONT_MAC(t0, t1, t2, u[24], m[17]);+	MONT_MAC(v0, v1, v2, u[25], m[16]);+	MONT_MAC(w0, w1, w2, u[26], m[15]);+	MONT_MAC(t0, t1, t2, u[27], m[14]);+	MONT_MAC(v0, v1, v2, u[28], m[13]);+	MONT_MAC(w0, w1, w2, u[29], m[12]);+	MONT_MAC(t0, t1, t2, u[30], m[11]);+	MONT_MAC(v0, v1, v2, u[31], m[10]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[9] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 42 */+	MONT_MAC(t0, t1, t2, a[11], b[31]);+	MONT_MUL(v0, v1, v2, a[12], b[30]);+	MONT_MUL(w0, w1, w2, a[13], b[29]);+	MONT_MAC(t0, t1, t2, a[14], b[28]);+	MONT_MAC(v0, v1, v2, a[15], b[27]);+	MONT_MAC(w0, w1, w2, a[16], b[26]);+	MONT_MAC(t0, t1, t2, a[17], b[25]);+	MONT_MAC(v0, v1, v2, a[18], b[24]);+	MONT_MAC(w0, w1, w2, a[19], b[23]);+	MONT_MAC(t0, t1, t2, a[20], b[22]);+	MONT_MAC(v0, v1, v2, a[21], b[21]);+	MONT_MAC(w0, w1, w2, a[22], b[20]);+	MONT_MAC(t0, t1, t2, a[23], b[19]);+	MONT_MAC(v0, v1, v2, a[24], b[18]);+	MONT_MAC(w0, w1, w2, a[25], b[17]);+	MONT_MAC(t0, t1, t2, a[26], b[16]);+	MONT_MAC(v0, v1, v2, a[27], b[15]);+	MONT_MAC(w0, w1, w2, a[28], b[14]);+	MONT_MAC(t0, t1, t2, a[29], b[13]);+	MONT_MAC(v0, v1, v2, a[30], b[12]);+	MONT_MAC(w0, w1, w2, a[31], b[11]);+	MONT_MAC(t0, t1, t2, u[11], m[31]);+	MONT_MAC(v0, v1, v2, u[12], m[30]);+	MONT_MAC(w0, w1, w2, u[13], m[29]);+	MONT_MAC(t0, t1, t2, u[14], m[28]);+	MONT_MAC(v0, v1, v2, u[15], m[27]);+	MONT_MAC(w0, w1, w2, u[16], m[26]);+	MONT_MAC(t0, t1, t2, u[17], m[25]);+	MONT_MAC(v0, v1, v2, u[18], m[24]);+	MONT_MAC(w0, w1, w2, u[19], m[23]);+	MONT_MAC(t0, t1, t2, u[20], m[22]);+	MONT_MAC(v0, v1, v2, u[21], m[21]);+	MONT_MAC(w0, w1, w2, u[22], m[20]);+	MONT_MAC(t0, t1, t2, u[23], m[19]);+	MONT_MAC(v0, v1, v2, u[24], m[18]);+	MONT_MAC(w0, w1, w2, u[25], m[17]);+	MONT_MAC(t0, t1, t2, u[26], m[16]);+	MONT_MAC(v0, v1, v2, u[27], m[15]);+	MONT_MAC(w0, w1, w2, u[28], m[14]);+	MONT_MAC(t0, t1, t2, u[29], m[13]);+	MONT_MAC(v0, v1, v2, u[30], m[12]);+	MONT_MAC(w0, w1, w2, u[31], m[11]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[10] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 43 */+	MONT_MAC(t0, t1, t2, a[12], b[31]);+	MONT_MUL(v0, v1, v2, a[13], b[30]);+	MONT_MUL(w0, w1, w2, a[14], b[29]);+	MONT_MAC(t0, t1, t2, a[15], b[28]);+	MONT_MAC(v0, v1, v2, a[16], b[27]);+	MONT_MAC(w0, w1, w2, a[17], b[26]);+	MONT_MAC(t0, t1, t2, a[18], b[25]);+	MONT_MAC(v0, v1, v2, a[19], b[24]);+	MONT_MAC(w0, w1, w2, a[20], b[23]);+	MONT_MAC(t0, t1, t2, a[21], b[22]);+	MONT_MAC(v0, v1, v2, a[22], b[21]);+	MONT_MAC(w0, w1, w2, a[23], b[20]);+	MONT_MAC(t0, t1, t2, a[24], b[19]);+	MONT_MAC(v0, v1, v2, a[25], b[18]);+	MONT_MAC(w0, w1, w2, a[26], b[17]);+	MONT_MAC(t0, t1, t2, a[27], b[16]);+	MONT_MAC(v0, v1, v2, a[28], b[15]);+	MONT_MAC(w0, w1, w2, a[29], b[14]);+	MONT_MAC(t0, t1, t2, a[30], b[13]);+	MONT_MAC(v0, v1, v2, a[31], b[12]);+	MONT_MAC(w0, w1, w2, u[12], m[31]);+	MONT_MAC(t0, t1, t2, u[13], m[30]);+	MONT_MAC(v0, v1, v2, u[14], m[29]);+	MONT_MAC(w0, w1, w2, u[15], m[28]);+	MONT_MAC(t0, t1, t2, u[16], m[27]);+	MONT_MAC(v0, v1, v2, u[17], m[26]);+	MONT_MAC(w0, w1, w2, u[18], m[25]);+	MONT_MAC(t0, t1, t2, u[19], m[24]);+	MONT_MAC(v0, v1, v2, u[20], m[23]);+	MONT_MAC(w0, w1, w2, u[21], m[22]);+	MONT_MAC(t0, t1, t2, u[22], m[21]);+	MONT_MAC(v0, v1, v2, u[23], m[20]);+	MONT_MAC(w0, w1, w2, u[24], m[19]);+	MONT_MAC(t0, t1, t2, u[25], m[18]);+	MONT_MAC(v0, v1, v2, u[26], m[17]);+	MONT_MAC(w0, w1, w2, u[27], m[16]);+	MONT_MAC(t0, t1, t2, u[28], m[15]);+	MONT_MAC(v0, v1, v2, u[29], m[14]);+	MONT_MAC(w0, w1, w2, u[30], m[13]);+	MONT_MAC(t0, t1, t2, u[31], m[12]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[11] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 44 */+	MONT_MAC(t0, t1, t2, a[13], b[31]);+	MONT_MUL(v0, v1, v2, a[14], b[30]);+	MONT_MUL(w0, w1, w2, a[15], b[29]);+	MONT_MAC(t0, t1, t2, a[16], b[28]);+	MONT_MAC(v0, v1, v2, a[17], b[27]);+	MONT_MAC(w0, w1, w2, a[18], b[26]);+	MONT_MAC(t0, t1, t2, a[19], b[25]);+	MONT_MAC(v0, v1, v2, a[20], b[24]);+	MONT_MAC(w0, w1, w2, a[21], b[23]);+	MONT_MAC(t0, t1, t2, a[22], b[22]);+	MONT_MAC(v0, v1, v2, a[23], b[21]);+	MONT_MAC(w0, w1, w2, a[24], b[20]);+	MONT_MAC(t0, t1, t2, a[25], b[19]);+	MONT_MAC(v0, v1, v2, a[26], b[18]);+	MONT_MAC(w0, w1, w2, a[27], b[17]);+	MONT_MAC(t0, t1, t2, a[28], b[16]);+	MONT_MAC(v0, v1, v2, a[29], b[15]);+	MONT_MAC(w0, w1, w2, a[30], b[14]);+	MONT_MAC(t0, t1, t2, a[31], b[13]);+	MONT_MAC(v0, v1, v2, u[13], m[31]);+	MONT_MAC(w0, w1, w2, u[14], m[30]);+	MONT_MAC(t0, t1, t2, u[15], m[29]);+	MONT_MAC(v0, v1, v2, u[16], m[28]);+	MONT_MAC(w0, w1, w2, u[17], m[27]);+	MONT_MAC(t0, t1, t2, u[18], m[26]);+	MONT_MAC(v0, v1, v2, u[19], m[25]);+	MONT_MAC(w0, w1, w2, u[20], m[24]);+	MONT_MAC(t0, t1, t2, u[21], m[23]);+	MONT_MAC(v0, v1, v2, u[22], m[22]);+	MONT_MAC(w0, w1, w2, u[23], m[21]);+	MONT_MAC(t0, t1, t2, u[24], m[20]);+	MONT_MAC(v0, v1, v2, u[25], m[19]);+	MONT_MAC(w0, w1, w2, u[26], m[18]);+	MONT_MAC(t0, t1, t2, u[27], m[17]);+	MONT_MAC(v0, v1, v2, u[28], m[16]);+	MONT_MAC(w0, w1, w2, u[29], m[15]);+	MONT_MAC(t0, t1, t2, u[30], m[14]);+	MONT_MAC(v0, v1, v2, u[31], m[13]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[12] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 45 */+	MONT_MAC(t0, t1, t2, a[14], b[31]);+	MONT_MUL(v0, v1, v2, a[15], b[30]);+	MONT_MUL(w0, w1, w2, a[16], b[29]);+	MONT_MAC(t0, t1, t2, a[17], b[28]);+	MONT_MAC(v0, v1, v2, a[18], b[27]);+	MONT_MAC(w0, w1, w2, a[19], b[26]);+	MONT_MAC(t0, t1, t2, a[20], b[25]);+	MONT_MAC(v0, v1, v2, a[21], b[24]);+	MONT_MAC(w0, w1, w2, a[22], b[23]);+	MONT_MAC(t0, t1, t2, a[23], b[22]);+	MONT_MAC(v0, v1, v2, a[24], b[21]);+	MONT_MAC(w0, w1, w2, a[25], b[20]);+	MONT_MAC(t0, t1, t2, a[26], b[19]);+	MONT_MAC(v0, v1, v2, a[27], b[18]);+	MONT_MAC(w0, w1, w2, a[28], b[17]);+	MONT_MAC(t0, t1, t2, a[29], b[16]);+	MONT_MAC(v0, v1, v2, a[30], b[15]);+	MONT_MAC(w0, w1, w2, a[31], b[14]);+	MONT_MAC(t0, t1, t2, u[14], m[31]);+	MONT_MAC(v0, v1, v2, u[15], m[30]);+	MONT_MAC(w0, w1, w2, u[16], m[29]);+	MONT_MAC(t0, t1, t2, u[17], m[28]);+	MONT_MAC(v0, v1, v2, u[18], m[27]);+	MONT_MAC(w0, w1, w2, u[19], m[26]);+	MONT_MAC(t0, t1, t2, u[20], m[25]);+	MONT_MAC(v0, v1, v2, u[21], m[24]);+	MONT_MAC(w0, w1, w2, u[22], m[23]);+	MONT_MAC(t0, t1, t2, u[23], m[22]);+	MONT_MAC(v0, v1, v2, u[24], m[21]);+	MONT_MAC(w0, w1, w2, u[25], m[20]);+	MONT_MAC(t0, t1, t2, u[26], m[19]);+	MONT_MAC(v0, v1, v2, u[27], m[18]);+	MONT_MAC(w0, w1, w2, u[28], m[17]);+	MONT_MAC(t0, t1, t2, u[29], m[16]);+	MONT_MAC(v0, v1, v2, u[30], m[15]);+	MONT_MAC(w0, w1, w2, u[31], m[14]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[13] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 46 */+	MONT_MAC(t0, t1, t2, a[15], b[31]);+	MONT_MUL(v0, v1, v2, a[16], b[30]);+	MONT_MUL(w0, w1, w2, a[17], b[29]);+	MONT_MAC(t0, t1, t2, a[18], b[28]);+	MONT_MAC(v0, v1, v2, a[19], b[27]);+	MONT_MAC(w0, w1, w2, a[20], b[26]);+	MONT_MAC(t0, t1, t2, a[21], b[25]);+	MONT_MAC(v0, v1, v2, a[22], b[24]);+	MONT_MAC(w0, w1, w2, a[23], b[23]);+	MONT_MAC(t0, t1, t2, a[24], b[22]);+	MONT_MAC(v0, v1, v2, a[25], b[21]);+	MONT_MAC(w0, w1, w2, a[26], b[20]);+	MONT_MAC(t0, t1, t2, a[27], b[19]);+	MONT_MAC(v0, v1, v2, a[28], b[18]);+	MONT_MAC(w0, w1, w2, a[29], b[17]);+	MONT_MAC(t0, t1, t2, a[30], b[16]);+	MONT_MAC(v0, v1, v2, a[31], b[15]);+	MONT_MAC(w0, w1, w2, u[15], m[31]);+	MONT_MAC(t0, t1, t2, u[16], m[30]);+	MONT_MAC(v0, v1, v2, u[17], m[29]);+	MONT_MAC(w0, w1, w2, u[18], m[28]);+	MONT_MAC(t0, t1, t2, u[19], m[27]);+	MONT_MAC(v0, v1, v2, u[20], m[26]);+	MONT_MAC(w0, w1, w2, u[21], m[25]);+	MONT_MAC(t0, t1, t2, u[22], m[24]);+	MONT_MAC(v0, v1, v2, u[23], m[23]);+	MONT_MAC(w0, w1, w2, u[24], m[22]);+	MONT_MAC(t0, t1, t2, u[25], m[21]);+	MONT_MAC(v0, v1, v2, u[26], m[20]);+	MONT_MAC(w0, w1, w2, u[27], m[19]);+	MONT_MAC(t0, t1, t2, u[28], m[18]);+	MONT_MAC(v0, v1, v2, u[29], m[17]);+	MONT_MAC(w0, w1, w2, u[30], m[16]);+	MONT_MAC(t0, t1, t2, u[31], m[15]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[14] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 47 */+	MONT_MAC(t0, t1, t2, a[16], b[31]);+	MONT_MUL(v0, v1, v2, a[17], b[30]);+	MONT_MUL(w0, w1, w2, a[18], b[29]);+	MONT_MAC(t0, t1, t2, a[19], b[28]);+	MONT_MAC(v0, v1, v2, a[20], b[27]);+	MONT_MAC(w0, w1, w2, a[21], b[26]);+	MONT_MAC(t0, t1, t2, a[22], b[25]);+	MONT_MAC(v0, v1, v2, a[23], b[24]);+	MONT_MAC(w0, w1, w2, a[24], b[23]);+	MONT_MAC(t0, t1, t2, a[25], b[22]);+	MONT_MAC(v0, v1, v2, a[26], b[21]);+	MONT_MAC(w0, w1, w2, a[27], b[20]);+	MONT_MAC(t0, t1, t2, a[28], b[19]);+	MONT_MAC(v0, v1, v2, a[29], b[18]);+	MONT_MAC(w0, w1, w2, a[30], b[17]);+	MONT_MAC(t0, t1, t2, a[31], b[16]);+	MONT_MAC(v0, v1, v2, u[16], m[31]);+	MONT_MAC(w0, w1, w2, u[17], m[30]);+	MONT_MAC(t0, t1, t2, u[18], m[29]);+	MONT_MAC(v0, v1, v2, u[19], m[28]);+	MONT_MAC(w0, w1, w2, u[20], m[27]);+	MONT_MAC(t0, t1, t2, u[21], m[26]);+	MONT_MAC(v0, v1, v2, u[22], m[25]);+	MONT_MAC(w0, w1, w2, u[23], m[24]);+	MONT_MAC(t0, t1, t2, u[24], m[23]);+	MONT_MAC(v0, v1, v2, u[25], m[22]);+	MONT_MAC(w0, w1, w2, u[26], m[21]);+	MONT_MAC(t0, t1, t2, u[27], m[20]);+	MONT_MAC(v0, v1, v2, u[28], m[19]);+	MONT_MAC(w0, w1, w2, u[29], m[18]);+	MONT_MAC(t0, t1, t2, u[30], m[17]);+	MONT_MAC(v0, v1, v2, u[31], m[16]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[15] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 48 */+	MONT_MAC(t0, t1, t2, a[17], b[31]);+	MONT_MUL(v0, v1, v2, a[18], b[30]);+	MONT_MUL(w0, w1, w2, a[19], b[29]);+	MONT_MAC(t0, t1, t2, a[20], b[28]);+	MONT_MAC(v0, v1, v2, a[21], b[27]);+	MONT_MAC(w0, w1, w2, a[22], b[26]);+	MONT_MAC(t0, t1, t2, a[23], b[25]);+	MONT_MAC(v0, v1, v2, a[24], b[24]);+	MONT_MAC(w0, w1, w2, a[25], b[23]);+	MONT_MAC(t0, t1, t2, a[26], b[22]);+	MONT_MAC(v0, v1, v2, a[27], b[21]);+	MONT_MAC(w0, w1, w2, a[28], b[20]);+	MONT_MAC(t0, t1, t2, a[29], b[19]);+	MONT_MAC(v0, v1, v2, a[30], b[18]);+	MONT_MAC(w0, w1, w2, a[31], b[17]);+	MONT_MAC(t0, t1, t2, u[17], m[31]);+	MONT_MAC(v0, v1, v2, u[18], m[30]);+	MONT_MAC(w0, w1, w2, u[19], m[29]);+	MONT_MAC(t0, t1, t2, u[20], m[28]);+	MONT_MAC(v0, v1, v2, u[21], m[27]);+	MONT_MAC(w0, w1, w2, u[22], m[26]);+	MONT_MAC(t0, t1, t2, u[23], m[25]);+	MONT_MAC(v0, v1, v2, u[24], m[24]);+	MONT_MAC(w0, w1, w2, u[25], m[23]);+	MONT_MAC(t0, t1, t2, u[26], m[22]);+	MONT_MAC(v0, v1, v2, u[27], m[21]);+	MONT_MAC(w0, w1, w2, u[28], m[20]);+	MONT_MAC(t0, t1, t2, u[29], m[19]);+	MONT_MAC(v0, v1, v2, u[30], m[18]);+	MONT_MAC(w0, w1, w2, u[31], m[17]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[16] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 49 */+	MONT_MAC(t0, t1, t2, a[18], b[31]);+	MONT_MUL(v0, v1, v2, a[19], b[30]);+	MONT_MUL(w0, w1, w2, a[20], b[29]);+	MONT_MAC(t0, t1, t2, a[21], b[28]);+	MONT_MAC(v0, v1, v2, a[22], b[27]);+	MONT_MAC(w0, w1, w2, a[23], b[26]);+	MONT_MAC(t0, t1, t2, a[24], b[25]);+	MONT_MAC(v0, v1, v2, a[25], b[24]);+	MONT_MAC(w0, w1, w2, a[26], b[23]);+	MONT_MAC(t0, t1, t2, a[27], b[22]);+	MONT_MAC(v0, v1, v2, a[28], b[21]);+	MONT_MAC(w0, w1, w2, a[29], b[20]);+	MONT_MAC(t0, t1, t2, a[30], b[19]);+	MONT_MAC(v0, v1, v2, a[31], b[18]);+	MONT_MAC(w0, w1, w2, u[18], m[31]);+	MONT_MAC(t0, t1, t2, u[19], m[30]);+	MONT_MAC(v0, v1, v2, u[20], m[29]);+	MONT_MAC(w0, w1, w2, u[21], m[28]);+	MONT_MAC(t0, t1, t2, u[22], m[27]);+	MONT_MAC(v0, v1, v2, u[23], m[26]);+	MONT_MAC(w0, w1, w2, u[24], m[25]);+	MONT_MAC(t0, t1, t2, u[25], m[24]);+	MONT_MAC(v0, v1, v2, u[26], m[23]);+	MONT_MAC(w0, w1, w2, u[27], m[22]);+	MONT_MAC(t0, t1, t2, u[28], m[21]);+	MONT_MAC(v0, v1, v2, u[29], m[20]);+	MONT_MAC(w0, w1, w2, u[30], m[19]);+	MONT_MAC(t0, t1, t2, u[31], m[18]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[17] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 50 */+	MONT_MAC(t0, t1, t2, a[19], b[31]);+	MONT_MUL(v0, v1, v2, a[20], b[30]);+	MONT_MUL(w0, w1, w2, a[21], b[29]);+	MONT_MAC(t0, t1, t2, a[22], b[28]);+	MONT_MAC(v0, v1, v2, a[23], b[27]);+	MONT_MAC(w0, w1, w2, a[24], b[26]);+	MONT_MAC(t0, t1, t2, a[25], b[25]);+	MONT_MAC(v0, v1, v2, a[26], b[24]);+	MONT_MAC(w0, w1, w2, a[27], b[23]);+	MONT_MAC(t0, t1, t2, a[28], b[22]);+	MONT_MAC(v0, v1, v2, a[29], b[21]);+	MONT_MAC(w0, w1, w2, a[30], b[20]);+	MONT_MAC(t0, t1, t2, a[31], b[19]);+	MONT_MAC(v0, v1, v2, u[19], m[31]);+	MONT_MAC(w0, w1, w2, u[20], m[30]);+	MONT_MAC(t0, t1, t2, u[21], m[29]);+	MONT_MAC(v0, v1, v2, u[22], m[28]);+	MONT_MAC(w0, w1, w2, u[23], m[27]);+	MONT_MAC(t0, t1, t2, u[24], m[26]);+	MONT_MAC(v0, v1, v2, u[25], m[25]);+	MONT_MAC(w0, w1, w2, u[26], m[24]);+	MONT_MAC(t0, t1, t2, u[27], m[23]);+	MONT_MAC(v0, v1, v2, u[28], m[22]);+	MONT_MAC(w0, w1, w2, u[29], m[21]);+	MONT_MAC(t0, t1, t2, u[30], m[20]);+	MONT_MAC(v0, v1, v2, u[31], m[19]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[18] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 51 */+	MONT_MAC(t0, t1, t2, a[20], b[31]);+	MONT_MUL(v0, v1, v2, a[21], b[30]);+	MONT_MUL(w0, w1, w2, a[22], b[29]);+	MONT_MAC(t0, t1, t2, a[23], b[28]);+	MONT_MAC(v0, v1, v2, a[24], b[27]);+	MONT_MAC(w0, w1, w2, a[25], b[26]);+	MONT_MAC(t0, t1, t2, a[26], b[25]);+	MONT_MAC(v0, v1, v2, a[27], b[24]);+	MONT_MAC(w0, w1, w2, a[28], b[23]);+	MONT_MAC(t0, t1, t2, a[29], b[22]);+	MONT_MAC(v0, v1, v2, a[30], b[21]);+	MONT_MAC(w0, w1, w2, a[31], b[20]);+	MONT_MAC(t0, t1, t2, u[20], m[31]);+	MONT_MAC(v0, v1, v2, u[21], m[30]);+	MONT_MAC(w0, w1, w2, u[22], m[29]);+	MONT_MAC(t0, t1, t2, u[23], m[28]);+	MONT_MAC(v0, v1, v2, u[24], m[27]);+	MONT_MAC(w0, w1, w2, u[25], m[26]);+	MONT_MAC(t0, t1, t2, u[26], m[25]);+	MONT_MAC(v0, v1, v2, u[27], m[24]);+	MONT_MAC(w0, w1, w2, u[28], m[23]);+	MONT_MAC(t0, t1, t2, u[29], m[22]);+	MONT_MAC(v0, v1, v2, u[30], m[21]);+	MONT_MAC(w0, w1, w2, u[31], m[20]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[19] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 52 */+	MONT_MAC(t0, t1, t2, a[21], b[31]);+	MONT_MUL(v0, v1, v2, a[22], b[30]);+	MONT_MUL(w0, w1, w2, a[23], b[29]);+	MONT_MAC(t0, t1, t2, a[24], b[28]);+	MONT_MAC(v0, v1, v2, a[25], b[27]);+	MONT_MAC(w0, w1, w2, a[26], b[26]);+	MONT_MAC(t0, t1, t2, a[27], b[25]);+	MONT_MAC(v0, v1, v2, a[28], b[24]);+	MONT_MAC(w0, w1, w2, a[29], b[23]);+	MONT_MAC(t0, t1, t2, a[30], b[22]);+	MONT_MAC(v0, v1, v2, a[31], b[21]);+	MONT_MAC(w0, w1, w2, u[21], m[31]);+	MONT_MAC(t0, t1, t2, u[22], m[30]);+	MONT_MAC(v0, v1, v2, u[23], m[29]);+	MONT_MAC(w0, w1, w2, u[24], m[28]);+	MONT_MAC(t0, t1, t2, u[25], m[27]);+	MONT_MAC(v0, v1, v2, u[26], m[26]);+	MONT_MAC(w0, w1, w2, u[27], m[25]);+	MONT_MAC(t0, t1, t2, u[28], m[24]);+	MONT_MAC(v0, v1, v2, u[29], m[23]);+	MONT_MAC(w0, w1, w2, u[30], m[22]);+	MONT_MAC(t0, t1, t2, u[31], m[21]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[20] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 53 */+	MONT_MAC(t0, t1, t2, a[22], b[31]);+	MONT_MUL(v0, v1, v2, a[23], b[30]);+	MONT_MUL(w0, w1, w2, a[24], b[29]);+	MONT_MAC(t0, t1, t2, a[25], b[28]);+	MONT_MAC(v0, v1, v2, a[26], b[27]);+	MONT_MAC(w0, w1, w2, a[27], b[26]);+	MONT_MAC(t0, t1, t2, a[28], b[25]);+	MONT_MAC(v0, v1, v2, a[29], b[24]);+	MONT_MAC(w0, w1, w2, a[30], b[23]);+	MONT_MAC(t0, t1, t2, a[31], b[22]);+	MONT_MAC(v0, v1, v2, u[22], m[31]);+	MONT_MAC(w0, w1, w2, u[23], m[30]);+	MONT_MAC(t0, t1, t2, u[24], m[29]);+	MONT_MAC(v0, v1, v2, u[25], m[28]);+	MONT_MAC(w0, w1, w2, u[26], m[27]);+	MONT_MAC(t0, t1, t2, u[27], m[26]);+	MONT_MAC(v0, v1, v2, u[28], m[25]);+	MONT_MAC(w0, w1, w2, u[29], m[24]);+	MONT_MAC(t0, t1, t2, u[30], m[23]);+	MONT_MAC(v0, v1, v2, u[31], m[22]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[21] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 54 */+	MONT_MAC(t0, t1, t2, a[23], b[31]);+	MONT_MUL(v0, v1, v2, a[24], b[30]);+	MONT_MUL(w0, w1, w2, a[25], b[29]);+	MONT_MAC(t0, t1, t2, a[26], b[28]);+	MONT_MAC(v0, v1, v2, a[27], b[27]);+	MONT_MAC(w0, w1, w2, a[28], b[26]);+	MONT_MAC(t0, t1, t2, a[29], b[25]);+	MONT_MAC(v0, v1, v2, a[30], b[24]);+	MONT_MAC(w0, w1, w2, a[31], b[23]);+	MONT_MAC(t0, t1, t2, u[23], m[31]);+	MONT_MAC(v0, v1, v2, u[24], m[30]);+	MONT_MAC(w0, w1, w2, u[25], m[29]);+	MONT_MAC(t0, t1, t2, u[26], m[28]);+	MONT_MAC(v0, v1, v2, u[27], m[27]);+	MONT_MAC(w0, w1, w2, u[28], m[26]);+	MONT_MAC(t0, t1, t2, u[29], m[25]);+	MONT_MAC(v0, v1, v2, u[30], m[24]);+	MONT_MAC(w0, w1, w2, u[31], m[23]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[22] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 55 */+	MONT_MAC(t0, t1, t2, a[24], b[31]);+	MONT_MUL(v0, v1, v2, a[25], b[30]);+	MONT_MUL(w0, w1, w2, a[26], b[29]);+	MONT_MAC(t0, t1, t2, a[27], b[28]);+	MONT_MAC(v0, v1, v2, a[28], b[27]);+	MONT_MAC(w0, w1, w2, a[29], b[26]);+	MONT_MAC(t0, t1, t2, a[30], b[25]);+	MONT_MAC(v0, v1, v2, a[31], b[24]);+	MONT_MAC(w0, w1, w2, u[24], m[31]);+	MONT_MAC(t0, t1, t2, u[25], m[30]);+	MONT_MAC(v0, v1, v2, u[26], m[29]);+	MONT_MAC(w0, w1, w2, u[27], m[28]);+	MONT_MAC(t0, t1, t2, u[28], m[27]);+	MONT_MAC(v0, v1, v2, u[29], m[26]);+	MONT_MAC(w0, w1, w2, u[30], m[25]);+	MONT_MAC(t0, t1, t2, u[31], m[24]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[23] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 56 */+	MONT_MAC(t0, t1, t2, a[25], b[31]);+	MONT_MUL(v0, v1, v2, a[26], b[30]);+	MONT_MUL(w0, w1, w2, a[27], b[29]);+	MONT_MAC(t0, t1, t2, a[28], b[28]);+	MONT_MAC(v0, v1, v2, a[29], b[27]);+	MONT_MAC(w0, w1, w2, a[30], b[26]);+	MONT_MAC(t0, t1, t2, a[31], b[25]);+	MONT_MAC(v0, v1, v2, u[25], m[31]);+	MONT_MAC(w0, w1, w2, u[26], m[30]);+	MONT_MAC(t0, t1, t2, u[27], m[29]);+	MONT_MAC(v0, v1, v2, u[28], m[28]);+	MONT_MAC(w0, w1, w2, u[29], m[27]);+	MONT_MAC(t0, t1, t2, u[30], m[26]);+	MONT_MAC(v0, v1, v2, u[31], m[25]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[24] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 57 */+	MONT_MAC(t0, t1, t2, a[26], b[31]);+	MONT_MUL(v0, v1, v2, a[27], b[30]);+	MONT_MUL(w0, w1, w2, a[28], b[29]);+	MONT_MAC(t0, t1, t2, a[29], b[28]);+	MONT_MAC(v0, v1, v2, a[30], b[27]);+	MONT_MAC(w0, w1, w2, a[31], b[26]);+	MONT_MAC(t0, t1, t2, u[26], m[31]);+	MONT_MAC(v0, v1, v2, u[27], m[30]);+	MONT_MAC(w0, w1, w2, u[28], m[29]);+	MONT_MAC(t0, t1, t2, u[29], m[28]);+	MONT_MAC(v0, v1, v2, u[30], m[27]);+	MONT_MAC(w0, w1, w2, u[31], m[26]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[25] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 58 */+	MONT_MAC(t0, t1, t2, a[27], b[31]);+	MONT_MUL(v0, v1, v2, a[28], b[30]);+	MONT_MUL(w0, w1, w2, a[29], b[29]);+	MONT_MAC(t0, t1, t2, a[30], b[28]);+	MONT_MAC(v0, v1, v2, a[31], b[27]);+	MONT_MAC(w0, w1, w2, u[27], m[31]);+	MONT_MAC(t0, t1, t2, u[28], m[30]);+	MONT_MAC(v0, v1, v2, u[29], m[29]);+	MONT_MAC(w0, w1, w2, u[30], m[28]);+	MONT_MAC(t0, t1, t2, u[31], m[27]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[26] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 59 */+	MONT_MAC(t0, t1, t2, a[28], b[31]);+	MONT_MUL(v0, v1, v2, a[29], b[30]);+	MONT_MUL(w0, w1, w2, a[30], b[29]);+	MONT_MAC(t0, t1, t2, a[31], b[28]);+	MONT_MAC(v0, v1, v2, u[28], m[31]);+	MONT_MAC(w0, w1, w2, u[29], m[30]);+	MONT_MAC(t0, t1, t2, u[30], m[29]);+	MONT_MAC(v0, v1, v2, u[31], m[28]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[27] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 60 */+	MONT_MAC(t0, t1, t2, a[29], b[31]);+	MONT_MUL(v0, v1, v2, a[30], b[30]);+	MONT_MUL(w0, w1, w2, a[31], b[29]);+	MONT_MAC(t0, t1, t2, u[29], m[31]);+	MONT_MAC(v0, v1, v2, u[30], m[30]);+	MONT_MAC(w0, w1, w2, u[31], m[29]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[28] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 61 */+	MONT_MAC(t0, t1, t2, a[30], b[31]);+	MONT_MUL(v0, v1, v2, a[31], b[30]);+	MONT_MUL(w0, w1, w2, u[30], m[31]);+	MONT_MAC(t0, t1, t2, u[31], m[30]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	z[29] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 62 */+	MONT_MAC(t0, t1, t2, a[31], b[31]);+	MONT_MUL(v0, v1, v2, u[31], m[31]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	z[30] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	z[31] = t0;++	/* under 2m, with t1 the bit above n limbs: at most one subtraction */+	borrow = sub_n(r, z, m, 32);+	take = t1 | (borrow ^ 1);+	select_n(r, r, z, take & 1, 32);+}++/* r = a * a * R^-1 mod m, for a under m; t is 2n limbs of scratch */+static void crypton_mont_sqr32(limb_t *r, const limb_t *a,+                               const limb_t *m, limb_t n0, limb_t *t)+{+	limb_t *u = t, *z = t + 32;+	limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take;+	limb_t s0, s1, s2, c0, c1, c2;++	/* column 0 */+	MONT_MAC(t0, t1, t2, a[0], a[0]);+	u[0] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[0], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 1 */+	MONT_MUL(s0, s1, s2, a[0], a[1]);+	MONT_MAC(t0, t1, t2, u[0], m[1]);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[1] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[1], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 2 */+	MONT_MUL(s0, s1, s2, a[0], a[2]);+	MONT_MAC(t0, t1, t2, a[1], a[1]);+	MONT_MUL(v0, v1, v2, u[0], m[2]);+	MONT_MUL(w0, w1, w2, u[1], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[2] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[2], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 3 */+	MONT_MUL(s0, s1, s2, a[0], a[3]);+	MONT_MUL(c0, c1, c2, a[1], a[2]);+	MONT_MAC(t0, t1, t2, u[0], m[3]);+	MONT_MUL(v0, v1, v2, u[1], m[2]);+	MONT_MUL(w0, w1, w2, u[2], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[3] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[3], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 4 */+	MONT_MUL(s0, s1, s2, a[0], a[4]);+	MONT_MUL(c0, c1, c2, a[1], a[3]);+	MONT_MAC(t0, t1, t2, a[2], a[2]);+	MONT_MUL(v0, v1, v2, u[0], m[4]);+	MONT_MUL(w0, w1, w2, u[1], m[3]);+	MONT_MAC(t0, t1, t2, u[2], m[2]);+	MONT_MAC(v0, v1, v2, u[3], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[4] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[4], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 5 */+	MONT_MUL(s0, s1, s2, a[0], a[5]);+	MONT_MUL(c0, c1, c2, a[1], a[4]);+	MONT_MAC(s0, s1, s2, a[2], a[3]);+	MONT_MAC(t0, t1, t2, u[0], m[5]);+	MONT_MUL(v0, v1, v2, u[1], m[4]);+	MONT_MUL(w0, w1, w2, u[2], m[3]);+	MONT_MAC(t0, t1, t2, u[3], m[2]);+	MONT_MAC(v0, v1, v2, u[4], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[5] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[5], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 6 */+	MONT_MUL(s0, s1, s2, a[0], a[6]);+	MONT_MUL(c0, c1, c2, a[1], a[5]);+	MONT_MAC(s0, s1, s2, a[2], a[4]);+	MONT_MAC(t0, t1, t2, a[3], a[3]);+	MONT_MUL(v0, v1, v2, u[0], m[6]);+	MONT_MUL(w0, w1, w2, u[1], m[5]);+	MONT_MAC(t0, t1, t2, u[2], m[4]);+	MONT_MAC(v0, v1, v2, u[3], m[3]);+	MONT_MAC(w0, w1, w2, u[4], m[2]);+	MONT_MAC(t0, t1, t2, u[5], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[6] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[6], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 7 */+	MONT_MUL(s0, s1, s2, a[0], a[7]);+	MONT_MUL(c0, c1, c2, a[1], a[6]);+	MONT_MAC(s0, s1, s2, a[2], a[5]);+	MONT_MAC(c0, c1, c2, a[3], a[4]);+	MONT_MAC(t0, t1, t2, u[0], m[7]);+	MONT_MUL(v0, v1, v2, u[1], m[6]);+	MONT_MUL(w0, w1, w2, u[2], m[5]);+	MONT_MAC(t0, t1, t2, u[3], m[4]);+	MONT_MAC(v0, v1, v2, u[4], m[3]);+	MONT_MAC(w0, w1, w2, u[5], m[2]);+	MONT_MAC(t0, t1, t2, u[6], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[7] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[7], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 8 */+	MONT_MUL(s0, s1, s2, a[0], a[8]);+	MONT_MUL(c0, c1, c2, a[1], a[7]);+	MONT_MAC(s0, s1, s2, a[2], a[6]);+	MONT_MAC(c0, c1, c2, a[3], a[5]);+	MONT_MAC(t0, t1, t2, a[4], a[4]);+	MONT_MUL(v0, v1, v2, u[0], m[8]);+	MONT_MUL(w0, w1, w2, u[1], m[7]);+	MONT_MAC(t0, t1, t2, u[2], m[6]);+	MONT_MAC(v0, v1, v2, u[3], m[5]);+	MONT_MAC(w0, w1, w2, u[4], m[4]);+	MONT_MAC(t0, t1, t2, u[5], m[3]);+	MONT_MAC(v0, v1, v2, u[6], m[2]);+	MONT_MAC(w0, w1, w2, u[7], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[8] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[8], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 9 */+	MONT_MUL(s0, s1, s2, a[0], a[9]);+	MONT_MUL(c0, c1, c2, a[1], a[8]);+	MONT_MAC(s0, s1, s2, a[2], a[7]);+	MONT_MAC(c0, c1, c2, a[3], a[6]);+	MONT_MAC(s0, s1, s2, a[4], a[5]);+	MONT_MAC(t0, t1, t2, u[0], m[9]);+	MONT_MUL(v0, v1, v2, u[1], m[8]);+	MONT_MUL(w0, w1, w2, u[2], m[7]);+	MONT_MAC(t0, t1, t2, u[3], m[6]);+	MONT_MAC(v0, v1, v2, u[4], m[5]);+	MONT_MAC(w0, w1, w2, u[5], m[4]);+	MONT_MAC(t0, t1, t2, u[6], m[3]);+	MONT_MAC(v0, v1, v2, u[7], m[2]);+	MONT_MAC(w0, w1, w2, u[8], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[9] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[9], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 10 */+	MONT_MUL(s0, s1, s2, a[0], a[10]);+	MONT_MUL(c0, c1, c2, a[1], a[9]);+	MONT_MAC(s0, s1, s2, a[2], a[8]);+	MONT_MAC(c0, c1, c2, a[3], a[7]);+	MONT_MAC(s0, s1, s2, a[4], a[6]);+	MONT_MAC(t0, t1, t2, a[5], a[5]);+	MONT_MUL(v0, v1, v2, u[0], m[10]);+	MONT_MUL(w0, w1, w2, u[1], m[9]);+	MONT_MAC(t0, t1, t2, u[2], m[8]);+	MONT_MAC(v0, v1, v2, u[3], m[7]);+	MONT_MAC(w0, w1, w2, u[4], m[6]);+	MONT_MAC(t0, t1, t2, u[5], m[5]);+	MONT_MAC(v0, v1, v2, u[6], m[4]);+	MONT_MAC(w0, w1, w2, u[7], m[3]);+	MONT_MAC(t0, t1, t2, u[8], m[2]);+	MONT_MAC(v0, v1, v2, u[9], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[10] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[10], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 11 */+	MONT_MUL(s0, s1, s2, a[0], a[11]);+	MONT_MUL(c0, c1, c2, a[1], a[10]);+	MONT_MAC(s0, s1, s2, a[2], a[9]);+	MONT_MAC(c0, c1, c2, a[3], a[8]);+	MONT_MAC(s0, s1, s2, a[4], a[7]);+	MONT_MAC(c0, c1, c2, a[5], a[6]);+	MONT_MAC(t0, t1, t2, u[0], m[11]);+	MONT_MUL(v0, v1, v2, u[1], m[10]);+	MONT_MUL(w0, w1, w2, u[2], m[9]);+	MONT_MAC(t0, t1, t2, u[3], m[8]);+	MONT_MAC(v0, v1, v2, u[4], m[7]);+	MONT_MAC(w0, w1, w2, u[5], m[6]);+	MONT_MAC(t0, t1, t2, u[6], m[5]);+	MONT_MAC(v0, v1, v2, u[7], m[4]);+	MONT_MAC(w0, w1, w2, u[8], m[3]);+	MONT_MAC(t0, t1, t2, u[9], m[2]);+	MONT_MAC(v0, v1, v2, u[10], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[11] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[11], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 12 */+	MONT_MUL(s0, s1, s2, a[0], a[12]);+	MONT_MUL(c0, c1, c2, a[1], a[11]);+	MONT_MAC(s0, s1, s2, a[2], a[10]);+	MONT_MAC(c0, c1, c2, a[3], a[9]);+	MONT_MAC(s0, s1, s2, a[4], a[8]);+	MONT_MAC(c0, c1, c2, a[5], a[7]);+	MONT_MAC(t0, t1, t2, a[6], a[6]);+	MONT_MUL(v0, v1, v2, u[0], m[12]);+	MONT_MUL(w0, w1, w2, u[1], m[11]);+	MONT_MAC(t0, t1, t2, u[2], m[10]);+	MONT_MAC(v0, v1, v2, u[3], m[9]);+	MONT_MAC(w0, w1, w2, u[4], m[8]);+	MONT_MAC(t0, t1, t2, u[5], m[7]);+	MONT_MAC(v0, v1, v2, u[6], m[6]);+	MONT_MAC(w0, w1, w2, u[7], m[5]);+	MONT_MAC(t0, t1, t2, u[8], m[4]);+	MONT_MAC(v0, v1, v2, u[9], m[3]);+	MONT_MAC(w0, w1, w2, u[10], m[2]);+	MONT_MAC(t0, t1, t2, u[11], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[12] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[12], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 13 */+	MONT_MUL(s0, s1, s2, a[0], a[13]);+	MONT_MUL(c0, c1, c2, a[1], a[12]);+	MONT_MAC(s0, s1, s2, a[2], a[11]);+	MONT_MAC(c0, c1, c2, a[3], a[10]);+	MONT_MAC(s0, s1, s2, a[4], a[9]);+	MONT_MAC(c0, c1, c2, a[5], a[8]);+	MONT_MAC(s0, s1, s2, a[6], a[7]);+	MONT_MAC(t0, t1, t2, u[0], m[13]);+	MONT_MUL(v0, v1, v2, u[1], m[12]);+	MONT_MUL(w0, w1, w2, u[2], m[11]);+	MONT_MAC(t0, t1, t2, u[3], m[10]);+	MONT_MAC(v0, v1, v2, u[4], m[9]);+	MONT_MAC(w0, w1, w2, u[5], m[8]);+	MONT_MAC(t0, t1, t2, u[6], m[7]);+	MONT_MAC(v0, v1, v2, u[7], m[6]);+	MONT_MAC(w0, w1, w2, u[8], m[5]);+	MONT_MAC(t0, t1, t2, u[9], m[4]);+	MONT_MAC(v0, v1, v2, u[10], m[3]);+	MONT_MAC(w0, w1, w2, u[11], m[2]);+	MONT_MAC(t0, t1, t2, u[12], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[13] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[13], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 14 */+	MONT_MUL(s0, s1, s2, a[0], a[14]);+	MONT_MUL(c0, c1, c2, a[1], a[13]);+	MONT_MAC(s0, s1, s2, a[2], a[12]);+	MONT_MAC(c0, c1, c2, a[3], a[11]);+	MONT_MAC(s0, s1, s2, a[4], a[10]);+	MONT_MAC(c0, c1, c2, a[5], a[9]);+	MONT_MAC(s0, s1, s2, a[6], a[8]);+	MONT_MAC(t0, t1, t2, a[7], a[7]);+	MONT_MUL(v0, v1, v2, u[0], m[14]);+	MONT_MUL(w0, w1, w2, u[1], m[13]);+	MONT_MAC(t0, t1, t2, u[2], m[12]);+	MONT_MAC(v0, v1, v2, u[3], m[11]);+	MONT_MAC(w0, w1, w2, u[4], m[10]);+	MONT_MAC(t0, t1, t2, u[5], m[9]);+	MONT_MAC(v0, v1, v2, u[6], m[8]);+	MONT_MAC(w0, w1, w2, u[7], m[7]);+	MONT_MAC(t0, t1, t2, u[8], m[6]);+	MONT_MAC(v0, v1, v2, u[9], m[5]);+	MONT_MAC(w0, w1, w2, u[10], m[4]);+	MONT_MAC(t0, t1, t2, u[11], m[3]);+	MONT_MAC(v0, v1, v2, u[12], m[2]);+	MONT_MAC(w0, w1, w2, u[13], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[14] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[14], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 15 */+	MONT_MUL(s0, s1, s2, a[0], a[15]);+	MONT_MUL(c0, c1, c2, a[1], a[14]);+	MONT_MAC(s0, s1, s2, a[2], a[13]);+	MONT_MAC(c0, c1, c2, a[3], a[12]);+	MONT_MAC(s0, s1, s2, a[4], a[11]);+	MONT_MAC(c0, c1, c2, a[5], a[10]);+	MONT_MAC(s0, s1, s2, a[6], a[9]);+	MONT_MAC(c0, c1, c2, a[7], a[8]);+	MONT_MAC(t0, t1, t2, u[0], m[15]);+	MONT_MUL(v0, v1, v2, u[1], m[14]);+	MONT_MUL(w0, w1, w2, u[2], m[13]);+	MONT_MAC(t0, t1, t2, u[3], m[12]);+	MONT_MAC(v0, v1, v2, u[4], m[11]);+	MONT_MAC(w0, w1, w2, u[5], m[10]);+	MONT_MAC(t0, t1, t2, u[6], m[9]);+	MONT_MAC(v0, v1, v2, u[7], m[8]);+	MONT_MAC(w0, w1, w2, u[8], m[7]);+	MONT_MAC(t0, t1, t2, u[9], m[6]);+	MONT_MAC(v0, v1, v2, u[10], m[5]);+	MONT_MAC(w0, w1, w2, u[11], m[4]);+	MONT_MAC(t0, t1, t2, u[12], m[3]);+	MONT_MAC(v0, v1, v2, u[13], m[2]);+	MONT_MAC(w0, w1, w2, u[14], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[15] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[15], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 16 */+	MONT_MUL(s0, s1, s2, a[0], a[16]);+	MONT_MUL(c0, c1, c2, a[1], a[15]);+	MONT_MAC(s0, s1, s2, a[2], a[14]);+	MONT_MAC(c0, c1, c2, a[3], a[13]);+	MONT_MAC(s0, s1, s2, a[4], a[12]);+	MONT_MAC(c0, c1, c2, a[5], a[11]);+	MONT_MAC(s0, s1, s2, a[6], a[10]);+	MONT_MAC(c0, c1, c2, a[7], a[9]);+	MONT_MAC(t0, t1, t2, a[8], a[8]);+	MONT_MUL(v0, v1, v2, u[0], m[16]);+	MONT_MUL(w0, w1, w2, u[1], m[15]);+	MONT_MAC(t0, t1, t2, u[2], m[14]);+	MONT_MAC(v0, v1, v2, u[3], m[13]);+	MONT_MAC(w0, w1, w2, u[4], m[12]);+	MONT_MAC(t0, t1, t2, u[5], m[11]);+	MONT_MAC(v0, v1, v2, u[6], m[10]);+	MONT_MAC(w0, w1, w2, u[7], m[9]);+	MONT_MAC(t0, t1, t2, u[8], m[8]);+	MONT_MAC(v0, v1, v2, u[9], m[7]);+	MONT_MAC(w0, w1, w2, u[10], m[6]);+	MONT_MAC(t0, t1, t2, u[11], m[5]);+	MONT_MAC(v0, v1, v2, u[12], m[4]);+	MONT_MAC(w0, w1, w2, u[13], m[3]);+	MONT_MAC(t0, t1, t2, u[14], m[2]);+	MONT_MAC(v0, v1, v2, u[15], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[16] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[16], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 17 */+	MONT_MUL(s0, s1, s2, a[0], a[17]);+	MONT_MUL(c0, c1, c2, a[1], a[16]);+	MONT_MAC(s0, s1, s2, a[2], a[15]);+	MONT_MAC(c0, c1, c2, a[3], a[14]);+	MONT_MAC(s0, s1, s2, a[4], a[13]);+	MONT_MAC(c0, c1, c2, a[5], a[12]);+	MONT_MAC(s0, s1, s2, a[6], a[11]);+	MONT_MAC(c0, c1, c2, a[7], a[10]);+	MONT_MAC(s0, s1, s2, a[8], a[9]);+	MONT_MAC(t0, t1, t2, u[0], m[17]);+	MONT_MUL(v0, v1, v2, u[1], m[16]);+	MONT_MUL(w0, w1, w2, u[2], m[15]);+	MONT_MAC(t0, t1, t2, u[3], m[14]);+	MONT_MAC(v0, v1, v2, u[4], m[13]);+	MONT_MAC(w0, w1, w2, u[5], m[12]);+	MONT_MAC(t0, t1, t2, u[6], m[11]);+	MONT_MAC(v0, v1, v2, u[7], m[10]);+	MONT_MAC(w0, w1, w2, u[8], m[9]);+	MONT_MAC(t0, t1, t2, u[9], m[8]);+	MONT_MAC(v0, v1, v2, u[10], m[7]);+	MONT_MAC(w0, w1, w2, u[11], m[6]);+	MONT_MAC(t0, t1, t2, u[12], m[5]);+	MONT_MAC(v0, v1, v2, u[13], m[4]);+	MONT_MAC(w0, w1, w2, u[14], m[3]);+	MONT_MAC(t0, t1, t2, u[15], m[2]);+	MONT_MAC(v0, v1, v2, u[16], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[17] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[17], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 18 */+	MONT_MUL(s0, s1, s2, a[0], a[18]);+	MONT_MUL(c0, c1, c2, a[1], a[17]);+	MONT_MAC(s0, s1, s2, a[2], a[16]);+	MONT_MAC(c0, c1, c2, a[3], a[15]);+	MONT_MAC(s0, s1, s2, a[4], a[14]);+	MONT_MAC(c0, c1, c2, a[5], a[13]);+	MONT_MAC(s0, s1, s2, a[6], a[12]);+	MONT_MAC(c0, c1, c2, a[7], a[11]);+	MONT_MAC(s0, s1, s2, a[8], a[10]);+	MONT_MAC(t0, t1, t2, a[9], a[9]);+	MONT_MUL(v0, v1, v2, u[0], m[18]);+	MONT_MUL(w0, w1, w2, u[1], m[17]);+	MONT_MAC(t0, t1, t2, u[2], m[16]);+	MONT_MAC(v0, v1, v2, u[3], m[15]);+	MONT_MAC(w0, w1, w2, u[4], m[14]);+	MONT_MAC(t0, t1, t2, u[5], m[13]);+	MONT_MAC(v0, v1, v2, u[6], m[12]);+	MONT_MAC(w0, w1, w2, u[7], m[11]);+	MONT_MAC(t0, t1, t2, u[8], m[10]);+	MONT_MAC(v0, v1, v2, u[9], m[9]);+	MONT_MAC(w0, w1, w2, u[10], m[8]);+	MONT_MAC(t0, t1, t2, u[11], m[7]);+	MONT_MAC(v0, v1, v2, u[12], m[6]);+	MONT_MAC(w0, w1, w2, u[13], m[5]);+	MONT_MAC(t0, t1, t2, u[14], m[4]);+	MONT_MAC(v0, v1, v2, u[15], m[3]);+	MONT_MAC(w0, w1, w2, u[16], m[2]);+	MONT_MAC(t0, t1, t2, u[17], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[18] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[18], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 19 */+	MONT_MUL(s0, s1, s2, a[0], a[19]);+	MONT_MUL(c0, c1, c2, a[1], a[18]);+	MONT_MAC(s0, s1, s2, a[2], a[17]);+	MONT_MAC(c0, c1, c2, a[3], a[16]);+	MONT_MAC(s0, s1, s2, a[4], a[15]);+	MONT_MAC(c0, c1, c2, a[5], a[14]);+	MONT_MAC(s0, s1, s2, a[6], a[13]);+	MONT_MAC(c0, c1, c2, a[7], a[12]);+	MONT_MAC(s0, s1, s2, a[8], a[11]);+	MONT_MAC(c0, c1, c2, a[9], a[10]);+	MONT_MAC(t0, t1, t2, u[0], m[19]);+	MONT_MUL(v0, v1, v2, u[1], m[18]);+	MONT_MUL(w0, w1, w2, u[2], m[17]);+	MONT_MAC(t0, t1, t2, u[3], m[16]);+	MONT_MAC(v0, v1, v2, u[4], m[15]);+	MONT_MAC(w0, w1, w2, u[5], m[14]);+	MONT_MAC(t0, t1, t2, u[6], m[13]);+	MONT_MAC(v0, v1, v2, u[7], m[12]);+	MONT_MAC(w0, w1, w2, u[8], m[11]);+	MONT_MAC(t0, t1, t2, u[9], m[10]);+	MONT_MAC(v0, v1, v2, u[10], m[9]);+	MONT_MAC(w0, w1, w2, u[11], m[8]);+	MONT_MAC(t0, t1, t2, u[12], m[7]);+	MONT_MAC(v0, v1, v2, u[13], m[6]);+	MONT_MAC(w0, w1, w2, u[14], m[5]);+	MONT_MAC(t0, t1, t2, u[15], m[4]);+	MONT_MAC(v0, v1, v2, u[16], m[3]);+	MONT_MAC(w0, w1, w2, u[17], m[2]);+	MONT_MAC(t0, t1, t2, u[18], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[19] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[19], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 20 */+	MONT_MUL(s0, s1, s2, a[0], a[20]);+	MONT_MUL(c0, c1, c2, a[1], a[19]);+	MONT_MAC(s0, s1, s2, a[2], a[18]);+	MONT_MAC(c0, c1, c2, a[3], a[17]);+	MONT_MAC(s0, s1, s2, a[4], a[16]);+	MONT_MAC(c0, c1, c2, a[5], a[15]);+	MONT_MAC(s0, s1, s2, a[6], a[14]);+	MONT_MAC(c0, c1, c2, a[7], a[13]);+	MONT_MAC(s0, s1, s2, a[8], a[12]);+	MONT_MAC(c0, c1, c2, a[9], a[11]);+	MONT_MAC(t0, t1, t2, a[10], a[10]);+	MONT_MUL(v0, v1, v2, u[0], m[20]);+	MONT_MUL(w0, w1, w2, u[1], m[19]);+	MONT_MAC(t0, t1, t2, u[2], m[18]);+	MONT_MAC(v0, v1, v2, u[3], m[17]);+	MONT_MAC(w0, w1, w2, u[4], m[16]);+	MONT_MAC(t0, t1, t2, u[5], m[15]);+	MONT_MAC(v0, v1, v2, u[6], m[14]);+	MONT_MAC(w0, w1, w2, u[7], m[13]);+	MONT_MAC(t0, t1, t2, u[8], m[12]);+	MONT_MAC(v0, v1, v2, u[9], m[11]);+	MONT_MAC(w0, w1, w2, u[10], m[10]);+	MONT_MAC(t0, t1, t2, u[11], m[9]);+	MONT_MAC(v0, v1, v2, u[12], m[8]);+	MONT_MAC(w0, w1, w2, u[13], m[7]);+	MONT_MAC(t0, t1, t2, u[14], m[6]);+	MONT_MAC(v0, v1, v2, u[15], m[5]);+	MONT_MAC(w0, w1, w2, u[16], m[4]);+	MONT_MAC(t0, t1, t2, u[17], m[3]);+	MONT_MAC(v0, v1, v2, u[18], m[2]);+	MONT_MAC(w0, w1, w2, u[19], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[20] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[20], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 21 */+	MONT_MUL(s0, s1, s2, a[0], a[21]);+	MONT_MUL(c0, c1, c2, a[1], a[20]);+	MONT_MAC(s0, s1, s2, a[2], a[19]);+	MONT_MAC(c0, c1, c2, a[3], a[18]);+	MONT_MAC(s0, s1, s2, a[4], a[17]);+	MONT_MAC(c0, c1, c2, a[5], a[16]);+	MONT_MAC(s0, s1, s2, a[6], a[15]);+	MONT_MAC(c0, c1, c2, a[7], a[14]);+	MONT_MAC(s0, s1, s2, a[8], a[13]);+	MONT_MAC(c0, c1, c2, a[9], a[12]);+	MONT_MAC(s0, s1, s2, a[10], a[11]);+	MONT_MAC(t0, t1, t2, u[0], m[21]);+	MONT_MUL(v0, v1, v2, u[1], m[20]);+	MONT_MUL(w0, w1, w2, u[2], m[19]);+	MONT_MAC(t0, t1, t2, u[3], m[18]);+	MONT_MAC(v0, v1, v2, u[4], m[17]);+	MONT_MAC(w0, w1, w2, u[5], m[16]);+	MONT_MAC(t0, t1, t2, u[6], m[15]);+	MONT_MAC(v0, v1, v2, u[7], m[14]);+	MONT_MAC(w0, w1, w2, u[8], m[13]);+	MONT_MAC(t0, t1, t2, u[9], m[12]);+	MONT_MAC(v0, v1, v2, u[10], m[11]);+	MONT_MAC(w0, w1, w2, u[11], m[10]);+	MONT_MAC(t0, t1, t2, u[12], m[9]);+	MONT_MAC(v0, v1, v2, u[13], m[8]);+	MONT_MAC(w0, w1, w2, u[14], m[7]);+	MONT_MAC(t0, t1, t2, u[15], m[6]);+	MONT_MAC(v0, v1, v2, u[16], m[5]);+	MONT_MAC(w0, w1, w2, u[17], m[4]);+	MONT_MAC(t0, t1, t2, u[18], m[3]);+	MONT_MAC(v0, v1, v2, u[19], m[2]);+	MONT_MAC(w0, w1, w2, u[20], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[21] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[21], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 22 */+	MONT_MUL(s0, s1, s2, a[0], a[22]);+	MONT_MUL(c0, c1, c2, a[1], a[21]);+	MONT_MAC(s0, s1, s2, a[2], a[20]);+	MONT_MAC(c0, c1, c2, a[3], a[19]);+	MONT_MAC(s0, s1, s2, a[4], a[18]);+	MONT_MAC(c0, c1, c2, a[5], a[17]);+	MONT_MAC(s0, s1, s2, a[6], a[16]);+	MONT_MAC(c0, c1, c2, a[7], a[15]);+	MONT_MAC(s0, s1, s2, a[8], a[14]);+	MONT_MAC(c0, c1, c2, a[9], a[13]);+	MONT_MAC(s0, s1, s2, a[10], a[12]);+	MONT_MAC(t0, t1, t2, a[11], a[11]);+	MONT_MUL(v0, v1, v2, u[0], m[22]);+	MONT_MUL(w0, w1, w2, u[1], m[21]);+	MONT_MAC(t0, t1, t2, u[2], m[20]);+	MONT_MAC(v0, v1, v2, u[3], m[19]);+	MONT_MAC(w0, w1, w2, u[4], m[18]);+	MONT_MAC(t0, t1, t2, u[5], m[17]);+	MONT_MAC(v0, v1, v2, u[6], m[16]);+	MONT_MAC(w0, w1, w2, u[7], m[15]);+	MONT_MAC(t0, t1, t2, u[8], m[14]);+	MONT_MAC(v0, v1, v2, u[9], m[13]);+	MONT_MAC(w0, w1, w2, u[10], m[12]);+	MONT_MAC(t0, t1, t2, u[11], m[11]);+	MONT_MAC(v0, v1, v2, u[12], m[10]);+	MONT_MAC(w0, w1, w2, u[13], m[9]);+	MONT_MAC(t0, t1, t2, u[14], m[8]);+	MONT_MAC(v0, v1, v2, u[15], m[7]);+	MONT_MAC(w0, w1, w2, u[16], m[6]);+	MONT_MAC(t0, t1, t2, u[17], m[5]);+	MONT_MAC(v0, v1, v2, u[18], m[4]);+	MONT_MAC(w0, w1, w2, u[19], m[3]);+	MONT_MAC(t0, t1, t2, u[20], m[2]);+	MONT_MAC(v0, v1, v2, u[21], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[22] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[22], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 23 */+	MONT_MUL(s0, s1, s2, a[0], a[23]);+	MONT_MUL(c0, c1, c2, a[1], a[22]);+	MONT_MAC(s0, s1, s2, a[2], a[21]);+	MONT_MAC(c0, c1, c2, a[3], a[20]);+	MONT_MAC(s0, s1, s2, a[4], a[19]);+	MONT_MAC(c0, c1, c2, a[5], a[18]);+	MONT_MAC(s0, s1, s2, a[6], a[17]);+	MONT_MAC(c0, c1, c2, a[7], a[16]);+	MONT_MAC(s0, s1, s2, a[8], a[15]);+	MONT_MAC(c0, c1, c2, a[9], a[14]);+	MONT_MAC(s0, s1, s2, a[10], a[13]);+	MONT_MAC(c0, c1, c2, a[11], a[12]);+	MONT_MAC(t0, t1, t2, u[0], m[23]);+	MONT_MUL(v0, v1, v2, u[1], m[22]);+	MONT_MUL(w0, w1, w2, u[2], m[21]);+	MONT_MAC(t0, t1, t2, u[3], m[20]);+	MONT_MAC(v0, v1, v2, u[4], m[19]);+	MONT_MAC(w0, w1, w2, u[5], m[18]);+	MONT_MAC(t0, t1, t2, u[6], m[17]);+	MONT_MAC(v0, v1, v2, u[7], m[16]);+	MONT_MAC(w0, w1, w2, u[8], m[15]);+	MONT_MAC(t0, t1, t2, u[9], m[14]);+	MONT_MAC(v0, v1, v2, u[10], m[13]);+	MONT_MAC(w0, w1, w2, u[11], m[12]);+	MONT_MAC(t0, t1, t2, u[12], m[11]);+	MONT_MAC(v0, v1, v2, u[13], m[10]);+	MONT_MAC(w0, w1, w2, u[14], m[9]);+	MONT_MAC(t0, t1, t2, u[15], m[8]);+	MONT_MAC(v0, v1, v2, u[16], m[7]);+	MONT_MAC(w0, w1, w2, u[17], m[6]);+	MONT_MAC(t0, t1, t2, u[18], m[5]);+	MONT_MAC(v0, v1, v2, u[19], m[4]);+	MONT_MAC(w0, w1, w2, u[20], m[3]);+	MONT_MAC(t0, t1, t2, u[21], m[2]);+	MONT_MAC(v0, v1, v2, u[22], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[23] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[23], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 24 */+	MONT_MUL(s0, s1, s2, a[0], a[24]);+	MONT_MUL(c0, c1, c2, a[1], a[23]);+	MONT_MAC(s0, s1, s2, a[2], a[22]);+	MONT_MAC(c0, c1, c2, a[3], a[21]);+	MONT_MAC(s0, s1, s2, a[4], a[20]);+	MONT_MAC(c0, c1, c2, a[5], a[19]);+	MONT_MAC(s0, s1, s2, a[6], a[18]);+	MONT_MAC(c0, c1, c2, a[7], a[17]);+	MONT_MAC(s0, s1, s2, a[8], a[16]);+	MONT_MAC(c0, c1, c2, a[9], a[15]);+	MONT_MAC(s0, s1, s2, a[10], a[14]);+	MONT_MAC(c0, c1, c2, a[11], a[13]);+	MONT_MAC(t0, t1, t2, a[12], a[12]);+	MONT_MUL(v0, v1, v2, u[0], m[24]);+	MONT_MUL(w0, w1, w2, u[1], m[23]);+	MONT_MAC(t0, t1, t2, u[2], m[22]);+	MONT_MAC(v0, v1, v2, u[3], m[21]);+	MONT_MAC(w0, w1, w2, u[4], m[20]);+	MONT_MAC(t0, t1, t2, u[5], m[19]);+	MONT_MAC(v0, v1, v2, u[6], m[18]);+	MONT_MAC(w0, w1, w2, u[7], m[17]);+	MONT_MAC(t0, t1, t2, u[8], m[16]);+	MONT_MAC(v0, v1, v2, u[9], m[15]);+	MONT_MAC(w0, w1, w2, u[10], m[14]);+	MONT_MAC(t0, t1, t2, u[11], m[13]);+	MONT_MAC(v0, v1, v2, u[12], m[12]);+	MONT_MAC(w0, w1, w2, u[13], m[11]);+	MONT_MAC(t0, t1, t2, u[14], m[10]);+	MONT_MAC(v0, v1, v2, u[15], m[9]);+	MONT_MAC(w0, w1, w2, u[16], m[8]);+	MONT_MAC(t0, t1, t2, u[17], m[7]);+	MONT_MAC(v0, v1, v2, u[18], m[6]);+	MONT_MAC(w0, w1, w2, u[19], m[5]);+	MONT_MAC(t0, t1, t2, u[20], m[4]);+	MONT_MAC(v0, v1, v2, u[21], m[3]);+	MONT_MAC(w0, w1, w2, u[22], m[2]);+	MONT_MAC(t0, t1, t2, u[23], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[24] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[24], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 25 */+	MONT_MUL(s0, s1, s2, a[0], a[25]);+	MONT_MUL(c0, c1, c2, a[1], a[24]);+	MONT_MAC(s0, s1, s2, a[2], a[23]);+	MONT_MAC(c0, c1, c2, a[3], a[22]);+	MONT_MAC(s0, s1, s2, a[4], a[21]);+	MONT_MAC(c0, c1, c2, a[5], a[20]);+	MONT_MAC(s0, s1, s2, a[6], a[19]);+	MONT_MAC(c0, c1, c2, a[7], a[18]);+	MONT_MAC(s0, s1, s2, a[8], a[17]);+	MONT_MAC(c0, c1, c2, a[9], a[16]);+	MONT_MAC(s0, s1, s2, a[10], a[15]);+	MONT_MAC(c0, c1, c2, a[11], a[14]);+	MONT_MAC(s0, s1, s2, a[12], a[13]);+	MONT_MAC(t0, t1, t2, u[0], m[25]);+	MONT_MUL(v0, v1, v2, u[1], m[24]);+	MONT_MUL(w0, w1, w2, u[2], m[23]);+	MONT_MAC(t0, t1, t2, u[3], m[22]);+	MONT_MAC(v0, v1, v2, u[4], m[21]);+	MONT_MAC(w0, w1, w2, u[5], m[20]);+	MONT_MAC(t0, t1, t2, u[6], m[19]);+	MONT_MAC(v0, v1, v2, u[7], m[18]);+	MONT_MAC(w0, w1, w2, u[8], m[17]);+	MONT_MAC(t0, t1, t2, u[9], m[16]);+	MONT_MAC(v0, v1, v2, u[10], m[15]);+	MONT_MAC(w0, w1, w2, u[11], m[14]);+	MONT_MAC(t0, t1, t2, u[12], m[13]);+	MONT_MAC(v0, v1, v2, u[13], m[12]);+	MONT_MAC(w0, w1, w2, u[14], m[11]);+	MONT_MAC(t0, t1, t2, u[15], m[10]);+	MONT_MAC(v0, v1, v2, u[16], m[9]);+	MONT_MAC(w0, w1, w2, u[17], m[8]);+	MONT_MAC(t0, t1, t2, u[18], m[7]);+	MONT_MAC(v0, v1, v2, u[19], m[6]);+	MONT_MAC(w0, w1, w2, u[20], m[5]);+	MONT_MAC(t0, t1, t2, u[21], m[4]);+	MONT_MAC(v0, v1, v2, u[22], m[3]);+	MONT_MAC(w0, w1, w2, u[23], m[2]);+	MONT_MAC(t0, t1, t2, u[24], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[25] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[25], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 26 */+	MONT_MUL(s0, s1, s2, a[0], a[26]);+	MONT_MUL(c0, c1, c2, a[1], a[25]);+	MONT_MAC(s0, s1, s2, a[2], a[24]);+	MONT_MAC(c0, c1, c2, a[3], a[23]);+	MONT_MAC(s0, s1, s2, a[4], a[22]);+	MONT_MAC(c0, c1, c2, a[5], a[21]);+	MONT_MAC(s0, s1, s2, a[6], a[20]);+	MONT_MAC(c0, c1, c2, a[7], a[19]);+	MONT_MAC(s0, s1, s2, a[8], a[18]);+	MONT_MAC(c0, c1, c2, a[9], a[17]);+	MONT_MAC(s0, s1, s2, a[10], a[16]);+	MONT_MAC(c0, c1, c2, a[11], a[15]);+	MONT_MAC(s0, s1, s2, a[12], a[14]);+	MONT_MAC(t0, t1, t2, a[13], a[13]);+	MONT_MUL(v0, v1, v2, u[0], m[26]);+	MONT_MUL(w0, w1, w2, u[1], m[25]);+	MONT_MAC(t0, t1, t2, u[2], m[24]);+	MONT_MAC(v0, v1, v2, u[3], m[23]);+	MONT_MAC(w0, w1, w2, u[4], m[22]);+	MONT_MAC(t0, t1, t2, u[5], m[21]);+	MONT_MAC(v0, v1, v2, u[6], m[20]);+	MONT_MAC(w0, w1, w2, u[7], m[19]);+	MONT_MAC(t0, t1, t2, u[8], m[18]);+	MONT_MAC(v0, v1, v2, u[9], m[17]);+	MONT_MAC(w0, w1, w2, u[10], m[16]);+	MONT_MAC(t0, t1, t2, u[11], m[15]);+	MONT_MAC(v0, v1, v2, u[12], m[14]);+	MONT_MAC(w0, w1, w2, u[13], m[13]);+	MONT_MAC(t0, t1, t2, u[14], m[12]);+	MONT_MAC(v0, v1, v2, u[15], m[11]);+	MONT_MAC(w0, w1, w2, u[16], m[10]);+	MONT_MAC(t0, t1, t2, u[17], m[9]);+	MONT_MAC(v0, v1, v2, u[18], m[8]);+	MONT_MAC(w0, w1, w2, u[19], m[7]);+	MONT_MAC(t0, t1, t2, u[20], m[6]);+	MONT_MAC(v0, v1, v2, u[21], m[5]);+	MONT_MAC(w0, w1, w2, u[22], m[4]);+	MONT_MAC(t0, t1, t2, u[23], m[3]);+	MONT_MAC(v0, v1, v2, u[24], m[2]);+	MONT_MAC(w0, w1, w2, u[25], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[26] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[26], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 27 */+	MONT_MUL(s0, s1, s2, a[0], a[27]);+	MONT_MUL(c0, c1, c2, a[1], a[26]);+	MONT_MAC(s0, s1, s2, a[2], a[25]);+	MONT_MAC(c0, c1, c2, a[3], a[24]);+	MONT_MAC(s0, s1, s2, a[4], a[23]);+	MONT_MAC(c0, c1, c2, a[5], a[22]);+	MONT_MAC(s0, s1, s2, a[6], a[21]);+	MONT_MAC(c0, c1, c2, a[7], a[20]);+	MONT_MAC(s0, s1, s2, a[8], a[19]);+	MONT_MAC(c0, c1, c2, a[9], a[18]);+	MONT_MAC(s0, s1, s2, a[10], a[17]);+	MONT_MAC(c0, c1, c2, a[11], a[16]);+	MONT_MAC(s0, s1, s2, a[12], a[15]);+	MONT_MAC(c0, c1, c2, a[13], a[14]);+	MONT_MAC(t0, t1, t2, u[0], m[27]);+	MONT_MUL(v0, v1, v2, u[1], m[26]);+	MONT_MUL(w0, w1, w2, u[2], m[25]);+	MONT_MAC(t0, t1, t2, u[3], m[24]);+	MONT_MAC(v0, v1, v2, u[4], m[23]);+	MONT_MAC(w0, w1, w2, u[5], m[22]);+	MONT_MAC(t0, t1, t2, u[6], m[21]);+	MONT_MAC(v0, v1, v2, u[7], m[20]);+	MONT_MAC(w0, w1, w2, u[8], m[19]);+	MONT_MAC(t0, t1, t2, u[9], m[18]);+	MONT_MAC(v0, v1, v2, u[10], m[17]);+	MONT_MAC(w0, w1, w2, u[11], m[16]);+	MONT_MAC(t0, t1, t2, u[12], m[15]);+	MONT_MAC(v0, v1, v2, u[13], m[14]);+	MONT_MAC(w0, w1, w2, u[14], m[13]);+	MONT_MAC(t0, t1, t2, u[15], m[12]);+	MONT_MAC(v0, v1, v2, u[16], m[11]);+	MONT_MAC(w0, w1, w2, u[17], m[10]);+	MONT_MAC(t0, t1, t2, u[18], m[9]);+	MONT_MAC(v0, v1, v2, u[19], m[8]);+	MONT_MAC(w0, w1, w2, u[20], m[7]);+	MONT_MAC(t0, t1, t2, u[21], m[6]);+	MONT_MAC(v0, v1, v2, u[22], m[5]);+	MONT_MAC(w0, w1, w2, u[23], m[4]);+	MONT_MAC(t0, t1, t2, u[24], m[3]);+	MONT_MAC(v0, v1, v2, u[25], m[2]);+	MONT_MAC(w0, w1, w2, u[26], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[27] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[27], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 28 */+	MONT_MUL(s0, s1, s2, a[0], a[28]);+	MONT_MUL(c0, c1, c2, a[1], a[27]);+	MONT_MAC(s0, s1, s2, a[2], a[26]);+	MONT_MAC(c0, c1, c2, a[3], a[25]);+	MONT_MAC(s0, s1, s2, a[4], a[24]);+	MONT_MAC(c0, c1, c2, a[5], a[23]);+	MONT_MAC(s0, s1, s2, a[6], a[22]);+	MONT_MAC(c0, c1, c2, a[7], a[21]);+	MONT_MAC(s0, s1, s2, a[8], a[20]);+	MONT_MAC(c0, c1, c2, a[9], a[19]);+	MONT_MAC(s0, s1, s2, a[10], a[18]);+	MONT_MAC(c0, c1, c2, a[11], a[17]);+	MONT_MAC(s0, s1, s2, a[12], a[16]);+	MONT_MAC(c0, c1, c2, a[13], a[15]);+	MONT_MAC(t0, t1, t2, a[14], a[14]);+	MONT_MUL(v0, v1, v2, u[0], m[28]);+	MONT_MUL(w0, w1, w2, u[1], m[27]);+	MONT_MAC(t0, t1, t2, u[2], m[26]);+	MONT_MAC(v0, v1, v2, u[3], m[25]);+	MONT_MAC(w0, w1, w2, u[4], m[24]);+	MONT_MAC(t0, t1, t2, u[5], m[23]);+	MONT_MAC(v0, v1, v2, u[6], m[22]);+	MONT_MAC(w0, w1, w2, u[7], m[21]);+	MONT_MAC(t0, t1, t2, u[8], m[20]);+	MONT_MAC(v0, v1, v2, u[9], m[19]);+	MONT_MAC(w0, w1, w2, u[10], m[18]);+	MONT_MAC(t0, t1, t2, u[11], m[17]);+	MONT_MAC(v0, v1, v2, u[12], m[16]);+	MONT_MAC(w0, w1, w2, u[13], m[15]);+	MONT_MAC(t0, t1, t2, u[14], m[14]);+	MONT_MAC(v0, v1, v2, u[15], m[13]);+	MONT_MAC(w0, w1, w2, u[16], m[12]);+	MONT_MAC(t0, t1, t2, u[17], m[11]);+	MONT_MAC(v0, v1, v2, u[18], m[10]);+	MONT_MAC(w0, w1, w2, u[19], m[9]);+	MONT_MAC(t0, t1, t2, u[20], m[8]);+	MONT_MAC(v0, v1, v2, u[21], m[7]);+	MONT_MAC(w0, w1, w2, u[22], m[6]);+	MONT_MAC(t0, t1, t2, u[23], m[5]);+	MONT_MAC(v0, v1, v2, u[24], m[4]);+	MONT_MAC(w0, w1, w2, u[25], m[3]);+	MONT_MAC(t0, t1, t2, u[26], m[2]);+	MONT_MAC(v0, v1, v2, u[27], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[28] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[28], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 29 */+	MONT_MUL(s0, s1, s2, a[0], a[29]);+	MONT_MUL(c0, c1, c2, a[1], a[28]);+	MONT_MAC(s0, s1, s2, a[2], a[27]);+	MONT_MAC(c0, c1, c2, a[3], a[26]);+	MONT_MAC(s0, s1, s2, a[4], a[25]);+	MONT_MAC(c0, c1, c2, a[5], a[24]);+	MONT_MAC(s0, s1, s2, a[6], a[23]);+	MONT_MAC(c0, c1, c2, a[7], a[22]);+	MONT_MAC(s0, s1, s2, a[8], a[21]);+	MONT_MAC(c0, c1, c2, a[9], a[20]);+	MONT_MAC(s0, s1, s2, a[10], a[19]);+	MONT_MAC(c0, c1, c2, a[11], a[18]);+	MONT_MAC(s0, s1, s2, a[12], a[17]);+	MONT_MAC(c0, c1, c2, a[13], a[16]);+	MONT_MAC(s0, s1, s2, a[14], a[15]);+	MONT_MAC(t0, t1, t2, u[0], m[29]);+	MONT_MUL(v0, v1, v2, u[1], m[28]);+	MONT_MUL(w0, w1, w2, u[2], m[27]);+	MONT_MAC(t0, t1, t2, u[3], m[26]);+	MONT_MAC(v0, v1, v2, u[4], m[25]);+	MONT_MAC(w0, w1, w2, u[5], m[24]);+	MONT_MAC(t0, t1, t2, u[6], m[23]);+	MONT_MAC(v0, v1, v2, u[7], m[22]);+	MONT_MAC(w0, w1, w2, u[8], m[21]);+	MONT_MAC(t0, t1, t2, u[9], m[20]);+	MONT_MAC(v0, v1, v2, u[10], m[19]);+	MONT_MAC(w0, w1, w2, u[11], m[18]);+	MONT_MAC(t0, t1, t2, u[12], m[17]);+	MONT_MAC(v0, v1, v2, u[13], m[16]);+	MONT_MAC(w0, w1, w2, u[14], m[15]);+	MONT_MAC(t0, t1, t2, u[15], m[14]);+	MONT_MAC(v0, v1, v2, u[16], m[13]);+	MONT_MAC(w0, w1, w2, u[17], m[12]);+	MONT_MAC(t0, t1, t2, u[18], m[11]);+	MONT_MAC(v0, v1, v2, u[19], m[10]);+	MONT_MAC(w0, w1, w2, u[20], m[9]);+	MONT_MAC(t0, t1, t2, u[21], m[8]);+	MONT_MAC(v0, v1, v2, u[22], m[7]);+	MONT_MAC(w0, w1, w2, u[23], m[6]);+	MONT_MAC(t0, t1, t2, u[24], m[5]);+	MONT_MAC(v0, v1, v2, u[25], m[4]);+	MONT_MAC(w0, w1, w2, u[26], m[3]);+	MONT_MAC(t0, t1, t2, u[27], m[2]);+	MONT_MAC(v0, v1, v2, u[28], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[29] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[29], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 30 */+	MONT_MUL(s0, s1, s2, a[0], a[30]);+	MONT_MUL(c0, c1, c2, a[1], a[29]);+	MONT_MAC(s0, s1, s2, a[2], a[28]);+	MONT_MAC(c0, c1, c2, a[3], a[27]);+	MONT_MAC(s0, s1, s2, a[4], a[26]);+	MONT_MAC(c0, c1, c2, a[5], a[25]);+	MONT_MAC(s0, s1, s2, a[6], a[24]);+	MONT_MAC(c0, c1, c2, a[7], a[23]);+	MONT_MAC(s0, s1, s2, a[8], a[22]);+	MONT_MAC(c0, c1, c2, a[9], a[21]);+	MONT_MAC(s0, s1, s2, a[10], a[20]);+	MONT_MAC(c0, c1, c2, a[11], a[19]);+	MONT_MAC(s0, s1, s2, a[12], a[18]);+	MONT_MAC(c0, c1, c2, a[13], a[17]);+	MONT_MAC(s0, s1, s2, a[14], a[16]);+	MONT_MAC(t0, t1, t2, a[15], a[15]);+	MONT_MUL(v0, v1, v2, u[0], m[30]);+	MONT_MUL(w0, w1, w2, u[1], m[29]);+	MONT_MAC(t0, t1, t2, u[2], m[28]);+	MONT_MAC(v0, v1, v2, u[3], m[27]);+	MONT_MAC(w0, w1, w2, u[4], m[26]);+	MONT_MAC(t0, t1, t2, u[5], m[25]);+	MONT_MAC(v0, v1, v2, u[6], m[24]);+	MONT_MAC(w0, w1, w2, u[7], m[23]);+	MONT_MAC(t0, t1, t2, u[8], m[22]);+	MONT_MAC(v0, v1, v2, u[9], m[21]);+	MONT_MAC(w0, w1, w2, u[10], m[20]);+	MONT_MAC(t0, t1, t2, u[11], m[19]);+	MONT_MAC(v0, v1, v2, u[12], m[18]);+	MONT_MAC(w0, w1, w2, u[13], m[17]);+	MONT_MAC(t0, t1, t2, u[14], m[16]);+	MONT_MAC(v0, v1, v2, u[15], m[15]);+	MONT_MAC(w0, w1, w2, u[16], m[14]);+	MONT_MAC(t0, t1, t2, u[17], m[13]);+	MONT_MAC(v0, v1, v2, u[18], m[12]);+	MONT_MAC(w0, w1, w2, u[19], m[11]);+	MONT_MAC(t0, t1, t2, u[20], m[10]);+	MONT_MAC(v0, v1, v2, u[21], m[9]);+	MONT_MAC(w0, w1, w2, u[22], m[8]);+	MONT_MAC(t0, t1, t2, u[23], m[7]);+	MONT_MAC(v0, v1, v2, u[24], m[6]);+	MONT_MAC(w0, w1, w2, u[25], m[5]);+	MONT_MAC(t0, t1, t2, u[26], m[4]);+	MONT_MAC(v0, v1, v2, u[27], m[3]);+	MONT_MAC(w0, w1, w2, u[28], m[2]);+	MONT_MAC(t0, t1, t2, u[29], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[30] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[30], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 31 */+	MONT_MUL(s0, s1, s2, a[0], a[31]);+	MONT_MUL(c0, c1, c2, a[1], a[30]);+	MONT_MAC(s0, s1, s2, a[2], a[29]);+	MONT_MAC(c0, c1, c2, a[3], a[28]);+	MONT_MAC(s0, s1, s2, a[4], a[27]);+	MONT_MAC(c0, c1, c2, a[5], a[26]);+	MONT_MAC(s0, s1, s2, a[6], a[25]);+	MONT_MAC(c0, c1, c2, a[7], a[24]);+	MONT_MAC(s0, s1, s2, a[8], a[23]);+	MONT_MAC(c0, c1, c2, a[9], a[22]);+	MONT_MAC(s0, s1, s2, a[10], a[21]);+	MONT_MAC(c0, c1, c2, a[11], a[20]);+	MONT_MAC(s0, s1, s2, a[12], a[19]);+	MONT_MAC(c0, c1, c2, a[13], a[18]);+	MONT_MAC(s0, s1, s2, a[14], a[17]);+	MONT_MAC(c0, c1, c2, a[15], a[16]);+	MONT_MAC(t0, t1, t2, u[0], m[31]);+	MONT_MUL(v0, v1, v2, u[1], m[30]);+	MONT_MUL(w0, w1, w2, u[2], m[29]);+	MONT_MAC(t0, t1, t2, u[3], m[28]);+	MONT_MAC(v0, v1, v2, u[4], m[27]);+	MONT_MAC(w0, w1, w2, u[5], m[26]);+	MONT_MAC(t0, t1, t2, u[6], m[25]);+	MONT_MAC(v0, v1, v2, u[7], m[24]);+	MONT_MAC(w0, w1, w2, u[8], m[23]);+	MONT_MAC(t0, t1, t2, u[9], m[22]);+	MONT_MAC(v0, v1, v2, u[10], m[21]);+	MONT_MAC(w0, w1, w2, u[11], m[20]);+	MONT_MAC(t0, t1, t2, u[12], m[19]);+	MONT_MAC(v0, v1, v2, u[13], m[18]);+	MONT_MAC(w0, w1, w2, u[14], m[17]);+	MONT_MAC(t0, t1, t2, u[15], m[16]);+	MONT_MAC(v0, v1, v2, u[16], m[15]);+	MONT_MAC(w0, w1, w2, u[17], m[14]);+	MONT_MAC(t0, t1, t2, u[18], m[13]);+	MONT_MAC(v0, v1, v2, u[19], m[12]);+	MONT_MAC(w0, w1, w2, u[20], m[11]);+	MONT_MAC(t0, t1, t2, u[21], m[10]);+	MONT_MAC(v0, v1, v2, u[22], m[9]);+	MONT_MAC(w0, w1, w2, u[23], m[8]);+	MONT_MAC(t0, t1, t2, u[24], m[7]);+	MONT_MAC(v0, v1, v2, u[25], m[6]);+	MONT_MAC(w0, w1, w2, u[26], m[5]);+	MONT_MAC(t0, t1, t2, u[27], m[4]);+	MONT_MAC(v0, v1, v2, u[28], m[3]);+	MONT_MAC(w0, w1, w2, u[29], m[2]);+	MONT_MAC(t0, t1, t2, u[30], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[31] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[31], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 32 */+	MONT_MUL(s0, s1, s2, a[1], a[31]);+	MONT_MUL(c0, c1, c2, a[2], a[30]);+	MONT_MAC(s0, s1, s2, a[3], a[29]);+	MONT_MAC(c0, c1, c2, a[4], a[28]);+	MONT_MAC(s0, s1, s2, a[5], a[27]);+	MONT_MAC(c0, c1, c2, a[6], a[26]);+	MONT_MAC(s0, s1, s2, a[7], a[25]);+	MONT_MAC(c0, c1, c2, a[8], a[24]);+	MONT_MAC(s0, s1, s2, a[9], a[23]);+	MONT_MAC(c0, c1, c2, a[10], a[22]);+	MONT_MAC(s0, s1, s2, a[11], a[21]);+	MONT_MAC(c0, c1, c2, a[12], a[20]);+	MONT_MAC(s0, s1, s2, a[13], a[19]);+	MONT_MAC(c0, c1, c2, a[14], a[18]);+	MONT_MAC(s0, s1, s2, a[15], a[17]);+	MONT_MAC(t0, t1, t2, a[16], a[16]);+	MONT_MUL(v0, v1, v2, u[1], m[31]);+	MONT_MUL(w0, w1, w2, u[2], m[30]);+	MONT_MAC(t0, t1, t2, u[3], m[29]);+	MONT_MAC(v0, v1, v2, u[4], m[28]);+	MONT_MAC(w0, w1, w2, u[5], m[27]);+	MONT_MAC(t0, t1, t2, u[6], m[26]);+	MONT_MAC(v0, v1, v2, u[7], m[25]);+	MONT_MAC(w0, w1, w2, u[8], m[24]);+	MONT_MAC(t0, t1, t2, u[9], m[23]);+	MONT_MAC(v0, v1, v2, u[10], m[22]);+	MONT_MAC(w0, w1, w2, u[11], m[21]);+	MONT_MAC(t0, t1, t2, u[12], m[20]);+	MONT_MAC(v0, v1, v2, u[13], m[19]);+	MONT_MAC(w0, w1, w2, u[14], m[18]);+	MONT_MAC(t0, t1, t2, u[15], m[17]);+	MONT_MAC(v0, v1, v2, u[16], m[16]);+	MONT_MAC(w0, w1, w2, u[17], m[15]);+	MONT_MAC(t0, t1, t2, u[18], m[14]);+	MONT_MAC(v0, v1, v2, u[19], m[13]);+	MONT_MAC(w0, w1, w2, u[20], m[12]);+	MONT_MAC(t0, t1, t2, u[21], m[11]);+	MONT_MAC(v0, v1, v2, u[22], m[10]);+	MONT_MAC(w0, w1, w2, u[23], m[9]);+	MONT_MAC(t0, t1, t2, u[24], m[8]);+	MONT_MAC(v0, v1, v2, u[25], m[7]);+	MONT_MAC(w0, w1, w2, u[26], m[6]);+	MONT_MAC(t0, t1, t2, u[27], m[5]);+	MONT_MAC(v0, v1, v2, u[28], m[4]);+	MONT_MAC(w0, w1, w2, u[29], m[3]);+	MONT_MAC(t0, t1, t2, u[30], m[2]);+	MONT_MAC(v0, v1, v2, u[31], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[0] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 33 */+	MONT_MUL(s0, s1, s2, a[2], a[31]);+	MONT_MUL(c0, c1, c2, a[3], a[30]);+	MONT_MAC(s0, s1, s2, a[4], a[29]);+	MONT_MAC(c0, c1, c2, a[5], a[28]);+	MONT_MAC(s0, s1, s2, a[6], a[27]);+	MONT_MAC(c0, c1, c2, a[7], a[26]);+	MONT_MAC(s0, s1, s2, a[8], a[25]);+	MONT_MAC(c0, c1, c2, a[9], a[24]);+	MONT_MAC(s0, s1, s2, a[10], a[23]);+	MONT_MAC(c0, c1, c2, a[11], a[22]);+	MONT_MAC(s0, s1, s2, a[12], a[21]);+	MONT_MAC(c0, c1, c2, a[13], a[20]);+	MONT_MAC(s0, s1, s2, a[14], a[19]);+	MONT_MAC(c0, c1, c2, a[15], a[18]);+	MONT_MAC(s0, s1, s2, a[16], a[17]);+	MONT_MAC(t0, t1, t2, u[2], m[31]);+	MONT_MUL(v0, v1, v2, u[3], m[30]);+	MONT_MUL(w0, w1, w2, u[4], m[29]);+	MONT_MAC(t0, t1, t2, u[5], m[28]);+	MONT_MAC(v0, v1, v2, u[6], m[27]);+	MONT_MAC(w0, w1, w2, u[7], m[26]);+	MONT_MAC(t0, t1, t2, u[8], m[25]);+	MONT_MAC(v0, v1, v2, u[9], m[24]);+	MONT_MAC(w0, w1, w2, u[10], m[23]);+	MONT_MAC(t0, t1, t2, u[11], m[22]);+	MONT_MAC(v0, v1, v2, u[12], m[21]);+	MONT_MAC(w0, w1, w2, u[13], m[20]);+	MONT_MAC(t0, t1, t2, u[14], m[19]);+	MONT_MAC(v0, v1, v2, u[15], m[18]);+	MONT_MAC(w0, w1, w2, u[16], m[17]);+	MONT_MAC(t0, t1, t2, u[17], m[16]);+	MONT_MAC(v0, v1, v2, u[18], m[15]);+	MONT_MAC(w0, w1, w2, u[19], m[14]);+	MONT_MAC(t0, t1, t2, u[20], m[13]);+	MONT_MAC(v0, v1, v2, u[21], m[12]);+	MONT_MAC(w0, w1, w2, u[22], m[11]);+	MONT_MAC(t0, t1, t2, u[23], m[10]);+	MONT_MAC(v0, v1, v2, u[24], m[9]);+	MONT_MAC(w0, w1, w2, u[25], m[8]);+	MONT_MAC(t0, t1, t2, u[26], m[7]);+	MONT_MAC(v0, v1, v2, u[27], m[6]);+	MONT_MAC(w0, w1, w2, u[28], m[5]);+	MONT_MAC(t0, t1, t2, u[29], m[4]);+	MONT_MAC(v0, v1, v2, u[30], m[3]);+	MONT_MAC(w0, w1, w2, u[31], m[2]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[1] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 34 */+	MONT_MUL(s0, s1, s2, a[3], a[31]);+	MONT_MUL(c0, c1, c2, a[4], a[30]);+	MONT_MAC(s0, s1, s2, a[5], a[29]);+	MONT_MAC(c0, c1, c2, a[6], a[28]);+	MONT_MAC(s0, s1, s2, a[7], a[27]);+	MONT_MAC(c0, c1, c2, a[8], a[26]);+	MONT_MAC(s0, s1, s2, a[9], a[25]);+	MONT_MAC(c0, c1, c2, a[10], a[24]);+	MONT_MAC(s0, s1, s2, a[11], a[23]);+	MONT_MAC(c0, c1, c2, a[12], a[22]);+	MONT_MAC(s0, s1, s2, a[13], a[21]);+	MONT_MAC(c0, c1, c2, a[14], a[20]);+	MONT_MAC(s0, s1, s2, a[15], a[19]);+	MONT_MAC(c0, c1, c2, a[16], a[18]);+	MONT_MAC(t0, t1, t2, a[17], a[17]);+	MONT_MUL(v0, v1, v2, u[3], m[31]);+	MONT_MUL(w0, w1, w2, u[4], m[30]);+	MONT_MAC(t0, t1, t2, u[5], m[29]);+	MONT_MAC(v0, v1, v2, u[6], m[28]);+	MONT_MAC(w0, w1, w2, u[7], m[27]);+	MONT_MAC(t0, t1, t2, u[8], m[26]);+	MONT_MAC(v0, v1, v2, u[9], m[25]);+	MONT_MAC(w0, w1, w2, u[10], m[24]);+	MONT_MAC(t0, t1, t2, u[11], m[23]);+	MONT_MAC(v0, v1, v2, u[12], m[22]);+	MONT_MAC(w0, w1, w2, u[13], m[21]);+	MONT_MAC(t0, t1, t2, u[14], m[20]);+	MONT_MAC(v0, v1, v2, u[15], m[19]);+	MONT_MAC(w0, w1, w2, u[16], m[18]);+	MONT_MAC(t0, t1, t2, u[17], m[17]);+	MONT_MAC(v0, v1, v2, u[18], m[16]);+	MONT_MAC(w0, w1, w2, u[19], m[15]);+	MONT_MAC(t0, t1, t2, u[20], m[14]);+	MONT_MAC(v0, v1, v2, u[21], m[13]);+	MONT_MAC(w0, w1, w2, u[22], m[12]);+	MONT_MAC(t0, t1, t2, u[23], m[11]);+	MONT_MAC(v0, v1, v2, u[24], m[10]);+	MONT_MAC(w0, w1, w2, u[25], m[9]);+	MONT_MAC(t0, t1, t2, u[26], m[8]);+	MONT_MAC(v0, v1, v2, u[27], m[7]);+	MONT_MAC(w0, w1, w2, u[28], m[6]);+	MONT_MAC(t0, t1, t2, u[29], m[5]);+	MONT_MAC(v0, v1, v2, u[30], m[4]);+	MONT_MAC(w0, w1, w2, u[31], m[3]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[2] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 35 */+	MONT_MUL(s0, s1, s2, a[4], a[31]);+	MONT_MUL(c0, c1, c2, a[5], a[30]);+	MONT_MAC(s0, s1, s2, a[6], a[29]);+	MONT_MAC(c0, c1, c2, a[7], a[28]);+	MONT_MAC(s0, s1, s2, a[8], a[27]);+	MONT_MAC(c0, c1, c2, a[9], a[26]);+	MONT_MAC(s0, s1, s2, a[10], a[25]);+	MONT_MAC(c0, c1, c2, a[11], a[24]);+	MONT_MAC(s0, s1, s2, a[12], a[23]);+	MONT_MAC(c0, c1, c2, a[13], a[22]);+	MONT_MAC(s0, s1, s2, a[14], a[21]);+	MONT_MAC(c0, c1, c2, a[15], a[20]);+	MONT_MAC(s0, s1, s2, a[16], a[19]);+	MONT_MAC(c0, c1, c2, a[17], a[18]);+	MONT_MAC(t0, t1, t2, u[4], m[31]);+	MONT_MUL(v0, v1, v2, u[5], m[30]);+	MONT_MUL(w0, w1, w2, u[6], m[29]);+	MONT_MAC(t0, t1, t2, u[7], m[28]);+	MONT_MAC(v0, v1, v2, u[8], m[27]);+	MONT_MAC(w0, w1, w2, u[9], m[26]);+	MONT_MAC(t0, t1, t2, u[10], m[25]);+	MONT_MAC(v0, v1, v2, u[11], m[24]);+	MONT_MAC(w0, w1, w2, u[12], m[23]);+	MONT_MAC(t0, t1, t2, u[13], m[22]);+	MONT_MAC(v0, v1, v2, u[14], m[21]);+	MONT_MAC(w0, w1, w2, u[15], m[20]);+	MONT_MAC(t0, t1, t2, u[16], m[19]);+	MONT_MAC(v0, v1, v2, u[17], m[18]);+	MONT_MAC(w0, w1, w2, u[18], m[17]);+	MONT_MAC(t0, t1, t2, u[19], m[16]);+	MONT_MAC(v0, v1, v2, u[20], m[15]);+	MONT_MAC(w0, w1, w2, u[21], m[14]);+	MONT_MAC(t0, t1, t2, u[22], m[13]);+	MONT_MAC(v0, v1, v2, u[23], m[12]);+	MONT_MAC(w0, w1, w2, u[24], m[11]);+	MONT_MAC(t0, t1, t2, u[25], m[10]);+	MONT_MAC(v0, v1, v2, u[26], m[9]);+	MONT_MAC(w0, w1, w2, u[27], m[8]);+	MONT_MAC(t0, t1, t2, u[28], m[7]);+	MONT_MAC(v0, v1, v2, u[29], m[6]);+	MONT_MAC(w0, w1, w2, u[30], m[5]);+	MONT_MAC(t0, t1, t2, u[31], m[4]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[3] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 36 */+	MONT_MUL(s0, s1, s2, a[5], a[31]);+	MONT_MUL(c0, c1, c2, a[6], a[30]);+	MONT_MAC(s0, s1, s2, a[7], a[29]);+	MONT_MAC(c0, c1, c2, a[8], a[28]);+	MONT_MAC(s0, s1, s2, a[9], a[27]);+	MONT_MAC(c0, c1, c2, a[10], a[26]);+	MONT_MAC(s0, s1, s2, a[11], a[25]);+	MONT_MAC(c0, c1, c2, a[12], a[24]);+	MONT_MAC(s0, s1, s2, a[13], a[23]);+	MONT_MAC(c0, c1, c2, a[14], a[22]);+	MONT_MAC(s0, s1, s2, a[15], a[21]);+	MONT_MAC(c0, c1, c2, a[16], a[20]);+	MONT_MAC(s0, s1, s2, a[17], a[19]);+	MONT_MAC(t0, t1, t2, a[18], a[18]);+	MONT_MUL(v0, v1, v2, u[5], m[31]);+	MONT_MUL(w0, w1, w2, u[6], m[30]);+	MONT_MAC(t0, t1, t2, u[7], m[29]);+	MONT_MAC(v0, v1, v2, u[8], m[28]);+	MONT_MAC(w0, w1, w2, u[9], m[27]);+	MONT_MAC(t0, t1, t2, u[10], m[26]);+	MONT_MAC(v0, v1, v2, u[11], m[25]);+	MONT_MAC(w0, w1, w2, u[12], m[24]);+	MONT_MAC(t0, t1, t2, u[13], m[23]);+	MONT_MAC(v0, v1, v2, u[14], m[22]);+	MONT_MAC(w0, w1, w2, u[15], m[21]);+	MONT_MAC(t0, t1, t2, u[16], m[20]);+	MONT_MAC(v0, v1, v2, u[17], m[19]);+	MONT_MAC(w0, w1, w2, u[18], m[18]);+	MONT_MAC(t0, t1, t2, u[19], m[17]);+	MONT_MAC(v0, v1, v2, u[20], m[16]);+	MONT_MAC(w0, w1, w2, u[21], m[15]);+	MONT_MAC(t0, t1, t2, u[22], m[14]);+	MONT_MAC(v0, v1, v2, u[23], m[13]);+	MONT_MAC(w0, w1, w2, u[24], m[12]);+	MONT_MAC(t0, t1, t2, u[25], m[11]);+	MONT_MAC(v0, v1, v2, u[26], m[10]);+	MONT_MAC(w0, w1, w2, u[27], m[9]);+	MONT_MAC(t0, t1, t2, u[28], m[8]);+	MONT_MAC(v0, v1, v2, u[29], m[7]);+	MONT_MAC(w0, w1, w2, u[30], m[6]);+	MONT_MAC(t0, t1, t2, u[31], m[5]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[4] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 37 */+	MONT_MUL(s0, s1, s2, a[6], a[31]);+	MONT_MUL(c0, c1, c2, a[7], a[30]);+	MONT_MAC(s0, s1, s2, a[8], a[29]);+	MONT_MAC(c0, c1, c2, a[9], a[28]);+	MONT_MAC(s0, s1, s2, a[10], a[27]);+	MONT_MAC(c0, c1, c2, a[11], a[26]);+	MONT_MAC(s0, s1, s2, a[12], a[25]);+	MONT_MAC(c0, c1, c2, a[13], a[24]);+	MONT_MAC(s0, s1, s2, a[14], a[23]);+	MONT_MAC(c0, c1, c2, a[15], a[22]);+	MONT_MAC(s0, s1, s2, a[16], a[21]);+	MONT_MAC(c0, c1, c2, a[17], a[20]);+	MONT_MAC(s0, s1, s2, a[18], a[19]);+	MONT_MAC(t0, t1, t2, u[6], m[31]);+	MONT_MUL(v0, v1, v2, u[7], m[30]);+	MONT_MUL(w0, w1, w2, u[8], m[29]);+	MONT_MAC(t0, t1, t2, u[9], m[28]);+	MONT_MAC(v0, v1, v2, u[10], m[27]);+	MONT_MAC(w0, w1, w2, u[11], m[26]);+	MONT_MAC(t0, t1, t2, u[12], m[25]);+	MONT_MAC(v0, v1, v2, u[13], m[24]);+	MONT_MAC(w0, w1, w2, u[14], m[23]);+	MONT_MAC(t0, t1, t2, u[15], m[22]);+	MONT_MAC(v0, v1, v2, u[16], m[21]);+	MONT_MAC(w0, w1, w2, u[17], m[20]);+	MONT_MAC(t0, t1, t2, u[18], m[19]);+	MONT_MAC(v0, v1, v2, u[19], m[18]);+	MONT_MAC(w0, w1, w2, u[20], m[17]);+	MONT_MAC(t0, t1, t2, u[21], m[16]);+	MONT_MAC(v0, v1, v2, u[22], m[15]);+	MONT_MAC(w0, w1, w2, u[23], m[14]);+	MONT_MAC(t0, t1, t2, u[24], m[13]);+	MONT_MAC(v0, v1, v2, u[25], m[12]);+	MONT_MAC(w0, w1, w2, u[26], m[11]);+	MONT_MAC(t0, t1, t2, u[27], m[10]);+	MONT_MAC(v0, v1, v2, u[28], m[9]);+	MONT_MAC(w0, w1, w2, u[29], m[8]);+	MONT_MAC(t0, t1, t2, u[30], m[7]);+	MONT_MAC(v0, v1, v2, u[31], m[6]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[5] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 38 */+	MONT_MUL(s0, s1, s2, a[7], a[31]);+	MONT_MUL(c0, c1, c2, a[8], a[30]);+	MONT_MAC(s0, s1, s2, a[9], a[29]);+	MONT_MAC(c0, c1, c2, a[10], a[28]);+	MONT_MAC(s0, s1, s2, a[11], a[27]);+	MONT_MAC(c0, c1, c2, a[12], a[26]);+	MONT_MAC(s0, s1, s2, a[13], a[25]);+	MONT_MAC(c0, c1, c2, a[14], a[24]);+	MONT_MAC(s0, s1, s2, a[15], a[23]);+	MONT_MAC(c0, c1, c2, a[16], a[22]);+	MONT_MAC(s0, s1, s2, a[17], a[21]);+	MONT_MAC(c0, c1, c2, a[18], a[20]);+	MONT_MAC(t0, t1, t2, a[19], a[19]);+	MONT_MUL(v0, v1, v2, u[7], m[31]);+	MONT_MUL(w0, w1, w2, u[8], m[30]);+	MONT_MAC(t0, t1, t2, u[9], m[29]);+	MONT_MAC(v0, v1, v2, u[10], m[28]);+	MONT_MAC(w0, w1, w2, u[11], m[27]);+	MONT_MAC(t0, t1, t2, u[12], m[26]);+	MONT_MAC(v0, v1, v2, u[13], m[25]);+	MONT_MAC(w0, w1, w2, u[14], m[24]);+	MONT_MAC(t0, t1, t2, u[15], m[23]);+	MONT_MAC(v0, v1, v2, u[16], m[22]);+	MONT_MAC(w0, w1, w2, u[17], m[21]);+	MONT_MAC(t0, t1, t2, u[18], m[20]);+	MONT_MAC(v0, v1, v2, u[19], m[19]);+	MONT_MAC(w0, w1, w2, u[20], m[18]);+	MONT_MAC(t0, t1, t2, u[21], m[17]);+	MONT_MAC(v0, v1, v2, u[22], m[16]);+	MONT_MAC(w0, w1, w2, u[23], m[15]);+	MONT_MAC(t0, t1, t2, u[24], m[14]);+	MONT_MAC(v0, v1, v2, u[25], m[13]);+	MONT_MAC(w0, w1, w2, u[26], m[12]);+	MONT_MAC(t0, t1, t2, u[27], m[11]);+	MONT_MAC(v0, v1, v2, u[28], m[10]);+	MONT_MAC(w0, w1, w2, u[29], m[9]);+	MONT_MAC(t0, t1, t2, u[30], m[8]);+	MONT_MAC(v0, v1, v2, u[31], m[7]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[6] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 39 */+	MONT_MUL(s0, s1, s2, a[8], a[31]);+	MONT_MUL(c0, c1, c2, a[9], a[30]);+	MONT_MAC(s0, s1, s2, a[10], a[29]);+	MONT_MAC(c0, c1, c2, a[11], a[28]);+	MONT_MAC(s0, s1, s2, a[12], a[27]);+	MONT_MAC(c0, c1, c2, a[13], a[26]);+	MONT_MAC(s0, s1, s2, a[14], a[25]);+	MONT_MAC(c0, c1, c2, a[15], a[24]);+	MONT_MAC(s0, s1, s2, a[16], a[23]);+	MONT_MAC(c0, c1, c2, a[17], a[22]);+	MONT_MAC(s0, s1, s2, a[18], a[21]);+	MONT_MAC(c0, c1, c2, a[19], a[20]);+	MONT_MAC(t0, t1, t2, u[8], m[31]);+	MONT_MUL(v0, v1, v2, u[9], m[30]);+	MONT_MUL(w0, w1, w2, u[10], m[29]);+	MONT_MAC(t0, t1, t2, u[11], m[28]);+	MONT_MAC(v0, v1, v2, u[12], m[27]);+	MONT_MAC(w0, w1, w2, u[13], m[26]);+	MONT_MAC(t0, t1, t2, u[14], m[25]);+	MONT_MAC(v0, v1, v2, u[15], m[24]);+	MONT_MAC(w0, w1, w2, u[16], m[23]);+	MONT_MAC(t0, t1, t2, u[17], m[22]);+	MONT_MAC(v0, v1, v2, u[18], m[21]);+	MONT_MAC(w0, w1, w2, u[19], m[20]);+	MONT_MAC(t0, t1, t2, u[20], m[19]);+	MONT_MAC(v0, v1, v2, u[21], m[18]);+	MONT_MAC(w0, w1, w2, u[22], m[17]);+	MONT_MAC(t0, t1, t2, u[23], m[16]);+	MONT_MAC(v0, v1, v2, u[24], m[15]);+	MONT_MAC(w0, w1, w2, u[25], m[14]);+	MONT_MAC(t0, t1, t2, u[26], m[13]);+	MONT_MAC(v0, v1, v2, u[27], m[12]);+	MONT_MAC(w0, w1, w2, u[28], m[11]);+	MONT_MAC(t0, t1, t2, u[29], m[10]);+	MONT_MAC(v0, v1, v2, u[30], m[9]);+	MONT_MAC(w0, w1, w2, u[31], m[8]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[7] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 40 */+	MONT_MUL(s0, s1, s2, a[9], a[31]);+	MONT_MUL(c0, c1, c2, a[10], a[30]);+	MONT_MAC(s0, s1, s2, a[11], a[29]);+	MONT_MAC(c0, c1, c2, a[12], a[28]);+	MONT_MAC(s0, s1, s2, a[13], a[27]);+	MONT_MAC(c0, c1, c2, a[14], a[26]);+	MONT_MAC(s0, s1, s2, a[15], a[25]);+	MONT_MAC(c0, c1, c2, a[16], a[24]);+	MONT_MAC(s0, s1, s2, a[17], a[23]);+	MONT_MAC(c0, c1, c2, a[18], a[22]);+	MONT_MAC(s0, s1, s2, a[19], a[21]);+	MONT_MAC(t0, t1, t2, a[20], a[20]);+	MONT_MUL(v0, v1, v2, u[9], m[31]);+	MONT_MUL(w0, w1, w2, u[10], m[30]);+	MONT_MAC(t0, t1, t2, u[11], m[29]);+	MONT_MAC(v0, v1, v2, u[12], m[28]);+	MONT_MAC(w0, w1, w2, u[13], m[27]);+	MONT_MAC(t0, t1, t2, u[14], m[26]);+	MONT_MAC(v0, v1, v2, u[15], m[25]);+	MONT_MAC(w0, w1, w2, u[16], m[24]);+	MONT_MAC(t0, t1, t2, u[17], m[23]);+	MONT_MAC(v0, v1, v2, u[18], m[22]);+	MONT_MAC(w0, w1, w2, u[19], m[21]);+	MONT_MAC(t0, t1, t2, u[20], m[20]);+	MONT_MAC(v0, v1, v2, u[21], m[19]);+	MONT_MAC(w0, w1, w2, u[22], m[18]);+	MONT_MAC(t0, t1, t2, u[23], m[17]);+	MONT_MAC(v0, v1, v2, u[24], m[16]);+	MONT_MAC(w0, w1, w2, u[25], m[15]);+	MONT_MAC(t0, t1, t2, u[26], m[14]);+	MONT_MAC(v0, v1, v2, u[27], m[13]);+	MONT_MAC(w0, w1, w2, u[28], m[12]);+	MONT_MAC(t0, t1, t2, u[29], m[11]);+	MONT_MAC(v0, v1, v2, u[30], m[10]);+	MONT_MAC(w0, w1, w2, u[31], m[9]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[8] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 41 */+	MONT_MUL(s0, s1, s2, a[10], a[31]);+	MONT_MUL(c0, c1, c2, a[11], a[30]);+	MONT_MAC(s0, s1, s2, a[12], a[29]);+	MONT_MAC(c0, c1, c2, a[13], a[28]);+	MONT_MAC(s0, s1, s2, a[14], a[27]);+	MONT_MAC(c0, c1, c2, a[15], a[26]);+	MONT_MAC(s0, s1, s2, a[16], a[25]);+	MONT_MAC(c0, c1, c2, a[17], a[24]);+	MONT_MAC(s0, s1, s2, a[18], a[23]);+	MONT_MAC(c0, c1, c2, a[19], a[22]);+	MONT_MAC(s0, s1, s2, a[20], a[21]);+	MONT_MAC(t0, t1, t2, u[10], m[31]);+	MONT_MUL(v0, v1, v2, u[11], m[30]);+	MONT_MUL(w0, w1, w2, u[12], m[29]);+	MONT_MAC(t0, t1, t2, u[13], m[28]);+	MONT_MAC(v0, v1, v2, u[14], m[27]);+	MONT_MAC(w0, w1, w2, u[15], m[26]);+	MONT_MAC(t0, t1, t2, u[16], m[25]);+	MONT_MAC(v0, v1, v2, u[17], m[24]);+	MONT_MAC(w0, w1, w2, u[18], m[23]);+	MONT_MAC(t0, t1, t2, u[19], m[22]);+	MONT_MAC(v0, v1, v2, u[20], m[21]);+	MONT_MAC(w0, w1, w2, u[21], m[20]);+	MONT_MAC(t0, t1, t2, u[22], m[19]);+	MONT_MAC(v0, v1, v2, u[23], m[18]);+	MONT_MAC(w0, w1, w2, u[24], m[17]);+	MONT_MAC(t0, t1, t2, u[25], m[16]);+	MONT_MAC(v0, v1, v2, u[26], m[15]);+	MONT_MAC(w0, w1, w2, u[27], m[14]);+	MONT_MAC(t0, t1, t2, u[28], m[13]);+	MONT_MAC(v0, v1, v2, u[29], m[12]);+	MONT_MAC(w0, w1, w2, u[30], m[11]);+	MONT_MAC(t0, t1, t2, u[31], m[10]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[9] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 42 */+	MONT_MUL(s0, s1, s2, a[11], a[31]);+	MONT_MUL(c0, c1, c2, a[12], a[30]);+	MONT_MAC(s0, s1, s2, a[13], a[29]);+	MONT_MAC(c0, c1, c2, a[14], a[28]);+	MONT_MAC(s0, s1, s2, a[15], a[27]);+	MONT_MAC(c0, c1, c2, a[16], a[26]);+	MONT_MAC(s0, s1, s2, a[17], a[25]);+	MONT_MAC(c0, c1, c2, a[18], a[24]);+	MONT_MAC(s0, s1, s2, a[19], a[23]);+	MONT_MAC(c0, c1, c2, a[20], a[22]);+	MONT_MAC(t0, t1, t2, a[21], a[21]);+	MONT_MUL(v0, v1, v2, u[11], m[31]);+	MONT_MUL(w0, w1, w2, u[12], m[30]);+	MONT_MAC(t0, t1, t2, u[13], m[29]);+	MONT_MAC(v0, v1, v2, u[14], m[28]);+	MONT_MAC(w0, w1, w2, u[15], m[27]);+	MONT_MAC(t0, t1, t2, u[16], m[26]);+	MONT_MAC(v0, v1, v2, u[17], m[25]);+	MONT_MAC(w0, w1, w2, u[18], m[24]);+	MONT_MAC(t0, t1, t2, u[19], m[23]);+	MONT_MAC(v0, v1, v2, u[20], m[22]);+	MONT_MAC(w0, w1, w2, u[21], m[21]);+	MONT_MAC(t0, t1, t2, u[22], m[20]);+	MONT_MAC(v0, v1, v2, u[23], m[19]);+	MONT_MAC(w0, w1, w2, u[24], m[18]);+	MONT_MAC(t0, t1, t2, u[25], m[17]);+	MONT_MAC(v0, v1, v2, u[26], m[16]);+	MONT_MAC(w0, w1, w2, u[27], m[15]);+	MONT_MAC(t0, t1, t2, u[28], m[14]);+	MONT_MAC(v0, v1, v2, u[29], m[13]);+	MONT_MAC(w0, w1, w2, u[30], m[12]);+	MONT_MAC(t0, t1, t2, u[31], m[11]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[10] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 43 */+	MONT_MUL(s0, s1, s2, a[12], a[31]);+	MONT_MUL(c0, c1, c2, a[13], a[30]);+	MONT_MAC(s0, s1, s2, a[14], a[29]);+	MONT_MAC(c0, c1, c2, a[15], a[28]);+	MONT_MAC(s0, s1, s2, a[16], a[27]);+	MONT_MAC(c0, c1, c2, a[17], a[26]);+	MONT_MAC(s0, s1, s2, a[18], a[25]);+	MONT_MAC(c0, c1, c2, a[19], a[24]);+	MONT_MAC(s0, s1, s2, a[20], a[23]);+	MONT_MAC(c0, c1, c2, a[21], a[22]);+	MONT_MAC(t0, t1, t2, u[12], m[31]);+	MONT_MUL(v0, v1, v2, u[13], m[30]);+	MONT_MUL(w0, w1, w2, u[14], m[29]);+	MONT_MAC(t0, t1, t2, u[15], m[28]);+	MONT_MAC(v0, v1, v2, u[16], m[27]);+	MONT_MAC(w0, w1, w2, u[17], m[26]);+	MONT_MAC(t0, t1, t2, u[18], m[25]);+	MONT_MAC(v0, v1, v2, u[19], m[24]);+	MONT_MAC(w0, w1, w2, u[20], m[23]);+	MONT_MAC(t0, t1, t2, u[21], m[22]);+	MONT_MAC(v0, v1, v2, u[22], m[21]);+	MONT_MAC(w0, w1, w2, u[23], m[20]);+	MONT_MAC(t0, t1, t2, u[24], m[19]);+	MONT_MAC(v0, v1, v2, u[25], m[18]);+	MONT_MAC(w0, w1, w2, u[26], m[17]);+	MONT_MAC(t0, t1, t2, u[27], m[16]);+	MONT_MAC(v0, v1, v2, u[28], m[15]);+	MONT_MAC(w0, w1, w2, u[29], m[14]);+	MONT_MAC(t0, t1, t2, u[30], m[13]);+	MONT_MAC(v0, v1, v2, u[31], m[12]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[11] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 44 */+	MONT_MUL(s0, s1, s2, a[13], a[31]);+	MONT_MUL(c0, c1, c2, a[14], a[30]);+	MONT_MAC(s0, s1, s2, a[15], a[29]);+	MONT_MAC(c0, c1, c2, a[16], a[28]);+	MONT_MAC(s0, s1, s2, a[17], a[27]);+	MONT_MAC(c0, c1, c2, a[18], a[26]);+	MONT_MAC(s0, s1, s2, a[19], a[25]);+	MONT_MAC(c0, c1, c2, a[20], a[24]);+	MONT_MAC(s0, s1, s2, a[21], a[23]);+	MONT_MAC(t0, t1, t2, a[22], a[22]);+	MONT_MUL(v0, v1, v2, u[13], m[31]);+	MONT_MUL(w0, w1, w2, u[14], m[30]);+	MONT_MAC(t0, t1, t2, u[15], m[29]);+	MONT_MAC(v0, v1, v2, u[16], m[28]);+	MONT_MAC(w0, w1, w2, u[17], m[27]);+	MONT_MAC(t0, t1, t2, u[18], m[26]);+	MONT_MAC(v0, v1, v2, u[19], m[25]);+	MONT_MAC(w0, w1, w2, u[20], m[24]);+	MONT_MAC(t0, t1, t2, u[21], m[23]);+	MONT_MAC(v0, v1, v2, u[22], m[22]);+	MONT_MAC(w0, w1, w2, u[23], m[21]);+	MONT_MAC(t0, t1, t2, u[24], m[20]);+	MONT_MAC(v0, v1, v2, u[25], m[19]);+	MONT_MAC(w0, w1, w2, u[26], m[18]);+	MONT_MAC(t0, t1, t2, u[27], m[17]);+	MONT_MAC(v0, v1, v2, u[28], m[16]);+	MONT_MAC(w0, w1, w2, u[29], m[15]);+	MONT_MAC(t0, t1, t2, u[30], m[14]);+	MONT_MAC(v0, v1, v2, u[31], m[13]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[12] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 45 */+	MONT_MUL(s0, s1, s2, a[14], a[31]);+	MONT_MUL(c0, c1, c2, a[15], a[30]);+	MONT_MAC(s0, s1, s2, a[16], a[29]);+	MONT_MAC(c0, c1, c2, a[17], a[28]);+	MONT_MAC(s0, s1, s2, a[18], a[27]);+	MONT_MAC(c0, c1, c2, a[19], a[26]);+	MONT_MAC(s0, s1, s2, a[20], a[25]);+	MONT_MAC(c0, c1, c2, a[21], a[24]);+	MONT_MAC(s0, s1, s2, a[22], a[23]);+	MONT_MAC(t0, t1, t2, u[14], m[31]);+	MONT_MUL(v0, v1, v2, u[15], m[30]);+	MONT_MUL(w0, w1, w2, u[16], m[29]);+	MONT_MAC(t0, t1, t2, u[17], m[28]);+	MONT_MAC(v0, v1, v2, u[18], m[27]);+	MONT_MAC(w0, w1, w2, u[19], m[26]);+	MONT_MAC(t0, t1, t2, u[20], m[25]);+	MONT_MAC(v0, v1, v2, u[21], m[24]);+	MONT_MAC(w0, w1, w2, u[22], m[23]);+	MONT_MAC(t0, t1, t2, u[23], m[22]);+	MONT_MAC(v0, v1, v2, u[24], m[21]);+	MONT_MAC(w0, w1, w2, u[25], m[20]);+	MONT_MAC(t0, t1, t2, u[26], m[19]);+	MONT_MAC(v0, v1, v2, u[27], m[18]);+	MONT_MAC(w0, w1, w2, u[28], m[17]);+	MONT_MAC(t0, t1, t2, u[29], m[16]);+	MONT_MAC(v0, v1, v2, u[30], m[15]);+	MONT_MAC(w0, w1, w2, u[31], m[14]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[13] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 46 */+	MONT_MUL(s0, s1, s2, a[15], a[31]);+	MONT_MUL(c0, c1, c2, a[16], a[30]);+	MONT_MAC(s0, s1, s2, a[17], a[29]);+	MONT_MAC(c0, c1, c2, a[18], a[28]);+	MONT_MAC(s0, s1, s2, a[19], a[27]);+	MONT_MAC(c0, c1, c2, a[20], a[26]);+	MONT_MAC(s0, s1, s2, a[21], a[25]);+	MONT_MAC(c0, c1, c2, a[22], a[24]);+	MONT_MAC(t0, t1, t2, a[23], a[23]);+	MONT_MUL(v0, v1, v2, u[15], m[31]);+	MONT_MUL(w0, w1, w2, u[16], m[30]);+	MONT_MAC(t0, t1, t2, u[17], m[29]);+	MONT_MAC(v0, v1, v2, u[18], m[28]);+	MONT_MAC(w0, w1, w2, u[19], m[27]);+	MONT_MAC(t0, t1, t2, u[20], m[26]);+	MONT_MAC(v0, v1, v2, u[21], m[25]);+	MONT_MAC(w0, w1, w2, u[22], m[24]);+	MONT_MAC(t0, t1, t2, u[23], m[23]);+	MONT_MAC(v0, v1, v2, u[24], m[22]);+	MONT_MAC(w0, w1, w2, u[25], m[21]);+	MONT_MAC(t0, t1, t2, u[26], m[20]);+	MONT_MAC(v0, v1, v2, u[27], m[19]);+	MONT_MAC(w0, w1, w2, u[28], m[18]);+	MONT_MAC(t0, t1, t2, u[29], m[17]);+	MONT_MAC(v0, v1, v2, u[30], m[16]);+	MONT_MAC(w0, w1, w2, u[31], m[15]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[14] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 47 */+	MONT_MUL(s0, s1, s2, a[16], a[31]);+	MONT_MUL(c0, c1, c2, a[17], a[30]);+	MONT_MAC(s0, s1, s2, a[18], a[29]);+	MONT_MAC(c0, c1, c2, a[19], a[28]);+	MONT_MAC(s0, s1, s2, a[20], a[27]);+	MONT_MAC(c0, c1, c2, a[21], a[26]);+	MONT_MAC(s0, s1, s2, a[22], a[25]);+	MONT_MAC(c0, c1, c2, a[23], a[24]);+	MONT_MAC(t0, t1, t2, u[16], m[31]);+	MONT_MUL(v0, v1, v2, u[17], m[30]);+	MONT_MUL(w0, w1, w2, u[18], m[29]);+	MONT_MAC(t0, t1, t2, u[19], m[28]);+	MONT_MAC(v0, v1, v2, u[20], m[27]);+	MONT_MAC(w0, w1, w2, u[21], m[26]);+	MONT_MAC(t0, t1, t2, u[22], m[25]);+	MONT_MAC(v0, v1, v2, u[23], m[24]);+	MONT_MAC(w0, w1, w2, u[24], m[23]);+	MONT_MAC(t0, t1, t2, u[25], m[22]);+	MONT_MAC(v0, v1, v2, u[26], m[21]);+	MONT_MAC(w0, w1, w2, u[27], m[20]);+	MONT_MAC(t0, t1, t2, u[28], m[19]);+	MONT_MAC(v0, v1, v2, u[29], m[18]);+	MONT_MAC(w0, w1, w2, u[30], m[17]);+	MONT_MAC(t0, t1, t2, u[31], m[16]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[15] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 48 */+	MONT_MUL(s0, s1, s2, a[17], a[31]);+	MONT_MUL(c0, c1, c2, a[18], a[30]);+	MONT_MAC(s0, s1, s2, a[19], a[29]);+	MONT_MAC(c0, c1, c2, a[20], a[28]);+	MONT_MAC(s0, s1, s2, a[21], a[27]);+	MONT_MAC(c0, c1, c2, a[22], a[26]);+	MONT_MAC(s0, s1, s2, a[23], a[25]);+	MONT_MAC(t0, t1, t2, a[24], a[24]);+	MONT_MUL(v0, v1, v2, u[17], m[31]);+	MONT_MUL(w0, w1, w2, u[18], m[30]);+	MONT_MAC(t0, t1, t2, u[19], m[29]);+	MONT_MAC(v0, v1, v2, u[20], m[28]);+	MONT_MAC(w0, w1, w2, u[21], m[27]);+	MONT_MAC(t0, t1, t2, u[22], m[26]);+	MONT_MAC(v0, v1, v2, u[23], m[25]);+	MONT_MAC(w0, w1, w2, u[24], m[24]);+	MONT_MAC(t0, t1, t2, u[25], m[23]);+	MONT_MAC(v0, v1, v2, u[26], m[22]);+	MONT_MAC(w0, w1, w2, u[27], m[21]);+	MONT_MAC(t0, t1, t2, u[28], m[20]);+	MONT_MAC(v0, v1, v2, u[29], m[19]);+	MONT_MAC(w0, w1, w2, u[30], m[18]);+	MONT_MAC(t0, t1, t2, u[31], m[17]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[16] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 49 */+	MONT_MUL(s0, s1, s2, a[18], a[31]);+	MONT_MUL(c0, c1, c2, a[19], a[30]);+	MONT_MAC(s0, s1, s2, a[20], a[29]);+	MONT_MAC(c0, c1, c2, a[21], a[28]);+	MONT_MAC(s0, s1, s2, a[22], a[27]);+	MONT_MAC(c0, c1, c2, a[23], a[26]);+	MONT_MAC(s0, s1, s2, a[24], a[25]);+	MONT_MAC(t0, t1, t2, u[18], m[31]);+	MONT_MUL(v0, v1, v2, u[19], m[30]);+	MONT_MUL(w0, w1, w2, u[20], m[29]);+	MONT_MAC(t0, t1, t2, u[21], m[28]);+	MONT_MAC(v0, v1, v2, u[22], m[27]);+	MONT_MAC(w0, w1, w2, u[23], m[26]);+	MONT_MAC(t0, t1, t2, u[24], m[25]);+	MONT_MAC(v0, v1, v2, u[25], m[24]);+	MONT_MAC(w0, w1, w2, u[26], m[23]);+	MONT_MAC(t0, t1, t2, u[27], m[22]);+	MONT_MAC(v0, v1, v2, u[28], m[21]);+	MONT_MAC(w0, w1, w2, u[29], m[20]);+	MONT_MAC(t0, t1, t2, u[30], m[19]);+	MONT_MAC(v0, v1, v2, u[31], m[18]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[17] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 50 */+	MONT_MUL(s0, s1, s2, a[19], a[31]);+	MONT_MUL(c0, c1, c2, a[20], a[30]);+	MONT_MAC(s0, s1, s2, a[21], a[29]);+	MONT_MAC(c0, c1, c2, a[22], a[28]);+	MONT_MAC(s0, s1, s2, a[23], a[27]);+	MONT_MAC(c0, c1, c2, a[24], a[26]);+	MONT_MAC(t0, t1, t2, a[25], a[25]);+	MONT_MUL(v0, v1, v2, u[19], m[31]);+	MONT_MUL(w0, w1, w2, u[20], m[30]);+	MONT_MAC(t0, t1, t2, u[21], m[29]);+	MONT_MAC(v0, v1, v2, u[22], m[28]);+	MONT_MAC(w0, w1, w2, u[23], m[27]);+	MONT_MAC(t0, t1, t2, u[24], m[26]);+	MONT_MAC(v0, v1, v2, u[25], m[25]);+	MONT_MAC(w0, w1, w2, u[26], m[24]);+	MONT_MAC(t0, t1, t2, u[27], m[23]);+	MONT_MAC(v0, v1, v2, u[28], m[22]);+	MONT_MAC(w0, w1, w2, u[29], m[21]);+	MONT_MAC(t0, t1, t2, u[30], m[20]);+	MONT_MAC(v0, v1, v2, u[31], m[19]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[18] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 51 */+	MONT_MUL(s0, s1, s2, a[20], a[31]);+	MONT_MUL(c0, c1, c2, a[21], a[30]);+	MONT_MAC(s0, s1, s2, a[22], a[29]);+	MONT_MAC(c0, c1, c2, a[23], a[28]);+	MONT_MAC(s0, s1, s2, a[24], a[27]);+	MONT_MAC(c0, c1, c2, a[25], a[26]);+	MONT_MAC(t0, t1, t2, u[20], m[31]);+	MONT_MUL(v0, v1, v2, u[21], m[30]);+	MONT_MUL(w0, w1, w2, u[22], m[29]);+	MONT_MAC(t0, t1, t2, u[23], m[28]);+	MONT_MAC(v0, v1, v2, u[24], m[27]);+	MONT_MAC(w0, w1, w2, u[25], m[26]);+	MONT_MAC(t0, t1, t2, u[26], m[25]);+	MONT_MAC(v0, v1, v2, u[27], m[24]);+	MONT_MAC(w0, w1, w2, u[28], m[23]);+	MONT_MAC(t0, t1, t2, u[29], m[22]);+	MONT_MAC(v0, v1, v2, u[30], m[21]);+	MONT_MAC(w0, w1, w2, u[31], m[20]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[19] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 52 */+	MONT_MUL(s0, s1, s2, a[21], a[31]);+	MONT_MUL(c0, c1, c2, a[22], a[30]);+	MONT_MAC(s0, s1, s2, a[23], a[29]);+	MONT_MAC(c0, c1, c2, a[24], a[28]);+	MONT_MAC(s0, s1, s2, a[25], a[27]);+	MONT_MAC(t0, t1, t2, a[26], a[26]);+	MONT_MUL(v0, v1, v2, u[21], m[31]);+	MONT_MUL(w0, w1, w2, u[22], m[30]);+	MONT_MAC(t0, t1, t2, u[23], m[29]);+	MONT_MAC(v0, v1, v2, u[24], m[28]);+	MONT_MAC(w0, w1, w2, u[25], m[27]);+	MONT_MAC(t0, t1, t2, u[26], m[26]);+	MONT_MAC(v0, v1, v2, u[27], m[25]);+	MONT_MAC(w0, w1, w2, u[28], m[24]);+	MONT_MAC(t0, t1, t2, u[29], m[23]);+	MONT_MAC(v0, v1, v2, u[30], m[22]);+	MONT_MAC(w0, w1, w2, u[31], m[21]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[20] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 53 */+	MONT_MUL(s0, s1, s2, a[22], a[31]);+	MONT_MUL(c0, c1, c2, a[23], a[30]);+	MONT_MAC(s0, s1, s2, a[24], a[29]);+	MONT_MAC(c0, c1, c2, a[25], a[28]);+	MONT_MAC(s0, s1, s2, a[26], a[27]);+	MONT_MAC(t0, t1, t2, u[22], m[31]);+	MONT_MUL(v0, v1, v2, u[23], m[30]);+	MONT_MUL(w0, w1, w2, u[24], m[29]);+	MONT_MAC(t0, t1, t2, u[25], m[28]);+	MONT_MAC(v0, v1, v2, u[26], m[27]);+	MONT_MAC(w0, w1, w2, u[27], m[26]);+	MONT_MAC(t0, t1, t2, u[28], m[25]);+	MONT_MAC(v0, v1, v2, u[29], m[24]);+	MONT_MAC(w0, w1, w2, u[30], m[23]);+	MONT_MAC(t0, t1, t2, u[31], m[22]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[21] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 54 */+	MONT_MUL(s0, s1, s2, a[23], a[31]);+	MONT_MUL(c0, c1, c2, a[24], a[30]);+	MONT_MAC(s0, s1, s2, a[25], a[29]);+	MONT_MAC(c0, c1, c2, a[26], a[28]);+	MONT_MAC(t0, t1, t2, a[27], a[27]);+	MONT_MUL(v0, v1, v2, u[23], m[31]);+	MONT_MUL(w0, w1, w2, u[24], m[30]);+	MONT_MAC(t0, t1, t2, u[25], m[29]);+	MONT_MAC(v0, v1, v2, u[26], m[28]);+	MONT_MAC(w0, w1, w2, u[27], m[27]);+	MONT_MAC(t0, t1, t2, u[28], m[26]);+	MONT_MAC(v0, v1, v2, u[29], m[25]);+	MONT_MAC(w0, w1, w2, u[30], m[24]);+	MONT_MAC(t0, t1, t2, u[31], m[23]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[22] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 55 */+	MONT_MUL(s0, s1, s2, a[24], a[31]);+	MONT_MUL(c0, c1, c2, a[25], a[30]);+	MONT_MAC(s0, s1, s2, a[26], a[29]);+	MONT_MAC(c0, c1, c2, a[27], a[28]);+	MONT_MAC(t0, t1, t2, u[24], m[31]);+	MONT_MUL(v0, v1, v2, u[25], m[30]);+	MONT_MUL(w0, w1, w2, u[26], m[29]);+	MONT_MAC(t0, t1, t2, u[27], m[28]);+	MONT_MAC(v0, v1, v2, u[28], m[27]);+	MONT_MAC(w0, w1, w2, u[29], m[26]);+	MONT_MAC(t0, t1, t2, u[30], m[25]);+	MONT_MAC(v0, v1, v2, u[31], m[24]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[23] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 56 */+	MONT_MUL(s0, s1, s2, a[25], a[31]);+	MONT_MUL(c0, c1, c2, a[26], a[30]);+	MONT_MAC(s0, s1, s2, a[27], a[29]);+	MONT_MAC(t0, t1, t2, a[28], a[28]);+	MONT_MUL(v0, v1, v2, u[25], m[31]);+	MONT_MUL(w0, w1, w2, u[26], m[30]);+	MONT_MAC(t0, t1, t2, u[27], m[29]);+	MONT_MAC(v0, v1, v2, u[28], m[28]);+	MONT_MAC(w0, w1, w2, u[29], m[27]);+	MONT_MAC(t0, t1, t2, u[30], m[26]);+	MONT_MAC(v0, v1, v2, u[31], m[25]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[24] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 57 */+	MONT_MUL(s0, s1, s2, a[26], a[31]);+	MONT_MUL(c0, c1, c2, a[27], a[30]);+	MONT_MAC(s0, s1, s2, a[28], a[29]);+	MONT_MAC(t0, t1, t2, u[26], m[31]);+	MONT_MUL(v0, v1, v2, u[27], m[30]);+	MONT_MUL(w0, w1, w2, u[28], m[29]);+	MONT_MAC(t0, t1, t2, u[29], m[28]);+	MONT_MAC(v0, v1, v2, u[30], m[27]);+	MONT_MAC(w0, w1, w2, u[31], m[26]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[25] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 58 */+	MONT_MUL(s0, s1, s2, a[27], a[31]);+	MONT_MUL(c0, c1, c2, a[28], a[30]);+	MONT_MAC(t0, t1, t2, a[29], a[29]);+	MONT_MUL(v0, v1, v2, u[27], m[31]);+	MONT_MUL(w0, w1, w2, u[28], m[30]);+	MONT_MAC(t0, t1, t2, u[29], m[29]);+	MONT_MAC(v0, v1, v2, u[30], m[28]);+	MONT_MAC(w0, w1, w2, u[31], m[27]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[26] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 59 */+	MONT_MUL(s0, s1, s2, a[28], a[31]);+	MONT_MUL(c0, c1, c2, a[29], a[30]);+	MONT_MAC(t0, t1, t2, u[28], m[31]);+	MONT_MUL(v0, v1, v2, u[29], m[30]);+	MONT_MUL(w0, w1, w2, u[30], m[29]);+	MONT_MAC(t0, t1, t2, u[31], m[28]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[27] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 60 */+	MONT_MUL(s0, s1, s2, a[29], a[31]);+	MONT_MAC(t0, t1, t2, a[30], a[30]);+	MONT_MUL(v0, v1, v2, u[29], m[31]);+	MONT_MUL(w0, w1, w2, u[30], m[30]);+	MONT_MAC(t0, t1, t2, u[31], m[29]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[28] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 61 */+	MONT_MUL(s0, s1, s2, a[30], a[31]);+	MONT_MAC(t0, t1, t2, u[30], m[31]);+	MONT_MUL(v0, v1, v2, u[31], m[30]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[29] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 62 */+	MONT_MAC(t0, t1, t2, a[31], a[31]);+	MONT_MUL(v0, v1, v2, u[31], m[31]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	z[30] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	z[31] = t0;++	/* under 2m, with t1 the bit above n limbs: at most one subtraction */+	borrow = sub_n(r, z, m, 32);+	take = t1 | (borrow ^ 1);+	select_n(r, r, z, take & 1, 32);+}++/* r = a * a * R^-1 mod m, for a under m; t is 2n limbs of scratch */+static void crypton_mont_sqr48(limb_t *r, const limb_t *a,+                               const limb_t *m, limb_t n0, limb_t *t)+{+	limb_t *u = t, *z = t + 48;+	limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take;+	limb_t s0, s1, s2, c0, c1, c2;++	/* column 0 */+	MONT_MAC(t0, t1, t2, a[0], a[0]);+	u[0] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[0], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 1 */+	MONT_MUL(s0, s1, s2, a[0], a[1]);+	MONT_MAC(t0, t1, t2, u[0], m[1]);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[1] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[1], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 2 */+	MONT_MUL(s0, s1, s2, a[0], a[2]);+	MONT_MAC(t0, t1, t2, a[1], a[1]);+	MONT_MUL(v0, v1, v2, u[0], m[2]);+	MONT_MUL(w0, w1, w2, u[1], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[2] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[2], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 3 */+	MONT_MUL(s0, s1, s2, a[0], a[3]);+	MONT_MUL(c0, c1, c2, a[1], a[2]);+	MONT_MAC(t0, t1, t2, u[0], m[3]);+	MONT_MUL(v0, v1, v2, u[1], m[2]);+	MONT_MUL(w0, w1, w2, u[2], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[3] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[3], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 4 */+	MONT_MUL(s0, s1, s2, a[0], a[4]);+	MONT_MUL(c0, c1, c2, a[1], a[3]);+	MONT_MAC(t0, t1, t2, a[2], a[2]);+	MONT_MUL(v0, v1, v2, u[0], m[4]);+	MONT_MUL(w0, w1, w2, u[1], m[3]);+	MONT_MAC(t0, t1, t2, u[2], m[2]);+	MONT_MAC(v0, v1, v2, u[3], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[4] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[4], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 5 */+	MONT_MUL(s0, s1, s2, a[0], a[5]);+	MONT_MUL(c0, c1, c2, a[1], a[4]);+	MONT_MAC(s0, s1, s2, a[2], a[3]);+	MONT_MAC(t0, t1, t2, u[0], m[5]);+	MONT_MUL(v0, v1, v2, u[1], m[4]);+	MONT_MUL(w0, w1, w2, u[2], m[3]);+	MONT_MAC(t0, t1, t2, u[3], m[2]);+	MONT_MAC(v0, v1, v2, u[4], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[5] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[5], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 6 */+	MONT_MUL(s0, s1, s2, a[0], a[6]);+	MONT_MUL(c0, c1, c2, a[1], a[5]);+	MONT_MAC(s0, s1, s2, a[2], a[4]);+	MONT_MAC(t0, t1, t2, a[3], a[3]);+	MONT_MUL(v0, v1, v2, u[0], m[6]);+	MONT_MUL(w0, w1, w2, u[1], m[5]);+	MONT_MAC(t0, t1, t2, u[2], m[4]);+	MONT_MAC(v0, v1, v2, u[3], m[3]);+	MONT_MAC(w0, w1, w2, u[4], m[2]);+	MONT_MAC(t0, t1, t2, u[5], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[6] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[6], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 7 */+	MONT_MUL(s0, s1, s2, a[0], a[7]);+	MONT_MUL(c0, c1, c2, a[1], a[6]);+	MONT_MAC(s0, s1, s2, a[2], a[5]);+	MONT_MAC(c0, c1, c2, a[3], a[4]);+	MONT_MAC(t0, t1, t2, u[0], m[7]);+	MONT_MUL(v0, v1, v2, u[1], m[6]);+	MONT_MUL(w0, w1, w2, u[2], m[5]);+	MONT_MAC(t0, t1, t2, u[3], m[4]);+	MONT_MAC(v0, v1, v2, u[4], m[3]);+	MONT_MAC(w0, w1, w2, u[5], m[2]);+	MONT_MAC(t0, t1, t2, u[6], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[7] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[7], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 8 */+	MONT_MUL(s0, s1, s2, a[0], a[8]);+	MONT_MUL(c0, c1, c2, a[1], a[7]);+	MONT_MAC(s0, s1, s2, a[2], a[6]);+	MONT_MAC(c0, c1, c2, a[3], a[5]);+	MONT_MAC(t0, t1, t2, a[4], a[4]);+	MONT_MUL(v0, v1, v2, u[0], m[8]);+	MONT_MUL(w0, w1, w2, u[1], m[7]);+	MONT_MAC(t0, t1, t2, u[2], m[6]);+	MONT_MAC(v0, v1, v2, u[3], m[5]);+	MONT_MAC(w0, w1, w2, u[4], m[4]);+	MONT_MAC(t0, t1, t2, u[5], m[3]);+	MONT_MAC(v0, v1, v2, u[6], m[2]);+	MONT_MAC(w0, w1, w2, u[7], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[8] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[8], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 9 */+	MONT_MUL(s0, s1, s2, a[0], a[9]);+	MONT_MUL(c0, c1, c2, a[1], a[8]);+	MONT_MAC(s0, s1, s2, a[2], a[7]);+	MONT_MAC(c0, c1, c2, a[3], a[6]);+	MONT_MAC(s0, s1, s2, a[4], a[5]);+	MONT_MAC(t0, t1, t2, u[0], m[9]);+	MONT_MUL(v0, v1, v2, u[1], m[8]);+	MONT_MUL(w0, w1, w2, u[2], m[7]);+	MONT_MAC(t0, t1, t2, u[3], m[6]);+	MONT_MAC(v0, v1, v2, u[4], m[5]);+	MONT_MAC(w0, w1, w2, u[5], m[4]);+	MONT_MAC(t0, t1, t2, u[6], m[3]);+	MONT_MAC(v0, v1, v2, u[7], m[2]);+	MONT_MAC(w0, w1, w2, u[8], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[9] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[9], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 10 */+	MONT_MUL(s0, s1, s2, a[0], a[10]);+	MONT_MUL(c0, c1, c2, a[1], a[9]);+	MONT_MAC(s0, s1, s2, a[2], a[8]);+	MONT_MAC(c0, c1, c2, a[3], a[7]);+	MONT_MAC(s0, s1, s2, a[4], a[6]);+	MONT_MAC(t0, t1, t2, a[5], a[5]);+	MONT_MUL(v0, v1, v2, u[0], m[10]);+	MONT_MUL(w0, w1, w2, u[1], m[9]);+	MONT_MAC(t0, t1, t2, u[2], m[8]);+	MONT_MAC(v0, v1, v2, u[3], m[7]);+	MONT_MAC(w0, w1, w2, u[4], m[6]);+	MONT_MAC(t0, t1, t2, u[5], m[5]);+	MONT_MAC(v0, v1, v2, u[6], m[4]);+	MONT_MAC(w0, w1, w2, u[7], m[3]);+	MONT_MAC(t0, t1, t2, u[8], m[2]);+	MONT_MAC(v0, v1, v2, u[9], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[10] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[10], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 11 */+	MONT_MUL(s0, s1, s2, a[0], a[11]);+	MONT_MUL(c0, c1, c2, a[1], a[10]);+	MONT_MAC(s0, s1, s2, a[2], a[9]);+	MONT_MAC(c0, c1, c2, a[3], a[8]);+	MONT_MAC(s0, s1, s2, a[4], a[7]);+	MONT_MAC(c0, c1, c2, a[5], a[6]);+	MONT_MAC(t0, t1, t2, u[0], m[11]);+	MONT_MUL(v0, v1, v2, u[1], m[10]);+	MONT_MUL(w0, w1, w2, u[2], m[9]);+	MONT_MAC(t0, t1, t2, u[3], m[8]);+	MONT_MAC(v0, v1, v2, u[4], m[7]);+	MONT_MAC(w0, w1, w2, u[5], m[6]);+	MONT_MAC(t0, t1, t2, u[6], m[5]);+	MONT_MAC(v0, v1, v2, u[7], m[4]);+	MONT_MAC(w0, w1, w2, u[8], m[3]);+	MONT_MAC(t0, t1, t2, u[9], m[2]);+	MONT_MAC(v0, v1, v2, u[10], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[11] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[11], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 12 */+	MONT_MUL(s0, s1, s2, a[0], a[12]);+	MONT_MUL(c0, c1, c2, a[1], a[11]);+	MONT_MAC(s0, s1, s2, a[2], a[10]);+	MONT_MAC(c0, c1, c2, a[3], a[9]);+	MONT_MAC(s0, s1, s2, a[4], a[8]);+	MONT_MAC(c0, c1, c2, a[5], a[7]);+	MONT_MAC(t0, t1, t2, a[6], a[6]);+	MONT_MUL(v0, v1, v2, u[0], m[12]);+	MONT_MUL(w0, w1, w2, u[1], m[11]);+	MONT_MAC(t0, t1, t2, u[2], m[10]);+	MONT_MAC(v0, v1, v2, u[3], m[9]);+	MONT_MAC(w0, w1, w2, u[4], m[8]);+	MONT_MAC(t0, t1, t2, u[5], m[7]);+	MONT_MAC(v0, v1, v2, u[6], m[6]);+	MONT_MAC(w0, w1, w2, u[7], m[5]);+	MONT_MAC(t0, t1, t2, u[8], m[4]);+	MONT_MAC(v0, v1, v2, u[9], m[3]);+	MONT_MAC(w0, w1, w2, u[10], m[2]);+	MONT_MAC(t0, t1, t2, u[11], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[12] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[12], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 13 */+	MONT_MUL(s0, s1, s2, a[0], a[13]);+	MONT_MUL(c0, c1, c2, a[1], a[12]);+	MONT_MAC(s0, s1, s2, a[2], a[11]);+	MONT_MAC(c0, c1, c2, a[3], a[10]);+	MONT_MAC(s0, s1, s2, a[4], a[9]);+	MONT_MAC(c0, c1, c2, a[5], a[8]);+	MONT_MAC(s0, s1, s2, a[6], a[7]);+	MONT_MAC(t0, t1, t2, u[0], m[13]);+	MONT_MUL(v0, v1, v2, u[1], m[12]);+	MONT_MUL(w0, w1, w2, u[2], m[11]);+	MONT_MAC(t0, t1, t2, u[3], m[10]);+	MONT_MAC(v0, v1, v2, u[4], m[9]);+	MONT_MAC(w0, w1, w2, u[5], m[8]);+	MONT_MAC(t0, t1, t2, u[6], m[7]);+	MONT_MAC(v0, v1, v2, u[7], m[6]);+	MONT_MAC(w0, w1, w2, u[8], m[5]);+	MONT_MAC(t0, t1, t2, u[9], m[4]);+	MONT_MAC(v0, v1, v2, u[10], m[3]);+	MONT_MAC(w0, w1, w2, u[11], m[2]);+	MONT_MAC(t0, t1, t2, u[12], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[13] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[13], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 14 */+	MONT_MUL(s0, s1, s2, a[0], a[14]);+	MONT_MUL(c0, c1, c2, a[1], a[13]);+	MONT_MAC(s0, s1, s2, a[2], a[12]);+	MONT_MAC(c0, c1, c2, a[3], a[11]);+	MONT_MAC(s0, s1, s2, a[4], a[10]);+	MONT_MAC(c0, c1, c2, a[5], a[9]);+	MONT_MAC(s0, s1, s2, a[6], a[8]);+	MONT_MAC(t0, t1, t2, a[7], a[7]);+	MONT_MUL(v0, v1, v2, u[0], m[14]);+	MONT_MUL(w0, w1, w2, u[1], m[13]);+	MONT_MAC(t0, t1, t2, u[2], m[12]);+	MONT_MAC(v0, v1, v2, u[3], m[11]);+	MONT_MAC(w0, w1, w2, u[4], m[10]);+	MONT_MAC(t0, t1, t2, u[5], m[9]);+	MONT_MAC(v0, v1, v2, u[6], m[8]);+	MONT_MAC(w0, w1, w2, u[7], m[7]);+	MONT_MAC(t0, t1, t2, u[8], m[6]);+	MONT_MAC(v0, v1, v2, u[9], m[5]);+	MONT_MAC(w0, w1, w2, u[10], m[4]);+	MONT_MAC(t0, t1, t2, u[11], m[3]);+	MONT_MAC(v0, v1, v2, u[12], m[2]);+	MONT_MAC(w0, w1, w2, u[13], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[14] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[14], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 15 */+	MONT_MUL(s0, s1, s2, a[0], a[15]);+	MONT_MUL(c0, c1, c2, a[1], a[14]);+	MONT_MAC(s0, s1, s2, a[2], a[13]);+	MONT_MAC(c0, c1, c2, a[3], a[12]);+	MONT_MAC(s0, s1, s2, a[4], a[11]);+	MONT_MAC(c0, c1, c2, a[5], a[10]);+	MONT_MAC(s0, s1, s2, a[6], a[9]);+	MONT_MAC(c0, c1, c2, a[7], a[8]);+	MONT_MAC(t0, t1, t2, u[0], m[15]);+	MONT_MUL(v0, v1, v2, u[1], m[14]);+	MONT_MUL(w0, w1, w2, u[2], m[13]);+	MONT_MAC(t0, t1, t2, u[3], m[12]);+	MONT_MAC(v0, v1, v2, u[4], m[11]);+	MONT_MAC(w0, w1, w2, u[5], m[10]);+	MONT_MAC(t0, t1, t2, u[6], m[9]);+	MONT_MAC(v0, v1, v2, u[7], m[8]);+	MONT_MAC(w0, w1, w2, u[8], m[7]);+	MONT_MAC(t0, t1, t2, u[9], m[6]);+	MONT_MAC(v0, v1, v2, u[10], m[5]);+	MONT_MAC(w0, w1, w2, u[11], m[4]);+	MONT_MAC(t0, t1, t2, u[12], m[3]);+	MONT_MAC(v0, v1, v2, u[13], m[2]);+	MONT_MAC(w0, w1, w2, u[14], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[15] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[15], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 16 */+	MONT_MUL(s0, s1, s2, a[0], a[16]);+	MONT_MUL(c0, c1, c2, a[1], a[15]);+	MONT_MAC(s0, s1, s2, a[2], a[14]);+	MONT_MAC(c0, c1, c2, a[3], a[13]);+	MONT_MAC(s0, s1, s2, a[4], a[12]);+	MONT_MAC(c0, c1, c2, a[5], a[11]);+	MONT_MAC(s0, s1, s2, a[6], a[10]);+	MONT_MAC(c0, c1, c2, a[7], a[9]);+	MONT_MAC(t0, t1, t2, a[8], a[8]);+	MONT_MUL(v0, v1, v2, u[0], m[16]);+	MONT_MUL(w0, w1, w2, u[1], m[15]);+	MONT_MAC(t0, t1, t2, u[2], m[14]);+	MONT_MAC(v0, v1, v2, u[3], m[13]);+	MONT_MAC(w0, w1, w2, u[4], m[12]);+	MONT_MAC(t0, t1, t2, u[5], m[11]);+	MONT_MAC(v0, v1, v2, u[6], m[10]);+	MONT_MAC(w0, w1, w2, u[7], m[9]);+	MONT_MAC(t0, t1, t2, u[8], m[8]);+	MONT_MAC(v0, v1, v2, u[9], m[7]);+	MONT_MAC(w0, w1, w2, u[10], m[6]);+	MONT_MAC(t0, t1, t2, u[11], m[5]);+	MONT_MAC(v0, v1, v2, u[12], m[4]);+	MONT_MAC(w0, w1, w2, u[13], m[3]);+	MONT_MAC(t0, t1, t2, u[14], m[2]);+	MONT_MAC(v0, v1, v2, u[15], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[16] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[16], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 17 */+	MONT_MUL(s0, s1, s2, a[0], a[17]);+	MONT_MUL(c0, c1, c2, a[1], a[16]);+	MONT_MAC(s0, s1, s2, a[2], a[15]);+	MONT_MAC(c0, c1, c2, a[3], a[14]);+	MONT_MAC(s0, s1, s2, a[4], a[13]);+	MONT_MAC(c0, c1, c2, a[5], a[12]);+	MONT_MAC(s0, s1, s2, a[6], a[11]);+	MONT_MAC(c0, c1, c2, a[7], a[10]);+	MONT_MAC(s0, s1, s2, a[8], a[9]);+	MONT_MAC(t0, t1, t2, u[0], m[17]);+	MONT_MUL(v0, v1, v2, u[1], m[16]);+	MONT_MUL(w0, w1, w2, u[2], m[15]);+	MONT_MAC(t0, t1, t2, u[3], m[14]);+	MONT_MAC(v0, v1, v2, u[4], m[13]);+	MONT_MAC(w0, w1, w2, u[5], m[12]);+	MONT_MAC(t0, t1, t2, u[6], m[11]);+	MONT_MAC(v0, v1, v2, u[7], m[10]);+	MONT_MAC(w0, w1, w2, u[8], m[9]);+	MONT_MAC(t0, t1, t2, u[9], m[8]);+	MONT_MAC(v0, v1, v2, u[10], m[7]);+	MONT_MAC(w0, w1, w2, u[11], m[6]);+	MONT_MAC(t0, t1, t2, u[12], m[5]);+	MONT_MAC(v0, v1, v2, u[13], m[4]);+	MONT_MAC(w0, w1, w2, u[14], m[3]);+	MONT_MAC(t0, t1, t2, u[15], m[2]);+	MONT_MAC(v0, v1, v2, u[16], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[17] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[17], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 18 */+	MONT_MUL(s0, s1, s2, a[0], a[18]);+	MONT_MUL(c0, c1, c2, a[1], a[17]);+	MONT_MAC(s0, s1, s2, a[2], a[16]);+	MONT_MAC(c0, c1, c2, a[3], a[15]);+	MONT_MAC(s0, s1, s2, a[4], a[14]);+	MONT_MAC(c0, c1, c2, a[5], a[13]);+	MONT_MAC(s0, s1, s2, a[6], a[12]);+	MONT_MAC(c0, c1, c2, a[7], a[11]);+	MONT_MAC(s0, s1, s2, a[8], a[10]);+	MONT_MAC(t0, t1, t2, a[9], a[9]);+	MONT_MUL(v0, v1, v2, u[0], m[18]);+	MONT_MUL(w0, w1, w2, u[1], m[17]);+	MONT_MAC(t0, t1, t2, u[2], m[16]);+	MONT_MAC(v0, v1, v2, u[3], m[15]);+	MONT_MAC(w0, w1, w2, u[4], m[14]);+	MONT_MAC(t0, t1, t2, u[5], m[13]);+	MONT_MAC(v0, v1, v2, u[6], m[12]);+	MONT_MAC(w0, w1, w2, u[7], m[11]);+	MONT_MAC(t0, t1, t2, u[8], m[10]);+	MONT_MAC(v0, v1, v2, u[9], m[9]);+	MONT_MAC(w0, w1, w2, u[10], m[8]);+	MONT_MAC(t0, t1, t2, u[11], m[7]);+	MONT_MAC(v0, v1, v2, u[12], m[6]);+	MONT_MAC(w0, w1, w2, u[13], m[5]);+	MONT_MAC(t0, t1, t2, u[14], m[4]);+	MONT_MAC(v0, v1, v2, u[15], m[3]);+	MONT_MAC(w0, w1, w2, u[16], m[2]);+	MONT_MAC(t0, t1, t2, u[17], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[18] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[18], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 19 */+	MONT_MUL(s0, s1, s2, a[0], a[19]);+	MONT_MUL(c0, c1, c2, a[1], a[18]);+	MONT_MAC(s0, s1, s2, a[2], a[17]);+	MONT_MAC(c0, c1, c2, a[3], a[16]);+	MONT_MAC(s0, s1, s2, a[4], a[15]);+	MONT_MAC(c0, c1, c2, a[5], a[14]);+	MONT_MAC(s0, s1, s2, a[6], a[13]);+	MONT_MAC(c0, c1, c2, a[7], a[12]);+	MONT_MAC(s0, s1, s2, a[8], a[11]);+	MONT_MAC(c0, c1, c2, a[9], a[10]);+	MONT_MAC(t0, t1, t2, u[0], m[19]);+	MONT_MUL(v0, v1, v2, u[1], m[18]);+	MONT_MUL(w0, w1, w2, u[2], m[17]);+	MONT_MAC(t0, t1, t2, u[3], m[16]);+	MONT_MAC(v0, v1, v2, u[4], m[15]);+	MONT_MAC(w0, w1, w2, u[5], m[14]);+	MONT_MAC(t0, t1, t2, u[6], m[13]);+	MONT_MAC(v0, v1, v2, u[7], m[12]);+	MONT_MAC(w0, w1, w2, u[8], m[11]);+	MONT_MAC(t0, t1, t2, u[9], m[10]);+	MONT_MAC(v0, v1, v2, u[10], m[9]);+	MONT_MAC(w0, w1, w2, u[11], m[8]);+	MONT_MAC(t0, t1, t2, u[12], m[7]);+	MONT_MAC(v0, v1, v2, u[13], m[6]);+	MONT_MAC(w0, w1, w2, u[14], m[5]);+	MONT_MAC(t0, t1, t2, u[15], m[4]);+	MONT_MAC(v0, v1, v2, u[16], m[3]);+	MONT_MAC(w0, w1, w2, u[17], m[2]);+	MONT_MAC(t0, t1, t2, u[18], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[19] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[19], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 20 */+	MONT_MUL(s0, s1, s2, a[0], a[20]);+	MONT_MUL(c0, c1, c2, a[1], a[19]);+	MONT_MAC(s0, s1, s2, a[2], a[18]);+	MONT_MAC(c0, c1, c2, a[3], a[17]);+	MONT_MAC(s0, s1, s2, a[4], a[16]);+	MONT_MAC(c0, c1, c2, a[5], a[15]);+	MONT_MAC(s0, s1, s2, a[6], a[14]);+	MONT_MAC(c0, c1, c2, a[7], a[13]);+	MONT_MAC(s0, s1, s2, a[8], a[12]);+	MONT_MAC(c0, c1, c2, a[9], a[11]);+	MONT_MAC(t0, t1, t2, a[10], a[10]);+	MONT_MUL(v0, v1, v2, u[0], m[20]);+	MONT_MUL(w0, w1, w2, u[1], m[19]);+	MONT_MAC(t0, t1, t2, u[2], m[18]);+	MONT_MAC(v0, v1, v2, u[3], m[17]);+	MONT_MAC(w0, w1, w2, u[4], m[16]);+	MONT_MAC(t0, t1, t2, u[5], m[15]);+	MONT_MAC(v0, v1, v2, u[6], m[14]);+	MONT_MAC(w0, w1, w2, u[7], m[13]);+	MONT_MAC(t0, t1, t2, u[8], m[12]);+	MONT_MAC(v0, v1, v2, u[9], m[11]);+	MONT_MAC(w0, w1, w2, u[10], m[10]);+	MONT_MAC(t0, t1, t2, u[11], m[9]);+	MONT_MAC(v0, v1, v2, u[12], m[8]);+	MONT_MAC(w0, w1, w2, u[13], m[7]);+	MONT_MAC(t0, t1, t2, u[14], m[6]);+	MONT_MAC(v0, v1, v2, u[15], m[5]);+	MONT_MAC(w0, w1, w2, u[16], m[4]);+	MONT_MAC(t0, t1, t2, u[17], m[3]);+	MONT_MAC(v0, v1, v2, u[18], m[2]);+	MONT_MAC(w0, w1, w2, u[19], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[20] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[20], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 21 */+	MONT_MUL(s0, s1, s2, a[0], a[21]);+	MONT_MUL(c0, c1, c2, a[1], a[20]);+	MONT_MAC(s0, s1, s2, a[2], a[19]);+	MONT_MAC(c0, c1, c2, a[3], a[18]);+	MONT_MAC(s0, s1, s2, a[4], a[17]);+	MONT_MAC(c0, c1, c2, a[5], a[16]);+	MONT_MAC(s0, s1, s2, a[6], a[15]);+	MONT_MAC(c0, c1, c2, a[7], a[14]);+	MONT_MAC(s0, s1, s2, a[8], a[13]);+	MONT_MAC(c0, c1, c2, a[9], a[12]);+	MONT_MAC(s0, s1, s2, a[10], a[11]);+	MONT_MAC(t0, t1, t2, u[0], m[21]);+	MONT_MUL(v0, v1, v2, u[1], m[20]);+	MONT_MUL(w0, w1, w2, u[2], m[19]);+	MONT_MAC(t0, t1, t2, u[3], m[18]);+	MONT_MAC(v0, v1, v2, u[4], m[17]);+	MONT_MAC(w0, w1, w2, u[5], m[16]);+	MONT_MAC(t0, t1, t2, u[6], m[15]);+	MONT_MAC(v0, v1, v2, u[7], m[14]);+	MONT_MAC(w0, w1, w2, u[8], m[13]);+	MONT_MAC(t0, t1, t2, u[9], m[12]);+	MONT_MAC(v0, v1, v2, u[10], m[11]);+	MONT_MAC(w0, w1, w2, u[11], m[10]);+	MONT_MAC(t0, t1, t2, u[12], m[9]);+	MONT_MAC(v0, v1, v2, u[13], m[8]);+	MONT_MAC(w0, w1, w2, u[14], m[7]);+	MONT_MAC(t0, t1, t2, u[15], m[6]);+	MONT_MAC(v0, v1, v2, u[16], m[5]);+	MONT_MAC(w0, w1, w2, u[17], m[4]);+	MONT_MAC(t0, t1, t2, u[18], m[3]);+	MONT_MAC(v0, v1, v2, u[19], m[2]);+	MONT_MAC(w0, w1, w2, u[20], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[21] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[21], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 22 */+	MONT_MUL(s0, s1, s2, a[0], a[22]);+	MONT_MUL(c0, c1, c2, a[1], a[21]);+	MONT_MAC(s0, s1, s2, a[2], a[20]);+	MONT_MAC(c0, c1, c2, a[3], a[19]);+	MONT_MAC(s0, s1, s2, a[4], a[18]);+	MONT_MAC(c0, c1, c2, a[5], a[17]);+	MONT_MAC(s0, s1, s2, a[6], a[16]);+	MONT_MAC(c0, c1, c2, a[7], a[15]);+	MONT_MAC(s0, s1, s2, a[8], a[14]);+	MONT_MAC(c0, c1, c2, a[9], a[13]);+	MONT_MAC(s0, s1, s2, a[10], a[12]);+	MONT_MAC(t0, t1, t2, a[11], a[11]);+	MONT_MUL(v0, v1, v2, u[0], m[22]);+	MONT_MUL(w0, w1, w2, u[1], m[21]);+	MONT_MAC(t0, t1, t2, u[2], m[20]);+	MONT_MAC(v0, v1, v2, u[3], m[19]);+	MONT_MAC(w0, w1, w2, u[4], m[18]);+	MONT_MAC(t0, t1, t2, u[5], m[17]);+	MONT_MAC(v0, v1, v2, u[6], m[16]);+	MONT_MAC(w0, w1, w2, u[7], m[15]);+	MONT_MAC(t0, t1, t2, u[8], m[14]);+	MONT_MAC(v0, v1, v2, u[9], m[13]);+	MONT_MAC(w0, w1, w2, u[10], m[12]);+	MONT_MAC(t0, t1, t2, u[11], m[11]);+	MONT_MAC(v0, v1, v2, u[12], m[10]);+	MONT_MAC(w0, w1, w2, u[13], m[9]);+	MONT_MAC(t0, t1, t2, u[14], m[8]);+	MONT_MAC(v0, v1, v2, u[15], m[7]);+	MONT_MAC(w0, w1, w2, u[16], m[6]);+	MONT_MAC(t0, t1, t2, u[17], m[5]);+	MONT_MAC(v0, v1, v2, u[18], m[4]);+	MONT_MAC(w0, w1, w2, u[19], m[3]);+	MONT_MAC(t0, t1, t2, u[20], m[2]);+	MONT_MAC(v0, v1, v2, u[21], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[22] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[22], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 23 */+	MONT_MUL(s0, s1, s2, a[0], a[23]);+	MONT_MUL(c0, c1, c2, a[1], a[22]);+	MONT_MAC(s0, s1, s2, a[2], a[21]);+	MONT_MAC(c0, c1, c2, a[3], a[20]);+	MONT_MAC(s0, s1, s2, a[4], a[19]);+	MONT_MAC(c0, c1, c2, a[5], a[18]);+	MONT_MAC(s0, s1, s2, a[6], a[17]);+	MONT_MAC(c0, c1, c2, a[7], a[16]);+	MONT_MAC(s0, s1, s2, a[8], a[15]);+	MONT_MAC(c0, c1, c2, a[9], a[14]);+	MONT_MAC(s0, s1, s2, a[10], a[13]);+	MONT_MAC(c0, c1, c2, a[11], a[12]);+	MONT_MAC(t0, t1, t2, u[0], m[23]);+	MONT_MUL(v0, v1, v2, u[1], m[22]);+	MONT_MUL(w0, w1, w2, u[2], m[21]);+	MONT_MAC(t0, t1, t2, u[3], m[20]);+	MONT_MAC(v0, v1, v2, u[4], m[19]);+	MONT_MAC(w0, w1, w2, u[5], m[18]);+	MONT_MAC(t0, t1, t2, u[6], m[17]);+	MONT_MAC(v0, v1, v2, u[7], m[16]);+	MONT_MAC(w0, w1, w2, u[8], m[15]);+	MONT_MAC(t0, t1, t2, u[9], m[14]);+	MONT_MAC(v0, v1, v2, u[10], m[13]);+	MONT_MAC(w0, w1, w2, u[11], m[12]);+	MONT_MAC(t0, t1, t2, u[12], m[11]);+	MONT_MAC(v0, v1, v2, u[13], m[10]);+	MONT_MAC(w0, w1, w2, u[14], m[9]);+	MONT_MAC(t0, t1, t2, u[15], m[8]);+	MONT_MAC(v0, v1, v2, u[16], m[7]);+	MONT_MAC(w0, w1, w2, u[17], m[6]);+	MONT_MAC(t0, t1, t2, u[18], m[5]);+	MONT_MAC(v0, v1, v2, u[19], m[4]);+	MONT_MAC(w0, w1, w2, u[20], m[3]);+	MONT_MAC(t0, t1, t2, u[21], m[2]);+	MONT_MAC(v0, v1, v2, u[22], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[23] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[23], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 24 */+	MONT_MUL(s0, s1, s2, a[0], a[24]);+	MONT_MUL(c0, c1, c2, a[1], a[23]);+	MONT_MAC(s0, s1, s2, a[2], a[22]);+	MONT_MAC(c0, c1, c2, a[3], a[21]);+	MONT_MAC(s0, s1, s2, a[4], a[20]);+	MONT_MAC(c0, c1, c2, a[5], a[19]);+	MONT_MAC(s0, s1, s2, a[6], a[18]);+	MONT_MAC(c0, c1, c2, a[7], a[17]);+	MONT_MAC(s0, s1, s2, a[8], a[16]);+	MONT_MAC(c0, c1, c2, a[9], a[15]);+	MONT_MAC(s0, s1, s2, a[10], a[14]);+	MONT_MAC(c0, c1, c2, a[11], a[13]);+	MONT_MAC(t0, t1, t2, a[12], a[12]);+	MONT_MUL(v0, v1, v2, u[0], m[24]);+	MONT_MUL(w0, w1, w2, u[1], m[23]);+	MONT_MAC(t0, t1, t2, u[2], m[22]);+	MONT_MAC(v0, v1, v2, u[3], m[21]);+	MONT_MAC(w0, w1, w2, u[4], m[20]);+	MONT_MAC(t0, t1, t2, u[5], m[19]);+	MONT_MAC(v0, v1, v2, u[6], m[18]);+	MONT_MAC(w0, w1, w2, u[7], m[17]);+	MONT_MAC(t0, t1, t2, u[8], m[16]);+	MONT_MAC(v0, v1, v2, u[9], m[15]);+	MONT_MAC(w0, w1, w2, u[10], m[14]);+	MONT_MAC(t0, t1, t2, u[11], m[13]);+	MONT_MAC(v0, v1, v2, u[12], m[12]);+	MONT_MAC(w0, w1, w2, u[13], m[11]);+	MONT_MAC(t0, t1, t2, u[14], m[10]);+	MONT_MAC(v0, v1, v2, u[15], m[9]);+	MONT_MAC(w0, w1, w2, u[16], m[8]);+	MONT_MAC(t0, t1, t2, u[17], m[7]);+	MONT_MAC(v0, v1, v2, u[18], m[6]);+	MONT_MAC(w0, w1, w2, u[19], m[5]);+	MONT_MAC(t0, t1, t2, u[20], m[4]);+	MONT_MAC(v0, v1, v2, u[21], m[3]);+	MONT_MAC(w0, w1, w2, u[22], m[2]);+	MONT_MAC(t0, t1, t2, u[23], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[24] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[24], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 25 */+	MONT_MUL(s0, s1, s2, a[0], a[25]);+	MONT_MUL(c0, c1, c2, a[1], a[24]);+	MONT_MAC(s0, s1, s2, a[2], a[23]);+	MONT_MAC(c0, c1, c2, a[3], a[22]);+	MONT_MAC(s0, s1, s2, a[4], a[21]);+	MONT_MAC(c0, c1, c2, a[5], a[20]);+	MONT_MAC(s0, s1, s2, a[6], a[19]);+	MONT_MAC(c0, c1, c2, a[7], a[18]);+	MONT_MAC(s0, s1, s2, a[8], a[17]);+	MONT_MAC(c0, c1, c2, a[9], a[16]);+	MONT_MAC(s0, s1, s2, a[10], a[15]);+	MONT_MAC(c0, c1, c2, a[11], a[14]);+	MONT_MAC(s0, s1, s2, a[12], a[13]);+	MONT_MAC(t0, t1, t2, u[0], m[25]);+	MONT_MUL(v0, v1, v2, u[1], m[24]);+	MONT_MUL(w0, w1, w2, u[2], m[23]);+	MONT_MAC(t0, t1, t2, u[3], m[22]);+	MONT_MAC(v0, v1, v2, u[4], m[21]);+	MONT_MAC(w0, w1, w2, u[5], m[20]);+	MONT_MAC(t0, t1, t2, u[6], m[19]);+	MONT_MAC(v0, v1, v2, u[7], m[18]);+	MONT_MAC(w0, w1, w2, u[8], m[17]);+	MONT_MAC(t0, t1, t2, u[9], m[16]);+	MONT_MAC(v0, v1, v2, u[10], m[15]);+	MONT_MAC(w0, w1, w2, u[11], m[14]);+	MONT_MAC(t0, t1, t2, u[12], m[13]);+	MONT_MAC(v0, v1, v2, u[13], m[12]);+	MONT_MAC(w0, w1, w2, u[14], m[11]);+	MONT_MAC(t0, t1, t2, u[15], m[10]);+	MONT_MAC(v0, v1, v2, u[16], m[9]);+	MONT_MAC(w0, w1, w2, u[17], m[8]);+	MONT_MAC(t0, t1, t2, u[18], m[7]);+	MONT_MAC(v0, v1, v2, u[19], m[6]);+	MONT_MAC(w0, w1, w2, u[20], m[5]);+	MONT_MAC(t0, t1, t2, u[21], m[4]);+	MONT_MAC(v0, v1, v2, u[22], m[3]);+	MONT_MAC(w0, w1, w2, u[23], m[2]);+	MONT_MAC(t0, t1, t2, u[24], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[25] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[25], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 26 */+	MONT_MUL(s0, s1, s2, a[0], a[26]);+	MONT_MUL(c0, c1, c2, a[1], a[25]);+	MONT_MAC(s0, s1, s2, a[2], a[24]);+	MONT_MAC(c0, c1, c2, a[3], a[23]);+	MONT_MAC(s0, s1, s2, a[4], a[22]);+	MONT_MAC(c0, c1, c2, a[5], a[21]);+	MONT_MAC(s0, s1, s2, a[6], a[20]);+	MONT_MAC(c0, c1, c2, a[7], a[19]);+	MONT_MAC(s0, s1, s2, a[8], a[18]);+	MONT_MAC(c0, c1, c2, a[9], a[17]);+	MONT_MAC(s0, s1, s2, a[10], a[16]);+	MONT_MAC(c0, c1, c2, a[11], a[15]);+	MONT_MAC(s0, s1, s2, a[12], a[14]);+	MONT_MAC(t0, t1, t2, a[13], a[13]);+	MONT_MUL(v0, v1, v2, u[0], m[26]);+	MONT_MUL(w0, w1, w2, u[1], m[25]);+	MONT_MAC(t0, t1, t2, u[2], m[24]);+	MONT_MAC(v0, v1, v2, u[3], m[23]);+	MONT_MAC(w0, w1, w2, u[4], m[22]);+	MONT_MAC(t0, t1, t2, u[5], m[21]);+	MONT_MAC(v0, v1, v2, u[6], m[20]);+	MONT_MAC(w0, w1, w2, u[7], m[19]);+	MONT_MAC(t0, t1, t2, u[8], m[18]);+	MONT_MAC(v0, v1, v2, u[9], m[17]);+	MONT_MAC(w0, w1, w2, u[10], m[16]);+	MONT_MAC(t0, t1, t2, u[11], m[15]);+	MONT_MAC(v0, v1, v2, u[12], m[14]);+	MONT_MAC(w0, w1, w2, u[13], m[13]);+	MONT_MAC(t0, t1, t2, u[14], m[12]);+	MONT_MAC(v0, v1, v2, u[15], m[11]);+	MONT_MAC(w0, w1, w2, u[16], m[10]);+	MONT_MAC(t0, t1, t2, u[17], m[9]);+	MONT_MAC(v0, v1, v2, u[18], m[8]);+	MONT_MAC(w0, w1, w2, u[19], m[7]);+	MONT_MAC(t0, t1, t2, u[20], m[6]);+	MONT_MAC(v0, v1, v2, u[21], m[5]);+	MONT_MAC(w0, w1, w2, u[22], m[4]);+	MONT_MAC(t0, t1, t2, u[23], m[3]);+	MONT_MAC(v0, v1, v2, u[24], m[2]);+	MONT_MAC(w0, w1, w2, u[25], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[26] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[26], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 27 */+	MONT_MUL(s0, s1, s2, a[0], a[27]);+	MONT_MUL(c0, c1, c2, a[1], a[26]);+	MONT_MAC(s0, s1, s2, a[2], a[25]);+	MONT_MAC(c0, c1, c2, a[3], a[24]);+	MONT_MAC(s0, s1, s2, a[4], a[23]);+	MONT_MAC(c0, c1, c2, a[5], a[22]);+	MONT_MAC(s0, s1, s2, a[6], a[21]);+	MONT_MAC(c0, c1, c2, a[7], a[20]);+	MONT_MAC(s0, s1, s2, a[8], a[19]);+	MONT_MAC(c0, c1, c2, a[9], a[18]);+	MONT_MAC(s0, s1, s2, a[10], a[17]);+	MONT_MAC(c0, c1, c2, a[11], a[16]);+	MONT_MAC(s0, s1, s2, a[12], a[15]);+	MONT_MAC(c0, c1, c2, a[13], a[14]);+	MONT_MAC(t0, t1, t2, u[0], m[27]);+	MONT_MUL(v0, v1, v2, u[1], m[26]);+	MONT_MUL(w0, w1, w2, u[2], m[25]);+	MONT_MAC(t0, t1, t2, u[3], m[24]);+	MONT_MAC(v0, v1, v2, u[4], m[23]);+	MONT_MAC(w0, w1, w2, u[5], m[22]);+	MONT_MAC(t0, t1, t2, u[6], m[21]);+	MONT_MAC(v0, v1, v2, u[7], m[20]);+	MONT_MAC(w0, w1, w2, u[8], m[19]);+	MONT_MAC(t0, t1, t2, u[9], m[18]);+	MONT_MAC(v0, v1, v2, u[10], m[17]);+	MONT_MAC(w0, w1, w2, u[11], m[16]);+	MONT_MAC(t0, t1, t2, u[12], m[15]);+	MONT_MAC(v0, v1, v2, u[13], m[14]);+	MONT_MAC(w0, w1, w2, u[14], m[13]);+	MONT_MAC(t0, t1, t2, u[15], m[12]);+	MONT_MAC(v0, v1, v2, u[16], m[11]);+	MONT_MAC(w0, w1, w2, u[17], m[10]);+	MONT_MAC(t0, t1, t2, u[18], m[9]);+	MONT_MAC(v0, v1, v2, u[19], m[8]);+	MONT_MAC(w0, w1, w2, u[20], m[7]);+	MONT_MAC(t0, t1, t2, u[21], m[6]);+	MONT_MAC(v0, v1, v2, u[22], m[5]);+	MONT_MAC(w0, w1, w2, u[23], m[4]);+	MONT_MAC(t0, t1, t2, u[24], m[3]);+	MONT_MAC(v0, v1, v2, u[25], m[2]);+	MONT_MAC(w0, w1, w2, u[26], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[27] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[27], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 28 */+	MONT_MUL(s0, s1, s2, a[0], a[28]);+	MONT_MUL(c0, c1, c2, a[1], a[27]);+	MONT_MAC(s0, s1, s2, a[2], a[26]);+	MONT_MAC(c0, c1, c2, a[3], a[25]);+	MONT_MAC(s0, s1, s2, a[4], a[24]);+	MONT_MAC(c0, c1, c2, a[5], a[23]);+	MONT_MAC(s0, s1, s2, a[6], a[22]);+	MONT_MAC(c0, c1, c2, a[7], a[21]);+	MONT_MAC(s0, s1, s2, a[8], a[20]);+	MONT_MAC(c0, c1, c2, a[9], a[19]);+	MONT_MAC(s0, s1, s2, a[10], a[18]);+	MONT_MAC(c0, c1, c2, a[11], a[17]);+	MONT_MAC(s0, s1, s2, a[12], a[16]);+	MONT_MAC(c0, c1, c2, a[13], a[15]);+	MONT_MAC(t0, t1, t2, a[14], a[14]);+	MONT_MUL(v0, v1, v2, u[0], m[28]);+	MONT_MUL(w0, w1, w2, u[1], m[27]);+	MONT_MAC(t0, t1, t2, u[2], m[26]);+	MONT_MAC(v0, v1, v2, u[3], m[25]);+	MONT_MAC(w0, w1, w2, u[4], m[24]);+	MONT_MAC(t0, t1, t2, u[5], m[23]);+	MONT_MAC(v0, v1, v2, u[6], m[22]);+	MONT_MAC(w0, w1, w2, u[7], m[21]);+	MONT_MAC(t0, t1, t2, u[8], m[20]);+	MONT_MAC(v0, v1, v2, u[9], m[19]);+	MONT_MAC(w0, w1, w2, u[10], m[18]);+	MONT_MAC(t0, t1, t2, u[11], m[17]);+	MONT_MAC(v0, v1, v2, u[12], m[16]);+	MONT_MAC(w0, w1, w2, u[13], m[15]);+	MONT_MAC(t0, t1, t2, u[14], m[14]);+	MONT_MAC(v0, v1, v2, u[15], m[13]);+	MONT_MAC(w0, w1, w2, u[16], m[12]);+	MONT_MAC(t0, t1, t2, u[17], m[11]);+	MONT_MAC(v0, v1, v2, u[18], m[10]);+	MONT_MAC(w0, w1, w2, u[19], m[9]);+	MONT_MAC(t0, t1, t2, u[20], m[8]);+	MONT_MAC(v0, v1, v2, u[21], m[7]);+	MONT_MAC(w0, w1, w2, u[22], m[6]);+	MONT_MAC(t0, t1, t2, u[23], m[5]);+	MONT_MAC(v0, v1, v2, u[24], m[4]);+	MONT_MAC(w0, w1, w2, u[25], m[3]);+	MONT_MAC(t0, t1, t2, u[26], m[2]);+	MONT_MAC(v0, v1, v2, u[27], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[28] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[28], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 29 */+	MONT_MUL(s0, s1, s2, a[0], a[29]);+	MONT_MUL(c0, c1, c2, a[1], a[28]);+	MONT_MAC(s0, s1, s2, a[2], a[27]);+	MONT_MAC(c0, c1, c2, a[3], a[26]);+	MONT_MAC(s0, s1, s2, a[4], a[25]);+	MONT_MAC(c0, c1, c2, a[5], a[24]);+	MONT_MAC(s0, s1, s2, a[6], a[23]);+	MONT_MAC(c0, c1, c2, a[7], a[22]);+	MONT_MAC(s0, s1, s2, a[8], a[21]);+	MONT_MAC(c0, c1, c2, a[9], a[20]);+	MONT_MAC(s0, s1, s2, a[10], a[19]);+	MONT_MAC(c0, c1, c2, a[11], a[18]);+	MONT_MAC(s0, s1, s2, a[12], a[17]);+	MONT_MAC(c0, c1, c2, a[13], a[16]);+	MONT_MAC(s0, s1, s2, a[14], a[15]);+	MONT_MAC(t0, t1, t2, u[0], m[29]);+	MONT_MUL(v0, v1, v2, u[1], m[28]);+	MONT_MUL(w0, w1, w2, u[2], m[27]);+	MONT_MAC(t0, t1, t2, u[3], m[26]);+	MONT_MAC(v0, v1, v2, u[4], m[25]);+	MONT_MAC(w0, w1, w2, u[5], m[24]);+	MONT_MAC(t0, t1, t2, u[6], m[23]);+	MONT_MAC(v0, v1, v2, u[7], m[22]);+	MONT_MAC(w0, w1, w2, u[8], m[21]);+	MONT_MAC(t0, t1, t2, u[9], m[20]);+	MONT_MAC(v0, v1, v2, u[10], m[19]);+	MONT_MAC(w0, w1, w2, u[11], m[18]);+	MONT_MAC(t0, t1, t2, u[12], m[17]);+	MONT_MAC(v0, v1, v2, u[13], m[16]);+	MONT_MAC(w0, w1, w2, u[14], m[15]);+	MONT_MAC(t0, t1, t2, u[15], m[14]);+	MONT_MAC(v0, v1, v2, u[16], m[13]);+	MONT_MAC(w0, w1, w2, u[17], m[12]);+	MONT_MAC(t0, t1, t2, u[18], m[11]);+	MONT_MAC(v0, v1, v2, u[19], m[10]);+	MONT_MAC(w0, w1, w2, u[20], m[9]);+	MONT_MAC(t0, t1, t2, u[21], m[8]);+	MONT_MAC(v0, v1, v2, u[22], m[7]);+	MONT_MAC(w0, w1, w2, u[23], m[6]);+	MONT_MAC(t0, t1, t2, u[24], m[5]);+	MONT_MAC(v0, v1, v2, u[25], m[4]);+	MONT_MAC(w0, w1, w2, u[26], m[3]);+	MONT_MAC(t0, t1, t2, u[27], m[2]);+	MONT_MAC(v0, v1, v2, u[28], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[29] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[29], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 30 */+	MONT_MUL(s0, s1, s2, a[0], a[30]);+	MONT_MUL(c0, c1, c2, a[1], a[29]);+	MONT_MAC(s0, s1, s2, a[2], a[28]);+	MONT_MAC(c0, c1, c2, a[3], a[27]);+	MONT_MAC(s0, s1, s2, a[4], a[26]);+	MONT_MAC(c0, c1, c2, a[5], a[25]);+	MONT_MAC(s0, s1, s2, a[6], a[24]);+	MONT_MAC(c0, c1, c2, a[7], a[23]);+	MONT_MAC(s0, s1, s2, a[8], a[22]);+	MONT_MAC(c0, c1, c2, a[9], a[21]);+	MONT_MAC(s0, s1, s2, a[10], a[20]);+	MONT_MAC(c0, c1, c2, a[11], a[19]);+	MONT_MAC(s0, s1, s2, a[12], a[18]);+	MONT_MAC(c0, c1, c2, a[13], a[17]);+	MONT_MAC(s0, s1, s2, a[14], a[16]);+	MONT_MAC(t0, t1, t2, a[15], a[15]);+	MONT_MUL(v0, v1, v2, u[0], m[30]);+	MONT_MUL(w0, w1, w2, u[1], m[29]);+	MONT_MAC(t0, t1, t2, u[2], m[28]);+	MONT_MAC(v0, v1, v2, u[3], m[27]);+	MONT_MAC(w0, w1, w2, u[4], m[26]);+	MONT_MAC(t0, t1, t2, u[5], m[25]);+	MONT_MAC(v0, v1, v2, u[6], m[24]);+	MONT_MAC(w0, w1, w2, u[7], m[23]);+	MONT_MAC(t0, t1, t2, u[8], m[22]);+	MONT_MAC(v0, v1, v2, u[9], m[21]);+	MONT_MAC(w0, w1, w2, u[10], m[20]);+	MONT_MAC(t0, t1, t2, u[11], m[19]);+	MONT_MAC(v0, v1, v2, u[12], m[18]);+	MONT_MAC(w0, w1, w2, u[13], m[17]);+	MONT_MAC(t0, t1, t2, u[14], m[16]);+	MONT_MAC(v0, v1, v2, u[15], m[15]);+	MONT_MAC(w0, w1, w2, u[16], m[14]);+	MONT_MAC(t0, t1, t2, u[17], m[13]);+	MONT_MAC(v0, v1, v2, u[18], m[12]);+	MONT_MAC(w0, w1, w2, u[19], m[11]);+	MONT_MAC(t0, t1, t2, u[20], m[10]);+	MONT_MAC(v0, v1, v2, u[21], m[9]);+	MONT_MAC(w0, w1, w2, u[22], m[8]);+	MONT_MAC(t0, t1, t2, u[23], m[7]);+	MONT_MAC(v0, v1, v2, u[24], m[6]);+	MONT_MAC(w0, w1, w2, u[25], m[5]);+	MONT_MAC(t0, t1, t2, u[26], m[4]);+	MONT_MAC(v0, v1, v2, u[27], m[3]);+	MONT_MAC(w0, w1, w2, u[28], m[2]);+	MONT_MAC(t0, t1, t2, u[29], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[30] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[30], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 31 */+	MONT_MUL(s0, s1, s2, a[0], a[31]);+	MONT_MUL(c0, c1, c2, a[1], a[30]);+	MONT_MAC(s0, s1, s2, a[2], a[29]);+	MONT_MAC(c0, c1, c2, a[3], a[28]);+	MONT_MAC(s0, s1, s2, a[4], a[27]);+	MONT_MAC(c0, c1, c2, a[5], a[26]);+	MONT_MAC(s0, s1, s2, a[6], a[25]);+	MONT_MAC(c0, c1, c2, a[7], a[24]);+	MONT_MAC(s0, s1, s2, a[8], a[23]);+	MONT_MAC(c0, c1, c2, a[9], a[22]);+	MONT_MAC(s0, s1, s2, a[10], a[21]);+	MONT_MAC(c0, c1, c2, a[11], a[20]);+	MONT_MAC(s0, s1, s2, a[12], a[19]);+	MONT_MAC(c0, c1, c2, a[13], a[18]);+	MONT_MAC(s0, s1, s2, a[14], a[17]);+	MONT_MAC(c0, c1, c2, a[15], a[16]);+	MONT_MAC(t0, t1, t2, u[0], m[31]);+	MONT_MUL(v0, v1, v2, u[1], m[30]);+	MONT_MUL(w0, w1, w2, u[2], m[29]);+	MONT_MAC(t0, t1, t2, u[3], m[28]);+	MONT_MAC(v0, v1, v2, u[4], m[27]);+	MONT_MAC(w0, w1, w2, u[5], m[26]);+	MONT_MAC(t0, t1, t2, u[6], m[25]);+	MONT_MAC(v0, v1, v2, u[7], m[24]);+	MONT_MAC(w0, w1, w2, u[8], m[23]);+	MONT_MAC(t0, t1, t2, u[9], m[22]);+	MONT_MAC(v0, v1, v2, u[10], m[21]);+	MONT_MAC(w0, w1, w2, u[11], m[20]);+	MONT_MAC(t0, t1, t2, u[12], m[19]);+	MONT_MAC(v0, v1, v2, u[13], m[18]);+	MONT_MAC(w0, w1, w2, u[14], m[17]);+	MONT_MAC(t0, t1, t2, u[15], m[16]);+	MONT_MAC(v0, v1, v2, u[16], m[15]);+	MONT_MAC(w0, w1, w2, u[17], m[14]);+	MONT_MAC(t0, t1, t2, u[18], m[13]);+	MONT_MAC(v0, v1, v2, u[19], m[12]);+	MONT_MAC(w0, w1, w2, u[20], m[11]);+	MONT_MAC(t0, t1, t2, u[21], m[10]);+	MONT_MAC(v0, v1, v2, u[22], m[9]);+	MONT_MAC(w0, w1, w2, u[23], m[8]);+	MONT_MAC(t0, t1, t2, u[24], m[7]);+	MONT_MAC(v0, v1, v2, u[25], m[6]);+	MONT_MAC(w0, w1, w2, u[26], m[5]);+	MONT_MAC(t0, t1, t2, u[27], m[4]);+	MONT_MAC(v0, v1, v2, u[28], m[3]);+	MONT_MAC(w0, w1, w2, u[29], m[2]);+	MONT_MAC(t0, t1, t2, u[30], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[31] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[31], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 32 */+	MONT_MUL(s0, s1, s2, a[0], a[32]);+	MONT_MUL(c0, c1, c2, a[1], a[31]);+	MONT_MAC(s0, s1, s2, a[2], a[30]);+	MONT_MAC(c0, c1, c2, a[3], a[29]);+	MONT_MAC(s0, s1, s2, a[4], a[28]);+	MONT_MAC(c0, c1, c2, a[5], a[27]);+	MONT_MAC(s0, s1, s2, a[6], a[26]);+	MONT_MAC(c0, c1, c2, a[7], a[25]);+	MONT_MAC(s0, s1, s2, a[8], a[24]);+	MONT_MAC(c0, c1, c2, a[9], a[23]);+	MONT_MAC(s0, s1, s2, a[10], a[22]);+	MONT_MAC(c0, c1, c2, a[11], a[21]);+	MONT_MAC(s0, s1, s2, a[12], a[20]);+	MONT_MAC(c0, c1, c2, a[13], a[19]);+	MONT_MAC(s0, s1, s2, a[14], a[18]);+	MONT_MAC(c0, c1, c2, a[15], a[17]);+	MONT_MAC(t0, t1, t2, a[16], a[16]);+	MONT_MUL(v0, v1, v2, u[0], m[32]);+	MONT_MUL(w0, w1, w2, u[1], m[31]);+	MONT_MAC(t0, t1, t2, u[2], m[30]);+	MONT_MAC(v0, v1, v2, u[3], m[29]);+	MONT_MAC(w0, w1, w2, u[4], m[28]);+	MONT_MAC(t0, t1, t2, u[5], m[27]);+	MONT_MAC(v0, v1, v2, u[6], m[26]);+	MONT_MAC(w0, w1, w2, u[7], m[25]);+	MONT_MAC(t0, t1, t2, u[8], m[24]);+	MONT_MAC(v0, v1, v2, u[9], m[23]);+	MONT_MAC(w0, w1, w2, u[10], m[22]);+	MONT_MAC(t0, t1, t2, u[11], m[21]);+	MONT_MAC(v0, v1, v2, u[12], m[20]);+	MONT_MAC(w0, w1, w2, u[13], m[19]);+	MONT_MAC(t0, t1, t2, u[14], m[18]);+	MONT_MAC(v0, v1, v2, u[15], m[17]);+	MONT_MAC(w0, w1, w2, u[16], m[16]);+	MONT_MAC(t0, t1, t2, u[17], m[15]);+	MONT_MAC(v0, v1, v2, u[18], m[14]);+	MONT_MAC(w0, w1, w2, u[19], m[13]);+	MONT_MAC(t0, t1, t2, u[20], m[12]);+	MONT_MAC(v0, v1, v2, u[21], m[11]);+	MONT_MAC(w0, w1, w2, u[22], m[10]);+	MONT_MAC(t0, t1, t2, u[23], m[9]);+	MONT_MAC(v0, v1, v2, u[24], m[8]);+	MONT_MAC(w0, w1, w2, u[25], m[7]);+	MONT_MAC(t0, t1, t2, u[26], m[6]);+	MONT_MAC(v0, v1, v2, u[27], m[5]);+	MONT_MAC(w0, w1, w2, u[28], m[4]);+	MONT_MAC(t0, t1, t2, u[29], m[3]);+	MONT_MAC(v0, v1, v2, u[30], m[2]);+	MONT_MAC(w0, w1, w2, u[31], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[32] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[32], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 33 */+	MONT_MUL(s0, s1, s2, a[0], a[33]);+	MONT_MUL(c0, c1, c2, a[1], a[32]);+	MONT_MAC(s0, s1, s2, a[2], a[31]);+	MONT_MAC(c0, c1, c2, a[3], a[30]);+	MONT_MAC(s0, s1, s2, a[4], a[29]);+	MONT_MAC(c0, c1, c2, a[5], a[28]);+	MONT_MAC(s0, s1, s2, a[6], a[27]);+	MONT_MAC(c0, c1, c2, a[7], a[26]);+	MONT_MAC(s0, s1, s2, a[8], a[25]);+	MONT_MAC(c0, c1, c2, a[9], a[24]);+	MONT_MAC(s0, s1, s2, a[10], a[23]);+	MONT_MAC(c0, c1, c2, a[11], a[22]);+	MONT_MAC(s0, s1, s2, a[12], a[21]);+	MONT_MAC(c0, c1, c2, a[13], a[20]);+	MONT_MAC(s0, s1, s2, a[14], a[19]);+	MONT_MAC(c0, c1, c2, a[15], a[18]);+	MONT_MAC(s0, s1, s2, a[16], a[17]);+	MONT_MAC(t0, t1, t2, u[0], m[33]);+	MONT_MUL(v0, v1, v2, u[1], m[32]);+	MONT_MUL(w0, w1, w2, u[2], m[31]);+	MONT_MAC(t0, t1, t2, u[3], m[30]);+	MONT_MAC(v0, v1, v2, u[4], m[29]);+	MONT_MAC(w0, w1, w2, u[5], m[28]);+	MONT_MAC(t0, t1, t2, u[6], m[27]);+	MONT_MAC(v0, v1, v2, u[7], m[26]);+	MONT_MAC(w0, w1, w2, u[8], m[25]);+	MONT_MAC(t0, t1, t2, u[9], m[24]);+	MONT_MAC(v0, v1, v2, u[10], m[23]);+	MONT_MAC(w0, w1, w2, u[11], m[22]);+	MONT_MAC(t0, t1, t2, u[12], m[21]);+	MONT_MAC(v0, v1, v2, u[13], m[20]);+	MONT_MAC(w0, w1, w2, u[14], m[19]);+	MONT_MAC(t0, t1, t2, u[15], m[18]);+	MONT_MAC(v0, v1, v2, u[16], m[17]);+	MONT_MAC(w0, w1, w2, u[17], m[16]);+	MONT_MAC(t0, t1, t2, u[18], m[15]);+	MONT_MAC(v0, v1, v2, u[19], m[14]);+	MONT_MAC(w0, w1, w2, u[20], m[13]);+	MONT_MAC(t0, t1, t2, u[21], m[12]);+	MONT_MAC(v0, v1, v2, u[22], m[11]);+	MONT_MAC(w0, w1, w2, u[23], m[10]);+	MONT_MAC(t0, t1, t2, u[24], m[9]);+	MONT_MAC(v0, v1, v2, u[25], m[8]);+	MONT_MAC(w0, w1, w2, u[26], m[7]);+	MONT_MAC(t0, t1, t2, u[27], m[6]);+	MONT_MAC(v0, v1, v2, u[28], m[5]);+	MONT_MAC(w0, w1, w2, u[29], m[4]);+	MONT_MAC(t0, t1, t2, u[30], m[3]);+	MONT_MAC(v0, v1, v2, u[31], m[2]);+	MONT_MAC(w0, w1, w2, u[32], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[33] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[33], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 34 */+	MONT_MUL(s0, s1, s2, a[0], a[34]);+	MONT_MUL(c0, c1, c2, a[1], a[33]);+	MONT_MAC(s0, s1, s2, a[2], a[32]);+	MONT_MAC(c0, c1, c2, a[3], a[31]);+	MONT_MAC(s0, s1, s2, a[4], a[30]);+	MONT_MAC(c0, c1, c2, a[5], a[29]);+	MONT_MAC(s0, s1, s2, a[6], a[28]);+	MONT_MAC(c0, c1, c2, a[7], a[27]);+	MONT_MAC(s0, s1, s2, a[8], a[26]);+	MONT_MAC(c0, c1, c2, a[9], a[25]);+	MONT_MAC(s0, s1, s2, a[10], a[24]);+	MONT_MAC(c0, c1, c2, a[11], a[23]);+	MONT_MAC(s0, s1, s2, a[12], a[22]);+	MONT_MAC(c0, c1, c2, a[13], a[21]);+	MONT_MAC(s0, s1, s2, a[14], a[20]);+	MONT_MAC(c0, c1, c2, a[15], a[19]);+	MONT_MAC(s0, s1, s2, a[16], a[18]);+	MONT_MAC(t0, t1, t2, a[17], a[17]);+	MONT_MUL(v0, v1, v2, u[0], m[34]);+	MONT_MUL(w0, w1, w2, u[1], m[33]);+	MONT_MAC(t0, t1, t2, u[2], m[32]);+	MONT_MAC(v0, v1, v2, u[3], m[31]);+	MONT_MAC(w0, w1, w2, u[4], m[30]);+	MONT_MAC(t0, t1, t2, u[5], m[29]);+	MONT_MAC(v0, v1, v2, u[6], m[28]);+	MONT_MAC(w0, w1, w2, u[7], m[27]);+	MONT_MAC(t0, t1, t2, u[8], m[26]);+	MONT_MAC(v0, v1, v2, u[9], m[25]);+	MONT_MAC(w0, w1, w2, u[10], m[24]);+	MONT_MAC(t0, t1, t2, u[11], m[23]);+	MONT_MAC(v0, v1, v2, u[12], m[22]);+	MONT_MAC(w0, w1, w2, u[13], m[21]);+	MONT_MAC(t0, t1, t2, u[14], m[20]);+	MONT_MAC(v0, v1, v2, u[15], m[19]);+	MONT_MAC(w0, w1, w2, u[16], m[18]);+	MONT_MAC(t0, t1, t2, u[17], m[17]);+	MONT_MAC(v0, v1, v2, u[18], m[16]);+	MONT_MAC(w0, w1, w2, u[19], m[15]);+	MONT_MAC(t0, t1, t2, u[20], m[14]);+	MONT_MAC(v0, v1, v2, u[21], m[13]);+	MONT_MAC(w0, w1, w2, u[22], m[12]);+	MONT_MAC(t0, t1, t2, u[23], m[11]);+	MONT_MAC(v0, v1, v2, u[24], m[10]);+	MONT_MAC(w0, w1, w2, u[25], m[9]);+	MONT_MAC(t0, t1, t2, u[26], m[8]);+	MONT_MAC(v0, v1, v2, u[27], m[7]);+	MONT_MAC(w0, w1, w2, u[28], m[6]);+	MONT_MAC(t0, t1, t2, u[29], m[5]);+	MONT_MAC(v0, v1, v2, u[30], m[4]);+	MONT_MAC(w0, w1, w2, u[31], m[3]);+	MONT_MAC(t0, t1, t2, u[32], m[2]);+	MONT_MAC(v0, v1, v2, u[33], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[34] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[34], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 35 */+	MONT_MUL(s0, s1, s2, a[0], a[35]);+	MONT_MUL(c0, c1, c2, a[1], a[34]);+	MONT_MAC(s0, s1, s2, a[2], a[33]);+	MONT_MAC(c0, c1, c2, a[3], a[32]);+	MONT_MAC(s0, s1, s2, a[4], a[31]);+	MONT_MAC(c0, c1, c2, a[5], a[30]);+	MONT_MAC(s0, s1, s2, a[6], a[29]);+	MONT_MAC(c0, c1, c2, a[7], a[28]);+	MONT_MAC(s0, s1, s2, a[8], a[27]);+	MONT_MAC(c0, c1, c2, a[9], a[26]);+	MONT_MAC(s0, s1, s2, a[10], a[25]);+	MONT_MAC(c0, c1, c2, a[11], a[24]);+	MONT_MAC(s0, s1, s2, a[12], a[23]);+	MONT_MAC(c0, c1, c2, a[13], a[22]);+	MONT_MAC(s0, s1, s2, a[14], a[21]);+	MONT_MAC(c0, c1, c2, a[15], a[20]);+	MONT_MAC(s0, s1, s2, a[16], a[19]);+	MONT_MAC(c0, c1, c2, a[17], a[18]);+	MONT_MAC(t0, t1, t2, u[0], m[35]);+	MONT_MUL(v0, v1, v2, u[1], m[34]);+	MONT_MUL(w0, w1, w2, u[2], m[33]);+	MONT_MAC(t0, t1, t2, u[3], m[32]);+	MONT_MAC(v0, v1, v2, u[4], m[31]);+	MONT_MAC(w0, w1, w2, u[5], m[30]);+	MONT_MAC(t0, t1, t2, u[6], m[29]);+	MONT_MAC(v0, v1, v2, u[7], m[28]);+	MONT_MAC(w0, w1, w2, u[8], m[27]);+	MONT_MAC(t0, t1, t2, u[9], m[26]);+	MONT_MAC(v0, v1, v2, u[10], m[25]);+	MONT_MAC(w0, w1, w2, u[11], m[24]);+	MONT_MAC(t0, t1, t2, u[12], m[23]);+	MONT_MAC(v0, v1, v2, u[13], m[22]);+	MONT_MAC(w0, w1, w2, u[14], m[21]);+	MONT_MAC(t0, t1, t2, u[15], m[20]);+	MONT_MAC(v0, v1, v2, u[16], m[19]);+	MONT_MAC(w0, w1, w2, u[17], m[18]);+	MONT_MAC(t0, t1, t2, u[18], m[17]);+	MONT_MAC(v0, v1, v2, u[19], m[16]);+	MONT_MAC(w0, w1, w2, u[20], m[15]);+	MONT_MAC(t0, t1, t2, u[21], m[14]);+	MONT_MAC(v0, v1, v2, u[22], m[13]);+	MONT_MAC(w0, w1, w2, u[23], m[12]);+	MONT_MAC(t0, t1, t2, u[24], m[11]);+	MONT_MAC(v0, v1, v2, u[25], m[10]);+	MONT_MAC(w0, w1, w2, u[26], m[9]);+	MONT_MAC(t0, t1, t2, u[27], m[8]);+	MONT_MAC(v0, v1, v2, u[28], m[7]);+	MONT_MAC(w0, w1, w2, u[29], m[6]);+	MONT_MAC(t0, t1, t2, u[30], m[5]);+	MONT_MAC(v0, v1, v2, u[31], m[4]);+	MONT_MAC(w0, w1, w2, u[32], m[3]);+	MONT_MAC(t0, t1, t2, u[33], m[2]);+	MONT_MAC(v0, v1, v2, u[34], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[35] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[35], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 36 */+	MONT_MUL(s0, s1, s2, a[0], a[36]);+	MONT_MUL(c0, c1, c2, a[1], a[35]);+	MONT_MAC(s0, s1, s2, a[2], a[34]);+	MONT_MAC(c0, c1, c2, a[3], a[33]);+	MONT_MAC(s0, s1, s2, a[4], a[32]);+	MONT_MAC(c0, c1, c2, a[5], a[31]);+	MONT_MAC(s0, s1, s2, a[6], a[30]);+	MONT_MAC(c0, c1, c2, a[7], a[29]);+	MONT_MAC(s0, s1, s2, a[8], a[28]);+	MONT_MAC(c0, c1, c2, a[9], a[27]);+	MONT_MAC(s0, s1, s2, a[10], a[26]);+	MONT_MAC(c0, c1, c2, a[11], a[25]);+	MONT_MAC(s0, s1, s2, a[12], a[24]);+	MONT_MAC(c0, c1, c2, a[13], a[23]);+	MONT_MAC(s0, s1, s2, a[14], a[22]);+	MONT_MAC(c0, c1, c2, a[15], a[21]);+	MONT_MAC(s0, s1, s2, a[16], a[20]);+	MONT_MAC(c0, c1, c2, a[17], a[19]);+	MONT_MAC(t0, t1, t2, a[18], a[18]);+	MONT_MUL(v0, v1, v2, u[0], m[36]);+	MONT_MUL(w0, w1, w2, u[1], m[35]);+	MONT_MAC(t0, t1, t2, u[2], m[34]);+	MONT_MAC(v0, v1, v2, u[3], m[33]);+	MONT_MAC(w0, w1, w2, u[4], m[32]);+	MONT_MAC(t0, t1, t2, u[5], m[31]);+	MONT_MAC(v0, v1, v2, u[6], m[30]);+	MONT_MAC(w0, w1, w2, u[7], m[29]);+	MONT_MAC(t0, t1, t2, u[8], m[28]);+	MONT_MAC(v0, v1, v2, u[9], m[27]);+	MONT_MAC(w0, w1, w2, u[10], m[26]);+	MONT_MAC(t0, t1, t2, u[11], m[25]);+	MONT_MAC(v0, v1, v2, u[12], m[24]);+	MONT_MAC(w0, w1, w2, u[13], m[23]);+	MONT_MAC(t0, t1, t2, u[14], m[22]);+	MONT_MAC(v0, v1, v2, u[15], m[21]);+	MONT_MAC(w0, w1, w2, u[16], m[20]);+	MONT_MAC(t0, t1, t2, u[17], m[19]);+	MONT_MAC(v0, v1, v2, u[18], m[18]);+	MONT_MAC(w0, w1, w2, u[19], m[17]);+	MONT_MAC(t0, t1, t2, u[20], m[16]);+	MONT_MAC(v0, v1, v2, u[21], m[15]);+	MONT_MAC(w0, w1, w2, u[22], m[14]);+	MONT_MAC(t0, t1, t2, u[23], m[13]);+	MONT_MAC(v0, v1, v2, u[24], m[12]);+	MONT_MAC(w0, w1, w2, u[25], m[11]);+	MONT_MAC(t0, t1, t2, u[26], m[10]);+	MONT_MAC(v0, v1, v2, u[27], m[9]);+	MONT_MAC(w0, w1, w2, u[28], m[8]);+	MONT_MAC(t0, t1, t2, u[29], m[7]);+	MONT_MAC(v0, v1, v2, u[30], m[6]);+	MONT_MAC(w0, w1, w2, u[31], m[5]);+	MONT_MAC(t0, t1, t2, u[32], m[4]);+	MONT_MAC(v0, v1, v2, u[33], m[3]);+	MONT_MAC(w0, w1, w2, u[34], m[2]);+	MONT_MAC(t0, t1, t2, u[35], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[36] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[36], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 37 */+	MONT_MUL(s0, s1, s2, a[0], a[37]);+	MONT_MUL(c0, c1, c2, a[1], a[36]);+	MONT_MAC(s0, s1, s2, a[2], a[35]);+	MONT_MAC(c0, c1, c2, a[3], a[34]);+	MONT_MAC(s0, s1, s2, a[4], a[33]);+	MONT_MAC(c0, c1, c2, a[5], a[32]);+	MONT_MAC(s0, s1, s2, a[6], a[31]);+	MONT_MAC(c0, c1, c2, a[7], a[30]);+	MONT_MAC(s0, s1, s2, a[8], a[29]);+	MONT_MAC(c0, c1, c2, a[9], a[28]);+	MONT_MAC(s0, s1, s2, a[10], a[27]);+	MONT_MAC(c0, c1, c2, a[11], a[26]);+	MONT_MAC(s0, s1, s2, a[12], a[25]);+	MONT_MAC(c0, c1, c2, a[13], a[24]);+	MONT_MAC(s0, s1, s2, a[14], a[23]);+	MONT_MAC(c0, c1, c2, a[15], a[22]);+	MONT_MAC(s0, s1, s2, a[16], a[21]);+	MONT_MAC(c0, c1, c2, a[17], a[20]);+	MONT_MAC(s0, s1, s2, a[18], a[19]);+	MONT_MAC(t0, t1, t2, u[0], m[37]);+	MONT_MUL(v0, v1, v2, u[1], m[36]);+	MONT_MUL(w0, w1, w2, u[2], m[35]);+	MONT_MAC(t0, t1, t2, u[3], m[34]);+	MONT_MAC(v0, v1, v2, u[4], m[33]);+	MONT_MAC(w0, w1, w2, u[5], m[32]);+	MONT_MAC(t0, t1, t2, u[6], m[31]);+	MONT_MAC(v0, v1, v2, u[7], m[30]);+	MONT_MAC(w0, w1, w2, u[8], m[29]);+	MONT_MAC(t0, t1, t2, u[9], m[28]);+	MONT_MAC(v0, v1, v2, u[10], m[27]);+	MONT_MAC(w0, w1, w2, u[11], m[26]);+	MONT_MAC(t0, t1, t2, u[12], m[25]);+	MONT_MAC(v0, v1, v2, u[13], m[24]);+	MONT_MAC(w0, w1, w2, u[14], m[23]);+	MONT_MAC(t0, t1, t2, u[15], m[22]);+	MONT_MAC(v0, v1, v2, u[16], m[21]);+	MONT_MAC(w0, w1, w2, u[17], m[20]);+	MONT_MAC(t0, t1, t2, u[18], m[19]);+	MONT_MAC(v0, v1, v2, u[19], m[18]);+	MONT_MAC(w0, w1, w2, u[20], m[17]);+	MONT_MAC(t0, t1, t2, u[21], m[16]);+	MONT_MAC(v0, v1, v2, u[22], m[15]);+	MONT_MAC(w0, w1, w2, u[23], m[14]);+	MONT_MAC(t0, t1, t2, u[24], m[13]);+	MONT_MAC(v0, v1, v2, u[25], m[12]);+	MONT_MAC(w0, w1, w2, u[26], m[11]);+	MONT_MAC(t0, t1, t2, u[27], m[10]);+	MONT_MAC(v0, v1, v2, u[28], m[9]);+	MONT_MAC(w0, w1, w2, u[29], m[8]);+	MONT_MAC(t0, t1, t2, u[30], m[7]);+	MONT_MAC(v0, v1, v2, u[31], m[6]);+	MONT_MAC(w0, w1, w2, u[32], m[5]);+	MONT_MAC(t0, t1, t2, u[33], m[4]);+	MONT_MAC(v0, v1, v2, u[34], m[3]);+	MONT_MAC(w0, w1, w2, u[35], m[2]);+	MONT_MAC(t0, t1, t2, u[36], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[37] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[37], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 38 */+	MONT_MUL(s0, s1, s2, a[0], a[38]);+	MONT_MUL(c0, c1, c2, a[1], a[37]);+	MONT_MAC(s0, s1, s2, a[2], a[36]);+	MONT_MAC(c0, c1, c2, a[3], a[35]);+	MONT_MAC(s0, s1, s2, a[4], a[34]);+	MONT_MAC(c0, c1, c2, a[5], a[33]);+	MONT_MAC(s0, s1, s2, a[6], a[32]);+	MONT_MAC(c0, c1, c2, a[7], a[31]);+	MONT_MAC(s0, s1, s2, a[8], a[30]);+	MONT_MAC(c0, c1, c2, a[9], a[29]);+	MONT_MAC(s0, s1, s2, a[10], a[28]);+	MONT_MAC(c0, c1, c2, a[11], a[27]);+	MONT_MAC(s0, s1, s2, a[12], a[26]);+	MONT_MAC(c0, c1, c2, a[13], a[25]);+	MONT_MAC(s0, s1, s2, a[14], a[24]);+	MONT_MAC(c0, c1, c2, a[15], a[23]);+	MONT_MAC(s0, s1, s2, a[16], a[22]);+	MONT_MAC(c0, c1, c2, a[17], a[21]);+	MONT_MAC(s0, s1, s2, a[18], a[20]);+	MONT_MAC(t0, t1, t2, a[19], a[19]);+	MONT_MUL(v0, v1, v2, u[0], m[38]);+	MONT_MUL(w0, w1, w2, u[1], m[37]);+	MONT_MAC(t0, t1, t2, u[2], m[36]);+	MONT_MAC(v0, v1, v2, u[3], m[35]);+	MONT_MAC(w0, w1, w2, u[4], m[34]);+	MONT_MAC(t0, t1, t2, u[5], m[33]);+	MONT_MAC(v0, v1, v2, u[6], m[32]);+	MONT_MAC(w0, w1, w2, u[7], m[31]);+	MONT_MAC(t0, t1, t2, u[8], m[30]);+	MONT_MAC(v0, v1, v2, u[9], m[29]);+	MONT_MAC(w0, w1, w2, u[10], m[28]);+	MONT_MAC(t0, t1, t2, u[11], m[27]);+	MONT_MAC(v0, v1, v2, u[12], m[26]);+	MONT_MAC(w0, w1, w2, u[13], m[25]);+	MONT_MAC(t0, t1, t2, u[14], m[24]);+	MONT_MAC(v0, v1, v2, u[15], m[23]);+	MONT_MAC(w0, w1, w2, u[16], m[22]);+	MONT_MAC(t0, t1, t2, u[17], m[21]);+	MONT_MAC(v0, v1, v2, u[18], m[20]);+	MONT_MAC(w0, w1, w2, u[19], m[19]);+	MONT_MAC(t0, t1, t2, u[20], m[18]);+	MONT_MAC(v0, v1, v2, u[21], m[17]);+	MONT_MAC(w0, w1, w2, u[22], m[16]);+	MONT_MAC(t0, t1, t2, u[23], m[15]);+	MONT_MAC(v0, v1, v2, u[24], m[14]);+	MONT_MAC(w0, w1, w2, u[25], m[13]);+	MONT_MAC(t0, t1, t2, u[26], m[12]);+	MONT_MAC(v0, v1, v2, u[27], m[11]);+	MONT_MAC(w0, w1, w2, u[28], m[10]);+	MONT_MAC(t0, t1, t2, u[29], m[9]);+	MONT_MAC(v0, v1, v2, u[30], m[8]);+	MONT_MAC(w0, w1, w2, u[31], m[7]);+	MONT_MAC(t0, t1, t2, u[32], m[6]);+	MONT_MAC(v0, v1, v2, u[33], m[5]);+	MONT_MAC(w0, w1, w2, u[34], m[4]);+	MONT_MAC(t0, t1, t2, u[35], m[3]);+	MONT_MAC(v0, v1, v2, u[36], m[2]);+	MONT_MAC(w0, w1, w2, u[37], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[38] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[38], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 39 */+	MONT_MUL(s0, s1, s2, a[0], a[39]);+	MONT_MUL(c0, c1, c2, a[1], a[38]);+	MONT_MAC(s0, s1, s2, a[2], a[37]);+	MONT_MAC(c0, c1, c2, a[3], a[36]);+	MONT_MAC(s0, s1, s2, a[4], a[35]);+	MONT_MAC(c0, c1, c2, a[5], a[34]);+	MONT_MAC(s0, s1, s2, a[6], a[33]);+	MONT_MAC(c0, c1, c2, a[7], a[32]);+	MONT_MAC(s0, s1, s2, a[8], a[31]);+	MONT_MAC(c0, c1, c2, a[9], a[30]);+	MONT_MAC(s0, s1, s2, a[10], a[29]);+	MONT_MAC(c0, c1, c2, a[11], a[28]);+	MONT_MAC(s0, s1, s2, a[12], a[27]);+	MONT_MAC(c0, c1, c2, a[13], a[26]);+	MONT_MAC(s0, s1, s2, a[14], a[25]);+	MONT_MAC(c0, c1, c2, a[15], a[24]);+	MONT_MAC(s0, s1, s2, a[16], a[23]);+	MONT_MAC(c0, c1, c2, a[17], a[22]);+	MONT_MAC(s0, s1, s2, a[18], a[21]);+	MONT_MAC(c0, c1, c2, a[19], a[20]);+	MONT_MAC(t0, t1, t2, u[0], m[39]);+	MONT_MUL(v0, v1, v2, u[1], m[38]);+	MONT_MUL(w0, w1, w2, u[2], m[37]);+	MONT_MAC(t0, t1, t2, u[3], m[36]);+	MONT_MAC(v0, v1, v2, u[4], m[35]);+	MONT_MAC(w0, w1, w2, u[5], m[34]);+	MONT_MAC(t0, t1, t2, u[6], m[33]);+	MONT_MAC(v0, v1, v2, u[7], m[32]);+	MONT_MAC(w0, w1, w2, u[8], m[31]);+	MONT_MAC(t0, t1, t2, u[9], m[30]);+	MONT_MAC(v0, v1, v2, u[10], m[29]);+	MONT_MAC(w0, w1, w2, u[11], m[28]);+	MONT_MAC(t0, t1, t2, u[12], m[27]);+	MONT_MAC(v0, v1, v2, u[13], m[26]);+	MONT_MAC(w0, w1, w2, u[14], m[25]);+	MONT_MAC(t0, t1, t2, u[15], m[24]);+	MONT_MAC(v0, v1, v2, u[16], m[23]);+	MONT_MAC(w0, w1, w2, u[17], m[22]);+	MONT_MAC(t0, t1, t2, u[18], m[21]);+	MONT_MAC(v0, v1, v2, u[19], m[20]);+	MONT_MAC(w0, w1, w2, u[20], m[19]);+	MONT_MAC(t0, t1, t2, u[21], m[18]);+	MONT_MAC(v0, v1, v2, u[22], m[17]);+	MONT_MAC(w0, w1, w2, u[23], m[16]);+	MONT_MAC(t0, t1, t2, u[24], m[15]);+	MONT_MAC(v0, v1, v2, u[25], m[14]);+	MONT_MAC(w0, w1, w2, u[26], m[13]);+	MONT_MAC(t0, t1, t2, u[27], m[12]);+	MONT_MAC(v0, v1, v2, u[28], m[11]);+	MONT_MAC(w0, w1, w2, u[29], m[10]);+	MONT_MAC(t0, t1, t2, u[30], m[9]);+	MONT_MAC(v0, v1, v2, u[31], m[8]);+	MONT_MAC(w0, w1, w2, u[32], m[7]);+	MONT_MAC(t0, t1, t2, u[33], m[6]);+	MONT_MAC(v0, v1, v2, u[34], m[5]);+	MONT_MAC(w0, w1, w2, u[35], m[4]);+	MONT_MAC(t0, t1, t2, u[36], m[3]);+	MONT_MAC(v0, v1, v2, u[37], m[2]);+	MONT_MAC(w0, w1, w2, u[38], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[39] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[39], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 40 */+	MONT_MUL(s0, s1, s2, a[0], a[40]);+	MONT_MUL(c0, c1, c2, a[1], a[39]);+	MONT_MAC(s0, s1, s2, a[2], a[38]);+	MONT_MAC(c0, c1, c2, a[3], a[37]);+	MONT_MAC(s0, s1, s2, a[4], a[36]);+	MONT_MAC(c0, c1, c2, a[5], a[35]);+	MONT_MAC(s0, s1, s2, a[6], a[34]);+	MONT_MAC(c0, c1, c2, a[7], a[33]);+	MONT_MAC(s0, s1, s2, a[8], a[32]);+	MONT_MAC(c0, c1, c2, a[9], a[31]);+	MONT_MAC(s0, s1, s2, a[10], a[30]);+	MONT_MAC(c0, c1, c2, a[11], a[29]);+	MONT_MAC(s0, s1, s2, a[12], a[28]);+	MONT_MAC(c0, c1, c2, a[13], a[27]);+	MONT_MAC(s0, s1, s2, a[14], a[26]);+	MONT_MAC(c0, c1, c2, a[15], a[25]);+	MONT_MAC(s0, s1, s2, a[16], a[24]);+	MONT_MAC(c0, c1, c2, a[17], a[23]);+	MONT_MAC(s0, s1, s2, a[18], a[22]);+	MONT_MAC(c0, c1, c2, a[19], a[21]);+	MONT_MAC(t0, t1, t2, a[20], a[20]);+	MONT_MUL(v0, v1, v2, u[0], m[40]);+	MONT_MUL(w0, w1, w2, u[1], m[39]);+	MONT_MAC(t0, t1, t2, u[2], m[38]);+	MONT_MAC(v0, v1, v2, u[3], m[37]);+	MONT_MAC(w0, w1, w2, u[4], m[36]);+	MONT_MAC(t0, t1, t2, u[5], m[35]);+	MONT_MAC(v0, v1, v2, u[6], m[34]);+	MONT_MAC(w0, w1, w2, u[7], m[33]);+	MONT_MAC(t0, t1, t2, u[8], m[32]);+	MONT_MAC(v0, v1, v2, u[9], m[31]);+	MONT_MAC(w0, w1, w2, u[10], m[30]);+	MONT_MAC(t0, t1, t2, u[11], m[29]);+	MONT_MAC(v0, v1, v2, u[12], m[28]);+	MONT_MAC(w0, w1, w2, u[13], m[27]);+	MONT_MAC(t0, t1, t2, u[14], m[26]);+	MONT_MAC(v0, v1, v2, u[15], m[25]);+	MONT_MAC(w0, w1, w2, u[16], m[24]);+	MONT_MAC(t0, t1, t2, u[17], m[23]);+	MONT_MAC(v0, v1, v2, u[18], m[22]);+	MONT_MAC(w0, w1, w2, u[19], m[21]);+	MONT_MAC(t0, t1, t2, u[20], m[20]);+	MONT_MAC(v0, v1, v2, u[21], m[19]);+	MONT_MAC(w0, w1, w2, u[22], m[18]);+	MONT_MAC(t0, t1, t2, u[23], m[17]);+	MONT_MAC(v0, v1, v2, u[24], m[16]);+	MONT_MAC(w0, w1, w2, u[25], m[15]);+	MONT_MAC(t0, t1, t2, u[26], m[14]);+	MONT_MAC(v0, v1, v2, u[27], m[13]);+	MONT_MAC(w0, w1, w2, u[28], m[12]);+	MONT_MAC(t0, t1, t2, u[29], m[11]);+	MONT_MAC(v0, v1, v2, u[30], m[10]);+	MONT_MAC(w0, w1, w2, u[31], m[9]);+	MONT_MAC(t0, t1, t2, u[32], m[8]);+	MONT_MAC(v0, v1, v2, u[33], m[7]);+	MONT_MAC(w0, w1, w2, u[34], m[6]);+	MONT_MAC(t0, t1, t2, u[35], m[5]);+	MONT_MAC(v0, v1, v2, u[36], m[4]);+	MONT_MAC(w0, w1, w2, u[37], m[3]);+	MONT_MAC(t0, t1, t2, u[38], m[2]);+	MONT_MAC(v0, v1, v2, u[39], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[40] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[40], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 41 */+	MONT_MUL(s0, s1, s2, a[0], a[41]);+	MONT_MUL(c0, c1, c2, a[1], a[40]);+	MONT_MAC(s0, s1, s2, a[2], a[39]);+	MONT_MAC(c0, c1, c2, a[3], a[38]);+	MONT_MAC(s0, s1, s2, a[4], a[37]);+	MONT_MAC(c0, c1, c2, a[5], a[36]);+	MONT_MAC(s0, s1, s2, a[6], a[35]);+	MONT_MAC(c0, c1, c2, a[7], a[34]);+	MONT_MAC(s0, s1, s2, a[8], a[33]);+	MONT_MAC(c0, c1, c2, a[9], a[32]);+	MONT_MAC(s0, s1, s2, a[10], a[31]);+	MONT_MAC(c0, c1, c2, a[11], a[30]);+	MONT_MAC(s0, s1, s2, a[12], a[29]);+	MONT_MAC(c0, c1, c2, a[13], a[28]);+	MONT_MAC(s0, s1, s2, a[14], a[27]);+	MONT_MAC(c0, c1, c2, a[15], a[26]);+	MONT_MAC(s0, s1, s2, a[16], a[25]);+	MONT_MAC(c0, c1, c2, a[17], a[24]);+	MONT_MAC(s0, s1, s2, a[18], a[23]);+	MONT_MAC(c0, c1, c2, a[19], a[22]);+	MONT_MAC(s0, s1, s2, a[20], a[21]);+	MONT_MAC(t0, t1, t2, u[0], m[41]);+	MONT_MUL(v0, v1, v2, u[1], m[40]);+	MONT_MUL(w0, w1, w2, u[2], m[39]);+	MONT_MAC(t0, t1, t2, u[3], m[38]);+	MONT_MAC(v0, v1, v2, u[4], m[37]);+	MONT_MAC(w0, w1, w2, u[5], m[36]);+	MONT_MAC(t0, t1, t2, u[6], m[35]);+	MONT_MAC(v0, v1, v2, u[7], m[34]);+	MONT_MAC(w0, w1, w2, u[8], m[33]);+	MONT_MAC(t0, t1, t2, u[9], m[32]);+	MONT_MAC(v0, v1, v2, u[10], m[31]);+	MONT_MAC(w0, w1, w2, u[11], m[30]);+	MONT_MAC(t0, t1, t2, u[12], m[29]);+	MONT_MAC(v0, v1, v2, u[13], m[28]);+	MONT_MAC(w0, w1, w2, u[14], m[27]);+	MONT_MAC(t0, t1, t2, u[15], m[26]);+	MONT_MAC(v0, v1, v2, u[16], m[25]);+	MONT_MAC(w0, w1, w2, u[17], m[24]);+	MONT_MAC(t0, t1, t2, u[18], m[23]);+	MONT_MAC(v0, v1, v2, u[19], m[22]);+	MONT_MAC(w0, w1, w2, u[20], m[21]);+	MONT_MAC(t0, t1, t2, u[21], m[20]);+	MONT_MAC(v0, v1, v2, u[22], m[19]);+	MONT_MAC(w0, w1, w2, u[23], m[18]);+	MONT_MAC(t0, t1, t2, u[24], m[17]);+	MONT_MAC(v0, v1, v2, u[25], m[16]);+	MONT_MAC(w0, w1, w2, u[26], m[15]);+	MONT_MAC(t0, t1, t2, u[27], m[14]);+	MONT_MAC(v0, v1, v2, u[28], m[13]);+	MONT_MAC(w0, w1, w2, u[29], m[12]);+	MONT_MAC(t0, t1, t2, u[30], m[11]);+	MONT_MAC(v0, v1, v2, u[31], m[10]);+	MONT_MAC(w0, w1, w2, u[32], m[9]);+	MONT_MAC(t0, t1, t2, u[33], m[8]);+	MONT_MAC(v0, v1, v2, u[34], m[7]);+	MONT_MAC(w0, w1, w2, u[35], m[6]);+	MONT_MAC(t0, t1, t2, u[36], m[5]);+	MONT_MAC(v0, v1, v2, u[37], m[4]);+	MONT_MAC(w0, w1, w2, u[38], m[3]);+	MONT_MAC(t0, t1, t2, u[39], m[2]);+	MONT_MAC(v0, v1, v2, u[40], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[41] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[41], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 42 */+	MONT_MUL(s0, s1, s2, a[0], a[42]);+	MONT_MUL(c0, c1, c2, a[1], a[41]);+	MONT_MAC(s0, s1, s2, a[2], a[40]);+	MONT_MAC(c0, c1, c2, a[3], a[39]);+	MONT_MAC(s0, s1, s2, a[4], a[38]);+	MONT_MAC(c0, c1, c2, a[5], a[37]);+	MONT_MAC(s0, s1, s2, a[6], a[36]);+	MONT_MAC(c0, c1, c2, a[7], a[35]);+	MONT_MAC(s0, s1, s2, a[8], a[34]);+	MONT_MAC(c0, c1, c2, a[9], a[33]);+	MONT_MAC(s0, s1, s2, a[10], a[32]);+	MONT_MAC(c0, c1, c2, a[11], a[31]);+	MONT_MAC(s0, s1, s2, a[12], a[30]);+	MONT_MAC(c0, c1, c2, a[13], a[29]);+	MONT_MAC(s0, s1, s2, a[14], a[28]);+	MONT_MAC(c0, c1, c2, a[15], a[27]);+	MONT_MAC(s0, s1, s2, a[16], a[26]);+	MONT_MAC(c0, c1, c2, a[17], a[25]);+	MONT_MAC(s0, s1, s2, a[18], a[24]);+	MONT_MAC(c0, c1, c2, a[19], a[23]);+	MONT_MAC(s0, s1, s2, a[20], a[22]);+	MONT_MAC(t0, t1, t2, a[21], a[21]);+	MONT_MUL(v0, v1, v2, u[0], m[42]);+	MONT_MUL(w0, w1, w2, u[1], m[41]);+	MONT_MAC(t0, t1, t2, u[2], m[40]);+	MONT_MAC(v0, v1, v2, u[3], m[39]);+	MONT_MAC(w0, w1, w2, u[4], m[38]);+	MONT_MAC(t0, t1, t2, u[5], m[37]);+	MONT_MAC(v0, v1, v2, u[6], m[36]);+	MONT_MAC(w0, w1, w2, u[7], m[35]);+	MONT_MAC(t0, t1, t2, u[8], m[34]);+	MONT_MAC(v0, v1, v2, u[9], m[33]);+	MONT_MAC(w0, w1, w2, u[10], m[32]);+	MONT_MAC(t0, t1, t2, u[11], m[31]);+	MONT_MAC(v0, v1, v2, u[12], m[30]);+	MONT_MAC(w0, w1, w2, u[13], m[29]);+	MONT_MAC(t0, t1, t2, u[14], m[28]);+	MONT_MAC(v0, v1, v2, u[15], m[27]);+	MONT_MAC(w0, w1, w2, u[16], m[26]);+	MONT_MAC(t0, t1, t2, u[17], m[25]);+	MONT_MAC(v0, v1, v2, u[18], m[24]);+	MONT_MAC(w0, w1, w2, u[19], m[23]);+	MONT_MAC(t0, t1, t2, u[20], m[22]);+	MONT_MAC(v0, v1, v2, u[21], m[21]);+	MONT_MAC(w0, w1, w2, u[22], m[20]);+	MONT_MAC(t0, t1, t2, u[23], m[19]);+	MONT_MAC(v0, v1, v2, u[24], m[18]);+	MONT_MAC(w0, w1, w2, u[25], m[17]);+	MONT_MAC(t0, t1, t2, u[26], m[16]);+	MONT_MAC(v0, v1, v2, u[27], m[15]);+	MONT_MAC(w0, w1, w2, u[28], m[14]);+	MONT_MAC(t0, t1, t2, u[29], m[13]);+	MONT_MAC(v0, v1, v2, u[30], m[12]);+	MONT_MAC(w0, w1, w2, u[31], m[11]);+	MONT_MAC(t0, t1, t2, u[32], m[10]);+	MONT_MAC(v0, v1, v2, u[33], m[9]);+	MONT_MAC(w0, w1, w2, u[34], m[8]);+	MONT_MAC(t0, t1, t2, u[35], m[7]);+	MONT_MAC(v0, v1, v2, u[36], m[6]);+	MONT_MAC(w0, w1, w2, u[37], m[5]);+	MONT_MAC(t0, t1, t2, u[38], m[4]);+	MONT_MAC(v0, v1, v2, u[39], m[3]);+	MONT_MAC(w0, w1, w2, u[40], m[2]);+	MONT_MAC(t0, t1, t2, u[41], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[42] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[42], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 43 */+	MONT_MUL(s0, s1, s2, a[0], a[43]);+	MONT_MUL(c0, c1, c2, a[1], a[42]);+	MONT_MAC(s0, s1, s2, a[2], a[41]);+	MONT_MAC(c0, c1, c2, a[3], a[40]);+	MONT_MAC(s0, s1, s2, a[4], a[39]);+	MONT_MAC(c0, c1, c2, a[5], a[38]);+	MONT_MAC(s0, s1, s2, a[6], a[37]);+	MONT_MAC(c0, c1, c2, a[7], a[36]);+	MONT_MAC(s0, s1, s2, a[8], a[35]);+	MONT_MAC(c0, c1, c2, a[9], a[34]);+	MONT_MAC(s0, s1, s2, a[10], a[33]);+	MONT_MAC(c0, c1, c2, a[11], a[32]);+	MONT_MAC(s0, s1, s2, a[12], a[31]);+	MONT_MAC(c0, c1, c2, a[13], a[30]);+	MONT_MAC(s0, s1, s2, a[14], a[29]);+	MONT_MAC(c0, c1, c2, a[15], a[28]);+	MONT_MAC(s0, s1, s2, a[16], a[27]);+	MONT_MAC(c0, c1, c2, a[17], a[26]);+	MONT_MAC(s0, s1, s2, a[18], a[25]);+	MONT_MAC(c0, c1, c2, a[19], a[24]);+	MONT_MAC(s0, s1, s2, a[20], a[23]);+	MONT_MAC(c0, c1, c2, a[21], a[22]);+	MONT_MAC(t0, t1, t2, u[0], m[43]);+	MONT_MUL(v0, v1, v2, u[1], m[42]);+	MONT_MUL(w0, w1, w2, u[2], m[41]);+	MONT_MAC(t0, t1, t2, u[3], m[40]);+	MONT_MAC(v0, v1, v2, u[4], m[39]);+	MONT_MAC(w0, w1, w2, u[5], m[38]);+	MONT_MAC(t0, t1, t2, u[6], m[37]);+	MONT_MAC(v0, v1, v2, u[7], m[36]);+	MONT_MAC(w0, w1, w2, u[8], m[35]);+	MONT_MAC(t0, t1, t2, u[9], m[34]);+	MONT_MAC(v0, v1, v2, u[10], m[33]);+	MONT_MAC(w0, w1, w2, u[11], m[32]);+	MONT_MAC(t0, t1, t2, u[12], m[31]);+	MONT_MAC(v0, v1, v2, u[13], m[30]);+	MONT_MAC(w0, w1, w2, u[14], m[29]);+	MONT_MAC(t0, t1, t2, u[15], m[28]);+	MONT_MAC(v0, v1, v2, u[16], m[27]);+	MONT_MAC(w0, w1, w2, u[17], m[26]);+	MONT_MAC(t0, t1, t2, u[18], m[25]);+	MONT_MAC(v0, v1, v2, u[19], m[24]);+	MONT_MAC(w0, w1, w2, u[20], m[23]);+	MONT_MAC(t0, t1, t2, u[21], m[22]);+	MONT_MAC(v0, v1, v2, u[22], m[21]);+	MONT_MAC(w0, w1, w2, u[23], m[20]);+	MONT_MAC(t0, t1, t2, u[24], m[19]);+	MONT_MAC(v0, v1, v2, u[25], m[18]);+	MONT_MAC(w0, w1, w2, u[26], m[17]);+	MONT_MAC(t0, t1, t2, u[27], m[16]);+	MONT_MAC(v0, v1, v2, u[28], m[15]);+	MONT_MAC(w0, w1, w2, u[29], m[14]);+	MONT_MAC(t0, t1, t2, u[30], m[13]);+	MONT_MAC(v0, v1, v2, u[31], m[12]);+	MONT_MAC(w0, w1, w2, u[32], m[11]);+	MONT_MAC(t0, t1, t2, u[33], m[10]);+	MONT_MAC(v0, v1, v2, u[34], m[9]);+	MONT_MAC(w0, w1, w2, u[35], m[8]);+	MONT_MAC(t0, t1, t2, u[36], m[7]);+	MONT_MAC(v0, v1, v2, u[37], m[6]);+	MONT_MAC(w0, w1, w2, u[38], m[5]);+	MONT_MAC(t0, t1, t2, u[39], m[4]);+	MONT_MAC(v0, v1, v2, u[40], m[3]);+	MONT_MAC(w0, w1, w2, u[41], m[2]);+	MONT_MAC(t0, t1, t2, u[42], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[43] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[43], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 44 */+	MONT_MUL(s0, s1, s2, a[0], a[44]);+	MONT_MUL(c0, c1, c2, a[1], a[43]);+	MONT_MAC(s0, s1, s2, a[2], a[42]);+	MONT_MAC(c0, c1, c2, a[3], a[41]);+	MONT_MAC(s0, s1, s2, a[4], a[40]);+	MONT_MAC(c0, c1, c2, a[5], a[39]);+	MONT_MAC(s0, s1, s2, a[6], a[38]);+	MONT_MAC(c0, c1, c2, a[7], a[37]);+	MONT_MAC(s0, s1, s2, a[8], a[36]);+	MONT_MAC(c0, c1, c2, a[9], a[35]);+	MONT_MAC(s0, s1, s2, a[10], a[34]);+	MONT_MAC(c0, c1, c2, a[11], a[33]);+	MONT_MAC(s0, s1, s2, a[12], a[32]);+	MONT_MAC(c0, c1, c2, a[13], a[31]);+	MONT_MAC(s0, s1, s2, a[14], a[30]);+	MONT_MAC(c0, c1, c2, a[15], a[29]);+	MONT_MAC(s0, s1, s2, a[16], a[28]);+	MONT_MAC(c0, c1, c2, a[17], a[27]);+	MONT_MAC(s0, s1, s2, a[18], a[26]);+	MONT_MAC(c0, c1, c2, a[19], a[25]);+	MONT_MAC(s0, s1, s2, a[20], a[24]);+	MONT_MAC(c0, c1, c2, a[21], a[23]);+	MONT_MAC(t0, t1, t2, a[22], a[22]);+	MONT_MUL(v0, v1, v2, u[0], m[44]);+	MONT_MUL(w0, w1, w2, u[1], m[43]);+	MONT_MAC(t0, t1, t2, u[2], m[42]);+	MONT_MAC(v0, v1, v2, u[3], m[41]);+	MONT_MAC(w0, w1, w2, u[4], m[40]);+	MONT_MAC(t0, t1, t2, u[5], m[39]);+	MONT_MAC(v0, v1, v2, u[6], m[38]);+	MONT_MAC(w0, w1, w2, u[7], m[37]);+	MONT_MAC(t0, t1, t2, u[8], m[36]);+	MONT_MAC(v0, v1, v2, u[9], m[35]);+	MONT_MAC(w0, w1, w2, u[10], m[34]);+	MONT_MAC(t0, t1, t2, u[11], m[33]);+	MONT_MAC(v0, v1, v2, u[12], m[32]);+	MONT_MAC(w0, w1, w2, u[13], m[31]);+	MONT_MAC(t0, t1, t2, u[14], m[30]);+	MONT_MAC(v0, v1, v2, u[15], m[29]);+	MONT_MAC(w0, w1, w2, u[16], m[28]);+	MONT_MAC(t0, t1, t2, u[17], m[27]);+	MONT_MAC(v0, v1, v2, u[18], m[26]);+	MONT_MAC(w0, w1, w2, u[19], m[25]);+	MONT_MAC(t0, t1, t2, u[20], m[24]);+	MONT_MAC(v0, v1, v2, u[21], m[23]);+	MONT_MAC(w0, w1, w2, u[22], m[22]);+	MONT_MAC(t0, t1, t2, u[23], m[21]);+	MONT_MAC(v0, v1, v2, u[24], m[20]);+	MONT_MAC(w0, w1, w2, u[25], m[19]);+	MONT_MAC(t0, t1, t2, u[26], m[18]);+	MONT_MAC(v0, v1, v2, u[27], m[17]);+	MONT_MAC(w0, w1, w2, u[28], m[16]);+	MONT_MAC(t0, t1, t2, u[29], m[15]);+	MONT_MAC(v0, v1, v2, u[30], m[14]);+	MONT_MAC(w0, w1, w2, u[31], m[13]);+	MONT_MAC(t0, t1, t2, u[32], m[12]);+	MONT_MAC(v0, v1, v2, u[33], m[11]);+	MONT_MAC(w0, w1, w2, u[34], m[10]);+	MONT_MAC(t0, t1, t2, u[35], m[9]);+	MONT_MAC(v0, v1, v2, u[36], m[8]);+	MONT_MAC(w0, w1, w2, u[37], m[7]);+	MONT_MAC(t0, t1, t2, u[38], m[6]);+	MONT_MAC(v0, v1, v2, u[39], m[5]);+	MONT_MAC(w0, w1, w2, u[40], m[4]);+	MONT_MAC(t0, t1, t2, u[41], m[3]);+	MONT_MAC(v0, v1, v2, u[42], m[2]);+	MONT_MAC(w0, w1, w2, u[43], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[44] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[44], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 45 */+	MONT_MUL(s0, s1, s2, a[0], a[45]);+	MONT_MUL(c0, c1, c2, a[1], a[44]);+	MONT_MAC(s0, s1, s2, a[2], a[43]);+	MONT_MAC(c0, c1, c2, a[3], a[42]);+	MONT_MAC(s0, s1, s2, a[4], a[41]);+	MONT_MAC(c0, c1, c2, a[5], a[40]);+	MONT_MAC(s0, s1, s2, a[6], a[39]);+	MONT_MAC(c0, c1, c2, a[7], a[38]);+	MONT_MAC(s0, s1, s2, a[8], a[37]);+	MONT_MAC(c0, c1, c2, a[9], a[36]);+	MONT_MAC(s0, s1, s2, a[10], a[35]);+	MONT_MAC(c0, c1, c2, a[11], a[34]);+	MONT_MAC(s0, s1, s2, a[12], a[33]);+	MONT_MAC(c0, c1, c2, a[13], a[32]);+	MONT_MAC(s0, s1, s2, a[14], a[31]);+	MONT_MAC(c0, c1, c2, a[15], a[30]);+	MONT_MAC(s0, s1, s2, a[16], a[29]);+	MONT_MAC(c0, c1, c2, a[17], a[28]);+	MONT_MAC(s0, s1, s2, a[18], a[27]);+	MONT_MAC(c0, c1, c2, a[19], a[26]);+	MONT_MAC(s0, s1, s2, a[20], a[25]);+	MONT_MAC(c0, c1, c2, a[21], a[24]);+	MONT_MAC(s0, s1, s2, a[22], a[23]);+	MONT_MAC(t0, t1, t2, u[0], m[45]);+	MONT_MUL(v0, v1, v2, u[1], m[44]);+	MONT_MUL(w0, w1, w2, u[2], m[43]);+	MONT_MAC(t0, t1, t2, u[3], m[42]);+	MONT_MAC(v0, v1, v2, u[4], m[41]);+	MONT_MAC(w0, w1, w2, u[5], m[40]);+	MONT_MAC(t0, t1, t2, u[6], m[39]);+	MONT_MAC(v0, v1, v2, u[7], m[38]);+	MONT_MAC(w0, w1, w2, u[8], m[37]);+	MONT_MAC(t0, t1, t2, u[9], m[36]);+	MONT_MAC(v0, v1, v2, u[10], m[35]);+	MONT_MAC(w0, w1, w2, u[11], m[34]);+	MONT_MAC(t0, t1, t2, u[12], m[33]);+	MONT_MAC(v0, v1, v2, u[13], m[32]);+	MONT_MAC(w0, w1, w2, u[14], m[31]);+	MONT_MAC(t0, t1, t2, u[15], m[30]);+	MONT_MAC(v0, v1, v2, u[16], m[29]);+	MONT_MAC(w0, w1, w2, u[17], m[28]);+	MONT_MAC(t0, t1, t2, u[18], m[27]);+	MONT_MAC(v0, v1, v2, u[19], m[26]);+	MONT_MAC(w0, w1, w2, u[20], m[25]);+	MONT_MAC(t0, t1, t2, u[21], m[24]);+	MONT_MAC(v0, v1, v2, u[22], m[23]);+	MONT_MAC(w0, w1, w2, u[23], m[22]);+	MONT_MAC(t0, t1, t2, u[24], m[21]);+	MONT_MAC(v0, v1, v2, u[25], m[20]);+	MONT_MAC(w0, w1, w2, u[26], m[19]);+	MONT_MAC(t0, t1, t2, u[27], m[18]);+	MONT_MAC(v0, v1, v2, u[28], m[17]);+	MONT_MAC(w0, w1, w2, u[29], m[16]);+	MONT_MAC(t0, t1, t2, u[30], m[15]);+	MONT_MAC(v0, v1, v2, u[31], m[14]);+	MONT_MAC(w0, w1, w2, u[32], m[13]);+	MONT_MAC(t0, t1, t2, u[33], m[12]);+	MONT_MAC(v0, v1, v2, u[34], m[11]);+	MONT_MAC(w0, w1, w2, u[35], m[10]);+	MONT_MAC(t0, t1, t2, u[36], m[9]);+	MONT_MAC(v0, v1, v2, u[37], m[8]);+	MONT_MAC(w0, w1, w2, u[38], m[7]);+	MONT_MAC(t0, t1, t2, u[39], m[6]);+	MONT_MAC(v0, v1, v2, u[40], m[5]);+	MONT_MAC(w0, w1, w2, u[41], m[4]);+	MONT_MAC(t0, t1, t2, u[42], m[3]);+	MONT_MAC(v0, v1, v2, u[43], m[2]);+	MONT_MAC(w0, w1, w2, u[44], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[45] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[45], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 46 */+	MONT_MUL(s0, s1, s2, a[0], a[46]);+	MONT_MUL(c0, c1, c2, a[1], a[45]);+	MONT_MAC(s0, s1, s2, a[2], a[44]);+	MONT_MAC(c0, c1, c2, a[3], a[43]);+	MONT_MAC(s0, s1, s2, a[4], a[42]);+	MONT_MAC(c0, c1, c2, a[5], a[41]);+	MONT_MAC(s0, s1, s2, a[6], a[40]);+	MONT_MAC(c0, c1, c2, a[7], a[39]);+	MONT_MAC(s0, s1, s2, a[8], a[38]);+	MONT_MAC(c0, c1, c2, a[9], a[37]);+	MONT_MAC(s0, s1, s2, a[10], a[36]);+	MONT_MAC(c0, c1, c2, a[11], a[35]);+	MONT_MAC(s0, s1, s2, a[12], a[34]);+	MONT_MAC(c0, c1, c2, a[13], a[33]);+	MONT_MAC(s0, s1, s2, a[14], a[32]);+	MONT_MAC(c0, c1, c2, a[15], a[31]);+	MONT_MAC(s0, s1, s2, a[16], a[30]);+	MONT_MAC(c0, c1, c2, a[17], a[29]);+	MONT_MAC(s0, s1, s2, a[18], a[28]);+	MONT_MAC(c0, c1, c2, a[19], a[27]);+	MONT_MAC(s0, s1, s2, a[20], a[26]);+	MONT_MAC(c0, c1, c2, a[21], a[25]);+	MONT_MAC(s0, s1, s2, a[22], a[24]);+	MONT_MAC(t0, t1, t2, a[23], a[23]);+	MONT_MUL(v0, v1, v2, u[0], m[46]);+	MONT_MUL(w0, w1, w2, u[1], m[45]);+	MONT_MAC(t0, t1, t2, u[2], m[44]);+	MONT_MAC(v0, v1, v2, u[3], m[43]);+	MONT_MAC(w0, w1, w2, u[4], m[42]);+	MONT_MAC(t0, t1, t2, u[5], m[41]);+	MONT_MAC(v0, v1, v2, u[6], m[40]);+	MONT_MAC(w0, w1, w2, u[7], m[39]);+	MONT_MAC(t0, t1, t2, u[8], m[38]);+	MONT_MAC(v0, v1, v2, u[9], m[37]);+	MONT_MAC(w0, w1, w2, u[10], m[36]);+	MONT_MAC(t0, t1, t2, u[11], m[35]);+	MONT_MAC(v0, v1, v2, u[12], m[34]);+	MONT_MAC(w0, w1, w2, u[13], m[33]);+	MONT_MAC(t0, t1, t2, u[14], m[32]);+	MONT_MAC(v0, v1, v2, u[15], m[31]);+	MONT_MAC(w0, w1, w2, u[16], m[30]);+	MONT_MAC(t0, t1, t2, u[17], m[29]);+	MONT_MAC(v0, v1, v2, u[18], m[28]);+	MONT_MAC(w0, w1, w2, u[19], m[27]);+	MONT_MAC(t0, t1, t2, u[20], m[26]);+	MONT_MAC(v0, v1, v2, u[21], m[25]);+	MONT_MAC(w0, w1, w2, u[22], m[24]);+	MONT_MAC(t0, t1, t2, u[23], m[23]);+	MONT_MAC(v0, v1, v2, u[24], m[22]);+	MONT_MAC(w0, w1, w2, u[25], m[21]);+	MONT_MAC(t0, t1, t2, u[26], m[20]);+	MONT_MAC(v0, v1, v2, u[27], m[19]);+	MONT_MAC(w0, w1, w2, u[28], m[18]);+	MONT_MAC(t0, t1, t2, u[29], m[17]);+	MONT_MAC(v0, v1, v2, u[30], m[16]);+	MONT_MAC(w0, w1, w2, u[31], m[15]);+	MONT_MAC(t0, t1, t2, u[32], m[14]);+	MONT_MAC(v0, v1, v2, u[33], m[13]);+	MONT_MAC(w0, w1, w2, u[34], m[12]);+	MONT_MAC(t0, t1, t2, u[35], m[11]);+	MONT_MAC(v0, v1, v2, u[36], m[10]);+	MONT_MAC(w0, w1, w2, u[37], m[9]);+	MONT_MAC(t0, t1, t2, u[38], m[8]);+	MONT_MAC(v0, v1, v2, u[39], m[7]);+	MONT_MAC(w0, w1, w2, u[40], m[6]);+	MONT_MAC(t0, t1, t2, u[41], m[5]);+	MONT_MAC(v0, v1, v2, u[42], m[4]);+	MONT_MAC(w0, w1, w2, u[43], m[3]);+	MONT_MAC(t0, t1, t2, u[44], m[2]);+	MONT_MAC(v0, v1, v2, u[45], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[46] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[46], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 47 */+	MONT_MUL(s0, s1, s2, a[0], a[47]);+	MONT_MUL(c0, c1, c2, a[1], a[46]);+	MONT_MAC(s0, s1, s2, a[2], a[45]);+	MONT_MAC(c0, c1, c2, a[3], a[44]);+	MONT_MAC(s0, s1, s2, a[4], a[43]);+	MONT_MAC(c0, c1, c2, a[5], a[42]);+	MONT_MAC(s0, s1, s2, a[6], a[41]);+	MONT_MAC(c0, c1, c2, a[7], a[40]);+	MONT_MAC(s0, s1, s2, a[8], a[39]);+	MONT_MAC(c0, c1, c2, a[9], a[38]);+	MONT_MAC(s0, s1, s2, a[10], a[37]);+	MONT_MAC(c0, c1, c2, a[11], a[36]);+	MONT_MAC(s0, s1, s2, a[12], a[35]);+	MONT_MAC(c0, c1, c2, a[13], a[34]);+	MONT_MAC(s0, s1, s2, a[14], a[33]);+	MONT_MAC(c0, c1, c2, a[15], a[32]);+	MONT_MAC(s0, s1, s2, a[16], a[31]);+	MONT_MAC(c0, c1, c2, a[17], a[30]);+	MONT_MAC(s0, s1, s2, a[18], a[29]);+	MONT_MAC(c0, c1, c2, a[19], a[28]);+	MONT_MAC(s0, s1, s2, a[20], a[27]);+	MONT_MAC(c0, c1, c2, a[21], a[26]);+	MONT_MAC(s0, s1, s2, a[22], a[25]);+	MONT_MAC(c0, c1, c2, a[23], a[24]);+	MONT_MAC(t0, t1, t2, u[0], m[47]);+	MONT_MUL(v0, v1, v2, u[1], m[46]);+	MONT_MUL(w0, w1, w2, u[2], m[45]);+	MONT_MAC(t0, t1, t2, u[3], m[44]);+	MONT_MAC(v0, v1, v2, u[4], m[43]);+	MONT_MAC(w0, w1, w2, u[5], m[42]);+	MONT_MAC(t0, t1, t2, u[6], m[41]);+	MONT_MAC(v0, v1, v2, u[7], m[40]);+	MONT_MAC(w0, w1, w2, u[8], m[39]);+	MONT_MAC(t0, t1, t2, u[9], m[38]);+	MONT_MAC(v0, v1, v2, u[10], m[37]);+	MONT_MAC(w0, w1, w2, u[11], m[36]);+	MONT_MAC(t0, t1, t2, u[12], m[35]);+	MONT_MAC(v0, v1, v2, u[13], m[34]);+	MONT_MAC(w0, w1, w2, u[14], m[33]);+	MONT_MAC(t0, t1, t2, u[15], m[32]);+	MONT_MAC(v0, v1, v2, u[16], m[31]);+	MONT_MAC(w0, w1, w2, u[17], m[30]);+	MONT_MAC(t0, t1, t2, u[18], m[29]);+	MONT_MAC(v0, v1, v2, u[19], m[28]);+	MONT_MAC(w0, w1, w2, u[20], m[27]);+	MONT_MAC(t0, t1, t2, u[21], m[26]);+	MONT_MAC(v0, v1, v2, u[22], m[25]);+	MONT_MAC(w0, w1, w2, u[23], m[24]);+	MONT_MAC(t0, t1, t2, u[24], m[23]);+	MONT_MAC(v0, v1, v2, u[25], m[22]);+	MONT_MAC(w0, w1, w2, u[26], m[21]);+	MONT_MAC(t0, t1, t2, u[27], m[20]);+	MONT_MAC(v0, v1, v2, u[28], m[19]);+	MONT_MAC(w0, w1, w2, u[29], m[18]);+	MONT_MAC(t0, t1, t2, u[30], m[17]);+	MONT_MAC(v0, v1, v2, u[31], m[16]);+	MONT_MAC(w0, w1, w2, u[32], m[15]);+	MONT_MAC(t0, t1, t2, u[33], m[14]);+	MONT_MAC(v0, v1, v2, u[34], m[13]);+	MONT_MAC(w0, w1, w2, u[35], m[12]);+	MONT_MAC(t0, t1, t2, u[36], m[11]);+	MONT_MAC(v0, v1, v2, u[37], m[10]);+	MONT_MAC(w0, w1, w2, u[38], m[9]);+	MONT_MAC(t0, t1, t2, u[39], m[8]);+	MONT_MAC(v0, v1, v2, u[40], m[7]);+	MONT_MAC(w0, w1, w2, u[41], m[6]);+	MONT_MAC(t0, t1, t2, u[42], m[5]);+	MONT_MAC(v0, v1, v2, u[43], m[4]);+	MONT_MAC(w0, w1, w2, u[44], m[3]);+	MONT_MAC(t0, t1, t2, u[45], m[2]);+	MONT_MAC(v0, v1, v2, u[46], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	u[47] = t0 * n0;+	MONT_MAC(t0, t1, t2, u[47], m[0]);+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 48 */+	MONT_MUL(s0, s1, s2, a[1], a[47]);+	MONT_MUL(c0, c1, c2, a[2], a[46]);+	MONT_MAC(s0, s1, s2, a[3], a[45]);+	MONT_MAC(c0, c1, c2, a[4], a[44]);+	MONT_MAC(s0, s1, s2, a[5], a[43]);+	MONT_MAC(c0, c1, c2, a[6], a[42]);+	MONT_MAC(s0, s1, s2, a[7], a[41]);+	MONT_MAC(c0, c1, c2, a[8], a[40]);+	MONT_MAC(s0, s1, s2, a[9], a[39]);+	MONT_MAC(c0, c1, c2, a[10], a[38]);+	MONT_MAC(s0, s1, s2, a[11], a[37]);+	MONT_MAC(c0, c1, c2, a[12], a[36]);+	MONT_MAC(s0, s1, s2, a[13], a[35]);+	MONT_MAC(c0, c1, c2, a[14], a[34]);+	MONT_MAC(s0, s1, s2, a[15], a[33]);+	MONT_MAC(c0, c1, c2, a[16], a[32]);+	MONT_MAC(s0, s1, s2, a[17], a[31]);+	MONT_MAC(c0, c1, c2, a[18], a[30]);+	MONT_MAC(s0, s1, s2, a[19], a[29]);+	MONT_MAC(c0, c1, c2, a[20], a[28]);+	MONT_MAC(s0, s1, s2, a[21], a[27]);+	MONT_MAC(c0, c1, c2, a[22], a[26]);+	MONT_MAC(s0, s1, s2, a[23], a[25]);+	MONT_MAC(t0, t1, t2, a[24], a[24]);+	MONT_MUL(v0, v1, v2, u[1], m[47]);+	MONT_MUL(w0, w1, w2, u[2], m[46]);+	MONT_MAC(t0, t1, t2, u[3], m[45]);+	MONT_MAC(v0, v1, v2, u[4], m[44]);+	MONT_MAC(w0, w1, w2, u[5], m[43]);+	MONT_MAC(t0, t1, t2, u[6], m[42]);+	MONT_MAC(v0, v1, v2, u[7], m[41]);+	MONT_MAC(w0, w1, w2, u[8], m[40]);+	MONT_MAC(t0, t1, t2, u[9], m[39]);+	MONT_MAC(v0, v1, v2, u[10], m[38]);+	MONT_MAC(w0, w1, w2, u[11], m[37]);+	MONT_MAC(t0, t1, t2, u[12], m[36]);+	MONT_MAC(v0, v1, v2, u[13], m[35]);+	MONT_MAC(w0, w1, w2, u[14], m[34]);+	MONT_MAC(t0, t1, t2, u[15], m[33]);+	MONT_MAC(v0, v1, v2, u[16], m[32]);+	MONT_MAC(w0, w1, w2, u[17], m[31]);+	MONT_MAC(t0, t1, t2, u[18], m[30]);+	MONT_MAC(v0, v1, v2, u[19], m[29]);+	MONT_MAC(w0, w1, w2, u[20], m[28]);+	MONT_MAC(t0, t1, t2, u[21], m[27]);+	MONT_MAC(v0, v1, v2, u[22], m[26]);+	MONT_MAC(w0, w1, w2, u[23], m[25]);+	MONT_MAC(t0, t1, t2, u[24], m[24]);+	MONT_MAC(v0, v1, v2, u[25], m[23]);+	MONT_MAC(w0, w1, w2, u[26], m[22]);+	MONT_MAC(t0, t1, t2, u[27], m[21]);+	MONT_MAC(v0, v1, v2, u[28], m[20]);+	MONT_MAC(w0, w1, w2, u[29], m[19]);+	MONT_MAC(t0, t1, t2, u[30], m[18]);+	MONT_MAC(v0, v1, v2, u[31], m[17]);+	MONT_MAC(w0, w1, w2, u[32], m[16]);+	MONT_MAC(t0, t1, t2, u[33], m[15]);+	MONT_MAC(v0, v1, v2, u[34], m[14]);+	MONT_MAC(w0, w1, w2, u[35], m[13]);+	MONT_MAC(t0, t1, t2, u[36], m[12]);+	MONT_MAC(v0, v1, v2, u[37], m[11]);+	MONT_MAC(w0, w1, w2, u[38], m[10]);+	MONT_MAC(t0, t1, t2, u[39], m[9]);+	MONT_MAC(v0, v1, v2, u[40], m[8]);+	MONT_MAC(w0, w1, w2, u[41], m[7]);+	MONT_MAC(t0, t1, t2, u[42], m[6]);+	MONT_MAC(v0, v1, v2, u[43], m[5]);+	MONT_MAC(w0, w1, w2, u[44], m[4]);+	MONT_MAC(t0, t1, t2, u[45], m[3]);+	MONT_MAC(v0, v1, v2, u[46], m[2]);+	MONT_MAC(w0, w1, w2, u[47], m[1]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[0] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 49 */+	MONT_MUL(s0, s1, s2, a[2], a[47]);+	MONT_MUL(c0, c1, c2, a[3], a[46]);+	MONT_MAC(s0, s1, s2, a[4], a[45]);+	MONT_MAC(c0, c1, c2, a[5], a[44]);+	MONT_MAC(s0, s1, s2, a[6], a[43]);+	MONT_MAC(c0, c1, c2, a[7], a[42]);+	MONT_MAC(s0, s1, s2, a[8], a[41]);+	MONT_MAC(c0, c1, c2, a[9], a[40]);+	MONT_MAC(s0, s1, s2, a[10], a[39]);+	MONT_MAC(c0, c1, c2, a[11], a[38]);+	MONT_MAC(s0, s1, s2, a[12], a[37]);+	MONT_MAC(c0, c1, c2, a[13], a[36]);+	MONT_MAC(s0, s1, s2, a[14], a[35]);+	MONT_MAC(c0, c1, c2, a[15], a[34]);+	MONT_MAC(s0, s1, s2, a[16], a[33]);+	MONT_MAC(c0, c1, c2, a[17], a[32]);+	MONT_MAC(s0, s1, s2, a[18], a[31]);+	MONT_MAC(c0, c1, c2, a[19], a[30]);+	MONT_MAC(s0, s1, s2, a[20], a[29]);+	MONT_MAC(c0, c1, c2, a[21], a[28]);+	MONT_MAC(s0, s1, s2, a[22], a[27]);+	MONT_MAC(c0, c1, c2, a[23], a[26]);+	MONT_MAC(s0, s1, s2, a[24], a[25]);+	MONT_MAC(t0, t1, t2, u[2], m[47]);+	MONT_MUL(v0, v1, v2, u[3], m[46]);+	MONT_MUL(w0, w1, w2, u[4], m[45]);+	MONT_MAC(t0, t1, t2, u[5], m[44]);+	MONT_MAC(v0, v1, v2, u[6], m[43]);+	MONT_MAC(w0, w1, w2, u[7], m[42]);+	MONT_MAC(t0, t1, t2, u[8], m[41]);+	MONT_MAC(v0, v1, v2, u[9], m[40]);+	MONT_MAC(w0, w1, w2, u[10], m[39]);+	MONT_MAC(t0, t1, t2, u[11], m[38]);+	MONT_MAC(v0, v1, v2, u[12], m[37]);+	MONT_MAC(w0, w1, w2, u[13], m[36]);+	MONT_MAC(t0, t1, t2, u[14], m[35]);+	MONT_MAC(v0, v1, v2, u[15], m[34]);+	MONT_MAC(w0, w1, w2, u[16], m[33]);+	MONT_MAC(t0, t1, t2, u[17], m[32]);+	MONT_MAC(v0, v1, v2, u[18], m[31]);+	MONT_MAC(w0, w1, w2, u[19], m[30]);+	MONT_MAC(t0, t1, t2, u[20], m[29]);+	MONT_MAC(v0, v1, v2, u[21], m[28]);+	MONT_MAC(w0, w1, w2, u[22], m[27]);+	MONT_MAC(t0, t1, t2, u[23], m[26]);+	MONT_MAC(v0, v1, v2, u[24], m[25]);+	MONT_MAC(w0, w1, w2, u[25], m[24]);+	MONT_MAC(t0, t1, t2, u[26], m[23]);+	MONT_MAC(v0, v1, v2, u[27], m[22]);+	MONT_MAC(w0, w1, w2, u[28], m[21]);+	MONT_MAC(t0, t1, t2, u[29], m[20]);+	MONT_MAC(v0, v1, v2, u[30], m[19]);+	MONT_MAC(w0, w1, w2, u[31], m[18]);+	MONT_MAC(t0, t1, t2, u[32], m[17]);+	MONT_MAC(v0, v1, v2, u[33], m[16]);+	MONT_MAC(w0, w1, w2, u[34], m[15]);+	MONT_MAC(t0, t1, t2, u[35], m[14]);+	MONT_MAC(v0, v1, v2, u[36], m[13]);+	MONT_MAC(w0, w1, w2, u[37], m[12]);+	MONT_MAC(t0, t1, t2, u[38], m[11]);+	MONT_MAC(v0, v1, v2, u[39], m[10]);+	MONT_MAC(w0, w1, w2, u[40], m[9]);+	MONT_MAC(t0, t1, t2, u[41], m[8]);+	MONT_MAC(v0, v1, v2, u[42], m[7]);+	MONT_MAC(w0, w1, w2, u[43], m[6]);+	MONT_MAC(t0, t1, t2, u[44], m[5]);+	MONT_MAC(v0, v1, v2, u[45], m[4]);+	MONT_MAC(w0, w1, w2, u[46], m[3]);+	MONT_MAC(t0, t1, t2, u[47], m[2]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[1] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 50 */+	MONT_MUL(s0, s1, s2, a[3], a[47]);+	MONT_MUL(c0, c1, c2, a[4], a[46]);+	MONT_MAC(s0, s1, s2, a[5], a[45]);+	MONT_MAC(c0, c1, c2, a[6], a[44]);+	MONT_MAC(s0, s1, s2, a[7], a[43]);+	MONT_MAC(c0, c1, c2, a[8], a[42]);+	MONT_MAC(s0, s1, s2, a[9], a[41]);+	MONT_MAC(c0, c1, c2, a[10], a[40]);+	MONT_MAC(s0, s1, s2, a[11], a[39]);+	MONT_MAC(c0, c1, c2, a[12], a[38]);+	MONT_MAC(s0, s1, s2, a[13], a[37]);+	MONT_MAC(c0, c1, c2, a[14], a[36]);+	MONT_MAC(s0, s1, s2, a[15], a[35]);+	MONT_MAC(c0, c1, c2, a[16], a[34]);+	MONT_MAC(s0, s1, s2, a[17], a[33]);+	MONT_MAC(c0, c1, c2, a[18], a[32]);+	MONT_MAC(s0, s1, s2, a[19], a[31]);+	MONT_MAC(c0, c1, c2, a[20], a[30]);+	MONT_MAC(s0, s1, s2, a[21], a[29]);+	MONT_MAC(c0, c1, c2, a[22], a[28]);+	MONT_MAC(s0, s1, s2, a[23], a[27]);+	MONT_MAC(c0, c1, c2, a[24], a[26]);+	MONT_MAC(t0, t1, t2, a[25], a[25]);+	MONT_MUL(v0, v1, v2, u[3], m[47]);+	MONT_MUL(w0, w1, w2, u[4], m[46]);+	MONT_MAC(t0, t1, t2, u[5], m[45]);+	MONT_MAC(v0, v1, v2, u[6], m[44]);+	MONT_MAC(w0, w1, w2, u[7], m[43]);+	MONT_MAC(t0, t1, t2, u[8], m[42]);+	MONT_MAC(v0, v1, v2, u[9], m[41]);+	MONT_MAC(w0, w1, w2, u[10], m[40]);+	MONT_MAC(t0, t1, t2, u[11], m[39]);+	MONT_MAC(v0, v1, v2, u[12], m[38]);+	MONT_MAC(w0, w1, w2, u[13], m[37]);+	MONT_MAC(t0, t1, t2, u[14], m[36]);+	MONT_MAC(v0, v1, v2, u[15], m[35]);+	MONT_MAC(w0, w1, w2, u[16], m[34]);+	MONT_MAC(t0, t1, t2, u[17], m[33]);+	MONT_MAC(v0, v1, v2, u[18], m[32]);+	MONT_MAC(w0, w1, w2, u[19], m[31]);+	MONT_MAC(t0, t1, t2, u[20], m[30]);+	MONT_MAC(v0, v1, v2, u[21], m[29]);+	MONT_MAC(w0, w1, w2, u[22], m[28]);+	MONT_MAC(t0, t1, t2, u[23], m[27]);+	MONT_MAC(v0, v1, v2, u[24], m[26]);+	MONT_MAC(w0, w1, w2, u[25], m[25]);+	MONT_MAC(t0, t1, t2, u[26], m[24]);+	MONT_MAC(v0, v1, v2, u[27], m[23]);+	MONT_MAC(w0, w1, w2, u[28], m[22]);+	MONT_MAC(t0, t1, t2, u[29], m[21]);+	MONT_MAC(v0, v1, v2, u[30], m[20]);+	MONT_MAC(w0, w1, w2, u[31], m[19]);+	MONT_MAC(t0, t1, t2, u[32], m[18]);+	MONT_MAC(v0, v1, v2, u[33], m[17]);+	MONT_MAC(w0, w1, w2, u[34], m[16]);+	MONT_MAC(t0, t1, t2, u[35], m[15]);+	MONT_MAC(v0, v1, v2, u[36], m[14]);+	MONT_MAC(w0, w1, w2, u[37], m[13]);+	MONT_MAC(t0, t1, t2, u[38], m[12]);+	MONT_MAC(v0, v1, v2, u[39], m[11]);+	MONT_MAC(w0, w1, w2, u[40], m[10]);+	MONT_MAC(t0, t1, t2, u[41], m[9]);+	MONT_MAC(v0, v1, v2, u[42], m[8]);+	MONT_MAC(w0, w1, w2, u[43], m[7]);+	MONT_MAC(t0, t1, t2, u[44], m[6]);+	MONT_MAC(v0, v1, v2, u[45], m[5]);+	MONT_MAC(w0, w1, w2, u[46], m[4]);+	MONT_MAC(t0, t1, t2, u[47], m[3]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[2] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 51 */+	MONT_MUL(s0, s1, s2, a[4], a[47]);+	MONT_MUL(c0, c1, c2, a[5], a[46]);+	MONT_MAC(s0, s1, s2, a[6], a[45]);+	MONT_MAC(c0, c1, c2, a[7], a[44]);+	MONT_MAC(s0, s1, s2, a[8], a[43]);+	MONT_MAC(c0, c1, c2, a[9], a[42]);+	MONT_MAC(s0, s1, s2, a[10], a[41]);+	MONT_MAC(c0, c1, c2, a[11], a[40]);+	MONT_MAC(s0, s1, s2, a[12], a[39]);+	MONT_MAC(c0, c1, c2, a[13], a[38]);+	MONT_MAC(s0, s1, s2, a[14], a[37]);+	MONT_MAC(c0, c1, c2, a[15], a[36]);+	MONT_MAC(s0, s1, s2, a[16], a[35]);+	MONT_MAC(c0, c1, c2, a[17], a[34]);+	MONT_MAC(s0, s1, s2, a[18], a[33]);+	MONT_MAC(c0, c1, c2, a[19], a[32]);+	MONT_MAC(s0, s1, s2, a[20], a[31]);+	MONT_MAC(c0, c1, c2, a[21], a[30]);+	MONT_MAC(s0, s1, s2, a[22], a[29]);+	MONT_MAC(c0, c1, c2, a[23], a[28]);+	MONT_MAC(s0, s1, s2, a[24], a[27]);+	MONT_MAC(c0, c1, c2, a[25], a[26]);+	MONT_MAC(t0, t1, t2, u[4], m[47]);+	MONT_MUL(v0, v1, v2, u[5], m[46]);+	MONT_MUL(w0, w1, w2, u[6], m[45]);+	MONT_MAC(t0, t1, t2, u[7], m[44]);+	MONT_MAC(v0, v1, v2, u[8], m[43]);+	MONT_MAC(w0, w1, w2, u[9], m[42]);+	MONT_MAC(t0, t1, t2, u[10], m[41]);+	MONT_MAC(v0, v1, v2, u[11], m[40]);+	MONT_MAC(w0, w1, w2, u[12], m[39]);+	MONT_MAC(t0, t1, t2, u[13], m[38]);+	MONT_MAC(v0, v1, v2, u[14], m[37]);+	MONT_MAC(w0, w1, w2, u[15], m[36]);+	MONT_MAC(t0, t1, t2, u[16], m[35]);+	MONT_MAC(v0, v1, v2, u[17], m[34]);+	MONT_MAC(w0, w1, w2, u[18], m[33]);+	MONT_MAC(t0, t1, t2, u[19], m[32]);+	MONT_MAC(v0, v1, v2, u[20], m[31]);+	MONT_MAC(w0, w1, w2, u[21], m[30]);+	MONT_MAC(t0, t1, t2, u[22], m[29]);+	MONT_MAC(v0, v1, v2, u[23], m[28]);+	MONT_MAC(w0, w1, w2, u[24], m[27]);+	MONT_MAC(t0, t1, t2, u[25], m[26]);+	MONT_MAC(v0, v1, v2, u[26], m[25]);+	MONT_MAC(w0, w1, w2, u[27], m[24]);+	MONT_MAC(t0, t1, t2, u[28], m[23]);+	MONT_MAC(v0, v1, v2, u[29], m[22]);+	MONT_MAC(w0, w1, w2, u[30], m[21]);+	MONT_MAC(t0, t1, t2, u[31], m[20]);+	MONT_MAC(v0, v1, v2, u[32], m[19]);+	MONT_MAC(w0, w1, w2, u[33], m[18]);+	MONT_MAC(t0, t1, t2, u[34], m[17]);+	MONT_MAC(v0, v1, v2, u[35], m[16]);+	MONT_MAC(w0, w1, w2, u[36], m[15]);+	MONT_MAC(t0, t1, t2, u[37], m[14]);+	MONT_MAC(v0, v1, v2, u[38], m[13]);+	MONT_MAC(w0, w1, w2, u[39], m[12]);+	MONT_MAC(t0, t1, t2, u[40], m[11]);+	MONT_MAC(v0, v1, v2, u[41], m[10]);+	MONT_MAC(w0, w1, w2, u[42], m[9]);+	MONT_MAC(t0, t1, t2, u[43], m[8]);+	MONT_MAC(v0, v1, v2, u[44], m[7]);+	MONT_MAC(w0, w1, w2, u[45], m[6]);+	MONT_MAC(t0, t1, t2, u[46], m[5]);+	MONT_MAC(v0, v1, v2, u[47], m[4]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[3] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 52 */+	MONT_MUL(s0, s1, s2, a[5], a[47]);+	MONT_MUL(c0, c1, c2, a[6], a[46]);+	MONT_MAC(s0, s1, s2, a[7], a[45]);+	MONT_MAC(c0, c1, c2, a[8], a[44]);+	MONT_MAC(s0, s1, s2, a[9], a[43]);+	MONT_MAC(c0, c1, c2, a[10], a[42]);+	MONT_MAC(s0, s1, s2, a[11], a[41]);+	MONT_MAC(c0, c1, c2, a[12], a[40]);+	MONT_MAC(s0, s1, s2, a[13], a[39]);+	MONT_MAC(c0, c1, c2, a[14], a[38]);+	MONT_MAC(s0, s1, s2, a[15], a[37]);+	MONT_MAC(c0, c1, c2, a[16], a[36]);+	MONT_MAC(s0, s1, s2, a[17], a[35]);+	MONT_MAC(c0, c1, c2, a[18], a[34]);+	MONT_MAC(s0, s1, s2, a[19], a[33]);+	MONT_MAC(c0, c1, c2, a[20], a[32]);+	MONT_MAC(s0, s1, s2, a[21], a[31]);+	MONT_MAC(c0, c1, c2, a[22], a[30]);+	MONT_MAC(s0, s1, s2, a[23], a[29]);+	MONT_MAC(c0, c1, c2, a[24], a[28]);+	MONT_MAC(s0, s1, s2, a[25], a[27]);+	MONT_MAC(t0, t1, t2, a[26], a[26]);+	MONT_MUL(v0, v1, v2, u[5], m[47]);+	MONT_MUL(w0, w1, w2, u[6], m[46]);+	MONT_MAC(t0, t1, t2, u[7], m[45]);+	MONT_MAC(v0, v1, v2, u[8], m[44]);+	MONT_MAC(w0, w1, w2, u[9], m[43]);+	MONT_MAC(t0, t1, t2, u[10], m[42]);+	MONT_MAC(v0, v1, v2, u[11], m[41]);+	MONT_MAC(w0, w1, w2, u[12], m[40]);+	MONT_MAC(t0, t1, t2, u[13], m[39]);+	MONT_MAC(v0, v1, v2, u[14], m[38]);+	MONT_MAC(w0, w1, w2, u[15], m[37]);+	MONT_MAC(t0, t1, t2, u[16], m[36]);+	MONT_MAC(v0, v1, v2, u[17], m[35]);+	MONT_MAC(w0, w1, w2, u[18], m[34]);+	MONT_MAC(t0, t1, t2, u[19], m[33]);+	MONT_MAC(v0, v1, v2, u[20], m[32]);+	MONT_MAC(w0, w1, w2, u[21], m[31]);+	MONT_MAC(t0, t1, t2, u[22], m[30]);+	MONT_MAC(v0, v1, v2, u[23], m[29]);+	MONT_MAC(w0, w1, w2, u[24], m[28]);+	MONT_MAC(t0, t1, t2, u[25], m[27]);+	MONT_MAC(v0, v1, v2, u[26], m[26]);+	MONT_MAC(w0, w1, w2, u[27], m[25]);+	MONT_MAC(t0, t1, t2, u[28], m[24]);+	MONT_MAC(v0, v1, v2, u[29], m[23]);+	MONT_MAC(w0, w1, w2, u[30], m[22]);+	MONT_MAC(t0, t1, t2, u[31], m[21]);+	MONT_MAC(v0, v1, v2, u[32], m[20]);+	MONT_MAC(w0, w1, w2, u[33], m[19]);+	MONT_MAC(t0, t1, t2, u[34], m[18]);+	MONT_MAC(v0, v1, v2, u[35], m[17]);+	MONT_MAC(w0, w1, w2, u[36], m[16]);+	MONT_MAC(t0, t1, t2, u[37], m[15]);+	MONT_MAC(v0, v1, v2, u[38], m[14]);+	MONT_MAC(w0, w1, w2, u[39], m[13]);+	MONT_MAC(t0, t1, t2, u[40], m[12]);+	MONT_MAC(v0, v1, v2, u[41], m[11]);+	MONT_MAC(w0, w1, w2, u[42], m[10]);+	MONT_MAC(t0, t1, t2, u[43], m[9]);+	MONT_MAC(v0, v1, v2, u[44], m[8]);+	MONT_MAC(w0, w1, w2, u[45], m[7]);+	MONT_MAC(t0, t1, t2, u[46], m[6]);+	MONT_MAC(v0, v1, v2, u[47], m[5]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[4] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 53 */+	MONT_MUL(s0, s1, s2, a[6], a[47]);+	MONT_MUL(c0, c1, c2, a[7], a[46]);+	MONT_MAC(s0, s1, s2, a[8], a[45]);+	MONT_MAC(c0, c1, c2, a[9], a[44]);+	MONT_MAC(s0, s1, s2, a[10], a[43]);+	MONT_MAC(c0, c1, c2, a[11], a[42]);+	MONT_MAC(s0, s1, s2, a[12], a[41]);+	MONT_MAC(c0, c1, c2, a[13], a[40]);+	MONT_MAC(s0, s1, s2, a[14], a[39]);+	MONT_MAC(c0, c1, c2, a[15], a[38]);+	MONT_MAC(s0, s1, s2, a[16], a[37]);+	MONT_MAC(c0, c1, c2, a[17], a[36]);+	MONT_MAC(s0, s1, s2, a[18], a[35]);+	MONT_MAC(c0, c1, c2, a[19], a[34]);+	MONT_MAC(s0, s1, s2, a[20], a[33]);+	MONT_MAC(c0, c1, c2, a[21], a[32]);+	MONT_MAC(s0, s1, s2, a[22], a[31]);+	MONT_MAC(c0, c1, c2, a[23], a[30]);+	MONT_MAC(s0, s1, s2, a[24], a[29]);+	MONT_MAC(c0, c1, c2, a[25], a[28]);+	MONT_MAC(s0, s1, s2, a[26], a[27]);+	MONT_MAC(t0, t1, t2, u[6], m[47]);+	MONT_MUL(v0, v1, v2, u[7], m[46]);+	MONT_MUL(w0, w1, w2, u[8], m[45]);+	MONT_MAC(t0, t1, t2, u[9], m[44]);+	MONT_MAC(v0, v1, v2, u[10], m[43]);+	MONT_MAC(w0, w1, w2, u[11], m[42]);+	MONT_MAC(t0, t1, t2, u[12], m[41]);+	MONT_MAC(v0, v1, v2, u[13], m[40]);+	MONT_MAC(w0, w1, w2, u[14], m[39]);+	MONT_MAC(t0, t1, t2, u[15], m[38]);+	MONT_MAC(v0, v1, v2, u[16], m[37]);+	MONT_MAC(w0, w1, w2, u[17], m[36]);+	MONT_MAC(t0, t1, t2, u[18], m[35]);+	MONT_MAC(v0, v1, v2, u[19], m[34]);+	MONT_MAC(w0, w1, w2, u[20], m[33]);+	MONT_MAC(t0, t1, t2, u[21], m[32]);+	MONT_MAC(v0, v1, v2, u[22], m[31]);+	MONT_MAC(w0, w1, w2, u[23], m[30]);+	MONT_MAC(t0, t1, t2, u[24], m[29]);+	MONT_MAC(v0, v1, v2, u[25], m[28]);+	MONT_MAC(w0, w1, w2, u[26], m[27]);+	MONT_MAC(t0, t1, t2, u[27], m[26]);+	MONT_MAC(v0, v1, v2, u[28], m[25]);+	MONT_MAC(w0, w1, w2, u[29], m[24]);+	MONT_MAC(t0, t1, t2, u[30], m[23]);+	MONT_MAC(v0, v1, v2, u[31], m[22]);+	MONT_MAC(w0, w1, w2, u[32], m[21]);+	MONT_MAC(t0, t1, t2, u[33], m[20]);+	MONT_MAC(v0, v1, v2, u[34], m[19]);+	MONT_MAC(w0, w1, w2, u[35], m[18]);+	MONT_MAC(t0, t1, t2, u[36], m[17]);+	MONT_MAC(v0, v1, v2, u[37], m[16]);+	MONT_MAC(w0, w1, w2, u[38], m[15]);+	MONT_MAC(t0, t1, t2, u[39], m[14]);+	MONT_MAC(v0, v1, v2, u[40], m[13]);+	MONT_MAC(w0, w1, w2, u[41], m[12]);+	MONT_MAC(t0, t1, t2, u[42], m[11]);+	MONT_MAC(v0, v1, v2, u[43], m[10]);+	MONT_MAC(w0, w1, w2, u[44], m[9]);+	MONT_MAC(t0, t1, t2, u[45], m[8]);+	MONT_MAC(v0, v1, v2, u[46], m[7]);+	MONT_MAC(w0, w1, w2, u[47], m[6]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[5] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 54 */+	MONT_MUL(s0, s1, s2, a[7], a[47]);+	MONT_MUL(c0, c1, c2, a[8], a[46]);+	MONT_MAC(s0, s1, s2, a[9], a[45]);+	MONT_MAC(c0, c1, c2, a[10], a[44]);+	MONT_MAC(s0, s1, s2, a[11], a[43]);+	MONT_MAC(c0, c1, c2, a[12], a[42]);+	MONT_MAC(s0, s1, s2, a[13], a[41]);+	MONT_MAC(c0, c1, c2, a[14], a[40]);+	MONT_MAC(s0, s1, s2, a[15], a[39]);+	MONT_MAC(c0, c1, c2, a[16], a[38]);+	MONT_MAC(s0, s1, s2, a[17], a[37]);+	MONT_MAC(c0, c1, c2, a[18], a[36]);+	MONT_MAC(s0, s1, s2, a[19], a[35]);+	MONT_MAC(c0, c1, c2, a[20], a[34]);+	MONT_MAC(s0, s1, s2, a[21], a[33]);+	MONT_MAC(c0, c1, c2, a[22], a[32]);+	MONT_MAC(s0, s1, s2, a[23], a[31]);+	MONT_MAC(c0, c1, c2, a[24], a[30]);+	MONT_MAC(s0, s1, s2, a[25], a[29]);+	MONT_MAC(c0, c1, c2, a[26], a[28]);+	MONT_MAC(t0, t1, t2, a[27], a[27]);+	MONT_MUL(v0, v1, v2, u[7], m[47]);+	MONT_MUL(w0, w1, w2, u[8], m[46]);+	MONT_MAC(t0, t1, t2, u[9], m[45]);+	MONT_MAC(v0, v1, v2, u[10], m[44]);+	MONT_MAC(w0, w1, w2, u[11], m[43]);+	MONT_MAC(t0, t1, t2, u[12], m[42]);+	MONT_MAC(v0, v1, v2, u[13], m[41]);+	MONT_MAC(w0, w1, w2, u[14], m[40]);+	MONT_MAC(t0, t1, t2, u[15], m[39]);+	MONT_MAC(v0, v1, v2, u[16], m[38]);+	MONT_MAC(w0, w1, w2, u[17], m[37]);+	MONT_MAC(t0, t1, t2, u[18], m[36]);+	MONT_MAC(v0, v1, v2, u[19], m[35]);+	MONT_MAC(w0, w1, w2, u[20], m[34]);+	MONT_MAC(t0, t1, t2, u[21], m[33]);+	MONT_MAC(v0, v1, v2, u[22], m[32]);+	MONT_MAC(w0, w1, w2, u[23], m[31]);+	MONT_MAC(t0, t1, t2, u[24], m[30]);+	MONT_MAC(v0, v1, v2, u[25], m[29]);+	MONT_MAC(w0, w1, w2, u[26], m[28]);+	MONT_MAC(t0, t1, t2, u[27], m[27]);+	MONT_MAC(v0, v1, v2, u[28], m[26]);+	MONT_MAC(w0, w1, w2, u[29], m[25]);+	MONT_MAC(t0, t1, t2, u[30], m[24]);+	MONT_MAC(v0, v1, v2, u[31], m[23]);+	MONT_MAC(w0, w1, w2, u[32], m[22]);+	MONT_MAC(t0, t1, t2, u[33], m[21]);+	MONT_MAC(v0, v1, v2, u[34], m[20]);+	MONT_MAC(w0, w1, w2, u[35], m[19]);+	MONT_MAC(t0, t1, t2, u[36], m[18]);+	MONT_MAC(v0, v1, v2, u[37], m[17]);+	MONT_MAC(w0, w1, w2, u[38], m[16]);+	MONT_MAC(t0, t1, t2, u[39], m[15]);+	MONT_MAC(v0, v1, v2, u[40], m[14]);+	MONT_MAC(w0, w1, w2, u[41], m[13]);+	MONT_MAC(t0, t1, t2, u[42], m[12]);+	MONT_MAC(v0, v1, v2, u[43], m[11]);+	MONT_MAC(w0, w1, w2, u[44], m[10]);+	MONT_MAC(t0, t1, t2, u[45], m[9]);+	MONT_MAC(v0, v1, v2, u[46], m[8]);+	MONT_MAC(w0, w1, w2, u[47], m[7]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[6] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 55 */+	MONT_MUL(s0, s1, s2, a[8], a[47]);+	MONT_MUL(c0, c1, c2, a[9], a[46]);+	MONT_MAC(s0, s1, s2, a[10], a[45]);+	MONT_MAC(c0, c1, c2, a[11], a[44]);+	MONT_MAC(s0, s1, s2, a[12], a[43]);+	MONT_MAC(c0, c1, c2, a[13], a[42]);+	MONT_MAC(s0, s1, s2, a[14], a[41]);+	MONT_MAC(c0, c1, c2, a[15], a[40]);+	MONT_MAC(s0, s1, s2, a[16], a[39]);+	MONT_MAC(c0, c1, c2, a[17], a[38]);+	MONT_MAC(s0, s1, s2, a[18], a[37]);+	MONT_MAC(c0, c1, c2, a[19], a[36]);+	MONT_MAC(s0, s1, s2, a[20], a[35]);+	MONT_MAC(c0, c1, c2, a[21], a[34]);+	MONT_MAC(s0, s1, s2, a[22], a[33]);+	MONT_MAC(c0, c1, c2, a[23], a[32]);+	MONT_MAC(s0, s1, s2, a[24], a[31]);+	MONT_MAC(c0, c1, c2, a[25], a[30]);+	MONT_MAC(s0, s1, s2, a[26], a[29]);+	MONT_MAC(c0, c1, c2, a[27], a[28]);+	MONT_MAC(t0, t1, t2, u[8], m[47]);+	MONT_MUL(v0, v1, v2, u[9], m[46]);+	MONT_MUL(w0, w1, w2, u[10], m[45]);+	MONT_MAC(t0, t1, t2, u[11], m[44]);+	MONT_MAC(v0, v1, v2, u[12], m[43]);+	MONT_MAC(w0, w1, w2, u[13], m[42]);+	MONT_MAC(t0, t1, t2, u[14], m[41]);+	MONT_MAC(v0, v1, v2, u[15], m[40]);+	MONT_MAC(w0, w1, w2, u[16], m[39]);+	MONT_MAC(t0, t1, t2, u[17], m[38]);+	MONT_MAC(v0, v1, v2, u[18], m[37]);+	MONT_MAC(w0, w1, w2, u[19], m[36]);+	MONT_MAC(t0, t1, t2, u[20], m[35]);+	MONT_MAC(v0, v1, v2, u[21], m[34]);+	MONT_MAC(w0, w1, w2, u[22], m[33]);+	MONT_MAC(t0, t1, t2, u[23], m[32]);+	MONT_MAC(v0, v1, v2, u[24], m[31]);+	MONT_MAC(w0, w1, w2, u[25], m[30]);+	MONT_MAC(t0, t1, t2, u[26], m[29]);+	MONT_MAC(v0, v1, v2, u[27], m[28]);+	MONT_MAC(w0, w1, w2, u[28], m[27]);+	MONT_MAC(t0, t1, t2, u[29], m[26]);+	MONT_MAC(v0, v1, v2, u[30], m[25]);+	MONT_MAC(w0, w1, w2, u[31], m[24]);+	MONT_MAC(t0, t1, t2, u[32], m[23]);+	MONT_MAC(v0, v1, v2, u[33], m[22]);+	MONT_MAC(w0, w1, w2, u[34], m[21]);+	MONT_MAC(t0, t1, t2, u[35], m[20]);+	MONT_MAC(v0, v1, v2, u[36], m[19]);+	MONT_MAC(w0, w1, w2, u[37], m[18]);+	MONT_MAC(t0, t1, t2, u[38], m[17]);+	MONT_MAC(v0, v1, v2, u[39], m[16]);+	MONT_MAC(w0, w1, w2, u[40], m[15]);+	MONT_MAC(t0, t1, t2, u[41], m[14]);+	MONT_MAC(v0, v1, v2, u[42], m[13]);+	MONT_MAC(w0, w1, w2, u[43], m[12]);+	MONT_MAC(t0, t1, t2, u[44], m[11]);+	MONT_MAC(v0, v1, v2, u[45], m[10]);+	MONT_MAC(w0, w1, w2, u[46], m[9]);+	MONT_MAC(t0, t1, t2, u[47], m[8]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[7] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 56 */+	MONT_MUL(s0, s1, s2, a[9], a[47]);+	MONT_MUL(c0, c1, c2, a[10], a[46]);+	MONT_MAC(s0, s1, s2, a[11], a[45]);+	MONT_MAC(c0, c1, c2, a[12], a[44]);+	MONT_MAC(s0, s1, s2, a[13], a[43]);+	MONT_MAC(c0, c1, c2, a[14], a[42]);+	MONT_MAC(s0, s1, s2, a[15], a[41]);+	MONT_MAC(c0, c1, c2, a[16], a[40]);+	MONT_MAC(s0, s1, s2, a[17], a[39]);+	MONT_MAC(c0, c1, c2, a[18], a[38]);+	MONT_MAC(s0, s1, s2, a[19], a[37]);+	MONT_MAC(c0, c1, c2, a[20], a[36]);+	MONT_MAC(s0, s1, s2, a[21], a[35]);+	MONT_MAC(c0, c1, c2, a[22], a[34]);+	MONT_MAC(s0, s1, s2, a[23], a[33]);+	MONT_MAC(c0, c1, c2, a[24], a[32]);+	MONT_MAC(s0, s1, s2, a[25], a[31]);+	MONT_MAC(c0, c1, c2, a[26], a[30]);+	MONT_MAC(s0, s1, s2, a[27], a[29]);+	MONT_MAC(t0, t1, t2, a[28], a[28]);+	MONT_MUL(v0, v1, v2, u[9], m[47]);+	MONT_MUL(w0, w1, w2, u[10], m[46]);+	MONT_MAC(t0, t1, t2, u[11], m[45]);+	MONT_MAC(v0, v1, v2, u[12], m[44]);+	MONT_MAC(w0, w1, w2, u[13], m[43]);+	MONT_MAC(t0, t1, t2, u[14], m[42]);+	MONT_MAC(v0, v1, v2, u[15], m[41]);+	MONT_MAC(w0, w1, w2, u[16], m[40]);+	MONT_MAC(t0, t1, t2, u[17], m[39]);+	MONT_MAC(v0, v1, v2, u[18], m[38]);+	MONT_MAC(w0, w1, w2, u[19], m[37]);+	MONT_MAC(t0, t1, t2, u[20], m[36]);+	MONT_MAC(v0, v1, v2, u[21], m[35]);+	MONT_MAC(w0, w1, w2, u[22], m[34]);+	MONT_MAC(t0, t1, t2, u[23], m[33]);+	MONT_MAC(v0, v1, v2, u[24], m[32]);+	MONT_MAC(w0, w1, w2, u[25], m[31]);+	MONT_MAC(t0, t1, t2, u[26], m[30]);+	MONT_MAC(v0, v1, v2, u[27], m[29]);+	MONT_MAC(w0, w1, w2, u[28], m[28]);+	MONT_MAC(t0, t1, t2, u[29], m[27]);+	MONT_MAC(v0, v1, v2, u[30], m[26]);+	MONT_MAC(w0, w1, w2, u[31], m[25]);+	MONT_MAC(t0, t1, t2, u[32], m[24]);+	MONT_MAC(v0, v1, v2, u[33], m[23]);+	MONT_MAC(w0, w1, w2, u[34], m[22]);+	MONT_MAC(t0, t1, t2, u[35], m[21]);+	MONT_MAC(v0, v1, v2, u[36], m[20]);+	MONT_MAC(w0, w1, w2, u[37], m[19]);+	MONT_MAC(t0, t1, t2, u[38], m[18]);+	MONT_MAC(v0, v1, v2, u[39], m[17]);+	MONT_MAC(w0, w1, w2, u[40], m[16]);+	MONT_MAC(t0, t1, t2, u[41], m[15]);+	MONT_MAC(v0, v1, v2, u[42], m[14]);+	MONT_MAC(w0, w1, w2, u[43], m[13]);+	MONT_MAC(t0, t1, t2, u[44], m[12]);+	MONT_MAC(v0, v1, v2, u[45], m[11]);+	MONT_MAC(w0, w1, w2, u[46], m[10]);+	MONT_MAC(t0, t1, t2, u[47], m[9]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[8] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 57 */+	MONT_MUL(s0, s1, s2, a[10], a[47]);+	MONT_MUL(c0, c1, c2, a[11], a[46]);+	MONT_MAC(s0, s1, s2, a[12], a[45]);+	MONT_MAC(c0, c1, c2, a[13], a[44]);+	MONT_MAC(s0, s1, s2, a[14], a[43]);+	MONT_MAC(c0, c1, c2, a[15], a[42]);+	MONT_MAC(s0, s1, s2, a[16], a[41]);+	MONT_MAC(c0, c1, c2, a[17], a[40]);+	MONT_MAC(s0, s1, s2, a[18], a[39]);+	MONT_MAC(c0, c1, c2, a[19], a[38]);+	MONT_MAC(s0, s1, s2, a[20], a[37]);+	MONT_MAC(c0, c1, c2, a[21], a[36]);+	MONT_MAC(s0, s1, s2, a[22], a[35]);+	MONT_MAC(c0, c1, c2, a[23], a[34]);+	MONT_MAC(s0, s1, s2, a[24], a[33]);+	MONT_MAC(c0, c1, c2, a[25], a[32]);+	MONT_MAC(s0, s1, s2, a[26], a[31]);+	MONT_MAC(c0, c1, c2, a[27], a[30]);+	MONT_MAC(s0, s1, s2, a[28], a[29]);+	MONT_MAC(t0, t1, t2, u[10], m[47]);+	MONT_MUL(v0, v1, v2, u[11], m[46]);+	MONT_MUL(w0, w1, w2, u[12], m[45]);+	MONT_MAC(t0, t1, t2, u[13], m[44]);+	MONT_MAC(v0, v1, v2, u[14], m[43]);+	MONT_MAC(w0, w1, w2, u[15], m[42]);+	MONT_MAC(t0, t1, t2, u[16], m[41]);+	MONT_MAC(v0, v1, v2, u[17], m[40]);+	MONT_MAC(w0, w1, w2, u[18], m[39]);+	MONT_MAC(t0, t1, t2, u[19], m[38]);+	MONT_MAC(v0, v1, v2, u[20], m[37]);+	MONT_MAC(w0, w1, w2, u[21], m[36]);+	MONT_MAC(t0, t1, t2, u[22], m[35]);+	MONT_MAC(v0, v1, v2, u[23], m[34]);+	MONT_MAC(w0, w1, w2, u[24], m[33]);+	MONT_MAC(t0, t1, t2, u[25], m[32]);+	MONT_MAC(v0, v1, v2, u[26], m[31]);+	MONT_MAC(w0, w1, w2, u[27], m[30]);+	MONT_MAC(t0, t1, t2, u[28], m[29]);+	MONT_MAC(v0, v1, v2, u[29], m[28]);+	MONT_MAC(w0, w1, w2, u[30], m[27]);+	MONT_MAC(t0, t1, t2, u[31], m[26]);+	MONT_MAC(v0, v1, v2, u[32], m[25]);+	MONT_MAC(w0, w1, w2, u[33], m[24]);+	MONT_MAC(t0, t1, t2, u[34], m[23]);+	MONT_MAC(v0, v1, v2, u[35], m[22]);+	MONT_MAC(w0, w1, w2, u[36], m[21]);+	MONT_MAC(t0, t1, t2, u[37], m[20]);+	MONT_MAC(v0, v1, v2, u[38], m[19]);+	MONT_MAC(w0, w1, w2, u[39], m[18]);+	MONT_MAC(t0, t1, t2, u[40], m[17]);+	MONT_MAC(v0, v1, v2, u[41], m[16]);+	MONT_MAC(w0, w1, w2, u[42], m[15]);+	MONT_MAC(t0, t1, t2, u[43], m[14]);+	MONT_MAC(v0, v1, v2, u[44], m[13]);+	MONT_MAC(w0, w1, w2, u[45], m[12]);+	MONT_MAC(t0, t1, t2, u[46], m[11]);+	MONT_MAC(v0, v1, v2, u[47], m[10]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[9] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 58 */+	MONT_MUL(s0, s1, s2, a[11], a[47]);+	MONT_MUL(c0, c1, c2, a[12], a[46]);+	MONT_MAC(s0, s1, s2, a[13], a[45]);+	MONT_MAC(c0, c1, c2, a[14], a[44]);+	MONT_MAC(s0, s1, s2, a[15], a[43]);+	MONT_MAC(c0, c1, c2, a[16], a[42]);+	MONT_MAC(s0, s1, s2, a[17], a[41]);+	MONT_MAC(c0, c1, c2, a[18], a[40]);+	MONT_MAC(s0, s1, s2, a[19], a[39]);+	MONT_MAC(c0, c1, c2, a[20], a[38]);+	MONT_MAC(s0, s1, s2, a[21], a[37]);+	MONT_MAC(c0, c1, c2, a[22], a[36]);+	MONT_MAC(s0, s1, s2, a[23], a[35]);+	MONT_MAC(c0, c1, c2, a[24], a[34]);+	MONT_MAC(s0, s1, s2, a[25], a[33]);+	MONT_MAC(c0, c1, c2, a[26], a[32]);+	MONT_MAC(s0, s1, s2, a[27], a[31]);+	MONT_MAC(c0, c1, c2, a[28], a[30]);+	MONT_MAC(t0, t1, t2, a[29], a[29]);+	MONT_MUL(v0, v1, v2, u[11], m[47]);+	MONT_MUL(w0, w1, w2, u[12], m[46]);+	MONT_MAC(t0, t1, t2, u[13], m[45]);+	MONT_MAC(v0, v1, v2, u[14], m[44]);+	MONT_MAC(w0, w1, w2, u[15], m[43]);+	MONT_MAC(t0, t1, t2, u[16], m[42]);+	MONT_MAC(v0, v1, v2, u[17], m[41]);+	MONT_MAC(w0, w1, w2, u[18], m[40]);+	MONT_MAC(t0, t1, t2, u[19], m[39]);+	MONT_MAC(v0, v1, v2, u[20], m[38]);+	MONT_MAC(w0, w1, w2, u[21], m[37]);+	MONT_MAC(t0, t1, t2, u[22], m[36]);+	MONT_MAC(v0, v1, v2, u[23], m[35]);+	MONT_MAC(w0, w1, w2, u[24], m[34]);+	MONT_MAC(t0, t1, t2, u[25], m[33]);+	MONT_MAC(v0, v1, v2, u[26], m[32]);+	MONT_MAC(w0, w1, w2, u[27], m[31]);+	MONT_MAC(t0, t1, t2, u[28], m[30]);+	MONT_MAC(v0, v1, v2, u[29], m[29]);+	MONT_MAC(w0, w1, w2, u[30], m[28]);+	MONT_MAC(t0, t1, t2, u[31], m[27]);+	MONT_MAC(v0, v1, v2, u[32], m[26]);+	MONT_MAC(w0, w1, w2, u[33], m[25]);+	MONT_MAC(t0, t1, t2, u[34], m[24]);+	MONT_MAC(v0, v1, v2, u[35], m[23]);+	MONT_MAC(w0, w1, w2, u[36], m[22]);+	MONT_MAC(t0, t1, t2, u[37], m[21]);+	MONT_MAC(v0, v1, v2, u[38], m[20]);+	MONT_MAC(w0, w1, w2, u[39], m[19]);+	MONT_MAC(t0, t1, t2, u[40], m[18]);+	MONT_MAC(v0, v1, v2, u[41], m[17]);+	MONT_MAC(w0, w1, w2, u[42], m[16]);+	MONT_MAC(t0, t1, t2, u[43], m[15]);+	MONT_MAC(v0, v1, v2, u[44], m[14]);+	MONT_MAC(w0, w1, w2, u[45], m[13]);+	MONT_MAC(t0, t1, t2, u[46], m[12]);+	MONT_MAC(v0, v1, v2, u[47], m[11]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[10] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 59 */+	MONT_MUL(s0, s1, s2, a[12], a[47]);+	MONT_MUL(c0, c1, c2, a[13], a[46]);+	MONT_MAC(s0, s1, s2, a[14], a[45]);+	MONT_MAC(c0, c1, c2, a[15], a[44]);+	MONT_MAC(s0, s1, s2, a[16], a[43]);+	MONT_MAC(c0, c1, c2, a[17], a[42]);+	MONT_MAC(s0, s1, s2, a[18], a[41]);+	MONT_MAC(c0, c1, c2, a[19], a[40]);+	MONT_MAC(s0, s1, s2, a[20], a[39]);+	MONT_MAC(c0, c1, c2, a[21], a[38]);+	MONT_MAC(s0, s1, s2, a[22], a[37]);+	MONT_MAC(c0, c1, c2, a[23], a[36]);+	MONT_MAC(s0, s1, s2, a[24], a[35]);+	MONT_MAC(c0, c1, c2, a[25], a[34]);+	MONT_MAC(s0, s1, s2, a[26], a[33]);+	MONT_MAC(c0, c1, c2, a[27], a[32]);+	MONT_MAC(s0, s1, s2, a[28], a[31]);+	MONT_MAC(c0, c1, c2, a[29], a[30]);+	MONT_MAC(t0, t1, t2, u[12], m[47]);+	MONT_MUL(v0, v1, v2, u[13], m[46]);+	MONT_MUL(w0, w1, w2, u[14], m[45]);+	MONT_MAC(t0, t1, t2, u[15], m[44]);+	MONT_MAC(v0, v1, v2, u[16], m[43]);+	MONT_MAC(w0, w1, w2, u[17], m[42]);+	MONT_MAC(t0, t1, t2, u[18], m[41]);+	MONT_MAC(v0, v1, v2, u[19], m[40]);+	MONT_MAC(w0, w1, w2, u[20], m[39]);+	MONT_MAC(t0, t1, t2, u[21], m[38]);+	MONT_MAC(v0, v1, v2, u[22], m[37]);+	MONT_MAC(w0, w1, w2, u[23], m[36]);+	MONT_MAC(t0, t1, t2, u[24], m[35]);+	MONT_MAC(v0, v1, v2, u[25], m[34]);+	MONT_MAC(w0, w1, w2, u[26], m[33]);+	MONT_MAC(t0, t1, t2, u[27], m[32]);+	MONT_MAC(v0, v1, v2, u[28], m[31]);+	MONT_MAC(w0, w1, w2, u[29], m[30]);+	MONT_MAC(t0, t1, t2, u[30], m[29]);+	MONT_MAC(v0, v1, v2, u[31], m[28]);+	MONT_MAC(w0, w1, w2, u[32], m[27]);+	MONT_MAC(t0, t1, t2, u[33], m[26]);+	MONT_MAC(v0, v1, v2, u[34], m[25]);+	MONT_MAC(w0, w1, w2, u[35], m[24]);+	MONT_MAC(t0, t1, t2, u[36], m[23]);+	MONT_MAC(v0, v1, v2, u[37], m[22]);+	MONT_MAC(w0, w1, w2, u[38], m[21]);+	MONT_MAC(t0, t1, t2, u[39], m[20]);+	MONT_MAC(v0, v1, v2, u[40], m[19]);+	MONT_MAC(w0, w1, w2, u[41], m[18]);+	MONT_MAC(t0, t1, t2, u[42], m[17]);+	MONT_MAC(v0, v1, v2, u[43], m[16]);+	MONT_MAC(w0, w1, w2, u[44], m[15]);+	MONT_MAC(t0, t1, t2, u[45], m[14]);+	MONT_MAC(v0, v1, v2, u[46], m[13]);+	MONT_MAC(w0, w1, w2, u[47], m[12]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[11] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 60 */+	MONT_MUL(s0, s1, s2, a[13], a[47]);+	MONT_MUL(c0, c1, c2, a[14], a[46]);+	MONT_MAC(s0, s1, s2, a[15], a[45]);+	MONT_MAC(c0, c1, c2, a[16], a[44]);+	MONT_MAC(s0, s1, s2, a[17], a[43]);+	MONT_MAC(c0, c1, c2, a[18], a[42]);+	MONT_MAC(s0, s1, s2, a[19], a[41]);+	MONT_MAC(c0, c1, c2, a[20], a[40]);+	MONT_MAC(s0, s1, s2, a[21], a[39]);+	MONT_MAC(c0, c1, c2, a[22], a[38]);+	MONT_MAC(s0, s1, s2, a[23], a[37]);+	MONT_MAC(c0, c1, c2, a[24], a[36]);+	MONT_MAC(s0, s1, s2, a[25], a[35]);+	MONT_MAC(c0, c1, c2, a[26], a[34]);+	MONT_MAC(s0, s1, s2, a[27], a[33]);+	MONT_MAC(c0, c1, c2, a[28], a[32]);+	MONT_MAC(s0, s1, s2, a[29], a[31]);+	MONT_MAC(t0, t1, t2, a[30], a[30]);+	MONT_MUL(v0, v1, v2, u[13], m[47]);+	MONT_MUL(w0, w1, w2, u[14], m[46]);+	MONT_MAC(t0, t1, t2, u[15], m[45]);+	MONT_MAC(v0, v1, v2, u[16], m[44]);+	MONT_MAC(w0, w1, w2, u[17], m[43]);+	MONT_MAC(t0, t1, t2, u[18], m[42]);+	MONT_MAC(v0, v1, v2, u[19], m[41]);+	MONT_MAC(w0, w1, w2, u[20], m[40]);+	MONT_MAC(t0, t1, t2, u[21], m[39]);+	MONT_MAC(v0, v1, v2, u[22], m[38]);+	MONT_MAC(w0, w1, w2, u[23], m[37]);+	MONT_MAC(t0, t1, t2, u[24], m[36]);+	MONT_MAC(v0, v1, v2, u[25], m[35]);+	MONT_MAC(w0, w1, w2, u[26], m[34]);+	MONT_MAC(t0, t1, t2, u[27], m[33]);+	MONT_MAC(v0, v1, v2, u[28], m[32]);+	MONT_MAC(w0, w1, w2, u[29], m[31]);+	MONT_MAC(t0, t1, t2, u[30], m[30]);+	MONT_MAC(v0, v1, v2, u[31], m[29]);+	MONT_MAC(w0, w1, w2, u[32], m[28]);+	MONT_MAC(t0, t1, t2, u[33], m[27]);+	MONT_MAC(v0, v1, v2, u[34], m[26]);+	MONT_MAC(w0, w1, w2, u[35], m[25]);+	MONT_MAC(t0, t1, t2, u[36], m[24]);+	MONT_MAC(v0, v1, v2, u[37], m[23]);+	MONT_MAC(w0, w1, w2, u[38], m[22]);+	MONT_MAC(t0, t1, t2, u[39], m[21]);+	MONT_MAC(v0, v1, v2, u[40], m[20]);+	MONT_MAC(w0, w1, w2, u[41], m[19]);+	MONT_MAC(t0, t1, t2, u[42], m[18]);+	MONT_MAC(v0, v1, v2, u[43], m[17]);+	MONT_MAC(w0, w1, w2, u[44], m[16]);+	MONT_MAC(t0, t1, t2, u[45], m[15]);+	MONT_MAC(v0, v1, v2, u[46], m[14]);+	MONT_MAC(w0, w1, w2, u[47], m[13]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[12] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 61 */+	MONT_MUL(s0, s1, s2, a[14], a[47]);+	MONT_MUL(c0, c1, c2, a[15], a[46]);+	MONT_MAC(s0, s1, s2, a[16], a[45]);+	MONT_MAC(c0, c1, c2, a[17], a[44]);+	MONT_MAC(s0, s1, s2, a[18], a[43]);+	MONT_MAC(c0, c1, c2, a[19], a[42]);+	MONT_MAC(s0, s1, s2, a[20], a[41]);+	MONT_MAC(c0, c1, c2, a[21], a[40]);+	MONT_MAC(s0, s1, s2, a[22], a[39]);+	MONT_MAC(c0, c1, c2, a[23], a[38]);+	MONT_MAC(s0, s1, s2, a[24], a[37]);+	MONT_MAC(c0, c1, c2, a[25], a[36]);+	MONT_MAC(s0, s1, s2, a[26], a[35]);+	MONT_MAC(c0, c1, c2, a[27], a[34]);+	MONT_MAC(s0, s1, s2, a[28], a[33]);+	MONT_MAC(c0, c1, c2, a[29], a[32]);+	MONT_MAC(s0, s1, s2, a[30], a[31]);+	MONT_MAC(t0, t1, t2, u[14], m[47]);+	MONT_MUL(v0, v1, v2, u[15], m[46]);+	MONT_MUL(w0, w1, w2, u[16], m[45]);+	MONT_MAC(t0, t1, t2, u[17], m[44]);+	MONT_MAC(v0, v1, v2, u[18], m[43]);+	MONT_MAC(w0, w1, w2, u[19], m[42]);+	MONT_MAC(t0, t1, t2, u[20], m[41]);+	MONT_MAC(v0, v1, v2, u[21], m[40]);+	MONT_MAC(w0, w1, w2, u[22], m[39]);+	MONT_MAC(t0, t1, t2, u[23], m[38]);+	MONT_MAC(v0, v1, v2, u[24], m[37]);+	MONT_MAC(w0, w1, w2, u[25], m[36]);+	MONT_MAC(t0, t1, t2, u[26], m[35]);+	MONT_MAC(v0, v1, v2, u[27], m[34]);+	MONT_MAC(w0, w1, w2, u[28], m[33]);+	MONT_MAC(t0, t1, t2, u[29], m[32]);+	MONT_MAC(v0, v1, v2, u[30], m[31]);+	MONT_MAC(w0, w1, w2, u[31], m[30]);+	MONT_MAC(t0, t1, t2, u[32], m[29]);+	MONT_MAC(v0, v1, v2, u[33], m[28]);+	MONT_MAC(w0, w1, w2, u[34], m[27]);+	MONT_MAC(t0, t1, t2, u[35], m[26]);+	MONT_MAC(v0, v1, v2, u[36], m[25]);+	MONT_MAC(w0, w1, w2, u[37], m[24]);+	MONT_MAC(t0, t1, t2, u[38], m[23]);+	MONT_MAC(v0, v1, v2, u[39], m[22]);+	MONT_MAC(w0, w1, w2, u[40], m[21]);+	MONT_MAC(t0, t1, t2, u[41], m[20]);+	MONT_MAC(v0, v1, v2, u[42], m[19]);+	MONT_MAC(w0, w1, w2, u[43], m[18]);+	MONT_MAC(t0, t1, t2, u[44], m[17]);+	MONT_MAC(v0, v1, v2, u[45], m[16]);+	MONT_MAC(w0, w1, w2, u[46], m[15]);+	MONT_MAC(t0, t1, t2, u[47], m[14]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[13] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 62 */+	MONT_MUL(s0, s1, s2, a[15], a[47]);+	MONT_MUL(c0, c1, c2, a[16], a[46]);+	MONT_MAC(s0, s1, s2, a[17], a[45]);+	MONT_MAC(c0, c1, c2, a[18], a[44]);+	MONT_MAC(s0, s1, s2, a[19], a[43]);+	MONT_MAC(c0, c1, c2, a[20], a[42]);+	MONT_MAC(s0, s1, s2, a[21], a[41]);+	MONT_MAC(c0, c1, c2, a[22], a[40]);+	MONT_MAC(s0, s1, s2, a[23], a[39]);+	MONT_MAC(c0, c1, c2, a[24], a[38]);+	MONT_MAC(s0, s1, s2, a[25], a[37]);+	MONT_MAC(c0, c1, c2, a[26], a[36]);+	MONT_MAC(s0, s1, s2, a[27], a[35]);+	MONT_MAC(c0, c1, c2, a[28], a[34]);+	MONT_MAC(s0, s1, s2, a[29], a[33]);+	MONT_MAC(c0, c1, c2, a[30], a[32]);+	MONT_MAC(t0, t1, t2, a[31], a[31]);+	MONT_MUL(v0, v1, v2, u[15], m[47]);+	MONT_MUL(w0, w1, w2, u[16], m[46]);+	MONT_MAC(t0, t1, t2, u[17], m[45]);+	MONT_MAC(v0, v1, v2, u[18], m[44]);+	MONT_MAC(w0, w1, w2, u[19], m[43]);+	MONT_MAC(t0, t1, t2, u[20], m[42]);+	MONT_MAC(v0, v1, v2, u[21], m[41]);+	MONT_MAC(w0, w1, w2, u[22], m[40]);+	MONT_MAC(t0, t1, t2, u[23], m[39]);+	MONT_MAC(v0, v1, v2, u[24], m[38]);+	MONT_MAC(w0, w1, w2, u[25], m[37]);+	MONT_MAC(t0, t1, t2, u[26], m[36]);+	MONT_MAC(v0, v1, v2, u[27], m[35]);+	MONT_MAC(w0, w1, w2, u[28], m[34]);+	MONT_MAC(t0, t1, t2, u[29], m[33]);+	MONT_MAC(v0, v1, v2, u[30], m[32]);+	MONT_MAC(w0, w1, w2, u[31], m[31]);+	MONT_MAC(t0, t1, t2, u[32], m[30]);+	MONT_MAC(v0, v1, v2, u[33], m[29]);+	MONT_MAC(w0, w1, w2, u[34], m[28]);+	MONT_MAC(t0, t1, t2, u[35], m[27]);+	MONT_MAC(v0, v1, v2, u[36], m[26]);+	MONT_MAC(w0, w1, w2, u[37], m[25]);+	MONT_MAC(t0, t1, t2, u[38], m[24]);+	MONT_MAC(v0, v1, v2, u[39], m[23]);+	MONT_MAC(w0, w1, w2, u[40], m[22]);+	MONT_MAC(t0, t1, t2, u[41], m[21]);+	MONT_MAC(v0, v1, v2, u[42], m[20]);+	MONT_MAC(w0, w1, w2, u[43], m[19]);+	MONT_MAC(t0, t1, t2, u[44], m[18]);+	MONT_MAC(v0, v1, v2, u[45], m[17]);+	MONT_MAC(w0, w1, w2, u[46], m[16]);+	MONT_MAC(t0, t1, t2, u[47], m[15]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[14] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 63 */+	MONT_MUL(s0, s1, s2, a[16], a[47]);+	MONT_MUL(c0, c1, c2, a[17], a[46]);+	MONT_MAC(s0, s1, s2, a[18], a[45]);+	MONT_MAC(c0, c1, c2, a[19], a[44]);+	MONT_MAC(s0, s1, s2, a[20], a[43]);+	MONT_MAC(c0, c1, c2, a[21], a[42]);+	MONT_MAC(s0, s1, s2, a[22], a[41]);+	MONT_MAC(c0, c1, c2, a[23], a[40]);+	MONT_MAC(s0, s1, s2, a[24], a[39]);+	MONT_MAC(c0, c1, c2, a[25], a[38]);+	MONT_MAC(s0, s1, s2, a[26], a[37]);+	MONT_MAC(c0, c1, c2, a[27], a[36]);+	MONT_MAC(s0, s1, s2, a[28], a[35]);+	MONT_MAC(c0, c1, c2, a[29], a[34]);+	MONT_MAC(s0, s1, s2, a[30], a[33]);+	MONT_MAC(c0, c1, c2, a[31], a[32]);+	MONT_MAC(t0, t1, t2, u[16], m[47]);+	MONT_MUL(v0, v1, v2, u[17], m[46]);+	MONT_MUL(w0, w1, w2, u[18], m[45]);+	MONT_MAC(t0, t1, t2, u[19], m[44]);+	MONT_MAC(v0, v1, v2, u[20], m[43]);+	MONT_MAC(w0, w1, w2, u[21], m[42]);+	MONT_MAC(t0, t1, t2, u[22], m[41]);+	MONT_MAC(v0, v1, v2, u[23], m[40]);+	MONT_MAC(w0, w1, w2, u[24], m[39]);+	MONT_MAC(t0, t1, t2, u[25], m[38]);+	MONT_MAC(v0, v1, v2, u[26], m[37]);+	MONT_MAC(w0, w1, w2, u[27], m[36]);+	MONT_MAC(t0, t1, t2, u[28], m[35]);+	MONT_MAC(v0, v1, v2, u[29], m[34]);+	MONT_MAC(w0, w1, w2, u[30], m[33]);+	MONT_MAC(t0, t1, t2, u[31], m[32]);+	MONT_MAC(v0, v1, v2, u[32], m[31]);+	MONT_MAC(w0, w1, w2, u[33], m[30]);+	MONT_MAC(t0, t1, t2, u[34], m[29]);+	MONT_MAC(v0, v1, v2, u[35], m[28]);+	MONT_MAC(w0, w1, w2, u[36], m[27]);+	MONT_MAC(t0, t1, t2, u[37], m[26]);+	MONT_MAC(v0, v1, v2, u[38], m[25]);+	MONT_MAC(w0, w1, w2, u[39], m[24]);+	MONT_MAC(t0, t1, t2, u[40], m[23]);+	MONT_MAC(v0, v1, v2, u[41], m[22]);+	MONT_MAC(w0, w1, w2, u[42], m[21]);+	MONT_MAC(t0, t1, t2, u[43], m[20]);+	MONT_MAC(v0, v1, v2, u[44], m[19]);+	MONT_MAC(w0, w1, w2, u[45], m[18]);+	MONT_MAC(t0, t1, t2, u[46], m[17]);+	MONT_MAC(v0, v1, v2, u[47], m[16]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[15] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 64 */+	MONT_MUL(s0, s1, s2, a[17], a[47]);+	MONT_MUL(c0, c1, c2, a[18], a[46]);+	MONT_MAC(s0, s1, s2, a[19], a[45]);+	MONT_MAC(c0, c1, c2, a[20], a[44]);+	MONT_MAC(s0, s1, s2, a[21], a[43]);+	MONT_MAC(c0, c1, c2, a[22], a[42]);+	MONT_MAC(s0, s1, s2, a[23], a[41]);+	MONT_MAC(c0, c1, c2, a[24], a[40]);+	MONT_MAC(s0, s1, s2, a[25], a[39]);+	MONT_MAC(c0, c1, c2, a[26], a[38]);+	MONT_MAC(s0, s1, s2, a[27], a[37]);+	MONT_MAC(c0, c1, c2, a[28], a[36]);+	MONT_MAC(s0, s1, s2, a[29], a[35]);+	MONT_MAC(c0, c1, c2, a[30], a[34]);+	MONT_MAC(s0, s1, s2, a[31], a[33]);+	MONT_MAC(t0, t1, t2, a[32], a[32]);+	MONT_MUL(v0, v1, v2, u[17], m[47]);+	MONT_MUL(w0, w1, w2, u[18], m[46]);+	MONT_MAC(t0, t1, t2, u[19], m[45]);+	MONT_MAC(v0, v1, v2, u[20], m[44]);+	MONT_MAC(w0, w1, w2, u[21], m[43]);+	MONT_MAC(t0, t1, t2, u[22], m[42]);+	MONT_MAC(v0, v1, v2, u[23], m[41]);+	MONT_MAC(w0, w1, w2, u[24], m[40]);+	MONT_MAC(t0, t1, t2, u[25], m[39]);+	MONT_MAC(v0, v1, v2, u[26], m[38]);+	MONT_MAC(w0, w1, w2, u[27], m[37]);+	MONT_MAC(t0, t1, t2, u[28], m[36]);+	MONT_MAC(v0, v1, v2, u[29], m[35]);+	MONT_MAC(w0, w1, w2, u[30], m[34]);+	MONT_MAC(t0, t1, t2, u[31], m[33]);+	MONT_MAC(v0, v1, v2, u[32], m[32]);+	MONT_MAC(w0, w1, w2, u[33], m[31]);+	MONT_MAC(t0, t1, t2, u[34], m[30]);+	MONT_MAC(v0, v1, v2, u[35], m[29]);+	MONT_MAC(w0, w1, w2, u[36], m[28]);+	MONT_MAC(t0, t1, t2, u[37], m[27]);+	MONT_MAC(v0, v1, v2, u[38], m[26]);+	MONT_MAC(w0, w1, w2, u[39], m[25]);+	MONT_MAC(t0, t1, t2, u[40], m[24]);+	MONT_MAC(v0, v1, v2, u[41], m[23]);+	MONT_MAC(w0, w1, w2, u[42], m[22]);+	MONT_MAC(t0, t1, t2, u[43], m[21]);+	MONT_MAC(v0, v1, v2, u[44], m[20]);+	MONT_MAC(w0, w1, w2, u[45], m[19]);+	MONT_MAC(t0, t1, t2, u[46], m[18]);+	MONT_MAC(v0, v1, v2, u[47], m[17]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[16] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 65 */+	MONT_MUL(s0, s1, s2, a[18], a[47]);+	MONT_MUL(c0, c1, c2, a[19], a[46]);+	MONT_MAC(s0, s1, s2, a[20], a[45]);+	MONT_MAC(c0, c1, c2, a[21], a[44]);+	MONT_MAC(s0, s1, s2, a[22], a[43]);+	MONT_MAC(c0, c1, c2, a[23], a[42]);+	MONT_MAC(s0, s1, s2, a[24], a[41]);+	MONT_MAC(c0, c1, c2, a[25], a[40]);+	MONT_MAC(s0, s1, s2, a[26], a[39]);+	MONT_MAC(c0, c1, c2, a[27], a[38]);+	MONT_MAC(s0, s1, s2, a[28], a[37]);+	MONT_MAC(c0, c1, c2, a[29], a[36]);+	MONT_MAC(s0, s1, s2, a[30], a[35]);+	MONT_MAC(c0, c1, c2, a[31], a[34]);+	MONT_MAC(s0, s1, s2, a[32], a[33]);+	MONT_MAC(t0, t1, t2, u[18], m[47]);+	MONT_MUL(v0, v1, v2, u[19], m[46]);+	MONT_MUL(w0, w1, w2, u[20], m[45]);+	MONT_MAC(t0, t1, t2, u[21], m[44]);+	MONT_MAC(v0, v1, v2, u[22], m[43]);+	MONT_MAC(w0, w1, w2, u[23], m[42]);+	MONT_MAC(t0, t1, t2, u[24], m[41]);+	MONT_MAC(v0, v1, v2, u[25], m[40]);+	MONT_MAC(w0, w1, w2, u[26], m[39]);+	MONT_MAC(t0, t1, t2, u[27], m[38]);+	MONT_MAC(v0, v1, v2, u[28], m[37]);+	MONT_MAC(w0, w1, w2, u[29], m[36]);+	MONT_MAC(t0, t1, t2, u[30], m[35]);+	MONT_MAC(v0, v1, v2, u[31], m[34]);+	MONT_MAC(w0, w1, w2, u[32], m[33]);+	MONT_MAC(t0, t1, t2, u[33], m[32]);+	MONT_MAC(v0, v1, v2, u[34], m[31]);+	MONT_MAC(w0, w1, w2, u[35], m[30]);+	MONT_MAC(t0, t1, t2, u[36], m[29]);+	MONT_MAC(v0, v1, v2, u[37], m[28]);+	MONT_MAC(w0, w1, w2, u[38], m[27]);+	MONT_MAC(t0, t1, t2, u[39], m[26]);+	MONT_MAC(v0, v1, v2, u[40], m[25]);+	MONT_MAC(w0, w1, w2, u[41], m[24]);+	MONT_MAC(t0, t1, t2, u[42], m[23]);+	MONT_MAC(v0, v1, v2, u[43], m[22]);+	MONT_MAC(w0, w1, w2, u[44], m[21]);+	MONT_MAC(t0, t1, t2, u[45], m[20]);+	MONT_MAC(v0, v1, v2, u[46], m[19]);+	MONT_MAC(w0, w1, w2, u[47], m[18]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[17] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 66 */+	MONT_MUL(s0, s1, s2, a[19], a[47]);+	MONT_MUL(c0, c1, c2, a[20], a[46]);+	MONT_MAC(s0, s1, s2, a[21], a[45]);+	MONT_MAC(c0, c1, c2, a[22], a[44]);+	MONT_MAC(s0, s1, s2, a[23], a[43]);+	MONT_MAC(c0, c1, c2, a[24], a[42]);+	MONT_MAC(s0, s1, s2, a[25], a[41]);+	MONT_MAC(c0, c1, c2, a[26], a[40]);+	MONT_MAC(s0, s1, s2, a[27], a[39]);+	MONT_MAC(c0, c1, c2, a[28], a[38]);+	MONT_MAC(s0, s1, s2, a[29], a[37]);+	MONT_MAC(c0, c1, c2, a[30], a[36]);+	MONT_MAC(s0, s1, s2, a[31], a[35]);+	MONT_MAC(c0, c1, c2, a[32], a[34]);+	MONT_MAC(t0, t1, t2, a[33], a[33]);+	MONT_MUL(v0, v1, v2, u[19], m[47]);+	MONT_MUL(w0, w1, w2, u[20], m[46]);+	MONT_MAC(t0, t1, t2, u[21], m[45]);+	MONT_MAC(v0, v1, v2, u[22], m[44]);+	MONT_MAC(w0, w1, w2, u[23], m[43]);+	MONT_MAC(t0, t1, t2, u[24], m[42]);+	MONT_MAC(v0, v1, v2, u[25], m[41]);+	MONT_MAC(w0, w1, w2, u[26], m[40]);+	MONT_MAC(t0, t1, t2, u[27], m[39]);+	MONT_MAC(v0, v1, v2, u[28], m[38]);+	MONT_MAC(w0, w1, w2, u[29], m[37]);+	MONT_MAC(t0, t1, t2, u[30], m[36]);+	MONT_MAC(v0, v1, v2, u[31], m[35]);+	MONT_MAC(w0, w1, w2, u[32], m[34]);+	MONT_MAC(t0, t1, t2, u[33], m[33]);+	MONT_MAC(v0, v1, v2, u[34], m[32]);+	MONT_MAC(w0, w1, w2, u[35], m[31]);+	MONT_MAC(t0, t1, t2, u[36], m[30]);+	MONT_MAC(v0, v1, v2, u[37], m[29]);+	MONT_MAC(w0, w1, w2, u[38], m[28]);+	MONT_MAC(t0, t1, t2, u[39], m[27]);+	MONT_MAC(v0, v1, v2, u[40], m[26]);+	MONT_MAC(w0, w1, w2, u[41], m[25]);+	MONT_MAC(t0, t1, t2, u[42], m[24]);+	MONT_MAC(v0, v1, v2, u[43], m[23]);+	MONT_MAC(w0, w1, w2, u[44], m[22]);+	MONT_MAC(t0, t1, t2, u[45], m[21]);+	MONT_MAC(v0, v1, v2, u[46], m[20]);+	MONT_MAC(w0, w1, w2, u[47], m[19]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[18] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 67 */+	MONT_MUL(s0, s1, s2, a[20], a[47]);+	MONT_MUL(c0, c1, c2, a[21], a[46]);+	MONT_MAC(s0, s1, s2, a[22], a[45]);+	MONT_MAC(c0, c1, c2, a[23], a[44]);+	MONT_MAC(s0, s1, s2, a[24], a[43]);+	MONT_MAC(c0, c1, c2, a[25], a[42]);+	MONT_MAC(s0, s1, s2, a[26], a[41]);+	MONT_MAC(c0, c1, c2, a[27], a[40]);+	MONT_MAC(s0, s1, s2, a[28], a[39]);+	MONT_MAC(c0, c1, c2, a[29], a[38]);+	MONT_MAC(s0, s1, s2, a[30], a[37]);+	MONT_MAC(c0, c1, c2, a[31], a[36]);+	MONT_MAC(s0, s1, s2, a[32], a[35]);+	MONT_MAC(c0, c1, c2, a[33], a[34]);+	MONT_MAC(t0, t1, t2, u[20], m[47]);+	MONT_MUL(v0, v1, v2, u[21], m[46]);+	MONT_MUL(w0, w1, w2, u[22], m[45]);+	MONT_MAC(t0, t1, t2, u[23], m[44]);+	MONT_MAC(v0, v1, v2, u[24], m[43]);+	MONT_MAC(w0, w1, w2, u[25], m[42]);+	MONT_MAC(t0, t1, t2, u[26], m[41]);+	MONT_MAC(v0, v1, v2, u[27], m[40]);+	MONT_MAC(w0, w1, w2, u[28], m[39]);+	MONT_MAC(t0, t1, t2, u[29], m[38]);+	MONT_MAC(v0, v1, v2, u[30], m[37]);+	MONT_MAC(w0, w1, w2, u[31], m[36]);+	MONT_MAC(t0, t1, t2, u[32], m[35]);+	MONT_MAC(v0, v1, v2, u[33], m[34]);+	MONT_MAC(w0, w1, w2, u[34], m[33]);+	MONT_MAC(t0, t1, t2, u[35], m[32]);+	MONT_MAC(v0, v1, v2, u[36], m[31]);+	MONT_MAC(w0, w1, w2, u[37], m[30]);+	MONT_MAC(t0, t1, t2, u[38], m[29]);+	MONT_MAC(v0, v1, v2, u[39], m[28]);+	MONT_MAC(w0, w1, w2, u[40], m[27]);+	MONT_MAC(t0, t1, t2, u[41], m[26]);+	MONT_MAC(v0, v1, v2, u[42], m[25]);+	MONT_MAC(w0, w1, w2, u[43], m[24]);+	MONT_MAC(t0, t1, t2, u[44], m[23]);+	MONT_MAC(v0, v1, v2, u[45], m[22]);+	MONT_MAC(w0, w1, w2, u[46], m[21]);+	MONT_MAC(t0, t1, t2, u[47], m[20]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[19] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 68 */+	MONT_MUL(s0, s1, s2, a[21], a[47]);+	MONT_MUL(c0, c1, c2, a[22], a[46]);+	MONT_MAC(s0, s1, s2, a[23], a[45]);+	MONT_MAC(c0, c1, c2, a[24], a[44]);+	MONT_MAC(s0, s1, s2, a[25], a[43]);+	MONT_MAC(c0, c1, c2, a[26], a[42]);+	MONT_MAC(s0, s1, s2, a[27], a[41]);+	MONT_MAC(c0, c1, c2, a[28], a[40]);+	MONT_MAC(s0, s1, s2, a[29], a[39]);+	MONT_MAC(c0, c1, c2, a[30], a[38]);+	MONT_MAC(s0, s1, s2, a[31], a[37]);+	MONT_MAC(c0, c1, c2, a[32], a[36]);+	MONT_MAC(s0, s1, s2, a[33], a[35]);+	MONT_MAC(t0, t1, t2, a[34], a[34]);+	MONT_MUL(v0, v1, v2, u[21], m[47]);+	MONT_MUL(w0, w1, w2, u[22], m[46]);+	MONT_MAC(t0, t1, t2, u[23], m[45]);+	MONT_MAC(v0, v1, v2, u[24], m[44]);+	MONT_MAC(w0, w1, w2, u[25], m[43]);+	MONT_MAC(t0, t1, t2, u[26], m[42]);+	MONT_MAC(v0, v1, v2, u[27], m[41]);+	MONT_MAC(w0, w1, w2, u[28], m[40]);+	MONT_MAC(t0, t1, t2, u[29], m[39]);+	MONT_MAC(v0, v1, v2, u[30], m[38]);+	MONT_MAC(w0, w1, w2, u[31], m[37]);+	MONT_MAC(t0, t1, t2, u[32], m[36]);+	MONT_MAC(v0, v1, v2, u[33], m[35]);+	MONT_MAC(w0, w1, w2, u[34], m[34]);+	MONT_MAC(t0, t1, t2, u[35], m[33]);+	MONT_MAC(v0, v1, v2, u[36], m[32]);+	MONT_MAC(w0, w1, w2, u[37], m[31]);+	MONT_MAC(t0, t1, t2, u[38], m[30]);+	MONT_MAC(v0, v1, v2, u[39], m[29]);+	MONT_MAC(w0, w1, w2, u[40], m[28]);+	MONT_MAC(t0, t1, t2, u[41], m[27]);+	MONT_MAC(v0, v1, v2, u[42], m[26]);+	MONT_MAC(w0, w1, w2, u[43], m[25]);+	MONT_MAC(t0, t1, t2, u[44], m[24]);+	MONT_MAC(v0, v1, v2, u[45], m[23]);+	MONT_MAC(w0, w1, w2, u[46], m[22]);+	MONT_MAC(t0, t1, t2, u[47], m[21]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[20] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 69 */+	MONT_MUL(s0, s1, s2, a[22], a[47]);+	MONT_MUL(c0, c1, c2, a[23], a[46]);+	MONT_MAC(s0, s1, s2, a[24], a[45]);+	MONT_MAC(c0, c1, c2, a[25], a[44]);+	MONT_MAC(s0, s1, s2, a[26], a[43]);+	MONT_MAC(c0, c1, c2, a[27], a[42]);+	MONT_MAC(s0, s1, s2, a[28], a[41]);+	MONT_MAC(c0, c1, c2, a[29], a[40]);+	MONT_MAC(s0, s1, s2, a[30], a[39]);+	MONT_MAC(c0, c1, c2, a[31], a[38]);+	MONT_MAC(s0, s1, s2, a[32], a[37]);+	MONT_MAC(c0, c1, c2, a[33], a[36]);+	MONT_MAC(s0, s1, s2, a[34], a[35]);+	MONT_MAC(t0, t1, t2, u[22], m[47]);+	MONT_MUL(v0, v1, v2, u[23], m[46]);+	MONT_MUL(w0, w1, w2, u[24], m[45]);+	MONT_MAC(t0, t1, t2, u[25], m[44]);+	MONT_MAC(v0, v1, v2, u[26], m[43]);+	MONT_MAC(w0, w1, w2, u[27], m[42]);+	MONT_MAC(t0, t1, t2, u[28], m[41]);+	MONT_MAC(v0, v1, v2, u[29], m[40]);+	MONT_MAC(w0, w1, w2, u[30], m[39]);+	MONT_MAC(t0, t1, t2, u[31], m[38]);+	MONT_MAC(v0, v1, v2, u[32], m[37]);+	MONT_MAC(w0, w1, w2, u[33], m[36]);+	MONT_MAC(t0, t1, t2, u[34], m[35]);+	MONT_MAC(v0, v1, v2, u[35], m[34]);+	MONT_MAC(w0, w1, w2, u[36], m[33]);+	MONT_MAC(t0, t1, t2, u[37], m[32]);+	MONT_MAC(v0, v1, v2, u[38], m[31]);+	MONT_MAC(w0, w1, w2, u[39], m[30]);+	MONT_MAC(t0, t1, t2, u[40], m[29]);+	MONT_MAC(v0, v1, v2, u[41], m[28]);+	MONT_MAC(w0, w1, w2, u[42], m[27]);+	MONT_MAC(t0, t1, t2, u[43], m[26]);+	MONT_MAC(v0, v1, v2, u[44], m[25]);+	MONT_MAC(w0, w1, w2, u[45], m[24]);+	MONT_MAC(t0, t1, t2, u[46], m[23]);+	MONT_MAC(v0, v1, v2, u[47], m[22]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[21] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 70 */+	MONT_MUL(s0, s1, s2, a[23], a[47]);+	MONT_MUL(c0, c1, c2, a[24], a[46]);+	MONT_MAC(s0, s1, s2, a[25], a[45]);+	MONT_MAC(c0, c1, c2, a[26], a[44]);+	MONT_MAC(s0, s1, s2, a[27], a[43]);+	MONT_MAC(c0, c1, c2, a[28], a[42]);+	MONT_MAC(s0, s1, s2, a[29], a[41]);+	MONT_MAC(c0, c1, c2, a[30], a[40]);+	MONT_MAC(s0, s1, s2, a[31], a[39]);+	MONT_MAC(c0, c1, c2, a[32], a[38]);+	MONT_MAC(s0, s1, s2, a[33], a[37]);+	MONT_MAC(c0, c1, c2, a[34], a[36]);+	MONT_MAC(t0, t1, t2, a[35], a[35]);+	MONT_MUL(v0, v1, v2, u[23], m[47]);+	MONT_MUL(w0, w1, w2, u[24], m[46]);+	MONT_MAC(t0, t1, t2, u[25], m[45]);+	MONT_MAC(v0, v1, v2, u[26], m[44]);+	MONT_MAC(w0, w1, w2, u[27], m[43]);+	MONT_MAC(t0, t1, t2, u[28], m[42]);+	MONT_MAC(v0, v1, v2, u[29], m[41]);+	MONT_MAC(w0, w1, w2, u[30], m[40]);+	MONT_MAC(t0, t1, t2, u[31], m[39]);+	MONT_MAC(v0, v1, v2, u[32], m[38]);+	MONT_MAC(w0, w1, w2, u[33], m[37]);+	MONT_MAC(t0, t1, t2, u[34], m[36]);+	MONT_MAC(v0, v1, v2, u[35], m[35]);+	MONT_MAC(w0, w1, w2, u[36], m[34]);+	MONT_MAC(t0, t1, t2, u[37], m[33]);+	MONT_MAC(v0, v1, v2, u[38], m[32]);+	MONT_MAC(w0, w1, w2, u[39], m[31]);+	MONT_MAC(t0, t1, t2, u[40], m[30]);+	MONT_MAC(v0, v1, v2, u[41], m[29]);+	MONT_MAC(w0, w1, w2, u[42], m[28]);+	MONT_MAC(t0, t1, t2, u[43], m[27]);+	MONT_MAC(v0, v1, v2, u[44], m[26]);+	MONT_MAC(w0, w1, w2, u[45], m[25]);+	MONT_MAC(t0, t1, t2, u[46], m[24]);+	MONT_MAC(v0, v1, v2, u[47], m[23]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[22] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 71 */+	MONT_MUL(s0, s1, s2, a[24], a[47]);+	MONT_MUL(c0, c1, c2, a[25], a[46]);+	MONT_MAC(s0, s1, s2, a[26], a[45]);+	MONT_MAC(c0, c1, c2, a[27], a[44]);+	MONT_MAC(s0, s1, s2, a[28], a[43]);+	MONT_MAC(c0, c1, c2, a[29], a[42]);+	MONT_MAC(s0, s1, s2, a[30], a[41]);+	MONT_MAC(c0, c1, c2, a[31], a[40]);+	MONT_MAC(s0, s1, s2, a[32], a[39]);+	MONT_MAC(c0, c1, c2, a[33], a[38]);+	MONT_MAC(s0, s1, s2, a[34], a[37]);+	MONT_MAC(c0, c1, c2, a[35], a[36]);+	MONT_MAC(t0, t1, t2, u[24], m[47]);+	MONT_MUL(v0, v1, v2, u[25], m[46]);+	MONT_MUL(w0, w1, w2, u[26], m[45]);+	MONT_MAC(t0, t1, t2, u[27], m[44]);+	MONT_MAC(v0, v1, v2, u[28], m[43]);+	MONT_MAC(w0, w1, w2, u[29], m[42]);+	MONT_MAC(t0, t1, t2, u[30], m[41]);+	MONT_MAC(v0, v1, v2, u[31], m[40]);+	MONT_MAC(w0, w1, w2, u[32], m[39]);+	MONT_MAC(t0, t1, t2, u[33], m[38]);+	MONT_MAC(v0, v1, v2, u[34], m[37]);+	MONT_MAC(w0, w1, w2, u[35], m[36]);+	MONT_MAC(t0, t1, t2, u[36], m[35]);+	MONT_MAC(v0, v1, v2, u[37], m[34]);+	MONT_MAC(w0, w1, w2, u[38], m[33]);+	MONT_MAC(t0, t1, t2, u[39], m[32]);+	MONT_MAC(v0, v1, v2, u[40], m[31]);+	MONT_MAC(w0, w1, w2, u[41], m[30]);+	MONT_MAC(t0, t1, t2, u[42], m[29]);+	MONT_MAC(v0, v1, v2, u[43], m[28]);+	MONT_MAC(w0, w1, w2, u[44], m[27]);+	MONT_MAC(t0, t1, t2, u[45], m[26]);+	MONT_MAC(v0, v1, v2, u[46], m[25]);+	MONT_MAC(w0, w1, w2, u[47], m[24]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[23] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 72 */+	MONT_MUL(s0, s1, s2, a[25], a[47]);+	MONT_MUL(c0, c1, c2, a[26], a[46]);+	MONT_MAC(s0, s1, s2, a[27], a[45]);+	MONT_MAC(c0, c1, c2, a[28], a[44]);+	MONT_MAC(s0, s1, s2, a[29], a[43]);+	MONT_MAC(c0, c1, c2, a[30], a[42]);+	MONT_MAC(s0, s1, s2, a[31], a[41]);+	MONT_MAC(c0, c1, c2, a[32], a[40]);+	MONT_MAC(s0, s1, s2, a[33], a[39]);+	MONT_MAC(c0, c1, c2, a[34], a[38]);+	MONT_MAC(s0, s1, s2, a[35], a[37]);+	MONT_MAC(t0, t1, t2, a[36], a[36]);+	MONT_MUL(v0, v1, v2, u[25], m[47]);+	MONT_MUL(w0, w1, w2, u[26], m[46]);+	MONT_MAC(t0, t1, t2, u[27], m[45]);+	MONT_MAC(v0, v1, v2, u[28], m[44]);+	MONT_MAC(w0, w1, w2, u[29], m[43]);+	MONT_MAC(t0, t1, t2, u[30], m[42]);+	MONT_MAC(v0, v1, v2, u[31], m[41]);+	MONT_MAC(w0, w1, w2, u[32], m[40]);+	MONT_MAC(t0, t1, t2, u[33], m[39]);+	MONT_MAC(v0, v1, v2, u[34], m[38]);+	MONT_MAC(w0, w1, w2, u[35], m[37]);+	MONT_MAC(t0, t1, t2, u[36], m[36]);+	MONT_MAC(v0, v1, v2, u[37], m[35]);+	MONT_MAC(w0, w1, w2, u[38], m[34]);+	MONT_MAC(t0, t1, t2, u[39], m[33]);+	MONT_MAC(v0, v1, v2, u[40], m[32]);+	MONT_MAC(w0, w1, w2, u[41], m[31]);+	MONT_MAC(t0, t1, t2, u[42], m[30]);+	MONT_MAC(v0, v1, v2, u[43], m[29]);+	MONT_MAC(w0, w1, w2, u[44], m[28]);+	MONT_MAC(t0, t1, t2, u[45], m[27]);+	MONT_MAC(v0, v1, v2, u[46], m[26]);+	MONT_MAC(w0, w1, w2, u[47], m[25]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[24] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 73 */+	MONT_MUL(s0, s1, s2, a[26], a[47]);+	MONT_MUL(c0, c1, c2, a[27], a[46]);+	MONT_MAC(s0, s1, s2, a[28], a[45]);+	MONT_MAC(c0, c1, c2, a[29], a[44]);+	MONT_MAC(s0, s1, s2, a[30], a[43]);+	MONT_MAC(c0, c1, c2, a[31], a[42]);+	MONT_MAC(s0, s1, s2, a[32], a[41]);+	MONT_MAC(c0, c1, c2, a[33], a[40]);+	MONT_MAC(s0, s1, s2, a[34], a[39]);+	MONT_MAC(c0, c1, c2, a[35], a[38]);+	MONT_MAC(s0, s1, s2, a[36], a[37]);+	MONT_MAC(t0, t1, t2, u[26], m[47]);+	MONT_MUL(v0, v1, v2, u[27], m[46]);+	MONT_MUL(w0, w1, w2, u[28], m[45]);+	MONT_MAC(t0, t1, t2, u[29], m[44]);+	MONT_MAC(v0, v1, v2, u[30], m[43]);+	MONT_MAC(w0, w1, w2, u[31], m[42]);+	MONT_MAC(t0, t1, t2, u[32], m[41]);+	MONT_MAC(v0, v1, v2, u[33], m[40]);+	MONT_MAC(w0, w1, w2, u[34], m[39]);+	MONT_MAC(t0, t1, t2, u[35], m[38]);+	MONT_MAC(v0, v1, v2, u[36], m[37]);+	MONT_MAC(w0, w1, w2, u[37], m[36]);+	MONT_MAC(t0, t1, t2, u[38], m[35]);+	MONT_MAC(v0, v1, v2, u[39], m[34]);+	MONT_MAC(w0, w1, w2, u[40], m[33]);+	MONT_MAC(t0, t1, t2, u[41], m[32]);+	MONT_MAC(v0, v1, v2, u[42], m[31]);+	MONT_MAC(w0, w1, w2, u[43], m[30]);+	MONT_MAC(t0, t1, t2, u[44], m[29]);+	MONT_MAC(v0, v1, v2, u[45], m[28]);+	MONT_MAC(w0, w1, w2, u[46], m[27]);+	MONT_MAC(t0, t1, t2, u[47], m[26]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[25] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 74 */+	MONT_MUL(s0, s1, s2, a[27], a[47]);+	MONT_MUL(c0, c1, c2, a[28], a[46]);+	MONT_MAC(s0, s1, s2, a[29], a[45]);+	MONT_MAC(c0, c1, c2, a[30], a[44]);+	MONT_MAC(s0, s1, s2, a[31], a[43]);+	MONT_MAC(c0, c1, c2, a[32], a[42]);+	MONT_MAC(s0, s1, s2, a[33], a[41]);+	MONT_MAC(c0, c1, c2, a[34], a[40]);+	MONT_MAC(s0, s1, s2, a[35], a[39]);+	MONT_MAC(c0, c1, c2, a[36], a[38]);+	MONT_MAC(t0, t1, t2, a[37], a[37]);+	MONT_MUL(v0, v1, v2, u[27], m[47]);+	MONT_MUL(w0, w1, w2, u[28], m[46]);+	MONT_MAC(t0, t1, t2, u[29], m[45]);+	MONT_MAC(v0, v1, v2, u[30], m[44]);+	MONT_MAC(w0, w1, w2, u[31], m[43]);+	MONT_MAC(t0, t1, t2, u[32], m[42]);+	MONT_MAC(v0, v1, v2, u[33], m[41]);+	MONT_MAC(w0, w1, w2, u[34], m[40]);+	MONT_MAC(t0, t1, t2, u[35], m[39]);+	MONT_MAC(v0, v1, v2, u[36], m[38]);+	MONT_MAC(w0, w1, w2, u[37], m[37]);+	MONT_MAC(t0, t1, t2, u[38], m[36]);+	MONT_MAC(v0, v1, v2, u[39], m[35]);+	MONT_MAC(w0, w1, w2, u[40], m[34]);+	MONT_MAC(t0, t1, t2, u[41], m[33]);+	MONT_MAC(v0, v1, v2, u[42], m[32]);+	MONT_MAC(w0, w1, w2, u[43], m[31]);+	MONT_MAC(t0, t1, t2, u[44], m[30]);+	MONT_MAC(v0, v1, v2, u[45], m[29]);+	MONT_MAC(w0, w1, w2, u[46], m[28]);+	MONT_MAC(t0, t1, t2, u[47], m[27]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[26] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 75 */+	MONT_MUL(s0, s1, s2, a[28], a[47]);+	MONT_MUL(c0, c1, c2, a[29], a[46]);+	MONT_MAC(s0, s1, s2, a[30], a[45]);+	MONT_MAC(c0, c1, c2, a[31], a[44]);+	MONT_MAC(s0, s1, s2, a[32], a[43]);+	MONT_MAC(c0, c1, c2, a[33], a[42]);+	MONT_MAC(s0, s1, s2, a[34], a[41]);+	MONT_MAC(c0, c1, c2, a[35], a[40]);+	MONT_MAC(s0, s1, s2, a[36], a[39]);+	MONT_MAC(c0, c1, c2, a[37], a[38]);+	MONT_MAC(t0, t1, t2, u[28], m[47]);+	MONT_MUL(v0, v1, v2, u[29], m[46]);+	MONT_MUL(w0, w1, w2, u[30], m[45]);+	MONT_MAC(t0, t1, t2, u[31], m[44]);+	MONT_MAC(v0, v1, v2, u[32], m[43]);+	MONT_MAC(w0, w1, w2, u[33], m[42]);+	MONT_MAC(t0, t1, t2, u[34], m[41]);+	MONT_MAC(v0, v1, v2, u[35], m[40]);+	MONT_MAC(w0, w1, w2, u[36], m[39]);+	MONT_MAC(t0, t1, t2, u[37], m[38]);+	MONT_MAC(v0, v1, v2, u[38], m[37]);+	MONT_MAC(w0, w1, w2, u[39], m[36]);+	MONT_MAC(t0, t1, t2, u[40], m[35]);+	MONT_MAC(v0, v1, v2, u[41], m[34]);+	MONT_MAC(w0, w1, w2, u[42], m[33]);+	MONT_MAC(t0, t1, t2, u[43], m[32]);+	MONT_MAC(v0, v1, v2, u[44], m[31]);+	MONT_MAC(w0, w1, w2, u[45], m[30]);+	MONT_MAC(t0, t1, t2, u[46], m[29]);+	MONT_MAC(v0, v1, v2, u[47], m[28]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[27] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 76 */+	MONT_MUL(s0, s1, s2, a[29], a[47]);+	MONT_MUL(c0, c1, c2, a[30], a[46]);+	MONT_MAC(s0, s1, s2, a[31], a[45]);+	MONT_MAC(c0, c1, c2, a[32], a[44]);+	MONT_MAC(s0, s1, s2, a[33], a[43]);+	MONT_MAC(c0, c1, c2, a[34], a[42]);+	MONT_MAC(s0, s1, s2, a[35], a[41]);+	MONT_MAC(c0, c1, c2, a[36], a[40]);+	MONT_MAC(s0, s1, s2, a[37], a[39]);+	MONT_MAC(t0, t1, t2, a[38], a[38]);+	MONT_MUL(v0, v1, v2, u[29], m[47]);+	MONT_MUL(w0, w1, w2, u[30], m[46]);+	MONT_MAC(t0, t1, t2, u[31], m[45]);+	MONT_MAC(v0, v1, v2, u[32], m[44]);+	MONT_MAC(w0, w1, w2, u[33], m[43]);+	MONT_MAC(t0, t1, t2, u[34], m[42]);+	MONT_MAC(v0, v1, v2, u[35], m[41]);+	MONT_MAC(w0, w1, w2, u[36], m[40]);+	MONT_MAC(t0, t1, t2, u[37], m[39]);+	MONT_MAC(v0, v1, v2, u[38], m[38]);+	MONT_MAC(w0, w1, w2, u[39], m[37]);+	MONT_MAC(t0, t1, t2, u[40], m[36]);+	MONT_MAC(v0, v1, v2, u[41], m[35]);+	MONT_MAC(w0, w1, w2, u[42], m[34]);+	MONT_MAC(t0, t1, t2, u[43], m[33]);+	MONT_MAC(v0, v1, v2, u[44], m[32]);+	MONT_MAC(w0, w1, w2, u[45], m[31]);+	MONT_MAC(t0, t1, t2, u[46], m[30]);+	MONT_MAC(v0, v1, v2, u[47], m[29]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[28] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 77 */+	MONT_MUL(s0, s1, s2, a[30], a[47]);+	MONT_MUL(c0, c1, c2, a[31], a[46]);+	MONT_MAC(s0, s1, s2, a[32], a[45]);+	MONT_MAC(c0, c1, c2, a[33], a[44]);+	MONT_MAC(s0, s1, s2, a[34], a[43]);+	MONT_MAC(c0, c1, c2, a[35], a[42]);+	MONT_MAC(s0, s1, s2, a[36], a[41]);+	MONT_MAC(c0, c1, c2, a[37], a[40]);+	MONT_MAC(s0, s1, s2, a[38], a[39]);+	MONT_MAC(t0, t1, t2, u[30], m[47]);+	MONT_MUL(v0, v1, v2, u[31], m[46]);+	MONT_MUL(w0, w1, w2, u[32], m[45]);+	MONT_MAC(t0, t1, t2, u[33], m[44]);+	MONT_MAC(v0, v1, v2, u[34], m[43]);+	MONT_MAC(w0, w1, w2, u[35], m[42]);+	MONT_MAC(t0, t1, t2, u[36], m[41]);+	MONT_MAC(v0, v1, v2, u[37], m[40]);+	MONT_MAC(w0, w1, w2, u[38], m[39]);+	MONT_MAC(t0, t1, t2, u[39], m[38]);+	MONT_MAC(v0, v1, v2, u[40], m[37]);+	MONT_MAC(w0, w1, w2, u[41], m[36]);+	MONT_MAC(t0, t1, t2, u[42], m[35]);+	MONT_MAC(v0, v1, v2, u[43], m[34]);+	MONT_MAC(w0, w1, w2, u[44], m[33]);+	MONT_MAC(t0, t1, t2, u[45], m[32]);+	MONT_MAC(v0, v1, v2, u[46], m[31]);+	MONT_MAC(w0, w1, w2, u[47], m[30]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[29] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 78 */+	MONT_MUL(s0, s1, s2, a[31], a[47]);+	MONT_MUL(c0, c1, c2, a[32], a[46]);+	MONT_MAC(s0, s1, s2, a[33], a[45]);+	MONT_MAC(c0, c1, c2, a[34], a[44]);+	MONT_MAC(s0, s1, s2, a[35], a[43]);+	MONT_MAC(c0, c1, c2, a[36], a[42]);+	MONT_MAC(s0, s1, s2, a[37], a[41]);+	MONT_MAC(c0, c1, c2, a[38], a[40]);+	MONT_MAC(t0, t1, t2, a[39], a[39]);+	MONT_MUL(v0, v1, v2, u[31], m[47]);+	MONT_MUL(w0, w1, w2, u[32], m[46]);+	MONT_MAC(t0, t1, t2, u[33], m[45]);+	MONT_MAC(v0, v1, v2, u[34], m[44]);+	MONT_MAC(w0, w1, w2, u[35], m[43]);+	MONT_MAC(t0, t1, t2, u[36], m[42]);+	MONT_MAC(v0, v1, v2, u[37], m[41]);+	MONT_MAC(w0, w1, w2, u[38], m[40]);+	MONT_MAC(t0, t1, t2, u[39], m[39]);+	MONT_MAC(v0, v1, v2, u[40], m[38]);+	MONT_MAC(w0, w1, w2, u[41], m[37]);+	MONT_MAC(t0, t1, t2, u[42], m[36]);+	MONT_MAC(v0, v1, v2, u[43], m[35]);+	MONT_MAC(w0, w1, w2, u[44], m[34]);+	MONT_MAC(t0, t1, t2, u[45], m[33]);+	MONT_MAC(v0, v1, v2, u[46], m[32]);+	MONT_MAC(w0, w1, w2, u[47], m[31]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[30] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 79 */+	MONT_MUL(s0, s1, s2, a[32], a[47]);+	MONT_MUL(c0, c1, c2, a[33], a[46]);+	MONT_MAC(s0, s1, s2, a[34], a[45]);+	MONT_MAC(c0, c1, c2, a[35], a[44]);+	MONT_MAC(s0, s1, s2, a[36], a[43]);+	MONT_MAC(c0, c1, c2, a[37], a[42]);+	MONT_MAC(s0, s1, s2, a[38], a[41]);+	MONT_MAC(c0, c1, c2, a[39], a[40]);+	MONT_MAC(t0, t1, t2, u[32], m[47]);+	MONT_MUL(v0, v1, v2, u[33], m[46]);+	MONT_MUL(w0, w1, w2, u[34], m[45]);+	MONT_MAC(t0, t1, t2, u[35], m[44]);+	MONT_MAC(v0, v1, v2, u[36], m[43]);+	MONT_MAC(w0, w1, w2, u[37], m[42]);+	MONT_MAC(t0, t1, t2, u[38], m[41]);+	MONT_MAC(v0, v1, v2, u[39], m[40]);+	MONT_MAC(w0, w1, w2, u[40], m[39]);+	MONT_MAC(t0, t1, t2, u[41], m[38]);+	MONT_MAC(v0, v1, v2, u[42], m[37]);+	MONT_MAC(w0, w1, w2, u[43], m[36]);+	MONT_MAC(t0, t1, t2, u[44], m[35]);+	MONT_MAC(v0, v1, v2, u[45], m[34]);+	MONT_MAC(w0, w1, w2, u[46], m[33]);+	MONT_MAC(t0, t1, t2, u[47], m[32]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[31] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 80 */+	MONT_MUL(s0, s1, s2, a[33], a[47]);+	MONT_MUL(c0, c1, c2, a[34], a[46]);+	MONT_MAC(s0, s1, s2, a[35], a[45]);+	MONT_MAC(c0, c1, c2, a[36], a[44]);+	MONT_MAC(s0, s1, s2, a[37], a[43]);+	MONT_MAC(c0, c1, c2, a[38], a[42]);+	MONT_MAC(s0, s1, s2, a[39], a[41]);+	MONT_MAC(t0, t1, t2, a[40], a[40]);+	MONT_MUL(v0, v1, v2, u[33], m[47]);+	MONT_MUL(w0, w1, w2, u[34], m[46]);+	MONT_MAC(t0, t1, t2, u[35], m[45]);+	MONT_MAC(v0, v1, v2, u[36], m[44]);+	MONT_MAC(w0, w1, w2, u[37], m[43]);+	MONT_MAC(t0, t1, t2, u[38], m[42]);+	MONT_MAC(v0, v1, v2, u[39], m[41]);+	MONT_MAC(w0, w1, w2, u[40], m[40]);+	MONT_MAC(t0, t1, t2, u[41], m[39]);+	MONT_MAC(v0, v1, v2, u[42], m[38]);+	MONT_MAC(w0, w1, w2, u[43], m[37]);+	MONT_MAC(t0, t1, t2, u[44], m[36]);+	MONT_MAC(v0, v1, v2, u[45], m[35]);+	MONT_MAC(w0, w1, w2, u[46], m[34]);+	MONT_MAC(t0, t1, t2, u[47], m[33]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[32] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 81 */+	MONT_MUL(s0, s1, s2, a[34], a[47]);+	MONT_MUL(c0, c1, c2, a[35], a[46]);+	MONT_MAC(s0, s1, s2, a[36], a[45]);+	MONT_MAC(c0, c1, c2, a[37], a[44]);+	MONT_MAC(s0, s1, s2, a[38], a[43]);+	MONT_MAC(c0, c1, c2, a[39], a[42]);+	MONT_MAC(s0, s1, s2, a[40], a[41]);+	MONT_MAC(t0, t1, t2, u[34], m[47]);+	MONT_MUL(v0, v1, v2, u[35], m[46]);+	MONT_MUL(w0, w1, w2, u[36], m[45]);+	MONT_MAC(t0, t1, t2, u[37], m[44]);+	MONT_MAC(v0, v1, v2, u[38], m[43]);+	MONT_MAC(w0, w1, w2, u[39], m[42]);+	MONT_MAC(t0, t1, t2, u[40], m[41]);+	MONT_MAC(v0, v1, v2, u[41], m[40]);+	MONT_MAC(w0, w1, w2, u[42], m[39]);+	MONT_MAC(t0, t1, t2, u[43], m[38]);+	MONT_MAC(v0, v1, v2, u[44], m[37]);+	MONT_MAC(w0, w1, w2, u[45], m[36]);+	MONT_MAC(t0, t1, t2, u[46], m[35]);+	MONT_MAC(v0, v1, v2, u[47], m[34]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[33] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 82 */+	MONT_MUL(s0, s1, s2, a[35], a[47]);+	MONT_MUL(c0, c1, c2, a[36], a[46]);+	MONT_MAC(s0, s1, s2, a[37], a[45]);+	MONT_MAC(c0, c1, c2, a[38], a[44]);+	MONT_MAC(s0, s1, s2, a[39], a[43]);+	MONT_MAC(c0, c1, c2, a[40], a[42]);+	MONT_MAC(t0, t1, t2, a[41], a[41]);+	MONT_MUL(v0, v1, v2, u[35], m[47]);+	MONT_MUL(w0, w1, w2, u[36], m[46]);+	MONT_MAC(t0, t1, t2, u[37], m[45]);+	MONT_MAC(v0, v1, v2, u[38], m[44]);+	MONT_MAC(w0, w1, w2, u[39], m[43]);+	MONT_MAC(t0, t1, t2, u[40], m[42]);+	MONT_MAC(v0, v1, v2, u[41], m[41]);+	MONT_MAC(w0, w1, w2, u[42], m[40]);+	MONT_MAC(t0, t1, t2, u[43], m[39]);+	MONT_MAC(v0, v1, v2, u[44], m[38]);+	MONT_MAC(w0, w1, w2, u[45], m[37]);+	MONT_MAC(t0, t1, t2, u[46], m[36]);+	MONT_MAC(v0, v1, v2, u[47], m[35]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[34] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 83 */+	MONT_MUL(s0, s1, s2, a[36], a[47]);+	MONT_MUL(c0, c1, c2, a[37], a[46]);+	MONT_MAC(s0, s1, s2, a[38], a[45]);+	MONT_MAC(c0, c1, c2, a[39], a[44]);+	MONT_MAC(s0, s1, s2, a[40], a[43]);+	MONT_MAC(c0, c1, c2, a[41], a[42]);+	MONT_MAC(t0, t1, t2, u[36], m[47]);+	MONT_MUL(v0, v1, v2, u[37], m[46]);+	MONT_MUL(w0, w1, w2, u[38], m[45]);+	MONT_MAC(t0, t1, t2, u[39], m[44]);+	MONT_MAC(v0, v1, v2, u[40], m[43]);+	MONT_MAC(w0, w1, w2, u[41], m[42]);+	MONT_MAC(t0, t1, t2, u[42], m[41]);+	MONT_MAC(v0, v1, v2, u[43], m[40]);+	MONT_MAC(w0, w1, w2, u[44], m[39]);+	MONT_MAC(t0, t1, t2, u[45], m[38]);+	MONT_MAC(v0, v1, v2, u[46], m[37]);+	MONT_MAC(w0, w1, w2, u[47], m[36]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[35] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 84 */+	MONT_MUL(s0, s1, s2, a[37], a[47]);+	MONT_MUL(c0, c1, c2, a[38], a[46]);+	MONT_MAC(s0, s1, s2, a[39], a[45]);+	MONT_MAC(c0, c1, c2, a[40], a[44]);+	MONT_MAC(s0, s1, s2, a[41], a[43]);+	MONT_MAC(t0, t1, t2, a[42], a[42]);+	MONT_MUL(v0, v1, v2, u[37], m[47]);+	MONT_MUL(w0, w1, w2, u[38], m[46]);+	MONT_MAC(t0, t1, t2, u[39], m[45]);+	MONT_MAC(v0, v1, v2, u[40], m[44]);+	MONT_MAC(w0, w1, w2, u[41], m[43]);+	MONT_MAC(t0, t1, t2, u[42], m[42]);+	MONT_MAC(v0, v1, v2, u[43], m[41]);+	MONT_MAC(w0, w1, w2, u[44], m[40]);+	MONT_MAC(t0, t1, t2, u[45], m[39]);+	MONT_MAC(v0, v1, v2, u[46], m[38]);+	MONT_MAC(w0, w1, w2, u[47], m[37]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[36] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 85 */+	MONT_MUL(s0, s1, s2, a[38], a[47]);+	MONT_MUL(c0, c1, c2, a[39], a[46]);+	MONT_MAC(s0, s1, s2, a[40], a[45]);+	MONT_MAC(c0, c1, c2, a[41], a[44]);+	MONT_MAC(s0, s1, s2, a[42], a[43]);+	MONT_MAC(t0, t1, t2, u[38], m[47]);+	MONT_MUL(v0, v1, v2, u[39], m[46]);+	MONT_MUL(w0, w1, w2, u[40], m[45]);+	MONT_MAC(t0, t1, t2, u[41], m[44]);+	MONT_MAC(v0, v1, v2, u[42], m[43]);+	MONT_MAC(w0, w1, w2, u[43], m[42]);+	MONT_MAC(t0, t1, t2, u[44], m[41]);+	MONT_MAC(v0, v1, v2, u[45], m[40]);+	MONT_MAC(w0, w1, w2, u[46], m[39]);+	MONT_MAC(t0, t1, t2, u[47], m[38]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[37] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 86 */+	MONT_MUL(s0, s1, s2, a[39], a[47]);+	MONT_MUL(c0, c1, c2, a[40], a[46]);+	MONT_MAC(s0, s1, s2, a[41], a[45]);+	MONT_MAC(c0, c1, c2, a[42], a[44]);+	MONT_MAC(t0, t1, t2, a[43], a[43]);+	MONT_MUL(v0, v1, v2, u[39], m[47]);+	MONT_MUL(w0, w1, w2, u[40], m[46]);+	MONT_MAC(t0, t1, t2, u[41], m[45]);+	MONT_MAC(v0, v1, v2, u[42], m[44]);+	MONT_MAC(w0, w1, w2, u[43], m[43]);+	MONT_MAC(t0, t1, t2, u[44], m[42]);+	MONT_MAC(v0, v1, v2, u[45], m[41]);+	MONT_MAC(w0, w1, w2, u[46], m[40]);+	MONT_MAC(t0, t1, t2, u[47], m[39]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[38] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 87 */+	MONT_MUL(s0, s1, s2, a[40], a[47]);+	MONT_MUL(c0, c1, c2, a[41], a[46]);+	MONT_MAC(s0, s1, s2, a[42], a[45]);+	MONT_MAC(c0, c1, c2, a[43], a[44]);+	MONT_MAC(t0, t1, t2, u[40], m[47]);+	MONT_MUL(v0, v1, v2, u[41], m[46]);+	MONT_MUL(w0, w1, w2, u[42], m[45]);+	MONT_MAC(t0, t1, t2, u[43], m[44]);+	MONT_MAC(v0, v1, v2, u[44], m[43]);+	MONT_MAC(w0, w1, w2, u[45], m[42]);+	MONT_MAC(t0, t1, t2, u[46], m[41]);+	MONT_MAC(v0, v1, v2, u[47], m[40]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[39] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 88 */+	MONT_MUL(s0, s1, s2, a[41], a[47]);+	MONT_MUL(c0, c1, c2, a[42], a[46]);+	MONT_MAC(s0, s1, s2, a[43], a[45]);+	MONT_MAC(t0, t1, t2, a[44], a[44]);+	MONT_MUL(v0, v1, v2, u[41], m[47]);+	MONT_MUL(w0, w1, w2, u[42], m[46]);+	MONT_MAC(t0, t1, t2, u[43], m[45]);+	MONT_MAC(v0, v1, v2, u[44], m[44]);+	MONT_MAC(w0, w1, w2, u[45], m[43]);+	MONT_MAC(t0, t1, t2, u[46], m[42]);+	MONT_MAC(v0, v1, v2, u[47], m[41]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[40] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 89 */+	MONT_MUL(s0, s1, s2, a[42], a[47]);+	MONT_MUL(c0, c1, c2, a[43], a[46]);+	MONT_MAC(s0, s1, s2, a[44], a[45]);+	MONT_MAC(t0, t1, t2, u[42], m[47]);+	MONT_MUL(v0, v1, v2, u[43], m[46]);+	MONT_MUL(w0, w1, w2, u[44], m[45]);+	MONT_MAC(t0, t1, t2, u[45], m[44]);+	MONT_MAC(v0, v1, v2, u[46], m[43]);+	MONT_MAC(w0, w1, w2, u[47], m[42]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[41] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 90 */+	MONT_MUL(s0, s1, s2, a[43], a[47]);+	MONT_MUL(c0, c1, c2, a[44], a[46]);+	MONT_MAC(t0, t1, t2, a[45], a[45]);+	MONT_MUL(v0, v1, v2, u[43], m[47]);+	MONT_MUL(w0, w1, w2, u[44], m[46]);+	MONT_MAC(t0, t1, t2, u[45], m[45]);+	MONT_MAC(v0, v1, v2, u[46], m[44]);+	MONT_MAC(w0, w1, w2, u[47], m[43]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[42] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 91 */+	MONT_MUL(s0, s1, s2, a[44], a[47]);+	MONT_MUL(c0, c1, c2, a[45], a[46]);+	MONT_MAC(t0, t1, t2, u[44], m[47]);+	MONT_MUL(v0, v1, v2, u[45], m[46]);+	MONT_MUL(w0, w1, w2, u[46], m[45]);+	MONT_MAC(t0, t1, t2, u[47], m[44]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_ADD(s0, s1, s2, c0, c1, c2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[43] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 92 */+	MONT_MUL(s0, s1, s2, a[45], a[47]);+	MONT_MAC(t0, t1, t2, a[46], a[46]);+	MONT_MUL(v0, v1, v2, u[45], m[47]);+	MONT_MUL(w0, w1, w2, u[46], m[46]);+	MONT_MAC(t0, t1, t2, u[47], m[45]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_ADD(t0, t1, t2, w0, w1, w2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[44] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 93 */+	MONT_MUL(s0, s1, s2, a[46], a[47]);+	MONT_MAC(t0, t1, t2, u[46], m[47]);+	MONT_MUL(v0, v1, v2, u[47], m[46]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	MONT_DBL(s0, s1, s2);+	MONT_ADD(t0, t1, t2, s0, s1, s2);+	z[45] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	/* column 94 */+	MONT_MAC(t0, t1, t2, a[47], a[47]);+	MONT_MUL(v0, v1, v2, u[47], m[47]);+	MONT_ADD(t0, t1, t2, v0, v1, v2);+	z[46] = t0;+	t0 = t1;+	t1 = t2;+	t2 = 0;+	z[47] = t0;++	/* under 2m, with t1 the bit above n limbs: at most one subtraction */+	borrow = sub_n(r, z, m, 48);+	take = t1 | (borrow ^ 1);+	select_n(r, r, z, take & 1, 48);+}++#endif
+ cbits/crypton_mont.py view
@@ -0,0 +1,173 @@+#!/usr/bin/env python3+"""Generate cbits/crypton_mont.h: Montgomery multiplication and squaring for+AArch64 at the limb counts of RSA's CRT halves, fully unrolled and+product-scanning.++Usage: python3 cbits/crypton_mont.py > cbits/crypton_mont.h++The result is a column at a time, with the reduction folded into the same+scan (the FIPS method), so the running sum stays in registers instead of+going through memory a row at a time.  A column's products are spread over+three accumulators and added together at its end, since one accumulator's+carry chain is what holds the core back, not its multipliers.  In a square,+each product off the diagonal is summed once, doubled, and added in.+"""++SIZES = (16, 24, 32)+# Squaring at 48 too, RSA-3072's modulus, for the verification that is+# seventeen squarings.  Its multiplication, at twice the code, is left out.+SQR_ONLY = (48,)++HEADER = """\+/* Generated by cbits/crypton_mont.py; do not edit.+ *+ * Montgomery multiplication and squaring for AArch64 at 16, 24 and 32 limbs,+ * the halves of RSA-2048, -3072 and -4096, and squaring at 48, RSA-3072's+ * modulus.  Each answers what mont_mul and mont_sqr answer. */+#ifndef CRYPTON_MONT_H+#define CRYPTON_MONT_H++#include "crypton_bignum.h"++/* (t2:t1:t0) += x * y */+#define MONT_MAC(t0, t1, t2, x, y)                                      \\+	do {                                                                \\+		limb_t lo_, hi_;                                                \\+		__asm__("mul\\t%0, %5, %6\\n\\t"                                   \\+		        "umulh\\t%1, %5, %6\\n\\t"                                 \\+		        "adds\\t%2, %2, %0\\n\\t"                                  \\+		        "adcs\\t%3, %3, %1\\n\\t"                                  \\+		        "adc\\t%4, %4, xzr"                                       \\+		        : "=&r"(lo_), "=&r"(hi_), "+r"(t0), "+r"(t1), "+r"(t2)  \\+		        : "r"(x), "r"(y)                                        \\+		        : "cc");                                                \\+	} while (0)++/* (s2:s1:s0) = x * y */+#define MONT_MUL(s0, s1, s2, x, y)                                      \\+	__asm__("mul\\t%0, %3, %4\\n\\t"                                       \\+	        "umulh\\t%1, %3, %4\\n\\t"                                     \\+	        "mov\\t%2, xzr"                                               \\+	        : "=&r"(s0), "=&r"(s1), "=r"(s2)                            \\+	        : "r"(x), "r"(y))++/* (t2:t1:t0) += (s2:s1:s0) */+#define MONT_ADD(t0, t1, t2, s0, s1, s2)                                \\+	__asm__("adds\\t%0, %0, %3\\n\\t"                                      \\+	        "adcs\\t%1, %1, %4\\n\\t"                                      \\+	        "adc\\t%2, %2, %5"                                            \\+	        : "+r"(t0), "+r"(t1), "+r"(t2)                              \\+	        : "r"(s0), "r"(s1), "r"(s2)                                 \\+	        : "cc")++/* (s2:s1:s0) *= 2 */+#define MONT_DBL(s0, s1, s2)                                            \\+	do {                                                                \\+		s2 = (s2 << 1) | (s1 >> 63);                                    \\+		s1 = (s1 << 1) | (s0 >> 63);                                    \\+		s0 <<= 1;                                                       \\+	} while (0)+"""++FOOTER = """\+#endif+"""+++class Column:+    """Products added to accumulators, each set up by its first product."""++    def __init__(self, w):+        self.w = w+        self.started = set()++    def add(self, acc, x, y):+        if acc == "t" or acc in self.started:+            self.w(f"\tMONT_MAC({acc}0, {acc}1, {acc}2, {x}, {y});")+        else:+            self.w(f"\tMONT_MUL({acc}0, {acc}1, {acc}2, {x}, {y});")+            self.started.add(acc)++    def fold(self, into, acc):+        if acc in self.started:+            self.w(f"\tMONT_ADD({into}0, {into}1, {into}2, {acc}0, {acc}1, {acc}2);")+++def column(w, singles, cross):+    """singles are added once, cross twice: s and c take the cross products,+    t, v and w take the rest, and all of them end up in t."""+    col = Column(w)+    for k, (x, y) in enumerate(cross):+        col.add("s" if k % 2 == 0 else "c", x, y)+    for k, (x, y) in enumerate(singles):+        col.add("tvw"[k % 3], x, y)+    col.fold("t", "v")+    col.fold("t", "w")+    if cross:+        col.fold("s", "c")+        w("\tMONT_DBL(s0, s1, s2);")+        w("\tMONT_ADD(t0, t1, t2, s0, s1, s2);")+++def gen(n, square):+    out = []+    w = out.append+    name = f"crypton_mont_{'sqr' if square else 'mul'}{n}"+    args = "const limb_t *a," if square else "const limb_t *a, const limb_t *b,"+    if square:+        w(f"/* r = a * a * R^-1 mod m, for a under m; t is 2n limbs of scratch */")+    else:+        w(f"/* r = a * b * R^-1 mod m, for a and b under m; t is 2n limbs of scratch */")+    w(f"static void {name}(limb_t *r, {args}")+    w(f"{' ' * len(f'static void {name}(')}const limb_t *m, limb_t n0, limb_t *t)")+    w("{")+    w(f"\tlimb_t *u = t, *z = t + {n};")+    w("\tlimb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take;")+    if square:+        w("\tlimb_t s0, s1, s2, c0, c1, c2;")+    w("")+    for i in range(2 * n - 1):+        w(f"\t/* column {i} */")+        singles, cross = [], []+        for j in range(max(0, i - n + 1), min(i, n - 1) + 1):+            k = i - j+            if not square:+                singles.append((f"a[{j}]", f"b[{k}]"))+            elif j < k:+                cross.append((f"a[{j}]", f"a[{k}]"))+            elif j == k:+                singles.append((f"a[{j}]", f"a[{j}]"))+        for j in range(max(0, i - n + 1), min(i, n)):+            singles.append((f"u[{j}]", f"m[{i - j}]"))+        column(w, singles, cross)+        if i < n:+            w(f"\tu[{i}] = t0 * n0;")+            w(f"\tMONT_MAC(t0, t1, t2, u[{i}], m[0]);")+        else:+            w(f"\tz[{i - n}] = t0;")+        w("\tt0 = t1;")+        w("\tt1 = t2;")+        w("\tt2 = 0;")+    w(f"\tz[{n - 1}] = t0;")+    w("")+    w("\t/* under 2m, with t1 the bit above n limbs: at most one subtraction */")+    w(f"\tborrow = sub_n(r, z, m, {n});")+    w("\ttake = t1 | (borrow ^ 1);")+    w(f"\tselect_n(r, r, z, take & 1, {n});")+    w("}")+    w("")+    return "\n".join(out)+++def main():+    print(HEADER)+    for n in SIZES:+        print(gen(n, False))+        print(gen(n, True))+    for n in SQR_ONLY:+        print(gen(n, True))+    print(FOOTER, end="")+++if __name__ == "__main__":+    main()
cbits/crypton_pbkdf2.c view
@@ -46,6 +46,7 @@  /* Internal function/type names for hash-specific things. */ #define HMAC_CTX(_name) HMAC_ ## _name ## _ctx+#define DIGEST_FITS(_name) pbkdf2_ ## _name ## _digest_fits_in_a_block #define HMAC_INIT(_name) HMAC_ ## _name ## _init #define HMAC_UPDATE(_name) HMAC_ ## _name ## _update #define HMAC_FINAL(_name) HMAC_ ## _name ## _final@@ -79,6 +80,16 @@  */ #define DECL_PBKDF2(_name, _blocksz, _hashsz, _ctx,                           \                     _init, _update, _xform, _final, _xcpy, _xtract, _xxor)    \+  /* HMAC_INIT below shortens a key longer than the block by hashing it,     \+   * which writes _hashsz bytes into a buffer of _blocksz.  An instantiation \+   * whose digest is larger than its block would overflow that buffer, and   \+   * would do it before any check inside the function could say so -- which  \+   * is where the check used to be.  Refuse such an instantiation here       \+   * instead, in front of the person writing it.  The three below are        \+   * SHA-1, SHA-256 and SHA-512, whose digests are 20, 32 and 64 bytes       \+   * against blocks of 64, 64 and 128. */                                    \+  typedef char DIGEST_FITS(_name)[(_hashsz) <= (_blocksz) ? 1 : -1];          \+                                                                              \   typedef struct {                                                            \     _ctx inner;                                                               \     _ctx outer;                                                               \@@ -101,9 +112,6 @@       nkey = _hashsz;                                                         \     }                                                                         \                                                                               \-    /* Standard doesn't cover case where blocksz < hashsz. */                 \-    assert(nkey <= _blocksz);                                                 \-                                                                              \     /* Right zero-pad short keys. */                                          \     if (k != key)                                                             \       memcpy(k, key, nkey);                                                   \@@ -192,7 +200,16 @@                      uint8_t *out, size_t nout)                               \   {                                                                           \     assert(iterations);                                                       \-    assert(out && nout);                                                      \+    assert(out);                                                              \+                                                                              \+    /* Zero bytes of derived key is zero bytes of work.  RFC 8018 asks for a  \+     * positive dkLen and the loop below would write a block regardless, so   \+     * this used to be `assert(out && nout)` -- which aborts the process, in  \+     * a library built without NDEBUG, on a length the caller chose.          \+     * Crypto.KDF.PBKDF2's own tryGenerate returns an empty result for this,  \+     * so return and let the two agree. */                                    \+    if (nout == 0)                                                            \+      return;                                                                 \                                                                               \     /* Starting point for inner loop. */                                      \     HMAC_CTX(_name) ctx;                                                      \
cbits/crypton_powm.c view
@@ -20,21 +20,536 @@ #include <stdlib.h> #include <crypton_bignum.h> #include <crypton_powm.h>+#include <crypton_bzero.h> -/* four bits of exponent per window, so a table of sixteen and no leftover- * bits: a byte holds exactly two windows */-#define WINDOW_BITS 4+/* On AArch64, multiplication and squaring at the RSA half sizes go through+ * generated code that scans a column at a time with the reduction folded in;+ * see cbits/crypton_mont.py.  CRYPTON_NO_MONT_FIXED keeps mont_mul and+ * mont_sqr, to compare. */+#if defined(__aarch64__) && LIMB_BITS == 64 && !defined(CRYPTON_NO_MONT_FIXED) \+    && (defined(__GNUC__) || defined(__clang__))+#define CRYPTON_MONT_FIXED 1+#include <crypton_mont.h>+#endif++/*+ * At RSA sizes on x86-64, the Montgomery multiplication below is the whole+ * cost, and s2n-bignum's is twice as fast because the C cannot form the two+ * carry chains ADCX and ADOX give.  The window, the table and its masked+ * scan are unchanged: only the multiply and the square are swapped, and+ * only for the sizes s2n-bignum has a Karatsuba multiplication for.+ *+ * Not on AArch64, where the C measures 5% faster than the assembly.+ * See cbits/s2n/README.md.+ */+/* Not on macOS, where crypton.cabal leaves these files out: Apple's assembler+ * cannot assemble bignum_emontredc_8n.  See the note beside them there. */+#if defined(CRYPTON_S2N_BIGNUM) && defined(__x86_64__) && !defined(__APPLE__)+#define CRYPTON_POWM_S2N 1+#include <crypton_cpu.h>++extern void bignum_kmul_16_32(uint64_t *z, const uint64_t *x,+                              const uint64_t *y, uint64_t *t);+extern void bignum_ksqr_16_32(uint64_t *z, const uint64_t *x, uint64_t *t);+extern void bignum_kmul_32_64(uint64_t *z, const uint64_t *x,+                              const uint64_t *y, uint64_t *t);+extern void bignum_ksqr_32_64(uint64_t *z, const uint64_t *x, uint64_t *t);+extern uint64_t bignum_emontredc_8n(uint64_t k, uint64_t *z,+                                    const uint64_t *m, uint64_t w);+extern void bignum_mul(uint64_t k, uint64_t *z, uint64_t m, const uint64_t *x,+                       uint64_t n, const uint64_t *y);+extern void bignum_sqr(uint64_t k, uint64_t *z, uint64_t n, const uint64_t *x);+extern void bignum_sqr_8_16(uint64_t *z, const uint64_t *x);+extern void bignum_mul_8_16(uint64_t *z, const uint64_t *x, const uint64_t *y);++/* 16 limbs is 1024 bits and 32 is 2048: the halves a CRT exponentiation+ * works in for RSA-2048 and RSA-4096, and the whole thing without CRT.  24+ * is 1536, the half of RSA-3072, which has no Karatsuba routine of its own:+ * its multiplication is the 32-limb one with the operands padded, and its+ * square is built from 8-limb pieces, which measured faster on a whole+ * exponentiation than the padded square.  Both are still nearly twice the+ * C, since the C cannot form ADCX and ADOX's two carry chains at any size.+ *+ * 48 and 64 are 3072 and 4096 bits: finite-field Diffie-Hellman over the+ * RFC 7919 groups of those sizes, and RSA without CRT.  There is no+ * Karatsuba routine at either, and padding to the next would be wasteful,+ * so the multiplication is s2n-bignum's schoolbook one of any size --+ * plain MUL and ADC, no ADX -- and the reduction is the same ADX one.  Not+ * the gain the Karatsuba sizes have, and still well ahead of the C.+ *+ * The reduction wants ADX, so the answer is a run-time one. */+static int powm_s2n_usable(uint32_t n)+{+	return (n == 16 || n == 24 || n == 32 || n == 48 || n == 64)+	    && (crypton_x86_simd_features() & CRYPTON_X86_ADX) != 0;+}++/* The scratch the routines ask for, in limbs.  kmul_16_32 and kmul_32_64+ * want three times their operand.  A 24-limb operand padded to 32 wants the+ * 32-limb routine's 64-limb product and its 96 limbs of its own, and the two+ * padded operands besides. */+#define POWM_S2N_PAD_Z 0+#define POWM_S2N_PAD_T 64+#define POWM_S2N_PAD_X (64 + 96)+#define POWM_S2N_PAD_Y (64 + 96 + 32)+static uint32_t powm_s2n_scratch(uint32_t n)+{+	return n == 24 ? 64 + 96 + 32 + 32 : 3 * n;+}++/* a 24-limb operand in the low limbs of 32, the rest zero: zeroed every+ * time rather than once, so that nothing else writing the scratch can+ * leave a high limb behind */+static void powm_s2n_pad(limb_t *to, const limb_t *from)+{+	memcpy(to, from, 24 * sizeof(limb_t));+	memset(to + 24, 0, 8 * sizeof(limb_t));+}++/* bignum_emontredc_8n leaves the result in the top half of z with one more+ * bit as its return value, and what is there is under twice the modulus --+ * the same place mont_reduce ends up, and finished the same way. */+static void powm_s2n_finish(limb_t *r, limb_t *z, const limb_t *m,+                            limb_t carry, uint32_t n)+{+	limb_t borrow = sub_n(r, z + n, m, n);+	limb_t take = carry | (borrow ^ 1);++	select_n(r, r, z + n, take & 1, n);+}++static void powm_s2n_mul(limb_t *r, const limb_t *a, const limb_t *b,+                         const limb_t *m, limb_t n0, uint32_t n, limb_t *z,+                         limb_t *scratch)+{+	if (n == 24) {+		/* the product of two 24-limb numbers is 48 limbs, so the top+		 * sixteen of the 64 written are zero and the reduction reads+		 * only what it should */+		z = scratch + POWM_S2N_PAD_Z;+		powm_s2n_pad(scratch + POWM_S2N_PAD_X, a);+		powm_s2n_pad(scratch + POWM_S2N_PAD_Y, b);+		bignum_kmul_32_64(z, scratch + POWM_S2N_PAD_X,+		                  scratch + POWM_S2N_PAD_Y,+		                  scratch + POWM_S2N_PAD_T);+	} else if (n == 48 || n == 64)+		bignum_mul(2 * n, z, n, a, n, b);+	else if (n == 16)+		bignum_kmul_16_32(z, a, b, scratch);+	else+		bignum_kmul_32_64(z, a, b, scratch);+	powm_s2n_finish(r, z, m, bignum_emontredc_8n(n, z, m, n0), n);+}++/* a^2 for a of 24 limbs, from three 8-limb squares and three 8-limb+ * products: a = a0 + a1 X + a2 X^2 with X = 2^512, and the cross terms, at+ * X to X^3 in c, go in doubled.  c is 48 limbs of scratch. */+static void powm_s2n_sqr24(limb_t *z, const limb_t *a, limb_t *c)+{+	limb_t k;+	uint32_t i;++	bignum_sqr_8_16(z, a);+	bignum_sqr_8_16(z + 16, a + 8);+	bignum_sqr_8_16(z + 32, a + 16);+	bignum_mul_8_16(c, a, a + 8);+	bignum_mul_8_16(c + 16, a + 8, a + 16);+	bignum_mul_8_16(c + 32, a, a + 16);+	k = add_n(c + 8, c + 8, c + 32, 16);+	for (i = 24; i < 32; i++) {+		c[i] += k;+		k = c[i] < k;+	}+	k = shl1(c, 32);+	k += add_n(z + 8, z + 8, c, 32);+	for (i = 40; i < 48; i++) {+		z[i] += k;+		k = z[i] < k;+	}+}++static void powm_s2n_sqr(limb_t *r, const limb_t *a, const limb_t *m,+                         limb_t n0, uint32_t n, limb_t *z, limb_t *scratch)+{+	if (n == 24) {+		z = scratch + POWM_S2N_PAD_Z;+		powm_s2n_sqr24(z, a, scratch + POWM_S2N_PAD_T);+	} else if (n == 48 || n == 64)+		bignum_sqr(2 * n, z, n, a);+	else if (n == 16)+		bignum_ksqr_16_32(z, a, scratch);+	else+		bignum_ksqr_32_64(z, a, scratch);+	powm_s2n_finish(r, z, m, bignum_emontredc_8n(n, z, m, n0), n);+}+#else+static uint32_t powm_s2n_scratch(uint32_t n)+{+	(void) n;+	return 0;+}+#endif++/* One or the other, decided once per call */+static void powm_mul(limb_t *r, const limb_t *a, const limb_t *b,+                     const limb_t *m, limb_t n0, uint32_t n, limb_t *t,+                     limb_t *scratch, int s2n)+{+#ifdef CRYPTON_POWM_S2N+	if (s2n) {+		powm_s2n_mul(r, a, b, m, n0, n, t, scratch);+		return;+	}+#else+	(void)scratch;+	(void)s2n;+#endif+#ifdef CRYPTON_MONT_FIXED+	switch (n) {+	case 16: crypton_mont_mul16(r, a, b, m, n0, t); return;+	case 24: crypton_mont_mul24(r, a, b, m, n0, t); return;+	case 32: crypton_mont_mul32(r, a, b, m, n0, t); return;+	}+#endif+	mont_mul(r, a, b, m, n0, n, t);+}++static void powm_sqr(limb_t *r, const limb_t *a, const limb_t *m, limb_t n0,+                     uint32_t n, limb_t *t, limb_t *scratch, int s2n)+{+#ifdef CRYPTON_POWM_S2N+	if (s2n) {+		powm_s2n_sqr(r, a, m, n0, n, t, scratch);+		return;+	}+#else+	(void)scratch;+	(void)s2n;+#endif+#ifdef CRYPTON_MONT_FIXED+	switch (n) {+	case 16: crypton_mont_sqr16(r, a, m, n0, t); return;+	case 24: crypton_mont_sqr24(r, a, m, n0, t); return;+	case 32: crypton_mont_sqr32(r, a, m, n0, t); return;+	case 48: crypton_mont_sqr48(r, a, m, n0, t); return;+	}+#endif+	mont_sqr(r, a, m, n0, n, t);+}++/* Four bits of exponent per window, so a table of sixteen.+ *+ * Five was written and measured and is not here.  A wider window saves+ * multiplications -- 205 of them against 256 at 1024 bits, with the same+ * 1024 squarings -- and pays for it in the masked scan of a table twice as+ * long.  When that was first measured the scan was the scalar one, and five+ * came out 3.5% better on an Apple M4, about 1% worse on an older x86-64 and+ * 7% worse where s2n-bignum's multiplication runs, since that makes the scan+ * the expensive half.  The conclusion written here was that what would make+ * a wider window pay is a cheaper scan.+ *+ * The cheaper scan arrived twenty-one minutes later, in the AVX2 form above,+ * and nobody measured the window again.  Measured since, on the machine the+ * x86-64 table is measured on and with both builds raced in one run: five is+ * 0.6% worse than four, beside 0.5% between two builds that differ in+ * nothing.  So the cheaper scan did not turn it round -- the window stays at+ * four, and now for a reason that was checked after the scan rather than+ * before it.+ *+ * The width is POWM_WINDOW_BITS so that it can be measured rather than+ * argued about.  It had to be renamed to be settable at all: WINDOW_BITS was+ * defined here, in cbits/crypton_ecc.c and in cbits/crypton_bignum.h, all+ * three to four, and a redefinition to the same value is silent -- so+ * -DWINDOW_BITS=5 was overridden by the header and three builds meant to+ * differ did not.  cbits/tests/powm builds the walk at four, five and six+ * and requires the same answers of all three. */+#ifndef POWM_WINDOW_BITS+#define POWM_WINDOW_BITS 4+#endif+#define WINDOW_BITS POWM_WINDOW_BITS #define TABLE_SIZE (1 << WINDOW_BITS) +/* Window number k counting from the bottom, where the bottom window holds+ * the exponent's least significant WINDOW_BITS bits.  The exponent is+ * big-endian bytes, and a window may straddle a byte boundary once the width+ * stops dividing eight, so this reads the two bytes it can fall across.+ *+ * Which window is being read is the loop counter, not the exponent, so the+ * shifts and the index here are public.+ */+static limb_t exp_window(const uint8_t *exp, uint32_t explen, uint32_t k)+{+	uint32_t lo = k * WINDOW_BITS;          /* bit offset from the bottom */+	uint32_t byte = lo / 8, off = lo % 8;+	uint32_t v;++	if (byte >= explen)+		return 0;+	v = exp[explen - 1 - byte];+	if (off + WINDOW_BITS > 8 && byte + 1 < explen)+		v |= (uint32_t) exp[explen - 2 - byte] << 8;+	return (v >> off) & (TABLE_SIZE - 1);+}++/* The masked scan of the table: every entry is read and a mask keeps the one+ * wanted, so that the address stream does not follow the exponent.  At+ * RSA-2048's CRT size that is two kilobytes read per window, and the window+ * loop runs 256 times per exponentiation, which is why it is worth a vector+ * register: removing the scan altogether measures 11% of an exponentiation+ * where s2n-bignum's multiplication runs, and the AVX2 form below gets+ * essentially all of it. */+static void scan_table(limb_t *sel, const limb_t *table, uint32_t n, limb_t w)+{+	uint32_t k, l;++	memset(sel, 0, n * sizeof(limb_t));+	for (k = 0; k < TABLE_SIZE; k++) {+		limb_t mask = eq_mask(k, w);++		for (l = 0; l < n; l++)+			sel[l] |= table[k * n + l] & mask;+	}+}++#if defined(__x86_64__) && defined(WITH_TARGET_ATTRIBUTES) && LIMB_BITS == 64+#define CRYPTON_POWM_SCAN_AVX2 1+#include <crypton_cpu.h>+#include <immintrin.h>++/* The same scan four limbs at a time.  The sixteen masks are worked out+ * once; after that each register of the answer is one pass over the table's+ * column, reading every entry exactly as the scalar form does. */+__attribute__((target("avx2")))+static void scan_table_avx2(limb_t *sel, const limb_t *table, uint32_t n,+                            limb_t w)+{+	__m256i masks[TABLE_SIZE];+	uint32_t k, l;++	for (k = 0; k < TABLE_SIZE; k++)+		masks[k] = _mm256_cmpeq_epi64(+			_mm256_set1_epi64x((long long) k),+			_mm256_set1_epi64x((long long) w));++	for (l = 0; l + 4 <= n; l += 4) {+		__m256i acc = _mm256_setzero_si256();++		for (k = 0; k < TABLE_SIZE; k++) {+			__m256i v = _mm256_loadu_si256(+				(const __m256i *) (table + k * n + l));++			acc = _mm256_or_si256(acc,+			                      _mm256_and_si256(v, masks[k]));+		}+		_mm256_storeu_si256((__m256i *) (sel + l), acc);+	}++	/* a modulus whose limbs do not come in fours ends here */+	for (; l < n; l++) {+		limb_t v = 0;++		for (k = 0; k < TABLE_SIZE; k++)+			v |= table[k * n + l] & eq_mask(k, w);+		sel[l] = v;+	}+}+#endif++/* Modular exponentiation for an exponent that is not a secret.+ *+ * RSA verification raises a signature to the public exponent, which is in+ * the public key and is usually 65537.  crypton_powm_sec below hides the+ * exponent and pays for it twice: a table of sixteen powers built before the+ * walk begins, and every entry of that table read at every window.  For a+ * seventeen-bit exponent the table alone costs more than the exponentiation.+ *+ * So this one reads the exponent's bits and branches on them, squaring at+ * every step and multiplying where a bit is set.  It is for public exponents+ * only, and nothing in this library may call it with a secret.+ *+ * It declines -- returning 1 without touching out -- in three cases, so a+ * caller must have something to fall back to.+ *+ * The first is a long exponent.  Walking it a bit at a time costs about one+ * and a half multiplications per bit where a windowed walk costs a quarter+ * less, and the table that buys the difference pays for itself after some+ * sixty-odd bits.  Past that the caller's own exponentiation is the better+ * algorithm and this one would have to grow a table to compete, which is+ * the thing it exists not to have.  RSA public exponents are seventeen+ * bits; the bound is set well above them and well below where a Diffie-+ * Hellman or DSA exponent starts.+ *+ * The second and third are a machine and a size where GMP, which the+ * caller has, was measured the faster; see powm_pub_wanted.+ *+ * Define CRYPTON_POWM_PUB_ALWAYS to make it accept anyway: the differential+ * test builds that way, so that this arithmetic is exercised on the+ * machines the assembly never reaches.+ */+#define POWM_PUB_MAX_EXP_BYTES 8++/* Which sizes this is worth taking at is not which sizes the assembly runs+ * at.  Against GMP, which is what the caller has, on an EPYC 7763 with ADX+ * on both sides: ahead at 2048 bits, and behind everywhere else measured --+ * slightly at 1024, by more at 1536, and by more again at 3072 and 4096,+ * which is the size where GMP's own setup and multiplication are the better+ * pair.  So 2048 alone, which is RSA-2048's modulus and the size nearly+ * every RSA public key has.  It once took every size the assembly ran at,+ * on the strength of 2048 alone, and was slower than GMP at 1024. */+static int powm_pub_wanted(uint32_t n)+{+#if defined(CRYPTON_POWM_PUB_ALWAYS)+	return n > 0;+#elif defined(CRYPTON_POWM_S2N)+	return n == 32 && powm_s2n_usable(n);+#elif defined(CRYPTON_MONT_FIXED)+	/* AArch64: the generated code is ahead of GMP at 2048 bits.  At the+	 * other RSA sizes Linux's GMP is level with the C or ahead of it.  The+	 * GMP GHC builds into itself on macOS is compiled without -O, and is+	 * behind the C from 1024 to 4096 bits on the same M4 */+#if defined(__APPLE__)+	return n >= 16 && n <= 64;+#else+	return n == 32;+#endif+#else+	(void) n;+	return 0;+#endif+}++int crypton_powm_window_bits(void)+{+	return WINDOW_BITS;+}++int crypton_powm_pub(uint8_t *out,+                     const uint8_t *base, uint32_t baselen,+                     const uint8_t *exp, uint32_t explen,+                     const uint8_t *mod, uint32_t modlen)+{+	return crypton_powm_pub_r2(out, base, baselen, exp, explen, mod, modlen, NULL);+}++int crypton_powm_r2(uint8_t *out, const uint8_t *mod, uint32_t modlen)+{+	uint32_t n = (modlen + LIMB_BYTES - 1) / LIMB_BYTES;+	limb_t *space, *m, *r2, *t;++	if (modlen == 0 || n == 0 || (mod[modlen - 1] & 1) == 0 || !powm_pub_wanted(n))+		return 1;+	space = calloc(5 * n, sizeof(limb_t));+	if (space == NULL)+		return 1;+	m = space;+	r2 = m + n;+	t = r2 + n;+	if (from_be(m, n, mod, modlen) != 0) {+		free(space);+		return 1;+	}+	mont_r2(r2, m, mont_n0(m[0]), n, t);+	to_be(out, modlen, r2, n);+	free(space);+	return 0;+}++int crypton_powm_pub_r2(uint8_t *out,+                        const uint8_t *base, uint32_t baselen,+                        const uint8_t *exp, uint32_t explen,+                        const uint8_t *mod, uint32_t modlen,+                        const uint8_t *r2be)+{+	uint32_t n = (modlen + LIMB_BYTES - 1) / LIMB_BYTES;+	uint32_t words = (6 + 2) * n + powm_s2n_scratch(n);+	limb_t *space, *m, *r2, *acc, *tmp, *bm, *one, *t, *scratch, n0;+	uint32_t top, i;+	int s2n = 0;++	if (modlen == 0 || n == 0 || (mod[modlen - 1] & 1) == 0)+		return 1;+	if (explen > POWM_PUB_MAX_EXP_BYTES)+		return 1;+	if (!powm_pub_wanted(n))+		return 1;+#ifdef CRYPTON_POWM_S2N+	s2n = powm_s2n_usable(n);+#endif++	/* the exponent's top set bit, counted from the bottom.  An exponent of+	 * zero is left to the caller, whose answer for it is already right. */+	top = explen * 8;+	while (top > 0 &&+	       ((exp[explen - 1 - (top - 1) / 8] >> ((top - 1) % 8)) & 1) == 0)+		top--;+	if (top == 0)+		return 1;++	space = calloc(words, sizeof(limb_t));+	if (space == NULL)+		return 1;+	m = space;+	r2 = m + n;+	acc = r2 + n;+	tmp = acc + n;+	bm = tmp + n;+	one = bm + n;+	t = one + n;+	scratch = t + 2 * n;++	if (from_be(m, n, mod, modlen) != 0)+		goto fail;+	n0 = mont_n0(m[0]);+	if (r2be == NULL)+		mont_r2(r2, m, n0, n, t);+	else if (from_be(r2, n, r2be, modlen) != 0)+		goto fail;++	/* the base into Montgomery form.  The accumulator starts there rather+	 * than at one, because the exponent's top bit is a one by+	 * construction and the first squaring would be of one. */+	if (from_be(tmp, n, base, baselen) != 0)+		goto fail;+	powm_mul(bm, tmp, r2, m, n0, n, t, scratch, s2n);+	memcpy(acc, bm, n * sizeof(limb_t));++	for (i = top - 1; i > 0; i--) {+		limb_t *swap;++		powm_sqr(tmp, acc, m, n0, n, t, scratch, s2n);+		swap = acc; acc = tmp; tmp = swap;+		if ((exp[explen - 1 - (i - 1) / 8] >> ((i - 1) % 8)) & 1) {+			powm_mul(tmp, acc, bm, m, n0, n, t, scratch, s2n);+			swap = acc; acc = tmp; tmp = swap;+		}+	}++	/* out of Montgomery form */+	one[0] = 1;+	powm_mul(tmp, acc, one, m, n0, n, t, scratch, s2n);+	to_be(out, modlen, tmp, n);++	free(space);+	return 0;++fail:+	free(space);+	return 1;+}+ int crypton_powm_sec(uint8_t *out,                      const uint8_t *base, uint32_t baselen,                      const uint8_t *exp, uint32_t explen,                      const uint8_t *mod, uint32_t modlen) { 	uint32_t n = (modlen + LIMB_BYTES - 1) / LIMB_BYTES;-	uint32_t words = (TABLE_SIZE + 7) * n;-	limb_t *space, *m, *r2, *acc, *sel, *prod, *table, *t, n0;+	uint32_t words = (TABLE_SIZE + 7) * n + powm_s2n_scratch(n);+	limb_t *space, *m, *r2, *acc, *sel, *prod, *table, *t, *scratch, n0; 	uint32_t i, j, k;+	int s2n = 0;+#ifdef CRYPTON_POWM_SCAN_AVX2+	int avx2 = (crypton_x86_simd_features() & CRYPTON_X86_AVX2) != 0;+#endif  	if (modlen == 0 || n == 0 || (mod[modlen - 1] & 1) == 0) 		return 1;@@ -50,62 +565,78 @@ 	prod = sel + n; 	t = prod + n; 	table = t + 2 * n;+	scratch = table + TABLE_SIZE * n; +#ifdef CRYPTON_POWM_S2N+	s2n = powm_s2n_usable(n);+#endif+ 	if (from_be(m, n, mod, modlen) != 0) 		goto fail;-	mont_r2(r2, m, n, t); 	n0 = mont_n0(m[0]);+	mont_r2(r2, m, n0, n, t);  	/* table[k] = base^k in Montgomery form, and table[0] = 1 there */ 	memset(table, 0, n * sizeof(limb_t)); 	table[0] = 1;-	mont_mul(acc, table, r2, m, n0, n, t);+	powm_mul(acc, table, r2, m, n0, n, t, scratch, s2n); 	memcpy(table, acc, n * sizeof(limb_t));  	if (from_be(sel, n, base, baselen) != 0) 		goto fail;-	mont_mul(table + n, sel, r2, m, n0, n, t);+	powm_mul(table + n, sel, r2, m, n0, n, t, scratch, s2n); 	for (k = 2; k < TABLE_SIZE; k++)-		mont_mul(table + k * n, table + (k - 1) * n, table + n, m, n0, n, t);+		powm_mul(table + k * n, table + (k - 1) * n, table + n, m, n0, n,+			         t, scratch, s2n);  	memcpy(acc, table, n * sizeof(limb_t)); -	for (i = explen * 2; i > 0; i--) {-		uint32_t nib = i - 1;-		limb_t w = (exp[explen - 1 - nib / 2] >> (4 * (nib % 2))) & 0xf;+	for (i = (explen * 8 + WINDOW_BITS - 1) / WINDOW_BITS; i > 0; i--) {+		limb_t w = exp_window(exp, explen, i - 1); +		/* squaring into the other buffer and swapping the two saves a+		 * copy of the modulus' width every time; which of the three+		 * buffers a pointer names is nobody's secret */ 		for (j = 0; j < WINDOW_BITS; j++) {-			mont_sqr(sel, acc, m, n0, n, t);-			memcpy(acc, sel, n * sizeof(limb_t));+			limb_t *swap;++			powm_sqr(sel, acc, m, n0, n, t, scratch, s2n);+			swap = acc;+			acc = sel;+			sel = swap; 		} -		/* every entry is read, and a mask keeps the one wanted */-		memset(sel, 0, n * sizeof(limb_t));-		for (k = 0; k < TABLE_SIZE; k++) {-			limb_t mask = eq_mask(k, w);-			uint32_t l;+#ifdef CRYPTON_POWM_SCAN_AVX2+		if (avx2)+			scan_table_avx2(sel, table, n, w);+		else+#endif+			scan_table(sel, table, n, w);+		powm_mul(prod, acc, sel, m, n0, n, t, scratch, s2n);+		{+			limb_t *swap = acc; -			for (l = 0; l < n; l++)-				sel[l] |= table[k * n + l] & mask;+			acc = prod;+			prod = swap; 		}-		mont_mul(prod, acc, sel, m, n0, n, t);-		memcpy(acc, prod, n * sizeof(limb_t)); 	}  	/* out of Montgomery form */ 	memset(sel, 0, n * sizeof(limb_t)); 	sel[0] = 1;-	mont_mul(prod, acc, sel, m, n0, n, t);+	powm_mul(prod, acc, sel, m, n0, n, t, scratch, s2n); 	to_be(out, modlen, prod, n);  	/* nothing here is the caller's secret, but the exponent's bits passed-	 * through the accumulators */-	memset(space, 0, words * sizeof(limb_t));+	 * through the accumulators.  crypton_bzero rather than memset, since+	 * this memory is freed on the next line and a store to memory about to+	 * die is one an optimizer may drop. */+	crypton_bzero(space, words * sizeof(limb_t)); 	free(space); 	return 0;  fail:-	memset(space, 0, words * sizeof(limb_t));+	crypton_bzero(space, words * sizeof(limb_t)); 	free(space); 	return 1; }
cbits/crypton_powm.h view
@@ -20,4 +20,53 @@                      const uint8_t *exp, uint32_t explen,                      const uint8_t *mod, uint32_t modlen); +/* The same, for an exponent that is not a secret: it reads the exponent's+ * bits and branches on them.  Nothing in this library may call it with a+ * secret exponent.+ *+ * It declines -- returning 1 with out untouched -- whenever it is not the+ * faster of the two things the caller could do: for an exponent of more+ * than eight bytes, where a windowed walk is the better algorithm;+ * everywhere the vendored assembly's Montgomery multiplication does not+ * run; and at every modulus size but 2048 bits, where GMP was measured the+ * faster even with the assembly.  So a caller must have something to fall+ * back to.+ */+int crypton_powm_pub(uint8_t *out,+                     const uint8_t *base, uint32_t baselen,+                     const uint8_t *exp, uint32_t explen,+                     const uint8_t *mod, uint32_t modlen);++/* crypton_powm_pub with R^2 mod m given, big-endian in modlen bytes, as+ * crypton_powm_r2 computes it; NULL computes it here as crypton_powm_pub does.+ * A caller verifying under one key many times keeps it. */+int crypton_powm_pub_r2(uint8_t *out,+                        const uint8_t *base, uint32_t baselen,+                        const uint8_t *exp, uint32_t explen,+                        const uint8_t *mod, uint32_t modlen,+                        const uint8_t *r2be);++/* R^2 mod m for crypton_powm_pub_r2, or nonzero where crypton_powm_pub would+ * decline this modulus, so that nothing is kept for it. */+int crypton_powm_r2(uint8_t *out, const uint8_t *mod, uint32_t modlen);++/* Two secure exponentiations at once, as the halves of an RSA private-key+ * operation by CRT are: each as crypton_powm_sec, with the same contract.+ *+ * Only on x86-64 with AVX-512 IFMA, for two moduli of 128, 192 or 256 bytes+ * and two exponents of the same length.  It declines everywhere else --+ * returning nonzero with both outputs untouched -- so a caller must have+ * crypton_powm_sec to fall back to. */+int crypton_powm_sec_x2(uint8_t *out1, const uint8_t *base1, uint32_t baselen1,+                        const uint8_t *exp1, uint32_t explen1,+                        const uint8_t *mod1, uint32_t modlen1,+                        uint8_t *out2, const uint8_t *base2, uint32_t baselen2,+                        const uint8_t *exp2, uint32_t explen2,+                        const uint8_t *mod2, uint32_t modlen2);++/* The window the secure exponentiation walks the exponent in, so that a+ * build which did not take -DPOWM_WINDOW_BITS cannot pass for one that+ * did. */+int crypton_powm_window_bits(void);+ #endif
+ cbits/crypton_powm_ifma.c view
@@ -0,0 +1,187 @@+/* Two secure exponentiations at once with AVX-512 IFMA, the CRT halves of+ * RSA-2048, -3072 and -4096: crypton_powm_sec's walk with a five-bit window,+ * over the multiplication generated by cbits/crypton_powm_ifma.py. */+#include <stdlib.h>+#include <string.h>+#include <crypton_bignum.h>+#include <crypton_bzero.h>+#include <crypton_powm.h>++#if defined(__x86_64__) && (defined(__GNUC__) || defined(__clang__))+#include <crypton_cpu.h>+#include "crypton_powm_ifma.h"++#define WIN 5++typedef void (*amm_fn)(uint64_t *, const uint64_t *, const uint64_t *, const uint64_t *,+                       uint64_t, uint64_t *, const uint64_t *, const uint64_t *,+                       const uint64_t *, uint64_t);+typedef void (*sel_fn)(uint64_t *, const uint64_t *, uint64_t, uint64_t *,+                       const uint64_t *, uint64_t);++/* limbs of 64 bits to digits of 52, d of them; the rest of the w words zero */+static void to_digits(uint64_t *dg, uint32_t w, const limb_t *l, uint32_t n, uint32_t d)+{+	uint32_t i;++	memset(dg, 0, w * sizeof(uint64_t));+	for (i = 0; i < d; i++) {+		uint32_t bit = 52 * i, li = bit / 64, sh = bit % 64;+		uint64_t v = li < n ? l[li] >> sh : 0;++		if (sh > 12 && li + 1 < n)+			v |= l[li + 1] << (64 - sh);+		dg[i] = v & IFMA_MASK52;+	}+}++static void from_digits(limb_t *l, uint32_t n, const uint64_t *dg, uint32_t d)+{+	uint32_t i;++	memset(l, 0, n * sizeof(limb_t));+	for (i = 0; i < d; i++) {+		uint32_t bit = 52 * i, li = bit / 64, sh = bit % 64;++		if (li < n)+			l[li] |= dg[i] << sh;+		if (sh > 12 && li + 1 < n)+			l[li + 1] |= dg[i] >> (64 - sh);+	}+}++/* Window k from the bottom of a big-endian exponent of len bytes.  Which+ * window is read is the loop counter's business, not the exponent's. */+static uint64_t window(const uint8_t *e, uint32_t len, uint32_t k)+{+	uint64_t v = 0;+	uint32_t j;++	for (j = 0; j < WIN; j++) {+		uint32_t bit = k * WIN + j;++		if (bit < 8 * len)+			v |= (uint64_t) ((e[len - 1 - bit / 8] >> (bit % 8)) & 1) << j;+	}+	return v;+}++/* m and k0 in digits, and R^2 mod m for R = 2^(52d): crypton's R^2 is for+ * 2^(64n), so it is doubled the rest of the way */+static int setup(const uint8_t *mod, uint32_t modlen, uint32_t n, uint32_t d, uint32_t w,+                 limb_t *ml, uint64_t *md, uint64_t *rr, uint64_t *k0, limb_t *t)+{+	limb_t *r2 = t + 4 * n;+	uint32_t j;++	if (from_be(ml, n, mod, modlen) != 0 || (ml[0] & 1) == 0)+		return 1;+	*k0 = mont_n0(ml[0]);+	mont_r2(r2, ml, *k0, n, t);+	for (j = 0; j < 2 * (52 * d - 64 * n); j++)+		dbl_mod(r2, ml, n, t);+	to_digits(md, w, ml, n, d);+	to_digits(rr, w, r2, n, d);+	return 0;+}++int crypton_powm_sec_x2(uint8_t *out1, const uint8_t *base1, uint32_t baselen1,+                        const uint8_t *exp1, uint32_t explen1,+                        const uint8_t *mod1, uint32_t modlen1,+                        uint8_t *out2, const uint8_t *base2, uint32_t baselen2,+                        const uint8_t *exp2, uint32_t explen2,+                        const uint8_t *mod2, uint32_t modlen2)+{+	uint32_t n, d, w, nwin, k, j;+	amm_fn amm;+	sel_fn sel;+	uint64_t *space, *m1, *m2, *rr1, *rr2, *x1, *x2, *s1, *s2, *one, *t1, *t2, k1, k2;+	limb_t *l1, *l2, *lt, *lt2;+	size_t words;++	if (modlen1 != modlen2 || explen1 != explen2)+		return 1;+	if ((crypton_x86_simd_features() & CRYPTON_X86_IFMA) == 0)+		return 1;+	switch (modlen1) {+	case 128: d = 20; amm = ifma_amm20_x2; sel = ifma_sel5; break;+	case 192: d = 30; amm = ifma_amm30_x2; sel = ifma_sel8; break;+	case 256: d = 40; amm = ifma_amm40_x2; sel = ifma_sel10; break;+	default: return 1;+	}+	n = modlen1 / 8;+	w = 4 * ((d + 3) / 4);++	/* nine numbers in digits, two tables of 32, and limbs for the setup */+	words = (size_t) w * (9 + 2 * 32) + (size_t) n * 12;+	space = calloc(words, sizeof(uint64_t));+	if (space == NULL)+		return 1;+	m1 = space; m2 = m1 + w; rr1 = m2 + w; rr2 = rr1 + w;+	x1 = rr2 + w; x2 = x1 + w; s1 = x2 + w; s2 = s1 + w; one = s2 + w;+	t1 = one + w; t2 = t1 + 32 * w;+	l1 = (limb_t *)(t2 + 32 * w); l2 = l1 + n; lt = l2 + n; lt2 = lt + n;++	if (setup(mod1, modlen1, n, d, w, l1, m1, rr1, &k1, lt) != 0+	    || setup(mod2, modlen2, n, d, w, l2, m2, rr2, &k2, lt) != 0)+		goto fail;++	/* the bases, under their moduli as the contract says, into digits */+	if (from_be(lt, n, base1, baselen1) != 0 || from_be(lt2, n, base2, baselen2) != 0)+		goto fail;+	to_digits(s1, w, lt, n, d);+	to_digits(s2, w, lt2, n, d);++	/* table[0] = R and table[1] = b R in Montgomery form, table[k] = b^k R */+	one[0] = 1;+	amm(t1, one, rr1, m1, k1, t2, one, rr2, m2, k2);+	amm(t1 + w, s1, rr1, m1, k1, t2 + w, s2, rr2, m2, k2);+	for (k = 2; k < 32; k++)+		amm(t1 + k * w, t1 + (k - 1) * w, t1 + w, m1, k1,+		    t2 + k * w, t2 + (k - 1) * w, t2 + w, m2, k2);++	memcpy(x1, t1, w * sizeof(uint64_t));+	memcpy(x2, t2, w * sizeof(uint64_t));+	nwin = (8 * explen1 + WIN - 1) / WIN;+	for (k = nwin; k > 0; k--) {+		for (j = 0; j < WIN; j++)+			amm(x1, x1, x1, m1, k1, x2, x2, x2, m2, k2);+		sel(s1, t1, window(exp1, explen1, k - 1), s2, t2, window(exp2, explen2, k - 1));+		amm(x1, x1, s1, m1, k1, x2, x2, s2, m2, k2);+	}++	/* out of Montgomery form, then under m with one masked subtraction */+	amm(x1, x1, one, m1, k1, x2, x2, one, m2, k2);+	from_digits(lt, n, x1, d);+	select_n(lt, lt2, lt, sub_n(lt2, lt, l1, n) ^ 1, n);+	to_be(out1, modlen1, lt, n);+	from_digits(lt, n, x2, d);+	select_n(lt, lt2, lt, sub_n(lt2, lt, l2, n) ^ 1, n);+	to_be(out2, modlen2, lt, n);++	crypton_bzero(space, words * sizeof(uint64_t));+	free(space);+	return 0;++fail:+	crypton_bzero(space, words * sizeof(uint64_t));+	free(space);+	return 1;+}++#else++int crypton_powm_sec_x2(uint8_t *out1, const uint8_t *base1, uint32_t baselen1,+                        const uint8_t *exp1, uint32_t explen1,+                        const uint8_t *mod1, uint32_t modlen1,+                        uint8_t *out2, const uint8_t *base2, uint32_t baselen2,+                        const uint8_t *exp2, uint32_t explen2,+                        const uint8_t *mod2, uint32_t modlen2)+{+	(void) out1; (void) base1; (void) baselen1; (void) exp1; (void) explen1;+	(void) mod1; (void) modlen1; (void) out2; (void) base2; (void) baselen2;+	(void) exp2; (void) explen2; (void) mod2; (void) modlen2;+	return 1;+}++#endif
+ cbits/crypton_powm_ifma.h view
@@ -0,0 +1,513 @@+/* Generated by cbits/crypton_powm_ifma.py; do not edit.+ *+ * Almost-Montgomery multiplication in radix 2^52, two at once: r = a b+ * 2^-52D mod m, under 2m for a and b under 2m.  Digits are 52 bits in 64-bit+ * words, each number padded with zero words to a whole number of 256-bit+ * registers.  m is read from memory; at 20 digits a is held in registers,+ * above that it is read from memory too. */+#ifndef CRYPTON_POWM_IFMA_H+#define CRYPTON_POWM_IFMA_H++#include <stdint.h>+#include <immintrin.h>++#define IFMA_MASK52 ((UINT64_C(1) << 52) - 1)+#define IFMA_TARGET __attribute__((target("avx2,avx512f,avx512vl,avx512ifma")))++/* 20 lanes to 52-bit digits: what is above 52 bits moves up a lane,+ * and the one-bit carries that can then remain are resolved together */+IFMA_TARGET static inline void ifma_norm5(uint64_t *r, __m256i X0, __m256i X1, __m256i X2, __m256i X3, __m256i X4)+{+	const __m256i mk = _mm256_set1_epi64x((long long) IFMA_MASK52), one = _mm256_set1_epi64x(1);+	const __m256i zero = _mm256_setzero_si256();+	uint64_t g = 0, p = 0, c;+	__m256i C0 = _mm256_srli_epi64(X0, 52);+	__m256i C1 = _mm256_srli_epi64(X1, 52);+	__m256i C2 = _mm256_srli_epi64(X2, 52);+	__m256i C3 = _mm256_srli_epi64(X3, 52);+	__m256i C4 = _mm256_srli_epi64(X4, 52);+	X0 = _mm256_add_epi64(_mm256_and_si256(X0, mk), _mm256_alignr_epi64(C0, zero, 3));+	X1 = _mm256_add_epi64(_mm256_and_si256(X1, mk), _mm256_alignr_epi64(C1, C0, 3));+	X2 = _mm256_add_epi64(_mm256_and_si256(X2, mk), _mm256_alignr_epi64(C2, C1, 3));+	X3 = _mm256_add_epi64(_mm256_and_si256(X3, mk), _mm256_alignr_epi64(C3, C2, 3));+	X4 = _mm256_add_epi64(_mm256_and_si256(X4, mk), _mm256_alignr_epi64(C4, C3, 3));+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X0, mk) << 0;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X0, mk) << 0;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X1, mk) << 4;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X1, mk) << 4;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X2, mk) << 8;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X2, mk) << 8;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X3, mk) << 12;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X3, mk) << 12;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X4, mk) << 16;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X4, mk) << 16;+	c = ((g << 1) + p) ^ p;+	X0 = _mm256_and_si256(_mm256_mask_add_epi64(X0, (__mmask8)(c >> 0 & 15), X0, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 0), X0);+	X1 = _mm256_and_si256(_mm256_mask_add_epi64(X1, (__mmask8)(c >> 4 & 15), X1, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 4), X1);+	X2 = _mm256_and_si256(_mm256_mask_add_epi64(X2, (__mmask8)(c >> 8 & 15), X2, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 8), X2);+	X3 = _mm256_and_si256(_mm256_mask_add_epi64(X3, (__mmask8)(c >> 12 & 15), X3, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 12), X3);+	X4 = _mm256_and_si256(_mm256_mask_add_epi64(X4, (__mmask8)(c >> 16 & 15), X4, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 16), X4);+}++/* both tables' entries k1 and k2 of 32, every entry read and a mask keeping+ * one; entries are 20 words apart */+IFMA_TARGET static void ifma_sel5(uint64_t *o1, const uint64_t *t1, uint64_t k1,+                                uint64_t *o2, const uint64_t *t2, uint64_t k2)+{+	const __m256i one = _mm256_set1_epi64x(1);+	__m256i v1 = _mm256_set1_epi64x((long long) k1), v2 = _mm256_set1_epi64x((long long) k2);+	__m256i kv = _mm256_setzero_si256();+	__m256i P0 = kv, Q0 = kv, P1 = kv, Q1 = kv, P2 = kv, Q2 = kv, P3 = kv, Q3 = kv, P4 = kv, Q4 = kv;+	int k;+	for (k = 0; k < 32; k++) {+		__mmask8 s1 = _mm256_cmpeq_epi64_mask(v1, kv), s2 = _mm256_cmpeq_epi64_mask(v2, kv);+		const uint64_t *e1 = t1 + 20 * k, *e2 = t2 + 20 * k;+		P0 = _mm256_mask_loadu_epi64(P0, s1, e1 + 0); Q0 = _mm256_mask_loadu_epi64(Q0, s2, e2 + 0);+		P1 = _mm256_mask_loadu_epi64(P1, s1, e1 + 4); Q1 = _mm256_mask_loadu_epi64(Q1, s2, e2 + 4);+		P2 = _mm256_mask_loadu_epi64(P2, s1, e1 + 8); Q2 = _mm256_mask_loadu_epi64(Q2, s2, e2 + 8);+		P3 = _mm256_mask_loadu_epi64(P3, s1, e1 + 12); Q3 = _mm256_mask_loadu_epi64(Q3, s2, e2 + 12);+		P4 = _mm256_mask_loadu_epi64(P4, s1, e1 + 16); Q4 = _mm256_mask_loadu_epi64(Q4, s2, e2 + 16);+		kv = _mm256_add_epi64(kv, one);+	}+	_mm256_storeu_si256((__m256i *)(o1 + 0), P0); _mm256_storeu_si256((__m256i *)(o2 + 0), Q0);+	_mm256_storeu_si256((__m256i *)(o1 + 4), P1); _mm256_storeu_si256((__m256i *)(o2 + 4), Q1);+	_mm256_storeu_si256((__m256i *)(o1 + 8), P2); _mm256_storeu_si256((__m256i *)(o2 + 8), Q2);+	_mm256_storeu_si256((__m256i *)(o1 + 12), P3); _mm256_storeu_si256((__m256i *)(o2 + 12), Q3);+	_mm256_storeu_si256((__m256i *)(o1 + 16), P4); _mm256_storeu_si256((__m256i *)(o2 + 16), Q4);+}++/* 32 lanes to 52-bit digits: what is above 52 bits moves up a lane,+ * and the one-bit carries that can then remain are resolved together */+IFMA_TARGET static inline void ifma_norm8(uint64_t *r, __m256i X0, __m256i X1, __m256i X2, __m256i X3, __m256i X4, __m256i X5, __m256i X6, __m256i X7)+{+	const __m256i mk = _mm256_set1_epi64x((long long) IFMA_MASK52), one = _mm256_set1_epi64x(1);+	const __m256i zero = _mm256_setzero_si256();+	uint64_t g = 0, p = 0, c;+	__m256i C0 = _mm256_srli_epi64(X0, 52);+	__m256i C1 = _mm256_srli_epi64(X1, 52);+	__m256i C2 = _mm256_srli_epi64(X2, 52);+	__m256i C3 = _mm256_srli_epi64(X3, 52);+	__m256i C4 = _mm256_srli_epi64(X4, 52);+	__m256i C5 = _mm256_srli_epi64(X5, 52);+	__m256i C6 = _mm256_srli_epi64(X6, 52);+	__m256i C7 = _mm256_srli_epi64(X7, 52);+	X0 = _mm256_add_epi64(_mm256_and_si256(X0, mk), _mm256_alignr_epi64(C0, zero, 3));+	X1 = _mm256_add_epi64(_mm256_and_si256(X1, mk), _mm256_alignr_epi64(C1, C0, 3));+	X2 = _mm256_add_epi64(_mm256_and_si256(X2, mk), _mm256_alignr_epi64(C2, C1, 3));+	X3 = _mm256_add_epi64(_mm256_and_si256(X3, mk), _mm256_alignr_epi64(C3, C2, 3));+	X4 = _mm256_add_epi64(_mm256_and_si256(X4, mk), _mm256_alignr_epi64(C4, C3, 3));+	X5 = _mm256_add_epi64(_mm256_and_si256(X5, mk), _mm256_alignr_epi64(C5, C4, 3));+	X6 = _mm256_add_epi64(_mm256_and_si256(X6, mk), _mm256_alignr_epi64(C6, C5, 3));+	X7 = _mm256_add_epi64(_mm256_and_si256(X7, mk), _mm256_alignr_epi64(C7, C6, 3));+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X0, mk) << 0;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X0, mk) << 0;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X1, mk) << 4;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X1, mk) << 4;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X2, mk) << 8;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X2, mk) << 8;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X3, mk) << 12;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X3, mk) << 12;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X4, mk) << 16;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X4, mk) << 16;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X5, mk) << 20;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X5, mk) << 20;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X6, mk) << 24;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X6, mk) << 24;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X7, mk) << 28;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X7, mk) << 28;+	c = ((g << 1) + p) ^ p;+	X0 = _mm256_and_si256(_mm256_mask_add_epi64(X0, (__mmask8)(c >> 0 & 15), X0, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 0), X0);+	X1 = _mm256_and_si256(_mm256_mask_add_epi64(X1, (__mmask8)(c >> 4 & 15), X1, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 4), X1);+	X2 = _mm256_and_si256(_mm256_mask_add_epi64(X2, (__mmask8)(c >> 8 & 15), X2, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 8), X2);+	X3 = _mm256_and_si256(_mm256_mask_add_epi64(X3, (__mmask8)(c >> 12 & 15), X3, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 12), X3);+	X4 = _mm256_and_si256(_mm256_mask_add_epi64(X4, (__mmask8)(c >> 16 & 15), X4, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 16), X4);+	X5 = _mm256_and_si256(_mm256_mask_add_epi64(X5, (__mmask8)(c >> 20 & 15), X5, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 20), X5);+	X6 = _mm256_and_si256(_mm256_mask_add_epi64(X6, (__mmask8)(c >> 24 & 15), X6, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 24), X6);+	X7 = _mm256_and_si256(_mm256_mask_add_epi64(X7, (__mmask8)(c >> 28 & 15), X7, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 28), X7);+}++/* both tables' entries k1 and k2 of 32, every entry read and a mask keeping+ * one; entries are 32 words apart */+IFMA_TARGET static void ifma_sel8(uint64_t *o1, const uint64_t *t1, uint64_t k1,+                                uint64_t *o2, const uint64_t *t2, uint64_t k2)+{+	const __m256i one = _mm256_set1_epi64x(1);+	__m256i v1 = _mm256_set1_epi64x((long long) k1), v2 = _mm256_set1_epi64x((long long) k2);+	__m256i kv = _mm256_setzero_si256();+	__m256i P0 = kv, Q0 = kv, P1 = kv, Q1 = kv, P2 = kv, Q2 = kv, P3 = kv, Q3 = kv, P4 = kv, Q4 = kv, P5 = kv, Q5 = kv, P6 = kv, Q6 = kv, P7 = kv, Q7 = kv;+	int k;+	for (k = 0; k < 32; k++) {+		__mmask8 s1 = _mm256_cmpeq_epi64_mask(v1, kv), s2 = _mm256_cmpeq_epi64_mask(v2, kv);+		const uint64_t *e1 = t1 + 32 * k, *e2 = t2 + 32 * k;+		P0 = _mm256_mask_loadu_epi64(P0, s1, e1 + 0); Q0 = _mm256_mask_loadu_epi64(Q0, s2, e2 + 0);+		P1 = _mm256_mask_loadu_epi64(P1, s1, e1 + 4); Q1 = _mm256_mask_loadu_epi64(Q1, s2, e2 + 4);+		P2 = _mm256_mask_loadu_epi64(P2, s1, e1 + 8); Q2 = _mm256_mask_loadu_epi64(Q2, s2, e2 + 8);+		P3 = _mm256_mask_loadu_epi64(P3, s1, e1 + 12); Q3 = _mm256_mask_loadu_epi64(Q3, s2, e2 + 12);+		P4 = _mm256_mask_loadu_epi64(P4, s1, e1 + 16); Q4 = _mm256_mask_loadu_epi64(Q4, s2, e2 + 16);+		P5 = _mm256_mask_loadu_epi64(P5, s1, e1 + 20); Q5 = _mm256_mask_loadu_epi64(Q5, s2, e2 + 20);+		P6 = _mm256_mask_loadu_epi64(P6, s1, e1 + 24); Q6 = _mm256_mask_loadu_epi64(Q6, s2, e2 + 24);+		P7 = _mm256_mask_loadu_epi64(P7, s1, e1 + 28); Q7 = _mm256_mask_loadu_epi64(Q7, s2, e2 + 28);+		kv = _mm256_add_epi64(kv, one);+	}+	_mm256_storeu_si256((__m256i *)(o1 + 0), P0); _mm256_storeu_si256((__m256i *)(o2 + 0), Q0);+	_mm256_storeu_si256((__m256i *)(o1 + 4), P1); _mm256_storeu_si256((__m256i *)(o2 + 4), Q1);+	_mm256_storeu_si256((__m256i *)(o1 + 8), P2); _mm256_storeu_si256((__m256i *)(o2 + 8), Q2);+	_mm256_storeu_si256((__m256i *)(o1 + 12), P3); _mm256_storeu_si256((__m256i *)(o2 + 12), Q3);+	_mm256_storeu_si256((__m256i *)(o1 + 16), P4); _mm256_storeu_si256((__m256i *)(o2 + 16), Q4);+	_mm256_storeu_si256((__m256i *)(o1 + 20), P5); _mm256_storeu_si256((__m256i *)(o2 + 20), Q5);+	_mm256_storeu_si256((__m256i *)(o1 + 24), P6); _mm256_storeu_si256((__m256i *)(o2 + 24), Q6);+	_mm256_storeu_si256((__m256i *)(o1 + 28), P7); _mm256_storeu_si256((__m256i *)(o2 + 28), Q7);+}++/* 40 lanes to 52-bit digits: what is above 52 bits moves up a lane,+ * and the one-bit carries that can then remain are resolved together */+IFMA_TARGET static inline void ifma_norm10(uint64_t *r, __m256i X0, __m256i X1, __m256i X2, __m256i X3, __m256i X4, __m256i X5, __m256i X6, __m256i X7, __m256i X8, __m256i X9)+{+	const __m256i mk = _mm256_set1_epi64x((long long) IFMA_MASK52), one = _mm256_set1_epi64x(1);+	const __m256i zero = _mm256_setzero_si256();+	uint64_t g = 0, p = 0, c;+	__m256i C0 = _mm256_srli_epi64(X0, 52);+	__m256i C1 = _mm256_srli_epi64(X1, 52);+	__m256i C2 = _mm256_srli_epi64(X2, 52);+	__m256i C3 = _mm256_srli_epi64(X3, 52);+	__m256i C4 = _mm256_srli_epi64(X4, 52);+	__m256i C5 = _mm256_srli_epi64(X5, 52);+	__m256i C6 = _mm256_srli_epi64(X6, 52);+	__m256i C7 = _mm256_srli_epi64(X7, 52);+	__m256i C8 = _mm256_srli_epi64(X8, 52);+	__m256i C9 = _mm256_srli_epi64(X9, 52);+	X0 = _mm256_add_epi64(_mm256_and_si256(X0, mk), _mm256_alignr_epi64(C0, zero, 3));+	X1 = _mm256_add_epi64(_mm256_and_si256(X1, mk), _mm256_alignr_epi64(C1, C0, 3));+	X2 = _mm256_add_epi64(_mm256_and_si256(X2, mk), _mm256_alignr_epi64(C2, C1, 3));+	X3 = _mm256_add_epi64(_mm256_and_si256(X3, mk), _mm256_alignr_epi64(C3, C2, 3));+	X4 = _mm256_add_epi64(_mm256_and_si256(X4, mk), _mm256_alignr_epi64(C4, C3, 3));+	X5 = _mm256_add_epi64(_mm256_and_si256(X5, mk), _mm256_alignr_epi64(C5, C4, 3));+	X6 = _mm256_add_epi64(_mm256_and_si256(X6, mk), _mm256_alignr_epi64(C6, C5, 3));+	X7 = _mm256_add_epi64(_mm256_and_si256(X7, mk), _mm256_alignr_epi64(C7, C6, 3));+	X8 = _mm256_add_epi64(_mm256_and_si256(X8, mk), _mm256_alignr_epi64(C8, C7, 3));+	X9 = _mm256_add_epi64(_mm256_and_si256(X9, mk), _mm256_alignr_epi64(C9, C8, 3));+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X0, mk) << 0;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X0, mk) << 0;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X1, mk) << 4;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X1, mk) << 4;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X2, mk) << 8;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X2, mk) << 8;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X3, mk) << 12;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X3, mk) << 12;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X4, mk) << 16;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X4, mk) << 16;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X5, mk) << 20;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X5, mk) << 20;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X6, mk) << 24;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X6, mk) << 24;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X7, mk) << 28;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X7, mk) << 28;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X8, mk) << 32;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X8, mk) << 32;+	g |= (uint64_t) _mm256_cmpgt_epu64_mask(X9, mk) << 36;+	p |= (uint64_t) _mm256_cmpeq_epu64_mask(X9, mk) << 36;+	c = ((g << 1) + p) ^ p;+	X0 = _mm256_and_si256(_mm256_mask_add_epi64(X0, (__mmask8)(c >> 0 & 15), X0, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 0), X0);+	X1 = _mm256_and_si256(_mm256_mask_add_epi64(X1, (__mmask8)(c >> 4 & 15), X1, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 4), X1);+	X2 = _mm256_and_si256(_mm256_mask_add_epi64(X2, (__mmask8)(c >> 8 & 15), X2, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 8), X2);+	X3 = _mm256_and_si256(_mm256_mask_add_epi64(X3, (__mmask8)(c >> 12 & 15), X3, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 12), X3);+	X4 = _mm256_and_si256(_mm256_mask_add_epi64(X4, (__mmask8)(c >> 16 & 15), X4, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 16), X4);+	X5 = _mm256_and_si256(_mm256_mask_add_epi64(X5, (__mmask8)(c >> 20 & 15), X5, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 20), X5);+	X6 = _mm256_and_si256(_mm256_mask_add_epi64(X6, (__mmask8)(c >> 24 & 15), X6, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 24), X6);+	X7 = _mm256_and_si256(_mm256_mask_add_epi64(X7, (__mmask8)(c >> 28 & 15), X7, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 28), X7);+	X8 = _mm256_and_si256(_mm256_mask_add_epi64(X8, (__mmask8)(c >> 32 & 15), X8, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 32), X8);+	X9 = _mm256_and_si256(_mm256_mask_add_epi64(X9, (__mmask8)(c >> 36 & 15), X9, one), mk);+	_mm256_storeu_si256((__m256i *)(r + 36), X9);+}++/* both tables' entries k1 and k2 of 32, every entry read and a mask keeping+ * one; entries are 40 words apart */+IFMA_TARGET static void ifma_sel10(uint64_t *o1, const uint64_t *t1, uint64_t k1,+                                uint64_t *o2, const uint64_t *t2, uint64_t k2)+{+	const __m256i one = _mm256_set1_epi64x(1);+	__m256i v1 = _mm256_set1_epi64x((long long) k1), v2 = _mm256_set1_epi64x((long long) k2);+	__m256i kv = _mm256_setzero_si256();+	__m256i P0 = kv, Q0 = kv, P1 = kv, Q1 = kv, P2 = kv, Q2 = kv, P3 = kv, Q3 = kv, P4 = kv, Q4 = kv, P5 = kv, Q5 = kv, P6 = kv, Q6 = kv, P7 = kv, Q7 = kv, P8 = kv, Q8 = kv, P9 = kv, Q9 = kv;+	int k;+	for (k = 0; k < 32; k++) {+		__mmask8 s1 = _mm256_cmpeq_epi64_mask(v1, kv), s2 = _mm256_cmpeq_epi64_mask(v2, kv);+		const uint64_t *e1 = t1 + 40 * k, *e2 = t2 + 40 * k;+		P0 = _mm256_mask_loadu_epi64(P0, s1, e1 + 0); Q0 = _mm256_mask_loadu_epi64(Q0, s2, e2 + 0);+		P1 = _mm256_mask_loadu_epi64(P1, s1, e1 + 4); Q1 = _mm256_mask_loadu_epi64(Q1, s2, e2 + 4);+		P2 = _mm256_mask_loadu_epi64(P2, s1, e1 + 8); Q2 = _mm256_mask_loadu_epi64(Q2, s2, e2 + 8);+		P3 = _mm256_mask_loadu_epi64(P3, s1, e1 + 12); Q3 = _mm256_mask_loadu_epi64(Q3, s2, e2 + 12);+		P4 = _mm256_mask_loadu_epi64(P4, s1, e1 + 16); Q4 = _mm256_mask_loadu_epi64(Q4, s2, e2 + 16);+		P5 = _mm256_mask_loadu_epi64(P5, s1, e1 + 20); Q5 = _mm256_mask_loadu_epi64(Q5, s2, e2 + 20);+		P6 = _mm256_mask_loadu_epi64(P6, s1, e1 + 24); Q6 = _mm256_mask_loadu_epi64(Q6, s2, e2 + 24);+		P7 = _mm256_mask_loadu_epi64(P7, s1, e1 + 28); Q7 = _mm256_mask_loadu_epi64(Q7, s2, e2 + 28);+		P8 = _mm256_mask_loadu_epi64(P8, s1, e1 + 32); Q8 = _mm256_mask_loadu_epi64(Q8, s2, e2 + 32);+		P9 = _mm256_mask_loadu_epi64(P9, s1, e1 + 36); Q9 = _mm256_mask_loadu_epi64(Q9, s2, e2 + 36);+		kv = _mm256_add_epi64(kv, one);+	}+	_mm256_storeu_si256((__m256i *)(o1 + 0), P0); _mm256_storeu_si256((__m256i *)(o2 + 0), Q0);+	_mm256_storeu_si256((__m256i *)(o1 + 4), P1); _mm256_storeu_si256((__m256i *)(o2 + 4), Q1);+	_mm256_storeu_si256((__m256i *)(o1 + 8), P2); _mm256_storeu_si256((__m256i *)(o2 + 8), Q2);+	_mm256_storeu_si256((__m256i *)(o1 + 12), P3); _mm256_storeu_si256((__m256i *)(o2 + 12), Q3);+	_mm256_storeu_si256((__m256i *)(o1 + 16), P4); _mm256_storeu_si256((__m256i *)(o2 + 16), Q4);+	_mm256_storeu_si256((__m256i *)(o1 + 20), P5); _mm256_storeu_si256((__m256i *)(o2 + 20), Q5);+	_mm256_storeu_si256((__m256i *)(o1 + 24), P6); _mm256_storeu_si256((__m256i *)(o2 + 24), Q6);+	_mm256_storeu_si256((__m256i *)(o1 + 28), P7); _mm256_storeu_si256((__m256i *)(o2 + 28), Q7);+	_mm256_storeu_si256((__m256i *)(o1 + 32), P8); _mm256_storeu_si256((__m256i *)(o2 + 32), Q8);+	_mm256_storeu_si256((__m256i *)(o1 + 36), P9); _mm256_storeu_si256((__m256i *)(o2 + 36), Q9);+}++/* 20 digits, 5 registers a number */+IFMA_TARGET static void ifma_amm20_x2(uint64_t *r1, const uint64_t *a1, const uint64_t *b1,+                                    const uint64_t *m1, uint64_t k1,+                                    uint64_t *r2, const uint64_t *a2, const uint64_t *b2,+                                    const uint64_t *m2, uint64_t k2)+{+	const __m256i zero = _mm256_setzero_si256();+#define L(p, k) _mm256_loadu_si256((const __m256i *)((p) + 4 * (k)))+	__m256i X0 = zero, X1 = zero, X2 = zero, X3 = zero, X4 = zero, Y0 = zero, Y1 = zero, Y2 = zero, Y3 = zero, Y4 = zero;+	__m256i A0 = L(a1, 0), A1 = L(a1, 1), A2 = L(a1, 2), A3 = L(a1, 3), A4 = L(a1, 4);+	__m256i B0 = L(a2, 0), B1 = L(a2, 1), B2 = L(a2, 2), B3 = L(a2, 3), B4 = L(a2, 4);+	uint64_t x1 = 0, x2 = 0;+	int i;++	for (i = 0; i < 20; i++) {+		uint64_t bi = b1[i], ci = b2[i], y, z;+		/* lane 0 in full: the products' high halves, which are lane 1's, too */+		unsigned __int128 s1 = (unsigned __int128) a1[0] * bi + x1;+		unsigned __int128 s2 = (unsigned __int128) a2[0] * ci + x2;+		__m256i bv = _mm256_set1_epi64x((long long) bi), cv = _mm256_set1_epi64x((long long) ci);+		__m256i yv, zv;++		y = ((uint64_t) s1 * k1) & IFMA_MASK52;+		z = ((uint64_t) s2 * k2) & IFMA_MASK52;+		s1 += (unsigned __int128) m1[0] * y;+		s2 += (unsigned __int128) m2[0] * z;+		yv = _mm256_set1_epi64x((long long) y);+		zv = _mm256_set1_epi64x((long long) z);+		X0 = _mm256_madd52lo_epu64(X0, A0, bv); Y0 = _mm256_madd52lo_epu64(Y0, B0, cv);+		X1 = _mm256_madd52lo_epu64(X1, A1, bv); Y1 = _mm256_madd52lo_epu64(Y1, B1, cv);+		X2 = _mm256_madd52lo_epu64(X2, A2, bv); Y2 = _mm256_madd52lo_epu64(Y2, B2, cv);+		X3 = _mm256_madd52lo_epu64(X3, A3, bv); Y3 = _mm256_madd52lo_epu64(Y3, B3, cv);+		X4 = _mm256_madd52lo_epu64(X4, A4, bv); Y4 = _mm256_madd52lo_epu64(Y4, B4, cv);+		X0 = _mm256_madd52lo_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52lo_epu64(Y0, L(m2, 0), zv);+		X1 = _mm256_madd52lo_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52lo_epu64(Y1, L(m2, 1), zv);+		X2 = _mm256_madd52lo_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52lo_epu64(Y2, L(m2, 2), zv);+		X3 = _mm256_madd52lo_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52lo_epu64(Y3, L(m2, 3), zv);+		X4 = _mm256_madd52lo_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52lo_epu64(Y4, L(m2, 4), zv);+		/* down a lane: lane 0 is a multiple of 2^52, and s carries it */+		X0 = _mm256_alignr_epi64(X1, X0, 1); Y0 = _mm256_alignr_epi64(Y1, Y0, 1);+		X1 = _mm256_alignr_epi64(X2, X1, 1); Y1 = _mm256_alignr_epi64(Y2, Y1, 1);+		X2 = _mm256_alignr_epi64(X3, X2, 1); Y2 = _mm256_alignr_epi64(Y3, Y2, 1);+		X3 = _mm256_alignr_epi64(X4, X3, 1); Y3 = _mm256_alignr_epi64(Y4, Y3, 1);+		X4 = _mm256_alignr_epi64(zero, X4, 1); Y4 = _mm256_alignr_epi64(zero, Y4, 1);+		x1 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(X0)) + (uint64_t)(s1 >> 52);+		x2 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(Y0)) + (uint64_t)(s2 >> 52);+		X0 = _mm256_madd52hi_epu64(X0, A0, bv); Y0 = _mm256_madd52hi_epu64(Y0, B0, cv);+		X1 = _mm256_madd52hi_epu64(X1, A1, bv); Y1 = _mm256_madd52hi_epu64(Y1, B1, cv);+		X2 = _mm256_madd52hi_epu64(X2, A2, bv); Y2 = _mm256_madd52hi_epu64(Y2, B2, cv);+		X3 = _mm256_madd52hi_epu64(X3, A3, bv); Y3 = _mm256_madd52hi_epu64(Y3, B3, cv);+		X4 = _mm256_madd52hi_epu64(X4, A4, bv); Y4 = _mm256_madd52hi_epu64(Y4, B4, cv);+		X0 = _mm256_madd52hi_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52hi_epu64(Y0, L(m2, 0), zv);+		X1 = _mm256_madd52hi_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52hi_epu64(Y1, L(m2, 1), zv);+		X2 = _mm256_madd52hi_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52hi_epu64(Y2, L(m2, 2), zv);+		X3 = _mm256_madd52hi_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52hi_epu64(Y3, L(m2, 3), zv);+		X4 = _mm256_madd52hi_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52hi_epu64(Y4, L(m2, 4), zv);+	}+#undef L+	/* lane 0, with the carry it was never given, is what the scalar holds */+	X0 = _mm256_mask_set1_epi64(X0, 1, (long long) x1);+	Y0 = _mm256_mask_set1_epi64(Y0, 1, (long long) x2);+	ifma_norm5(r1, X0, X1, X2, X3, X4);+	ifma_norm5(r2, Y0, Y1, Y2, Y3, Y4);+}++/* 30 digits, 8 registers a number */+IFMA_TARGET static void ifma_amm30_x2(uint64_t *r1, const uint64_t *a1, const uint64_t *b1,+                                    const uint64_t *m1, uint64_t k1,+                                    uint64_t *r2, const uint64_t *a2, const uint64_t *b2,+                                    const uint64_t *m2, uint64_t k2)+{+	const __m256i zero = _mm256_setzero_si256();+#define L(p, k) _mm256_loadu_si256((const __m256i *)((p) + 4 * (k)))+	__m256i X0 = zero, X1 = zero, X2 = zero, X3 = zero, X4 = zero, X5 = zero, X6 = zero, X7 = zero, Y0 = zero, Y1 = zero, Y2 = zero, Y3 = zero, Y4 = zero, Y5 = zero, Y6 = zero, Y7 = zero;+	uint64_t x1 = 0, x2 = 0;+	int i;++	for (i = 0; i < 30; i++) {+		uint64_t bi = b1[i], ci = b2[i], y, z;+		/* lane 0 in full: the products' high halves, which are lane 1's, too */+		unsigned __int128 s1 = (unsigned __int128) a1[0] * bi + x1;+		unsigned __int128 s2 = (unsigned __int128) a2[0] * ci + x2;+		__m256i bv = _mm256_set1_epi64x((long long) bi), cv = _mm256_set1_epi64x((long long) ci);+		__m256i yv, zv;++		y = ((uint64_t) s1 * k1) & IFMA_MASK52;+		z = ((uint64_t) s2 * k2) & IFMA_MASK52;+		s1 += (unsigned __int128) m1[0] * y;+		s2 += (unsigned __int128) m2[0] * z;+		yv = _mm256_set1_epi64x((long long) y);+		zv = _mm256_set1_epi64x((long long) z);+		X0 = _mm256_madd52lo_epu64(X0, L(a1, 0), bv); Y0 = _mm256_madd52lo_epu64(Y0, L(a2, 0), cv);+		X1 = _mm256_madd52lo_epu64(X1, L(a1, 1), bv); Y1 = _mm256_madd52lo_epu64(Y1, L(a2, 1), cv);+		X2 = _mm256_madd52lo_epu64(X2, L(a1, 2), bv); Y2 = _mm256_madd52lo_epu64(Y2, L(a2, 2), cv);+		X3 = _mm256_madd52lo_epu64(X3, L(a1, 3), bv); Y3 = _mm256_madd52lo_epu64(Y3, L(a2, 3), cv);+		X4 = _mm256_madd52lo_epu64(X4, L(a1, 4), bv); Y4 = _mm256_madd52lo_epu64(Y4, L(a2, 4), cv);+		X5 = _mm256_madd52lo_epu64(X5, L(a1, 5), bv); Y5 = _mm256_madd52lo_epu64(Y5, L(a2, 5), cv);+		X6 = _mm256_madd52lo_epu64(X6, L(a1, 6), bv); Y6 = _mm256_madd52lo_epu64(Y6, L(a2, 6), cv);+		X7 = _mm256_madd52lo_epu64(X7, L(a1, 7), bv); Y7 = _mm256_madd52lo_epu64(Y7, L(a2, 7), cv);+		X0 = _mm256_madd52lo_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52lo_epu64(Y0, L(m2, 0), zv);+		X1 = _mm256_madd52lo_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52lo_epu64(Y1, L(m2, 1), zv);+		X2 = _mm256_madd52lo_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52lo_epu64(Y2, L(m2, 2), zv);+		X3 = _mm256_madd52lo_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52lo_epu64(Y3, L(m2, 3), zv);+		X4 = _mm256_madd52lo_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52lo_epu64(Y4, L(m2, 4), zv);+		X5 = _mm256_madd52lo_epu64(X5, L(m1, 5), yv); Y5 = _mm256_madd52lo_epu64(Y5, L(m2, 5), zv);+		X6 = _mm256_madd52lo_epu64(X6, L(m1, 6), yv); Y6 = _mm256_madd52lo_epu64(Y6, L(m2, 6), zv);+		X7 = _mm256_madd52lo_epu64(X7, L(m1, 7), yv); Y7 = _mm256_madd52lo_epu64(Y7, L(m2, 7), zv);+		/* down a lane: lane 0 is a multiple of 2^52, and s carries it */+		X0 = _mm256_alignr_epi64(X1, X0, 1); Y0 = _mm256_alignr_epi64(Y1, Y0, 1);+		X1 = _mm256_alignr_epi64(X2, X1, 1); Y1 = _mm256_alignr_epi64(Y2, Y1, 1);+		X2 = _mm256_alignr_epi64(X3, X2, 1); Y2 = _mm256_alignr_epi64(Y3, Y2, 1);+		X3 = _mm256_alignr_epi64(X4, X3, 1); Y3 = _mm256_alignr_epi64(Y4, Y3, 1);+		X4 = _mm256_alignr_epi64(X5, X4, 1); Y4 = _mm256_alignr_epi64(Y5, Y4, 1);+		X5 = _mm256_alignr_epi64(X6, X5, 1); Y5 = _mm256_alignr_epi64(Y6, Y5, 1);+		X6 = _mm256_alignr_epi64(X7, X6, 1); Y6 = _mm256_alignr_epi64(Y7, Y6, 1);+		X7 = _mm256_alignr_epi64(zero, X7, 1); Y7 = _mm256_alignr_epi64(zero, Y7, 1);+		x1 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(X0)) + (uint64_t)(s1 >> 52);+		x2 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(Y0)) + (uint64_t)(s2 >> 52);+		X0 = _mm256_madd52hi_epu64(X0, L(a1, 0), bv); Y0 = _mm256_madd52hi_epu64(Y0, L(a2, 0), cv);+		X1 = _mm256_madd52hi_epu64(X1, L(a1, 1), bv); Y1 = _mm256_madd52hi_epu64(Y1, L(a2, 1), cv);+		X2 = _mm256_madd52hi_epu64(X2, L(a1, 2), bv); Y2 = _mm256_madd52hi_epu64(Y2, L(a2, 2), cv);+		X3 = _mm256_madd52hi_epu64(X3, L(a1, 3), bv); Y3 = _mm256_madd52hi_epu64(Y3, L(a2, 3), cv);+		X4 = _mm256_madd52hi_epu64(X4, L(a1, 4), bv); Y4 = _mm256_madd52hi_epu64(Y4, L(a2, 4), cv);+		X5 = _mm256_madd52hi_epu64(X5, L(a1, 5), bv); Y5 = _mm256_madd52hi_epu64(Y5, L(a2, 5), cv);+		X6 = _mm256_madd52hi_epu64(X6, L(a1, 6), bv); Y6 = _mm256_madd52hi_epu64(Y6, L(a2, 6), cv);+		X7 = _mm256_madd52hi_epu64(X7, L(a1, 7), bv); Y7 = _mm256_madd52hi_epu64(Y7, L(a2, 7), cv);+		X0 = _mm256_madd52hi_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52hi_epu64(Y0, L(m2, 0), zv);+		X1 = _mm256_madd52hi_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52hi_epu64(Y1, L(m2, 1), zv);+		X2 = _mm256_madd52hi_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52hi_epu64(Y2, L(m2, 2), zv);+		X3 = _mm256_madd52hi_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52hi_epu64(Y3, L(m2, 3), zv);+		X4 = _mm256_madd52hi_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52hi_epu64(Y4, L(m2, 4), zv);+		X5 = _mm256_madd52hi_epu64(X5, L(m1, 5), yv); Y5 = _mm256_madd52hi_epu64(Y5, L(m2, 5), zv);+		X6 = _mm256_madd52hi_epu64(X6, L(m1, 6), yv); Y6 = _mm256_madd52hi_epu64(Y6, L(m2, 6), zv);+		X7 = _mm256_madd52hi_epu64(X7, L(m1, 7), yv); Y7 = _mm256_madd52hi_epu64(Y7, L(m2, 7), zv);+	}+#undef L+	/* lane 0, with the carry it was never given, is what the scalar holds */+	X0 = _mm256_mask_set1_epi64(X0, 1, (long long) x1);+	Y0 = _mm256_mask_set1_epi64(Y0, 1, (long long) x2);+	ifma_norm8(r1, X0, X1, X2, X3, X4, X5, X6, X7);+	ifma_norm8(r2, Y0, Y1, Y2, Y3, Y4, Y5, Y6, Y7);+}++/* 40 digits, 10 registers a number */+IFMA_TARGET static void ifma_amm40_x2(uint64_t *r1, const uint64_t *a1, const uint64_t *b1,+                                    const uint64_t *m1, uint64_t k1,+                                    uint64_t *r2, const uint64_t *a2, const uint64_t *b2,+                                    const uint64_t *m2, uint64_t k2)+{+	const __m256i zero = _mm256_setzero_si256();+#define L(p, k) _mm256_loadu_si256((const __m256i *)((p) + 4 * (k)))+	__m256i X0 = zero, X1 = zero, X2 = zero, X3 = zero, X4 = zero, X5 = zero, X6 = zero, X7 = zero, X8 = zero, X9 = zero, Y0 = zero, Y1 = zero, Y2 = zero, Y3 = zero, Y4 = zero, Y5 = zero, Y6 = zero, Y7 = zero, Y8 = zero, Y9 = zero;+	uint64_t x1 = 0, x2 = 0;+	int i;++	for (i = 0; i < 40; i++) {+		uint64_t bi = b1[i], ci = b2[i], y, z;+		/* lane 0 in full: the products' high halves, which are lane 1's, too */+		unsigned __int128 s1 = (unsigned __int128) a1[0] * bi + x1;+		unsigned __int128 s2 = (unsigned __int128) a2[0] * ci + x2;+		__m256i bv = _mm256_set1_epi64x((long long) bi), cv = _mm256_set1_epi64x((long long) ci);+		__m256i yv, zv;++		y = ((uint64_t) s1 * k1) & IFMA_MASK52;+		z = ((uint64_t) s2 * k2) & IFMA_MASK52;+		s1 += (unsigned __int128) m1[0] * y;+		s2 += (unsigned __int128) m2[0] * z;+		yv = _mm256_set1_epi64x((long long) y);+		zv = _mm256_set1_epi64x((long long) z);+		X0 = _mm256_madd52lo_epu64(X0, L(a1, 0), bv); Y0 = _mm256_madd52lo_epu64(Y0, L(a2, 0), cv);+		X1 = _mm256_madd52lo_epu64(X1, L(a1, 1), bv); Y1 = _mm256_madd52lo_epu64(Y1, L(a2, 1), cv);+		X2 = _mm256_madd52lo_epu64(X2, L(a1, 2), bv); Y2 = _mm256_madd52lo_epu64(Y2, L(a2, 2), cv);+		X3 = _mm256_madd52lo_epu64(X3, L(a1, 3), bv); Y3 = _mm256_madd52lo_epu64(Y3, L(a2, 3), cv);+		X4 = _mm256_madd52lo_epu64(X4, L(a1, 4), bv); Y4 = _mm256_madd52lo_epu64(Y4, L(a2, 4), cv);+		X5 = _mm256_madd52lo_epu64(X5, L(a1, 5), bv); Y5 = _mm256_madd52lo_epu64(Y5, L(a2, 5), cv);+		X6 = _mm256_madd52lo_epu64(X6, L(a1, 6), bv); Y6 = _mm256_madd52lo_epu64(Y6, L(a2, 6), cv);+		X7 = _mm256_madd52lo_epu64(X7, L(a1, 7), bv); Y7 = _mm256_madd52lo_epu64(Y7, L(a2, 7), cv);+		X8 = _mm256_madd52lo_epu64(X8, L(a1, 8), bv); Y8 = _mm256_madd52lo_epu64(Y8, L(a2, 8), cv);+		X9 = _mm256_madd52lo_epu64(X9, L(a1, 9), bv); Y9 = _mm256_madd52lo_epu64(Y9, L(a2, 9), cv);+		X0 = _mm256_madd52lo_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52lo_epu64(Y0, L(m2, 0), zv);+		X1 = _mm256_madd52lo_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52lo_epu64(Y1, L(m2, 1), zv);+		X2 = _mm256_madd52lo_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52lo_epu64(Y2, L(m2, 2), zv);+		X3 = _mm256_madd52lo_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52lo_epu64(Y3, L(m2, 3), zv);+		X4 = _mm256_madd52lo_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52lo_epu64(Y4, L(m2, 4), zv);+		X5 = _mm256_madd52lo_epu64(X5, L(m1, 5), yv); Y5 = _mm256_madd52lo_epu64(Y5, L(m2, 5), zv);+		X6 = _mm256_madd52lo_epu64(X6, L(m1, 6), yv); Y6 = _mm256_madd52lo_epu64(Y6, L(m2, 6), zv);+		X7 = _mm256_madd52lo_epu64(X7, L(m1, 7), yv); Y7 = _mm256_madd52lo_epu64(Y7, L(m2, 7), zv);+		X8 = _mm256_madd52lo_epu64(X8, L(m1, 8), yv); Y8 = _mm256_madd52lo_epu64(Y8, L(m2, 8), zv);+		X9 = _mm256_madd52lo_epu64(X9, L(m1, 9), yv); Y9 = _mm256_madd52lo_epu64(Y9, L(m2, 9), zv);+		/* down a lane: lane 0 is a multiple of 2^52, and s carries it */+		X0 = _mm256_alignr_epi64(X1, X0, 1); Y0 = _mm256_alignr_epi64(Y1, Y0, 1);+		X1 = _mm256_alignr_epi64(X2, X1, 1); Y1 = _mm256_alignr_epi64(Y2, Y1, 1);+		X2 = _mm256_alignr_epi64(X3, X2, 1); Y2 = _mm256_alignr_epi64(Y3, Y2, 1);+		X3 = _mm256_alignr_epi64(X4, X3, 1); Y3 = _mm256_alignr_epi64(Y4, Y3, 1);+		X4 = _mm256_alignr_epi64(X5, X4, 1); Y4 = _mm256_alignr_epi64(Y5, Y4, 1);+		X5 = _mm256_alignr_epi64(X6, X5, 1); Y5 = _mm256_alignr_epi64(Y6, Y5, 1);+		X6 = _mm256_alignr_epi64(X7, X6, 1); Y6 = _mm256_alignr_epi64(Y7, Y6, 1);+		X7 = _mm256_alignr_epi64(X8, X7, 1); Y7 = _mm256_alignr_epi64(Y8, Y7, 1);+		X8 = _mm256_alignr_epi64(X9, X8, 1); Y8 = _mm256_alignr_epi64(Y9, Y8, 1);+		X9 = _mm256_alignr_epi64(zero, X9, 1); Y9 = _mm256_alignr_epi64(zero, Y9, 1);+		x1 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(X0)) + (uint64_t)(s1 >> 52);+		x2 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(Y0)) + (uint64_t)(s2 >> 52);+		X0 = _mm256_madd52hi_epu64(X0, L(a1, 0), bv); Y0 = _mm256_madd52hi_epu64(Y0, L(a2, 0), cv);+		X1 = _mm256_madd52hi_epu64(X1, L(a1, 1), bv); Y1 = _mm256_madd52hi_epu64(Y1, L(a2, 1), cv);+		X2 = _mm256_madd52hi_epu64(X2, L(a1, 2), bv); Y2 = _mm256_madd52hi_epu64(Y2, L(a2, 2), cv);+		X3 = _mm256_madd52hi_epu64(X3, L(a1, 3), bv); Y3 = _mm256_madd52hi_epu64(Y3, L(a2, 3), cv);+		X4 = _mm256_madd52hi_epu64(X4, L(a1, 4), bv); Y4 = _mm256_madd52hi_epu64(Y4, L(a2, 4), cv);+		X5 = _mm256_madd52hi_epu64(X5, L(a1, 5), bv); Y5 = _mm256_madd52hi_epu64(Y5, L(a2, 5), cv);+		X6 = _mm256_madd52hi_epu64(X6, L(a1, 6), bv); Y6 = _mm256_madd52hi_epu64(Y6, L(a2, 6), cv);+		X7 = _mm256_madd52hi_epu64(X7, L(a1, 7), bv); Y7 = _mm256_madd52hi_epu64(Y7, L(a2, 7), cv);+		X8 = _mm256_madd52hi_epu64(X8, L(a1, 8), bv); Y8 = _mm256_madd52hi_epu64(Y8, L(a2, 8), cv);+		X9 = _mm256_madd52hi_epu64(X9, L(a1, 9), bv); Y9 = _mm256_madd52hi_epu64(Y9, L(a2, 9), cv);+		X0 = _mm256_madd52hi_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52hi_epu64(Y0, L(m2, 0), zv);+		X1 = _mm256_madd52hi_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52hi_epu64(Y1, L(m2, 1), zv);+		X2 = _mm256_madd52hi_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52hi_epu64(Y2, L(m2, 2), zv);+		X3 = _mm256_madd52hi_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52hi_epu64(Y3, L(m2, 3), zv);+		X4 = _mm256_madd52hi_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52hi_epu64(Y4, L(m2, 4), zv);+		X5 = _mm256_madd52hi_epu64(X5, L(m1, 5), yv); Y5 = _mm256_madd52hi_epu64(Y5, L(m2, 5), zv);+		X6 = _mm256_madd52hi_epu64(X6, L(m1, 6), yv); Y6 = _mm256_madd52hi_epu64(Y6, L(m2, 6), zv);+		X7 = _mm256_madd52hi_epu64(X7, L(m1, 7), yv); Y7 = _mm256_madd52hi_epu64(Y7, L(m2, 7), zv);+		X8 = _mm256_madd52hi_epu64(X8, L(m1, 8), yv); Y8 = _mm256_madd52hi_epu64(Y8, L(m2, 8), zv);+		X9 = _mm256_madd52hi_epu64(X9, L(m1, 9), yv); Y9 = _mm256_madd52hi_epu64(Y9, L(m2, 9), zv);+	}+#undef L+	/* lane 0, with the carry it was never given, is what the scalar holds */+	X0 = _mm256_mask_set1_epi64(X0, 1, (long long) x1);+	Y0 = _mm256_mask_set1_epi64(Y0, 1, (long long) x2);+	ifma_norm10(r1, X0, X1, X2, X3, X4, X5, X6, X7, X8, X9);+	ifma_norm10(r2, Y0, Y1, Y2, Y3, Y4, Y5, Y6, Y7, Y8, Y9);+}++#endif
+ cbits/crypton_powm_ifma.py view
@@ -0,0 +1,178 @@+#!/usr/bin/env python3+"""Generate cbits/crypton_powm_ifma.h: two almost-Montgomery multiplications+at once in radix 2^52 with AVX-512 IFMA on 256-bit registers, at 20, 30 and+40 digits.  Usage: python3 cbits/crypton_powm_ifma.py > cbits/crypton_powm_ifma.h++Gueron and Krasnov's method, a digit of b at a time.  Lane 0 is followed in+full in a scalar, so it is read before the high halves go in and they stay+off the chain the next digit waits on.+"""++SIZES = (20, 30, 40)++HEADER = """\+/* Generated by cbits/crypton_powm_ifma.py; do not edit.+ *+ * Almost-Montgomery multiplication in radix 2^52, two at once: r = a b+ * 2^-52D mod m, under 2m for a and b under 2m.  Digits are 52 bits in 64-bit+ * words, each number padded with zero words to a whole number of 256-bit+ * registers.  m is read from memory; at 20 digits a is held in registers,+ * above that it is read from memory too. */+#ifndef CRYPTON_POWM_IFMA_H+#define CRYPTON_POWM_IFMA_H++#include <stdint.h>+#include <immintrin.h>++#define IFMA_MASK52 ((UINT64_C(1) << 52) - 1)+#define IFMA_TARGET __attribute__((target("avx2,avx512f,avx512vl,avx512ifma")))+"""++FOOTER = """\+#endif+"""+++def regs(d):+    return (d + 3) // 4+++def gen_norm(R):+    o = []+    w = o.append+    xs = [f"X{j}" for j in range(R)]+    w(f"/* {4 * R} lanes to 52-bit digits: what is above 52 bits moves up a lane,")+    w(f" * and the one-bit carries that can then remain are resolved together */")+    w(f"IFMA_TARGET static inline void ifma_norm{R}(uint64_t *r, " ++      ", ".join(f"__m256i {x}" for x in xs) + ")")+    w("{")+    w("\tconst __m256i mk = _mm256_set1_epi64x((long long) IFMA_MASK52), one = _mm256_set1_epi64x(1);")+    w("\tconst __m256i zero = _mm256_setzero_si256();")+    w("\tuint64_t g = 0, p = 0, c;")+    for j in range(R):+        w(f"\t__m256i C{j} = _mm256_srli_epi64(X{j}, 52);")+    for j in range(R):+        prev = f"C{j - 1}" if j else "zero"+        w(f"\tX{j} = _mm256_add_epi64(_mm256_and_si256(X{j}, mk), _mm256_alignr_epi64(C{j}, {prev}, 3));")+    for j in range(R):+        w(f"\tg |= (uint64_t) _mm256_cmpgt_epu64_mask(X{j}, mk) << {4 * j};")+        w(f"\tp |= (uint64_t) _mm256_cmpeq_epu64_mask(X{j}, mk) << {4 * j};")+    w("\tc = ((g << 1) + p) ^ p;")+    for j in range(R):+        w(f"\tX{j} = _mm256_and_si256(_mm256_mask_add_epi64(X{j}, (__mmask8)(c >> {4 * j} & 15), X{j}, one), mk);")+        w(f"\t_mm256_storeu_si256((__m256i *)(r + {4 * j}), X{j});")+    w("}")+    w("")+    return "\n".join(o)+++def gen_amm(d):+    R = regs(d)+    in_regs = R <= 5+    o = []+    w = o.append+    w(f"/* {d} digits, {R} registers a number */")+    w(f"IFMA_TARGET static void ifma_amm{d}_x2(uint64_t *r1, const uint64_t *a1, const uint64_t *b1,")+    w(f"                                    const uint64_t *m1, uint64_t k1,")+    w(f"                                    uint64_t *r2, const uint64_t *a2, const uint64_t *b2,")+    w(f"                                    const uint64_t *m2, uint64_t k2)")+    w("{")+    w("\tconst __m256i zero = _mm256_setzero_si256();")+    w("#define L(p, k) _mm256_loadu_si256((const __m256i *)((p) + 4 * (k)))")+    X = [f"X{j}" for j in range(R)]+    Y = [f"Y{j}" for j in range(R)]+    w("\t__m256i " + ", ".join(f"{v} = zero" for v in X + Y) + ";")+    if in_regs:+        w("\t__m256i " + ", ".join(f"A{j} = L(a1, {j})" for j in range(R)) + ";")+        w("\t__m256i " + ", ".join(f"B{j} = L(a2, {j})" for j in range(R)) + ";")+        A = [f"A{j}" for j in range(R)]+        B = [f"B{j}" for j in range(R)]+    else:+        A = [f"L(a1, {j})" for j in range(R)]+        B = [f"L(a2, {j})" for j in range(R)]+    M = [f"L(m1, {j})" for j in range(R)]+    N = [f"L(m2, {j})" for j in range(R)]+    w("\tuint64_t x1 = 0, x2 = 0;")+    w("\tint i;")+    w("")+    w(f"\tfor (i = 0; i < {d}; i++) {{")+    w("\t\tuint64_t bi = b1[i], ci = b2[i], y, z;")+    w("\t\t/* lane 0 in full: the products' high halves, which are lane 1's, too */")+    w("\t\tunsigned __int128 s1 = (unsigned __int128) a1[0] * bi + x1;")+    w("\t\tunsigned __int128 s2 = (unsigned __int128) a2[0] * ci + x2;")+    w("\t\t__m256i bv = _mm256_set1_epi64x((long long) bi), cv = _mm256_set1_epi64x((long long) ci);")+    w("\t\t__m256i yv, zv;")+    w("")+    w("\t\ty = ((uint64_t) s1 * k1) & IFMA_MASK52;")+    w("\t\tz = ((uint64_t) s2 * k2) & IFMA_MASK52;")+    w("\t\ts1 += (unsigned __int128) m1[0] * y;")+    w("\t\ts2 += (unsigned __int128) m2[0] * z;")+    w("\t\tyv = _mm256_set1_epi64x((long long) y);")+    w("\t\tzv = _mm256_set1_epi64x((long long) z);")+    for j in range(R):+        w(f"\t\tX{j} = _mm256_madd52lo_epu64(X{j}, {A[j]}, bv); Y{j} = _mm256_madd52lo_epu64(Y{j}, {B[j]}, cv);")+    for j in range(R):+        w(f"\t\tX{j} = _mm256_madd52lo_epu64(X{j}, {M[j]}, yv); Y{j} = _mm256_madd52lo_epu64(Y{j}, {N[j]}, zv);")+    w("\t\t/* down a lane: lane 0 is a multiple of 2^52, and s carries it */")+    for j in range(R):+        nx = f"X{j + 1}" if j + 1 < R else "zero"+        ny = f"Y{j + 1}" if j + 1 < R else "zero"+        w(f"\t\tX{j} = _mm256_alignr_epi64({nx}, X{j}, 1); Y{j} = _mm256_alignr_epi64({ny}, Y{j}, 1);")+    w("\t\tx1 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(X0)) + (uint64_t)(s1 >> 52);")+    w("\t\tx2 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(Y0)) + (uint64_t)(s2 >> 52);")+    for j in range(R):+        w(f"\t\tX{j} = _mm256_madd52hi_epu64(X{j}, {A[j]}, bv); Y{j} = _mm256_madd52hi_epu64(Y{j}, {B[j]}, cv);")+    for j in range(R):+        w(f"\t\tX{j} = _mm256_madd52hi_epu64(X{j}, {M[j]}, yv); Y{j} = _mm256_madd52hi_epu64(Y{j}, {N[j]}, zv);")+    w("\t}")+    w("#undef L")+    w("\t/* lane 0, with the carry it was never given, is what the scalar holds */")+    w("\tX0 = _mm256_mask_set1_epi64(X0, 1, (long long) x1);")+    w("\tY0 = _mm256_mask_set1_epi64(Y0, 1, (long long) x2);")+    w(f"\tifma_norm{R}(r1, " + ", ".join(X) + ");")+    w(f"\tifma_norm{R}(r2, " + ", ".join(Y) + ");")+    w("}")+    w("")+    return "\n".join(o)+++def gen_sel(R):+    o = []+    w = o.append+    w(f"/* both tables' entries k1 and k2 of 32, every entry read and a mask keeping")+    w(f" * one; entries are {4 * R} words apart */")+    w(f"IFMA_TARGET static void ifma_sel{R}(uint64_t *o1, const uint64_t *t1, uint64_t k1,")+    w(f"                                uint64_t *o2, const uint64_t *t2, uint64_t k2)")+    w("{")+    w("\tconst __m256i one = _mm256_set1_epi64x(1);")+    w("\t__m256i v1 = _mm256_set1_epi64x((long long) k1), v2 = _mm256_set1_epi64x((long long) k2);")+    w("\t__m256i kv = _mm256_setzero_si256();")+    w("\t__m256i " + ", ".join(f"P{j} = kv, Q{j} = kv" for j in range(R)) + ";")+    w("\tint k;")+    w("\tfor (k = 0; k < 32; k++) {")+    w("\t\t__mmask8 s1 = _mm256_cmpeq_epi64_mask(v1, kv), s2 = _mm256_cmpeq_epi64_mask(v2, kv);")+    w(f"\t\tconst uint64_t *e1 = t1 + {4 * R} * k, *e2 = t2 + {4 * R} * k;")+    for j in range(R):+        w(f"\t\tP{j} = _mm256_mask_loadu_epi64(P{j}, s1, e1 + {4 * j}); Q{j} = _mm256_mask_loadu_epi64(Q{j}, s2, e2 + {4 * j});")+    w("\t\tkv = _mm256_add_epi64(kv, one);")+    w("\t}")+    for j in range(R):+        w(f"\t_mm256_storeu_si256((__m256i *)(o1 + {4 * j}), P{j}); _mm256_storeu_si256((__m256i *)(o2 + {4 * j}), Q{j});")+    w("}")+    w("")+    return "\n".join(o)+++def main():+    print(HEADER)+    rs = sorted({regs(d) for d in SIZES})+    for R in rs:+        print(gen_norm(R))+        print(gen_sel(R))+    for d in SIZES:+        print(gen_amm(d))+    print(FOOTER, end="")+++if __name__ == "__main__":+    main()
− cbits/crypton_rdrand.c 
@@ -1,126 +0,0 @@-/*- * Copyright (C) Thomas DuBuisson- * Copyright (C) 2013 Vincent Hanquez <tab@snarc.org>- *- * All rights reserved.- * - * Redistribution and use in source and binary forms, with or without- * modification, are permitted provided that the following conditions- * are met:- * 1. Redistributions of source code must retain the above copyright- *    notice, this list of conditions and the following disclaimer.- * 2. Redistributions in binary form must reproduce the above copyright- *    notice, this list of conditions and the following disclaimer in the- *    documentation and/or other materials provided with the distribution.- * 3. Neither the name of the author nor the names of his contributors- *    may be used to endorse or promote products derived from this software- *    without specific prior written permission.- * - * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE- * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF- * SUCH DAMAGE.- */--#include <stdint.h>-#include <stdlib.h>-#include <stdio.h>-#include <string.h>--int crypton_cpu_has_rdrand()-{-	uint32_t ax,bx,cx,dx,func=1;-#if defined(__PIC__) && defined(__i386__)-	__asm__ volatile ("mov %%ebx, %%edi;" "cpuid;" "xchgl %%ebx, %%edi;"-		: "=a" (ax), "=D" (bx), "=c" (cx), "=d" (dx) : "a" (func));-#else-	__asm__ volatile ("cpuid": "=a" (ax), "=b" (bx), "=c" (cx), "=d" (dx) : "a" (func));-#endif-	return (cx & 0x40000000);-}--/* inline encoding of 'rdrand %rax' to cover old binutils- * - no inputs- * - 'cc' to the clobber list as we modify condition code.- * - output of rdrand in rax and have a 8 bit error condition- */-#define inline_rdrand_rax(val, err) \-	asm(".byte 0x48,0x0f,0xc7,0xf0; setc %1" \-	   : "=a" (val), "=q" (err) \-	   : \-	   : "cc")--/* inline encoding of 'rdrand %eax' to cover old binutils- * - no inputs- * - 'cc' to the clobber list as we modify condition code.- * - output of rdrand in eax and have a 8 bit error condition- */-#define inline_rdrand_eax(val, err) \-	asm(".byte 0x0f,0xc7,0xf0; setc %1" \-	   : "=a" (val), "=q" (err) \-	   : \-	   : "cc")--#ifdef __x86_64__-# define RDRAND_SZ 8-# define RDRAND_T  uint64_t-#define inline_rdrand(val, err) err = crypton_rdrand_step(&val)-#else-# define RDRAND_SZ 4-# define RDRAND_T  uint32_t-#define inline_rdrand(val, err) err = crypton_rdrand_step(&val)-#endif--/* sadly many people are still using an old binutils,- * leading to report that instruction is not recognized.- */-#if 1-/* Returns 1 on success */-static inline int crypton_rdrand_step(RDRAND_T *buffer)-{-	unsigned char err;-	asm volatile ("rdrand %0; setc %1" : "=r" (*buffer), "=qm" (err));-	return (int) err;-}-#endif--/* Returns the number of bytes successfully generated */-int crypton_get_rand_bytes(uint8_t *buffer, size_t len)-{-	RDRAND_T tmp;-	int aligned = (intptr_t) buffer % RDRAND_SZ;-	int orig_len = len;-	int to_alignment = RDRAND_SZ - aligned;-	uint8_t ok;--	if (aligned != 0) {-		inline_rdrand(tmp, ok);-		if (!ok)-			return 0;-		memcpy(buffer, (uint8_t *) &tmp, to_alignment);-		buffer += to_alignment;-		len -= to_alignment;-	}--	for (; len >= RDRAND_SZ; buffer += RDRAND_SZ, len -= RDRAND_SZ) {-		inline_rdrand(tmp, ok);-		if (!ok)-			return (orig_len - len);-		*((RDRAND_T *) buffer) = tmp;-	}--	if (len > 0) {-		inline_rdrand(tmp, ok);-		if (!ok)-			return (orig_len - len);-		memcpy(buffer, (uint8_t *) &tmp, len);-	}-	return orig_len;-}
cbits/crypton_ripemd.c view
@@ -57,16 +57,17 @@ #define R(a, b, c, d, e, f, k, i, s)	\ 	a += f(b, c, d) + w[i] + k; a = rol32(a, s) + e; c = rol32(c, 10) -static void ripemd160_do_chunk(struct ripemd160_ctx *ctx, uint32_t *buf)+/* The words are read out of the block rather than the block being pointed at+ * as though it were an array of them; see crypton_md5.c. */+static void ripemd160_do_chunk(struct ripemd160_ctx *ctx, const uint8_t *buf) { 	uint32_t a1, b1, c1, d1, e1, a2, b2, c2, d2, e2;-#ifdef ARCH_IS_BIG_ENDIAN 	uint32_t w[16];-	cpu_to_le32_array(w, buf, 16);-#else-	uint32_t *w = buf;-#endif+	int wi; +	for (wi = 0; wi < 16; wi++)+		w[wi] = load_le32(buf + 4 * wi);+ 	a1 = ctx->h[0]; b1 = ctx->h[1]; c1 = ctx->h[2]; d1 = ctx->h[3]; e1 = ctx->h[4]; 	a2 = ctx->h[0]; b2 = ctx->h[1]; c2 = ctx->h[2]; d2 = ctx->h[3]; e2 = ctx->h[4]; @@ -260,24 +261,15 @@ 	ctx->sz += len; 	if (index && len >= to_fill) { 		memcpy(ctx->buf + index, data, to_fill);-		ripemd160_do_chunk(ctx, (uint32_t *) ctx->buf);+		ripemd160_do_chunk(ctx, ctx->buf); 		len -= to_fill; 		data += to_fill; 		index = 0; 	} -	if (need_alignment(data, 4)) {-		uint32_t tramp[16];-		ASSERT_ALIGNMENT(tramp, 4);-		for (; len >= 64; len -= 64, data += 64) {-			memcpy(tramp, data, 64);-			ripemd160_do_chunk(ctx, tramp);-		}-	} else {-		/* process as much 64-block as possible */-		for (; len >= 64; len -= 64, data += 64)-			ripemd160_do_chunk(ctx, (uint32_t *) data);-	}+	/* No trampoline: load_le32 does not ask for a boundary. */+	for (; len >= 64; len -= 64, data += 64)+		ripemd160_do_chunk(ctx, data);  	/* append data into buf */ 	if (len)
cbits/crypton_salsa.c view
@@ -59,7 +59,7 @@ 		QR (x15,x12,x13,x14); \ 	} -static void salsa_core(int rounds, block *out, const crypton_salsa_state *in)+static void salsa_core(int rounds, crypton_salsa_block *out, const crypton_salsa_state *in) { 	uint32_t x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13, x14, x15; 	int i;@@ -94,7 +94,7 @@ 	out->d[15] = cpu_to_le32(x15); } -void crypton_salsa_core_xor(int rounds, block *out, block *in)+void crypton_salsa_core_xor(int rounds, crypton_salsa_block *out, crypton_salsa_block *in) { 	uint32_t x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13, x14, x15; 	int i;@@ -165,7 +165,7 @@  void crypton_salsa_combine(uint8_t *dst, crypton_salsa_context *ctx, const uint8_t *src, uint32_t bytes) {-	block out;+	crypton_salsa_block out; 	crypton_salsa_state *st; 	int i; @@ -225,7 +225,7 @@ void crypton_salsa_generate(uint8_t *dst, crypton_salsa_context *ctx, uint32_t bytes) { 	crypton_salsa_state *st;-	block out;+	crypton_salsa_block out; 	int i;  	if (!bytes)@@ -252,7 +252,7 @@ 		/* xor new 64-bytes chunks and store the left over if any */ 		for (; bytes >= 64; bytes -= 64, dst += 64) { 			/* generate new chunk and update state */-			salsa_core(ctx->nb_rounds, (block *) dst, st);+			salsa_core(ctx->nb_rounds, (crypton_salsa_block *) dst, st); 			st->d[8] += 1; 			if (st->d[8] == 0) 				st->d[9] += 1;
cbits/crypton_salsa.h view
@@ -34,9 +34,9 @@ 	uint64_t q[8]; 	uint32_t d[16]; 	uint8_t  b[64];-} block;+} crypton_salsa_block; -typedef block crypton_salsa_state;+typedef crypton_salsa_block crypton_salsa_state;  typedef struct { 	crypton_salsa_state st;@@ -47,7 +47,7 @@ } crypton_salsa_context;  /* for scrypt */-void crypton_salsa_core_xor(int rounds, block *out, block *in);+void crypton_salsa_core_xor(int rounds, crypton_salsa_block *out, crypton_salsa_block *in);  void crypton_salsa_init_core(crypton_salsa_state *st, uint32_t keylen, const uint8_t *key, uint32_t ivlen, const uint8_t *iv); void crypton_salsa_init(crypton_salsa_context *ctx, uint8_t nb_rounds, uint32_t keylen, const uint8_t *key, uint32_t ivlen, const uint8_t *iv);
cbits/crypton_scrypt.c view
@@ -37,10 +37,10 @@ 	array_copy32(X, &in[(2 * r - 1) * 16], 16);  	for (i = 0; i < 2 * r; i += 2) {-		crypton_salsa_core_xor(8, (block *) X, (block *) &in[i*16]);+		crypton_salsa_core_xor(8, (crypton_salsa_block *) X, (crypton_salsa_block *) &in[i*16]); 		array_copy32(&out[i * 8], X, 16); -		crypton_salsa_core_xor(8, (block *) X, (block *) &in[i*16+16]);+		crypton_salsa_core_xor(8, (crypton_salsa_block *) X, (crypton_salsa_block *) &in[i*16+16]); 		array_copy32(&out[i * 8 + r * 16], X, 16); 	} }
cbits/crypton_sha1.c view
@@ -32,7 +32,7 @@  * Two threads racing to answer here both write the same value.  */ #ifdef WITH_ARMV8_SHA1-extern void crypton_sha1_armv8_do_chunk(uint32_t state[5], const uint32_t buf[16]);+extern void crypton_sha1_armv8_do_chunk(uint32_t state[5], const uint8_t buf[64]); extern void crypton_sha1_armv8_do_chunks(uint32_t state[5], const uint8_t *data,                                          uint32_t blocks); extern int crypton_sha1_armv8_available(void);@@ -55,8 +55,20 @@                                               const void *data, size_t blocks); #endif +/* Resolved before there is a second thread; see the constructor in+ * cbits/crypton_aes.c for why.  The test below then only ever reads. */ static int sha1_use_armv8 = -1;++__attribute__((constructor))+static void sha1_armv8_ctor(void)+{+	sha1_use_armv8 = crypton_sha1_armv8_available();+#ifdef SHA1_ASM+	if (sha1_use_armv8)+		crypton_armcap_P |= CRYPTON_ARMCAP_SHA1; #endif+}+#endif  #ifdef WITH_X86_SHA_NI #include "crypton_cpu.h"@@ -89,11 +101,13 @@ #define M(i)  (w[i & 0x0f] = rol32(w[i & 0x0f] ^ w[(i - 14) & 0x0f] \               ^ w[(i - 8) & 0x0f] ^ w[(i - 3) & 0x0f], 1)) -static void sha1_do_chunk_generic(struct sha1_ctx *ctx, uint32_t *buf)+/* The words are read out of the block rather than the block being pointed at+ * as though it were an array of them; see crypton_md5.c. */+static void sha1_do_chunk_generic(struct sha1_ctx *ctx, const uint8_t *buf) { 	uint32_t a, b, c, d, e; 	uint32_t w[16];-#define CPY(i)	w[i] = be32_to_cpu(buf[i])+#define CPY(i)	w[i] = load_be32(buf + 4 * (i)) 	CPY(0); CPY(1); CPY(2); CPY(3); CPY(4); CPY(5); CPY(6); CPY(7); 	CPY(8); CPY(9); CPY(10); CPY(11); CPY(12); CPY(13); CPY(14); CPY(15); #undef CPY@@ -197,14 +211,14 @@  * They arrived long after the x86-64 baseline, so ask before using them.  * Two threads racing to answer here both write the same value.  */-extern void crypton_sha1_x86_do_chunk(uint32_t state[5], const uint32_t buf[16]);+extern void crypton_sha1_x86_do_chunk(uint32_t state[5], const uint8_t buf[64]); extern void crypton_sha1_x86_do_chunks(uint32_t state[5], const uint8_t *data,                                        uint32_t blocks);  static int sha1_use_x86 = -1; #endif -static inline void sha1_do_chunk(struct sha1_ctx *ctx, uint32_t *buf)+static inline void sha1_do_chunk(struct sha1_ctx *ctx, const uint8_t *buf) { #ifdef WITH_ARMV8_SHA1 	if (sha1_use_armv8 < 0) {@@ -247,7 +261,7 @@ 	/* process partial buffer if there's enough data to make a block */ 	if (index && len >= to_fill) { 		memcpy(ctx->buf + index, data, to_fill);-		sha1_do_chunk(ctx, (uint32_t *) ctx->buf);+		sha1_do_chunk(ctx, ctx->buf); 		len -= to_fill; 		data += to_fill; 		index = 0;@@ -292,18 +306,9 @@ 	} #endif -	if (need_alignment(data, 4)) {-		uint32_t tramp[16];-		ASSERT_ALIGNMENT(tramp, 4);-		for (; len >= 64; len -= 64, data += 64) {-			memcpy(tramp, data, 64);-			sha1_do_chunk(ctx, tramp);-		}-	} else {-		/* process as much 64-block as possible */-		for (; len >= 64; len -= 64, data += 64)-			sha1_do_chunk(ctx, (uint32_t *) data);-	}+	/* No trampoline: load_be32 does not ask for a boundary. */+	for (; len >= 64; len -= 64, data += 64)+		sha1_do_chunk(ctx, data);  	/* append data into buf */ 	if (len)
cbits/crypton_sha256.c view
@@ -78,13 +78,16 @@ #define s0(x)       (ror32(x, 7) ^ ror32(x,18) ^ (x >> 3)) #define s1(x)       (ror32(x,17) ^ ror32(x,19) ^ (x >> 10)) -static void sha256_do_chunk_generic(struct sha256_ctx *ctx, uint32_t buf[])+/* The sixteen words are read out of the block rather than the block being+ * pointed at as though it were an array of them; see crypton_md5.c. */+static void sha256_do_chunk_generic(struct sha256_ctx *ctx, const uint8_t *buf) { 	uint32_t a, b, c, d, e, f, g, h, t1, t2; 	int i; 	uint32_t w[64]; -	cpu_to_be32_array(w, buf, 16);+	for (i = 0; i < 16; i++)+		w[i] = load_be32(buf + 4 * i); 	for (i = 16; i < 64; i++) 		w[i] = s1(w[i - 2]) + w[i - 7] + s0(w[i - 15]) + w[i - 16]; @@ -120,10 +123,18 @@  * sha256_armv8.c.  They are optional in ARMv8.0, so ask before using them.  * Two threads racing to answer here both write the same value.  */-extern void crypton_sha256_armv8_do_chunk(uint32_t state[8], const uint32_t buf[16]);+extern void crypton_sha256_armv8_do_chunk(uint32_t state[8], const uint8_t buf[64]); extern int crypton_sha256_armv8_available(void); +/* Resolved before there is a second thread; see the constructor in+ * cbits/crypton_aes.c for why.  The test below then only ever reads. */ static int sha256_use_armv8 = -1;++__attribute__((constructor))+static void sha256_armv8_ctor(void)+{+	sha256_use_armv8 = crypton_sha256_armv8_available();+} #endif  #if (defined(WITH_ARMV8_SHA256_ASM) && defined(WITH_ARMV8_SHA2)) \@@ -141,22 +152,34 @@ extern void crypton_sha256_asm_block_data_order(uint32_t state[8],                                                 const void *data, size_t blocks); +#ifdef WITH_ARMV8_SHA256_ASM+/* The assembly picks its path from crypton_armcap_P, so the answer to the+ * runtime check has to reach that word rather than the flag above.  It is+ * set here, before there is a second thread, for the reason the constructor+ * in cbits/crypton_aes.c gives.+ *+ * This ran from sha256_asm_ready below until 2.1.3, guarded by the flag+ * still being unresolved -- which stopped happening when the constructor+ * above was added, so the bit was never set and the assembly took its+ * generic path.  SHA-256 was 5.7 times slower on an Apple M4 for it. */+__attribute__((constructor))+static void sha256_armcap_ctor(void)+{+	if (crypton_sha256_armv8_available())+		crypton_armcap_P |= CRYPTON_ARMCAP_SHA256;+}+#endif+ static void sha256_asm_ready(void) {-#ifdef WITH_ARMV8_SHA256_ASM-	if (sha256_use_armv8 < 0) {-		if (crypton_sha256_armv8_available())-			crypton_armcap_P |= CRYPTON_ARMCAP_SHA256;-		sha256_use_armv8 = 1;-	}-#else+#ifndef WITH_ARMV8_SHA256_ASM 	crypton_x86_ia32cap_resolve(); #endif } #endif  -static void sha256_do_chunk(struct sha256_ctx *ctx, uint32_t buf[])+static void sha256_do_chunk(struct sha256_ctx *ctx, const uint8_t *buf) { #ifdef SHA256_ASM 	sha256_asm_ready();@@ -192,7 +215,7 @@ 	/* process partial buffer if there's enough data to make a block */ 	if (index && len >= to_fill) { 		memcpy(ctx->buf + index, data, to_fill);-		sha256_do_chunk(ctx, (uint32_t *) ctx->buf);+		sha256_do_chunk(ctx, ctx->buf); 		len -= to_fill; 		data += to_fill; 		index = 0;@@ -211,18 +234,9 @@ 		len -= (uint32_t) blocks * 64; 	} #else-	if (need_alignment(data, 4)) {-		uint32_t tramp[16];-		ASSERT_ALIGNMENT(tramp, 4);-		for (; len >= 64; len -= 64, data += 64) {-			memcpy(tramp, data, 64);-			sha256_do_chunk(ctx, tramp);-		}-	} else {-		/* process as much 64-block as possible */-		for (; len >= 64; len -= 64, data += 64)-			sha256_do_chunk(ctx, (uint32_t *) data);-	}+	/* No trampoline: load_be32 does not ask for a boundary. */+	for (; len >= 64; len -= 64, data += 64)+		sha256_do_chunk(ctx, data); #endif  	/* append data into buf */
cbits/crypton_sha256.h view
@@ -51,6 +51,18 @@  void crypton_sha256_init(struct sha256_ctx *ctx); void crypton_sha256_update(struct sha256_ctx *ctx, const uint8_t *data, uint32_t len);+/* The pointers are all required to be non-null, which is said here so that+ * the compiler knows it too.  Both of these write their digest through a+ * loop -- store_be32(out + 4 * i, ...) -- where sha1 and md5 write theirs at+ * constant offsets, and that is the difference that makes gcc's+ * -Wstringop-overflow reason about out being null: with+ * -fsanitize=undefined, UndefinedBehaviorSanitizer inserts a null check+ * before memcpy, because glibc declares memcpy nonnull, and the check puts a+ * null path in front of the warning pass, which then reports writing into+ * "a region of size 0" at "address zero".  Saying the pointer is never null+ * removes the path rather than the warning.  It costs nothing: compiled as+ * the package compiles it, the assembly is identical with and without. */+__attribute__((nonnull)) void crypton_sha256_finalize(struct sha256_ctx *ctx, uint8_t *out); void crypton_sha256_finalize_prefix(struct sha256_ctx *ctx, const uint8_t *data, uint32_t len, uint32_t n, uint8_t *out);
cbits/crypton_sha3.c view
@@ -102,14 +102,16 @@ #define sha3_asm_absorb crypton_keccak_asm_absorb #endif -static inline void sha3_do_chunk(uint64_t state[25], uint64_t buf[], int bufsz)+/* The words are read out of the block rather than the block being pointed at+ * as though it were an array of them; see crypton_md5.c. */+static inline void sha3_do_chunk(uint64_t state[25], const uint8_t *buf, int bufsz) { 	int i, j, r; 	uint64_t tmp, bc[5];  	/* merge buf with state */ 	for (i = 0; i < bufsz; i++)-		state[i] ^= le64_to_cpu(buf[i]);+		state[i] ^= load_le64(buf + 8 * i);  #ifdef WITH_ARMV8_SHA3 	if (sha3_armv8_ok()) {@@ -180,14 +182,14 @@ 	to_fill = ctx->bufsz - ctx->bufindex;  	if (ctx->bufindex == ctx->bufsz) {-		sha3_do_chunk(ctx->state, (uint64_t *) ctx->buf, ctx->bufsz / 8);+		sha3_do_chunk(ctx->state, ctx->buf, ctx->bufsz / 8); 		ctx->bufindex = 0; 	}  	/* process partial buffer if there's enough data to make a block */ 	if (ctx->bufindex && len >= to_fill) { 		memcpy(ctx->buf + ctx->bufindex, data, to_fill);-		sha3_do_chunk(ctx->state, (uint64_t *) ctx->buf, ctx->bufsz / 8);+		sha3_do_chunk(ctx->state, ctx->buf, ctx->bufsz / 8); 		len -= to_fill; 		data += to_fill; 		ctx->bufindex = 0;@@ -207,18 +209,9 @@ 	} #endif -	if (need_alignment(data, 8)) {-		uint64_t tramp[SHA3_BUF_SIZE_MAX/8];-		ASSERT_ALIGNMENT(tramp, 8);-		for (; len >= ctx->bufsz; len -= ctx->bufsz, data += ctx->bufsz) {-			memcpy(tramp, data, ctx->bufsz);-			sha3_do_chunk(ctx->state, tramp, ctx->bufsz / 8);-		}-	} else {-		/* process as much ctx->bufsz-block */-		for (; len >= ctx->bufsz; len -= ctx->bufsz, data += ctx->bufsz)-			sha3_do_chunk(ctx->state, (uint64_t *) data, ctx->bufsz / 8);-	}+	/* No trampoline: load_le64 does not ask for a boundary. */+	for (; len >= ctx->bufsz; len -= ctx->bufsz, data += ctx->bufsz)+		sha3_do_chunk(ctx->state, data, ctx->bufsz / 8);   	/* append data into buf */@@ -232,7 +225,7 @@ { 	/* process full buffer if needed */ 	if (ctx->bufindex == ctx->bufsz) {-		sha3_do_chunk(ctx->state, (uint64_t *) ctx->buf, ctx->bufsz / 8);+		sha3_do_chunk(ctx->state, ctx->buf, ctx->bufsz / 8); 		ctx->bufindex = 0; 	} @@ -242,7 +235,7 @@ 	ctx->buf[ctx->bufsz - 1] |= 0x80;  	/* process */-	sha3_do_chunk(ctx->state, (uint64_t *) ctx->buf, ctx->bufsz / 8);+	sha3_do_chunk(ctx->state, ctx->buf, ctx->bufsz / 8); 	ctx->bufindex = 0; }
cbits/crypton_sha512.c view
@@ -91,13 +91,16 @@ #define s0(x)       (ror64(x, 1) ^ ror64(x, 8) ^ (x >> 7)) #define s1(x)       (ror64(x, 19) ^ ror64(x, 61) ^ (x >> 6)) -static void sha512_do_chunk_generic(struct sha512_ctx *ctx, uint64_t *buf)+/* The words are read out of the block rather than the block being pointed at+ * as though it were an array of them; see crypton_md5.c. */+static void sha512_do_chunk_generic(struct sha512_ctx *ctx, const uint8_t *buf) { 	uint64_t a, b, c, d, e, f, g, h, t1, t2; 	int i; 	uint64_t w[80]; -	cpu_to_be64_array(w, buf, 16);+	for (i = 0; i < 16; i++)+		w[i] = load_be64(buf + 8 * i);  	for (i = 16; i < 80; i++) 		w[i] = s1(w[i - 2]) + w[i - 7] + s0(w[i - 15]) + w[i - 16];@@ -135,10 +138,18 @@  * widespread than the SHA-256 ones, so ask before using them.  Two threads  * racing to answer here both write the same value.  */-extern void crypton_sha512_armv8_do_chunk(uint64_t state[8], const uint64_t buf[16]);+extern void crypton_sha512_armv8_do_chunk(uint64_t state[8], const uint8_t buf[128]); extern int crypton_sha512_armv8_available(void); +/* Resolved before there is a second thread; see the constructor in+ * cbits/crypton_aes.c for why.  The test below then only ever reads. */ static int sha512_use_armv8 = -1;++__attribute__((constructor))+static void sha512_armv8_ctor(void)+{+	sha512_use_armv8 = crypton_sha512_armv8_available();+} #endif  @@ -154,7 +165,7 @@                                                 const void *data, size_t blocks); #endif -static void sha512_do_chunk(struct sha512_ctx *ctx, uint64_t *buf)+static void sha512_do_chunk(struct sha512_ctx *ctx, const uint8_t *buf) { #ifdef SHA512_ASM 	crypton_x86_ia32cap_resolve();@@ -192,7 +203,7 @@ 	/* process partial buffer if there's enough data to make a block */ 	if (index && len >= to_fill) { 		memcpy(ctx->buf + index, data, to_fill);-		sha512_do_chunk(ctx, (uint64_t *) ctx->buf);+		sha512_do_chunk(ctx, ctx->buf); 		len -= to_fill; 		data += to_fill; 		index = 0;@@ -210,18 +221,9 @@ 		len -= (uint32_t) blocks * 128; 	} #else-	if (need_alignment(data, 8)) {-		uint64_t tramp[16];-		ASSERT_ALIGNMENT(tramp, 8);-		for (; len >= 128; len -= 128, data += 128) {-			memcpy(tramp, data, 128);-			sha512_do_chunk(ctx, tramp);-		}-	} else {-		/* process as much 128-block as possible */-		for (; len >= 128; len -= 128, data += 128)-			sha512_do_chunk(ctx, (uint64_t *) data);-	}+	/* No trampoline: load_be64 does not ask for a boundary. */+	for (; len >= 128; len -= 128, data += 128)+		sha512_do_chunk(ctx, data); #endif  	/* append data into buf */@@ -344,7 +346,7 @@ 		/* re-init the context, otherwise len is changed */ 		memset(ctx, 0, sizeof(*ctx)); 		for (i = 0; i < 8; i++)-			ctx->h[i] = cpu_to_be64(((uint64_t *) out)[i]);+			ctx->h[i] = load_be64(out + 8 * i); 		} 	} }
cbits/crypton_sha512.h view
@@ -50,6 +50,18 @@  void crypton_sha512_init(struct sha512_ctx *ctx); void crypton_sha512_update(struct sha512_ctx *ctx, const uint8_t *data, uint32_t len);+/* The pointers are all required to be non-null, which is said here so that+ * the compiler knows it too.  Both of these write their digest through a+ * loop -- store_be32(out + 4 * i, ...) -- where sha1 and md5 write theirs at+ * constant offsets, and that is the difference that makes gcc's+ * -Wstringop-overflow reason about out being null: with+ * -fsanitize=undefined, UndefinedBehaviorSanitizer inserts a null check+ * before memcpy, because glibc declares memcpy nonnull, and the check puts a+ * null path in front of the warning pass, which then reports writing into+ * "a region of size 0" at "address zero".  Saying the pointer is never null+ * removes the path rather than the warning.  It costs nothing: compiled as+ * the package compiles it, the assembly is identical with and without. */+__attribute__((nonnull)) void crypton_sha512_finalize(struct sha512_ctx *ctx, uint8_t *out); void crypton_sha512_finalize_prefix(struct sha512_ctx *ctx, const uint8_t *data, uint32_t len, uint32_t n, uint8_t *out);
cbits/crypton_skein256.c view
@@ -37,7 +37,9 @@ static const uint8_t K256_6[2] = { 58, 22, }; static const uint8_t K256_7[2] = { 32, 32, }; -static inline void skein256_do_chunk(struct skein256_ctx *ctx, uint64_t *buf, uint32_t len)+/* The four words are read out of the block rather than the block being+ * pointed at as though it were an array of them; see crypton_md5.c. */+static inline void skein256_do_chunk(struct skein256_ctx *ctx, const uint8_t *bufp, uint32_t len) { 	uint64_t x[4]; 	uint64_t ts[3];@@ -78,11 +80,18 @@ 	ROUND(0,3,2,1,K256_7); \ 	INJECTKEY((i*2) + 2) -	x[0] = le64_to_cpu(buf[0]) + ks[0];-	x[1] = le64_to_cpu(buf[1]) + ks[1] + ts[0];-	x[2] = le64_to_cpu(buf[2]) + ks[2] + ts[1];-	x[3] = le64_to_cpu(buf[3]) + ks[3];+	uint64_t buf[4]; +	buf[0] = load_le64(bufp);+	buf[1] = load_le64(bufp + 8);+	buf[2] = load_le64(bufp + 16);+	buf[3] = load_le64(bufp + 24);++	x[0] = buf[0] + ks[0];+	x[1] = buf[1] + ks[1] + ts[0];+	x[2] = buf[2] + ks[2] + ts[1];+	x[3] = buf[3] + ks[3];+ 	/* 9 pass of 8 rounds = 72 rounds */ 	PASS(0); 	PASS(1);@@ -98,10 +107,10 @@ 	ctx->t0 = ts[0]; 	ctx->t1 = ts[1]; -	ctx->h[0] = x[0] ^ cpu_to_le64(buf[0]);-        ctx->h[1] = x[1] ^ cpu_to_le64(buf[1]);-        ctx->h[2] = x[2] ^ cpu_to_le64(buf[2]);-        ctx->h[3] = x[3] ^ cpu_to_le64(buf[3]);+	ctx->h[0] = x[0] ^ buf[0];+	ctx->h[1] = x[1] ^ buf[1];+	ctx->h[2] = x[2] ^ buf[2];+	ctx->h[3] = x[3] ^ buf[3]; }  void crypton_skein256_init(struct skein256_ctx *ctx, uint32_t hashlen)@@ -115,7 +124,7 @@ 	buf[0] = cpu_to_le64((SKEIN_VERSION << 32) | SKEIN_IDSTRING); 	buf[1] = cpu_to_le64(hashlen); 	buf[2] = 0; /* tree info, not implemented */-	skein256_do_chunk(ctx, buf, 4*8);+	skein256_do_chunk(ctx, (const uint8_t *) buf, 4*8);  	SET_TYPE(ctx, FLAG_FIRST | FLAG_TYPE(TYPE_MSG)); }@@ -130,7 +139,7 @@ 	to_fill = 32 - ctx->bufindex;  	if (ctx->bufindex == 32) {-		skein256_do_chunk(ctx, (uint64_t *) ctx->buf, 32);+		skein256_do_chunk(ctx, ctx->buf, 32); 		ctx->bufindex = 0; 	} @@ -138,24 +147,17 @@ 	 * and there's without doubt further blocks */ 	if (ctx->bufindex && len > to_fill) { 		memcpy(ctx->buf + ctx->bufindex, data, to_fill);-		skein256_do_chunk(ctx, (uint64_t *) ctx->buf, 32);+		skein256_do_chunk(ctx, ctx->buf, 32); 		len -= to_fill; 		data += to_fill; 		ctx->bufindex = 0; 	} -	if (need_alignment(data, 8)) {-		uint64_t tramp[4];-		ASSERT_ALIGNMENT(tramp, 8);-		for (; len > 32; len -= 32, data += 32) {-			memcpy(tramp, data, 32);-			skein256_do_chunk(ctx, tramp, 32);-		}-	} else {-		/* process as much 32-block as possible except the last one in case we finalize */-		for (; len > 32; len -= 32, data += 32)-			skein256_do_chunk(ctx, (uint64_t *) data, 32);-	}+	/* No trampoline for a block that is not on an eight-byte boundary: the+	 * words are read with load_le64 now, which does not ask.  The last+	 * block is left for the finalisation. */+	for (; len > 32; len -= 32, data += 32)+		skein256_do_chunk(ctx, data, 32);  	/* append data into buf */ 	if (len) {@@ -174,7 +176,7 @@ 	/* if buf is not complete pad with 0 bytes */ 	if (ctx->bufindex < 32) 		memset(ctx->buf + ctx->bufindex, '\0', 32 - ctx->bufindex);-	skein256_do_chunk(ctx, (uint64_t *) ctx->buf, ctx->bufindex);+	skein256_do_chunk(ctx, ctx->buf, ctx->bufindex);  	memset(ctx->buf, '\0', 32); @@ -187,9 +189,9 @@ 	/* threefish in counter mode, 0 for 1st 64 bytes, 1 for 2nd 64 bytes, .. */ 	for (i = 0; i*32 < outsize; i++) { 		uint64_t w[4];-		*((uint64_t *) ctx->buf) = cpu_to_le64(i);+		store_le64(ctx->buf, i); 		SET_TYPE(ctx, FLAG_FIRST | FLAG_FINAL | FLAG_TYPE(TYPE_OUT));-		skein256_do_chunk(ctx, (uint64_t *) ctx->buf, sizeof(uint64_t));+		skein256_do_chunk(ctx, ctx->buf, sizeof(uint64_t));  		n = outsize - i * 32; 		if (n >= 32) n = 32;
cbits/crypton_skein256.h view
@@ -37,8 +37,8 @@  #define SKEIN256_CTX_SIZE		sizeof(struct skein256_ctx) -void cryponite_skein256_init(struct skein256_ctx *ctx, uint32_t hashlen);-void cryponite_skein256_update(struct skein256_ctx *ctx, const uint8_t *data, uint32_t len);-void cryponite_skein256_finalize(struct skein256_ctx *ctx, uint32_t hashlen, uint8_t *out);+void crypton_skein256_init(struct skein256_ctx *ctx, uint32_t hashlen);+void crypton_skein256_update(struct skein256_ctx *ctx, const uint8_t *data, uint32_t len);+void crypton_skein256_finalize(struct skein256_ctx *ctx, uint32_t hashlen, uint8_t *out);  #endif
cbits/crypton_skein512.c view
@@ -37,7 +37,9 @@ static const uint8_t K512_6[4] = { 25, 29, 39, 43, }; static const uint8_t K512_7[4] = {  8, 35, 56, 22, }; -static inline void skein512_do_chunk(struct skein512_ctx *ctx, uint64_t *buf, uint32_t len)+/* The words are read out of the block rather than the block being pointed at+ * as though it were an array of them; see crypton_md5.c. */+static inline void skein512_do_chunk(struct skein512_ctx *ctx, const uint8_t *bufp, uint32_t len) { 	uint64_t x[8]; 	uint64_t ts[3];@@ -88,15 +90,21 @@ 	ROUND(6,1,0,7,2,5,4,3,K512_7); \ 	INJECTKEY((i*2) + 2) -	x[0] = le64_to_cpu(buf[0]) + ks[0];-	x[1] = le64_to_cpu(buf[1]) + ks[1];-	x[2] = le64_to_cpu(buf[2]) + ks[2];-	x[3] = le64_to_cpu(buf[3]) + ks[3];-	x[4] = le64_to_cpu(buf[4]) + ks[4];-	x[5] = le64_to_cpu(buf[5]) + ks[5] + ts[0];-	x[6] = le64_to_cpu(buf[6]) + ks[6] + ts[1];-	x[7] = le64_to_cpu(buf[7]) + ks[7];+	uint64_t buf[8];+	int bi; +	for (bi = 0; bi < 8; bi++)+		buf[bi] = load_le64(bufp + 8 * bi);++	x[0] = buf[0] + ks[0];+	x[1] = buf[1] + ks[1];+	x[2] = buf[2] + ks[2];+	x[3] = buf[3] + ks[3];+	x[4] = buf[4] + ks[4];+	x[5] = buf[5] + ks[5] + ts[0];+	x[6] = buf[6] + ks[6] + ts[1];+	x[7] = buf[7] + ks[7];+ 	/* 9 pass of 8 rounds = 72 rounds */ 	PASS(0); 	PASS(1);@@ -112,14 +120,14 @@ 	ctx->t0 = ts[0]; 	ctx->t1 = ts[1]; -	ctx->h[0] = x[0] ^ cpu_to_le64(buf[0]);-        ctx->h[1] = x[1] ^ cpu_to_le64(buf[1]);-        ctx->h[2] = x[2] ^ cpu_to_le64(buf[2]);-        ctx->h[3] = x[3] ^ cpu_to_le64(buf[3]);-        ctx->h[4] = x[4] ^ cpu_to_le64(buf[4]);-        ctx->h[5] = x[5] ^ cpu_to_le64(buf[5]);-        ctx->h[6] = x[6] ^ cpu_to_le64(buf[6]);-        ctx->h[7] = x[7] ^ cpu_to_le64(buf[7]);+	ctx->h[0] = x[0] ^ buf[0];+        ctx->h[1] = x[1] ^ buf[1];+        ctx->h[2] = x[2] ^ buf[2];+        ctx->h[3] = x[3] ^ buf[3];+        ctx->h[4] = x[4] ^ buf[4];+        ctx->h[5] = x[5] ^ buf[5];+        ctx->h[6] = x[6] ^ buf[6];+        ctx->h[7] = x[7] ^ buf[7]; }  void crypton_skein512_init(struct skein512_ctx *ctx, uint32_t hashlen)@@ -133,7 +141,7 @@ 	buf[0] = cpu_to_le64((SKEIN_VERSION << 32) | SKEIN_IDSTRING); 	buf[1] = cpu_to_le64(hashlen); 	buf[2] = 0; /* tree info, not implemented */-	skein512_do_chunk(ctx, buf, 4*8);+	skein512_do_chunk(ctx, (const uint8_t *) buf, 4*8);  	SET_TYPE(ctx, FLAG_FIRST | FLAG_TYPE(TYPE_MSG)); }@@ -148,7 +156,7 @@ 	to_fill = 64 - ctx->bufindex;  	if (ctx->bufindex == 64) {-		skein512_do_chunk(ctx, (uint64_t *) ctx->buf, 64);+		skein512_do_chunk(ctx, ctx->buf, 64); 		ctx->bufindex = 0; 	} @@ -156,24 +164,15 @@ 	 * and there's without doubt further blocks */ 	if (ctx->bufindex && len > to_fill) { 		memcpy(ctx->buf + ctx->bufindex, data, to_fill);-		skein512_do_chunk(ctx, (uint64_t *) ctx->buf, 64);+		skein512_do_chunk(ctx, ctx->buf, 64); 		len -= to_fill; 		data += to_fill; 		ctx->bufindex = 0; 	} -	if (need_alignment(data, 8)) {-		uint64_t tramp[8];-		ASSERT_ALIGNMENT(tramp, 8);-		for (; len > 64; len -= 64, data += 64) {-			memcpy(tramp, data, 64);-			skein512_do_chunk(ctx, tramp, 64);-		}-	} else {-		/* process as much 64-block as possible except the last one in case we finalize */-		for (; len > 64; len -= 64, data += 64)-			skein512_do_chunk(ctx, (uint64_t *) data, 64);-	}+	/* No trampoline: load_le64 does not ask for a boundary. */+	for (; len > 64; len -= 64, data += 64)+		skein512_do_chunk(ctx, data, 64);  	/* append data into buf */ 	if (len) {@@ -192,7 +191,7 @@ 	/* if buf is not complete pad with 0 bytes */ 	if (ctx->bufindex < 64) 		memset(ctx->buf + ctx->bufindex, '\0', 64 - ctx->bufindex);-	skein512_do_chunk(ctx, (uint64_t *) ctx->buf, ctx->bufindex);+	skein512_do_chunk(ctx, ctx->buf, ctx->bufindex);  	memset(ctx->buf, '\0', 64); @@ -205,9 +204,9 @@ 	/* threefish in counter mode, 0 for 1st 64 bytes, 1 for 2nd 64 bytes, .. */ 	for (i = 0; i*64 < outsize; i++) { 		uint64_t w[8];-		*((uint64_t *) ctx->buf) = cpu_to_le64(i);+		store_le64(ctx->buf, i); 		SET_TYPE(ctx, FLAG_FIRST | FLAG_FINAL | FLAG_TYPE(TYPE_OUT));-		skein512_do_chunk(ctx, (uint64_t *) ctx->buf, sizeof(uint64_t));+		skein512_do_chunk(ctx, ctx->buf, sizeof(uint64_t));  		n = outsize - i * 64; 		if (n >= 64) n = 64;
cbits/crypton_skein512.h view
@@ -37,8 +37,8 @@  #define SKEIN512_CTX_SIZE		sizeof(struct skein512_ctx) -void cryponite_skein512_init(struct skein512_ctx *ctx, uint32_t hashlen);-void cryponite_skein512_update(struct skein512_ctx *ctx, const uint8_t *data, uint32_t len);-void cryponite_skein512_finalize(struct skein512_ctx *ctx, uint32_t hashlen, uint8_t *out);+void crypton_skein512_init(struct skein512_ctx *ctx, uint32_t hashlen);+void crypton_skein512_update(struct skein512_ctx *ctx, const uint8_t *data, uint32_t len);+void crypton_skein512_finalize(struct skein512_ctx *ctx, uint32_t hashlen, uint8_t *out);  #endif
+ cbits/crypton_sysdrg.c view
@@ -0,0 +1,388 @@+/*+ * The generator behind MonadRandom IO.+ *+ * A ChaCha20 DRBG per operating system thread, seeded from a process-wide+ * DRBG, which is itself seeded from the system entropy pool.  This is the+ * shape RFC 9180's neighbours and the other libraries have settled on, and+ * it was asked for in #298.+ *+ * Per operating system thread and not per Haskell thread: a forkIO thread+ * moves between capabilities, so state kept against it would be shared by+ * threads running at the same time.  That is why the state is here and+ * reached through pthread_getspecific rather than held in Haskell.+ *+ * Three things force a reseed:+ *+ *   - a thread has produced CRYPTON_THREAD_RESEED bytes,+ *   - the process DRBG has issued CRYPTON_GLOBAL_RESEED bytes of seed,+ *   - the process has forked.+ *+ * The last one is the one that bites.  A child inherits its parent's state+ * and would otherwise produce the same stream; the generation counter below+ * is bumped in the child by a pthread_atfork handler, and every generator+ * compares against it before it answers.+ */++#include <stdint.h>+#include <string.h>+#include <stdlib.h>++#include "crypton_chacha.h"+#include "crypton_sha512.h"++#ifdef _WIN32+#include <windows.h>+#else+#include <pthread.h>+#include <unistd.h>+#endif++/* from crypton_sysrandom.c */+int crypton_sysrandom_available(void);+int crypton_sysrandom_bytes(uint8_t *buf, int len);++#define CHACHA_ROUNDS      20+#define SEED_KEY           32+#define SEED_IV             8+#define SEED_LEN           (SEED_KEY + SEED_IV)++#define CRYPTON_THREAD_RESEED  (1u << 20)+#define CRYPTON_GLOBAL_RESEED  (1u << 20)++typedef struct {+	crypton_chacha_context ctx;+	uint64_t used;+	uint32_t generation;+	int seeded;+} drg_t;++static drg_t global_drg;+static volatile uint32_t fork_generation = 0;++#ifdef _WIN32+static CRITICAL_SECTION global_lock;+/* Fls and not Tls: TlsAlloc has no destructor, so the state of every thread+ * that ever drew a byte would be left allocated and unscrubbed when the+ * thread ended.  FlsAlloc takes the callback that pthread_key_create does. */+static DWORD thread_slot;+static INIT_ONCE init_once = INIT_ONCE_STATIC_INIT;+#else+static pthread_mutex_t global_lock = PTHREAD_MUTEX_INITIALIZER;+static pthread_key_t thread_slot;+static pthread_once_t init_once = PTHREAD_ONCE_INIT;+#endif++static void scrub(void *p, size_t n)+{+	volatile uint8_t *q = (volatile uint8_t *) p;+	while (n--) *q++ = 0;+}++/*+ * Seed material for the process DRBG.+ *+ * What the system gives goes through SHA-512 rather than into the key+ * directly, so that the key is a fixed size whatever the call returns, and+ * so that a second source could be added without any of this changing shape.+ * There is one source today and the note below says why.+ */+static int seed_from_system(uint8_t out[SEED_LEN])+{+	struct sha512_ctx h;+	uint8_t buf[64];+	uint8_t digest[64];+	int got;++	crypton_sha512_init(&h);++	/*+	 * The system call only.  Where there is none -- an old kernel, a BSD+	 * this does not know -- seeding fails and the caller keeps the path+	 * it has, rather than this file growing a second copy of the device+	 * reading that Crypto.Random.Entropy.Unix already does.+	 *+	 * And RDRAND is not mixed in beside it, which it was until+	 * @vdukhovni pointed out that it cannot add anything: every system+	 * this code runs on already feeds RDRAND into the pool the call below+	 * draws from.  Linux does that whatever random.trust_cpu says -- that+	 * setting decides whether the contribution is *credited*, not whether+	 * it is mixed -- so the instruction's output is in this buffer+	 * already.+	 *+	 * The argument for keeping it was defence in depth against that pool+	 * having gone wrong.  It does not survive the above: a pool that has+	 * gone wrong has gone wrong with RDRAND already in it, so asking the+	 * instruction a second time covers only the case where the kernel's+	 * mixing is broken and the instruction is not.  That is not worth a+	 * second code path, a flag, and the standing invitation to read this+	 * as a second source when it is the same one twice.+	 */+	got = crypton_sysrandom_available() ? crypton_sysrandom_bytes(buf, 64) : 0;+	if (got <= 0) {+		/* Nothing else here is a seed on its own, so there is nothing to+		 * go on with: return before drawing anything that would only be+		 * thrown away. */+		scrub(&h, sizeof h);+		scrub(buf, sizeof buf);+		return 0;+	}+	crypton_sha512_update(&h, buf, (uint32_t) got);+++	crypton_sha512_finalize(&h, digest);+	memcpy(out, digest, SEED_LEN);++	scrub(&h, sizeof h);+	scrub(buf, sizeof buf);+	scrub(digest, sizeof digest);+	return 1;+}++static void drg_seed(drg_t *d, const uint8_t seed[SEED_LEN])+{+	crypton_chacha_init(&d->ctx, CHACHA_ROUNDS, SEED_KEY, seed,+	                    SEED_IV, seed + SEED_KEY);+	d->used = 0;+	d->generation = fork_generation;+	d->seeded = 1;+}++/*+ * Forget the key that produced the bytes just handed out.+ *+ * Without this the key stands until the next reseed, and anyone who reads a+ * generator's state can wind the counter back and reproduce everything it+ * has issued since -- up to CRYPTON_THREAD_RESEED bytes that were meant to+ * be secret.  Taking the next forty bytes of keystream as the new key and+ * nonce, and dropping the old ones, puts that out of reach one step after+ * it is issued: ChaCha20 does not run backwards, and the key that would+ * have been needed is gone.  It is what arc4random does.+ *+ * crypton_chacha_init memsets the whole context, so the counter returns to+ * zero and the tail of a part-used block goes with the old key rather than+ * being handed out under the new one.+ *+ * d->used is not touched.  It counts what callers were given, which is what+ * the reseed interval is written in terms of; these forty bytes are the+ * cost of the rekey and not an answer to anybody.+ */+static void drg_rekey(drg_t *d)+{+	uint8_t next[SEED_LEN];++	crypton_chacha_generate(next, &d->ctx, SEED_LEN);+	crypton_chacha_init(&d->ctx, CHACHA_ROUNDS, SEED_KEY, next,+	                    SEED_IV, next + SEED_KEY);+	scrub(next, sizeof next);+}++static int drg_stale(const drg_t *d, uint64_t limit)+{+	return !d->seeded || d->used >= limit || d->generation != fork_generation;+}++/* Bytes from the process DRBG, which is only ever asked for seed material. */+static int global_bytes(uint8_t *out, uint32_t len)+{+	int ok = 1;++#ifdef _WIN32+	EnterCriticalSection(&global_lock);+#else+	pthread_mutex_lock(&global_lock);+#endif+	if (drg_stale(&global_drg, CRYPTON_GLOBAL_RESEED)) {+		uint8_t seed[SEED_LEN];+		ok = seed_from_system(seed);+		if (ok)+			drg_seed(&global_drg, seed);+		scrub(seed, sizeof seed);+	}+	if (ok) {+		crypton_chacha_generate(out, &global_drg.ctx, len);+		global_drg.used += len;+		drg_rekey(&global_drg);+	}+#ifdef _WIN32+	LeaveCriticalSection(&global_lock);+#else+	pthread_mutex_unlock(&global_lock);+#endif+	return ok;+}++#ifdef _WIN32+static void WINAPI thread_free(void *p)+#else+static void thread_free(void *p)+#endif+{+	if (p) {+		scrub(p, sizeof(drg_t));+		free(p);+	}+}++#ifndef _WIN32+/* All three handlers, not just the child's.  The child's first draw has to+ * reseed -- the generation has changed -- and reseeding takes global_lock.+ * A fork made while another thread held it would hand the child a mutex+ * locked by a thread that did not come across, and the child would wait on+ * it for ever.  So the lock is taken before the fork and released on both+ * sides of it, which is the state the child needs it in. */+static void before_fork(void)+{+	pthread_mutex_lock(&global_lock);+}++static void after_fork_in_parent(void)+{+	pthread_mutex_unlock(&global_lock);+}++static void after_fork_in_child(void)+{+	pthread_mutex_unlock(&global_lock);+	fork_generation++;+}+#endif++#ifdef CRYPTON_SYSDRG_TESTING+/* For cbits/tests/sysdrg and nothing else: hold and release the process+ * generator's lock, so that a fork can be made to happen while another+ * thread holds it.  Nothing outside that test declares these, and the+ * library is never built with this defined. */+void crypton_sysdrg_test_lock(void);+void crypton_sysdrg_test_unlock(void);++static drg_t *this_thread(void);++/* The calling thread's ChaCha key, which is d[4..11] of the state.  A test+ * uses it to ask whether the key that produced a draw is still there+ * afterwards; see "a draw replaces the key that made it" in+ * cbits/tests/sysdrg. */+void crypton_sysdrg_test_key(uint8_t out[32]);++void crypton_sysdrg_test_key(uint8_t out[32])+{+	drg_t *d = this_thread();+	int i;++	if (!d)+		return;+	for (i = 0; i < 8; i++) {+		uint32_t w = d->ctx.st.d[4 + i];+		out[i * 4 + 0] = (uint8_t) (w);+		out[i * 4 + 1] = (uint8_t) (w >> 8);+		out[i * 4 + 2] = (uint8_t) (w >> 16);+		out[i * 4 + 3] = (uint8_t) (w >> 24);+	}+}++void crypton_sysdrg_test_lock(void)+{+#ifndef _WIN32+	pthread_mutex_lock(&global_lock);+#endif+}++void crypton_sysdrg_test_unlock(void)+{+#ifndef _WIN32+	pthread_mutex_unlock(&global_lock);+#endif+}+#endif++#ifdef _WIN32+static BOOL CALLBACK init_slot(PINIT_ONCE o, PVOID p, PVOID *c)+{+	(void) o; (void) p; (void) c;+	InitializeCriticalSection(&global_lock);+	thread_slot = FlsAlloc(thread_free);+	return TRUE;+}+#else+static void init_slot(void)+{+	pthread_key_create(&thread_slot, thread_free);+	pthread_atfork(before_fork, after_fork_in_parent, after_fork_in_child);+}+#endif++static drg_t *this_thread(void)+{+	drg_t *d;++#ifdef _WIN32+	InitOnceExecuteOnce(&init_once, init_slot, NULL, NULL);+	if (thread_slot == FLS_OUT_OF_INDEXES)+		return NULL;+	d = (drg_t *) FlsGetValue(thread_slot);+#else+	pthread_once(&init_once, init_slot);+	d = (drg_t *) pthread_getspecific(thread_slot);+#endif+	if (!d) {+		d = (drg_t *) calloc(1, sizeof(drg_t));+		if (!d)+			return NULL;+#ifdef _WIN32+		if (!FlsSetValue(thread_slot, d)) {+			free(d);+			return NULL;+		}+#else+		if (pthread_setspecific(thread_slot, d) != 0) {+			free(d);+			return NULL;+		}+#endif+	}+	return d;+}++/* Returns the number of bytes written, which is len unless there was no+ * seed to be had -- and then it is 0, so that the caller can say so rather+ * than hand back a buffer it cannot vouch for. */+int crypton_sysdrg_bytes(uint8_t *out, int len)+{+	drg_t *d;++	if (len < 0)+		return 0;+	d = this_thread();+	if (!d)+		return 0;++	if (drg_stale(d, CRYPTON_THREAD_RESEED)) {+		uint8_t seed[SEED_LEN];+		int ok = global_bytes(seed, SEED_LEN);+		if (ok)+			drg_seed(d, seed);+		scrub(seed, sizeof seed);+		if (!ok)+			return 0;+	}++	crypton_chacha_generate(out, &d->ctx, (uint32_t) len);+	d->used += (uint64_t) len;+	drg_rekey(d);+	return len;+}++/* For the tests: how many times the process has been seen to fork. */+uint32_t crypton_sysdrg_generation(void)+{+	return fork_generation;+}++/* For the tests: bytes this thread's generator has produced since it was+ * last seeded.  A generator shared between threads would carry the first+ * thread's count into the second; a per-thread one starts again at zero,+ * and that is the only way from outside to tell the two apart. */+uint64_t crypton_sysdrg_thread_used(void)+{+	drg_t *d = this_thread();+	return d ? d->used : 0;+}
+ cbits/crypton_sysrandom.c view
@@ -0,0 +1,119 @@+/*+ * The kernel's own random number generator, reached without a file+ * descriptor: getrandom(2) on Linux and FreeBSD, getentropy(3) on the+ * systems that have that instead.+ *+ * This is preferred over reading /dev/urandom because it needs no path and+ * no descriptor, so it still works where /dev is not mounted or not+ * populated -- a minimal container, a chroot, a sandbox -- and it cannot be+ * defeated by exhausting the descriptor table.+ *+ * Availability is decided at run time as well as at compile time: a binary+ * built against headers that declare getrandom can still run on a kernel+ * that does not implement it, and says so with ENOSYS.+ */++#include <stddef.h>+#include <stdint.h>+#include <errno.h>++#if defined(__linux__)+#include <unistd.h>+#include <sys/syscall.h>+#ifdef SYS_getrandom+#define CRYPTON_SYSRANDOM_GETRANDOM 1+#endif+#elif defined(__FreeBSD__)+#include <sys/param.h>+#if __FreeBSD_version >= 1200000+#include <sys/random.h>+#define CRYPTON_SYSRANDOM_GETRANDOM 1+#endif+#elif defined(__APPLE__)+/* getentropy(3) is declared in sys/random.h on Darwin, since 10.12 -- but+ * macOS only.  Crypto.Random says iOS does not allow it, which is why that+ * platform builds with INSECURE_ENTROPY; until someone can try it there,+ * iOS keeps the path it has rather than gaining an untested one. */+#include <TargetConditionals.h>+#if defined(TARGET_OS_OSX) && TARGET_OS_OSX+#include <sys/random.h>+#define CRYPTON_SYSRANDOM_GETENTROPY 1+#endif+#elif defined(__OpenBSD__)+#include <unistd.h>+#define CRYPTON_SYSRANDOM_GETENTROPY 1+#elif defined(__NetBSD__)+#include <sys/param.h>+#if __NetBSD_Version__ >= 1000000000+#include <sys/random.h>+#define CRYPTON_SYSRANDOM_GETENTROPY 1+#endif+#endif++/* getentropy(3) refuses more than 256 bytes in one call. */+#define CRYPTON_GETENTROPY_MAX 256++#if defined(CRYPTON_SYSRANDOM_GETRANDOM)++static int sysrandom_once(uint8_t *buf, size_t n)+{+#if defined(__linux__)+	return (int) syscall(SYS_getrandom, buf, n, 0);+#else+	return (int) getrandom(buf, n, 0);+#endif+}++#elif defined(CRYPTON_SYSRANDOM_GETENTROPY)++static int sysrandom_once(uint8_t *buf, size_t n)+{+	if (n > CRYPTON_GETENTROPY_MAX)+		n = CRYPTON_GETENTROPY_MAX;+	if (getentropy(buf, n) != 0)+		return -1;+	return (int) n;+}++#else++static int sysrandom_once(uint8_t *buf, size_t n)+{+	(void) buf; (void) n;+	errno = ENOSYS;+	return -1;+}++#endif++/* Is the call there, on this kernel, right now?  A zero-length request+ * answers that without consuming anything. */+int crypton_sysrandom_available(void)+{+	uint8_t b;+	int r = sysrandom_once(&b, 0);+	return r < 0 ? 0 : 1;+}++/* Fill the buffer.  Returns the number of bytes written, which is n unless+ * the call failed for a reason other than being interrupted. */+int crypton_sysrandom_bytes(uint8_t *buf, int len)+{+	size_t n = (size_t) len;+	size_t done = 0;++	if (len < 0)+		return 0;+	while (done < n) {+		int r = sysrandom_once(buf + done, n - done);+		if (r < 0) {+			if (errno == EINTR)+				continue;+			break;+		}+		if (r == 0)+			break;+		done += (size_t) r;+	}+	return (int) done;+}
cbits/crypton_tiger.c view
@@ -306,7 +306,9 @@ 	ctx->h[2] = 0xf096a5b4c3b2e187ULL; } -static inline void tiger_do_chunk(struct tiger_ctx *ctx, uint64_t *buf)+/* The words are read out of the block rather than the block being pointed at+ * as though it were an array of them; see crypton_md5.c. */+static inline void tiger_do_chunk(struct tiger_ctx *ctx, const uint8_t *buf) { 	uint64_t x0, x1, x2, x3, x4, x5, x6, x7; 	uint64_t a,b,c;@@ -314,8 +316,8 @@ 	b = ctx->h[1]; 	c = ctx->h[2]; -	x0 = cpu_to_le64(buf[0]); x1 = cpu_to_le64(buf[1]); x2 = cpu_to_le64(buf[2]); x3 = cpu_to_le64(buf[3]);-	x4 = cpu_to_le64(buf[4]); x5 = cpu_to_le64(buf[5]); x6 = cpu_to_le64(buf[6]); x7 = cpu_to_le64(buf[7]);+	x0 = load_le64(buf     ); x1 = load_le64(buf +  8); x2 = load_le64(buf + 16); x3 = load_le64(buf + 24);+	x4 = load_le64(buf + 32); x5 = load_le64(buf + 40); x6 = load_le64(buf + 48); x7 = load_le64(buf + 56);  #define BYTEOF(c, n) ((uint8_t) (c >> ((n * 8)))) @@ -376,24 +378,15 @@ 	/* process partial buffer if there's enough data to make a block */ 	if (index && len >= to_fill) { 		memcpy(ctx->buf + index, data, to_fill);-		tiger_do_chunk(ctx, (uint64_t *) ctx->buf);+		tiger_do_chunk(ctx, ctx->buf); 		len -= to_fill; 		data += to_fill; 		index = 0; 	} -	if (need_alignment(data, 8)) {-		uint64_t tramp[8];-		ASSERT_ALIGNMENT(tramp, 8);-		for (; len >= 64; len -= 64, data += 64) {-			memcpy(tramp, data, 64);-			tiger_do_chunk(ctx, tramp);-		}-	} else {-		/* process as much 64-block as possible */-		for (; len >= 64; len -= 64, data += 64)-			tiger_do_chunk(ctx, (uint64_t *) data);-	}+	/* No trampoline: load_le64 does not ask for a boundary. */+	for (; len >= 64; len -= 64, data += 64)+		tiger_do_chunk(ctx, data);  	/* append data into buf */ 	if (len)
cbits/crypton_xsalsa.c view
@@ -41,7 +41,7 @@   memset(ctx, 0, sizeof(*ctx));   ctx->nb_rounds = nb_rounds; -  /* Create initial 512-bit input block:+  /* Create initial 512-bit input crypton_salsa_block:        (x0, x5, x10, x15) is the Salsa20 constant        (x1, x2, x3, x4, x11, x12, x13, x14) is a 256-bit key        (x6, x7, x8, x9) is the first 128 bits of a 192-bit nonce@@ -56,7 +56,7 @@ void crypton_xsalsa_derive(crypton_salsa_context *ctx,                               uint32_t ivlen, const uint8_t *iv) {-  /* Finish creating initial 512-bit input block:+  /* Finish creating initial 512-bit input crypton_salsa_block:        (x6, x7, x8, x9) is the first 128 bits of a 192-bit nonce       Except iv has been shifted by 64 bits so there are now only 128 bits ahead.@@ -65,15 +65,15 @@   ctx->st.d[ 9] += load_le32(iv + 4);    /* Compute (z0, z1, . . . , z15) = doubleround ^(r/2) (x0, x1, . . . , x15) */-  block hSalsa;-  memset(&hSalsa, 0, sizeof(block));+  crypton_salsa_block hSalsa;+  memset(&hSalsa, 0, sizeof(crypton_salsa_block));   crypton_salsa_core_xor(ctx->nb_rounds, &hSalsa, &ctx->st);  -  /* Build a new 512-bit input block (x′0, x′1, . . . , x′15):+  /* Build a new 512-bit input crypton_salsa_block (x′0, x′1, . . . , x′15):        (x′0, x′5, x′10, x′15) is the Salsa20 constant        (x′1,x′2,x′3,x′4,x′11,x′12,x′13,x′14) = (z0,z5,z10,z15,z6,z7,z8,z9)        (x′6,x′7) is the last 64 bits of the 192-bit nonce-       (x′8, x′9) is a 64-bit block counter.+       (x′8, x′9) is a 64-bit crypton_salsa_block counter.   */   ctx->st.d[ 1] = hSalsa.d[ 0] - ctx->st.d[ 0];   ctx->st.d[ 2] = hSalsa.d[ 5] - ctx->st.d[ 5];
+ cbits/curve25519/x25519.c view
@@ -0,0 +1,88 @@+/*+ * X25519 through the vendored s2n-bignum where it is built, and through+ * curve25519-donna where it is not.+ *+ * The fixed-base routine is the interesting half: crypton had none, and asked+ * for the public key by multiplying the base point 9 the general way.  With a+ * table it is four to five times less work, and a TLS handshake generates a+ * key every time.  Measured on an Apple M4:+ *+ *                     donna     s2n+ *     shared secret   18.15 us  12.35+ *     key generation  18.15     3.65+ *+ * and on an x86-64, 41.19 to 26.96 and 41.18 to 8.54.+ */+#include <string.h>++#include "curve25519/x25519.h"++void crypton_curve25519_donna(uint8_t *mypublic, const uint8_t *secret,+                              const uint8_t *basepoint);++/* The assembly takes four little-endian 64-bit words, which is the same bits+ * as the 32 little-endian bytes RFC 7748 sends, so the two cross by copying+ * -- on a little-endian machine, which is the only kind s2n-bignum is for. */+#if defined(CRYPTON_S2N_BIGNUM) && defined(__BYTE_ORDER__) \+    && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__+#define CRYPTON_X25519_S2N 1+#include "crypton_cpu.h"++extern void curve25519_x25519(uint64_t res[4], const uint64_t scalar[4],+                              const uint64_t point[4]);+extern void curve25519_x25519_alt(uint64_t res[4], const uint64_t scalar[4],+                                  const uint64_t point[4]);+extern void curve25519_x25519base(uint64_t res[4], const uint64_t scalar[4]);+extern void curve25519_x25519base_alt(uint64_t res[4], const uint64_t scalar[4]);++/* The same question as everywhere else in cbits/s2n: a microarchitecture one+ * on ARM that no feature bit answers, and exactly a feature bit on x86-64. */+static int use_alt(void)+{+#if defined(__aarch64__) || defined(__arm64__)+#ifdef __APPLE__+	return 1;+#else+	return 0;+#endif+#else+	return (crypton_x86_simd_features() & CRYPTON_X86_ADX) == 0;+#endif+}+#endif++void crypton_x25519(uint8_t out[32], const uint8_t secret[32],+                    const uint8_t point[32])+{+#ifdef CRYPTON_X25519_S2N+	uint64_t r[4], s[4], p[4];++	memcpy(s, secret, 32);+	memcpy(p, point, 32);+	if (use_alt())+		curve25519_x25519_alt(r, s, p);+	else+		curve25519_x25519(r, s, p);+	memcpy(out, r, 32);+#else+	crypton_curve25519_donna(out, secret, point);+#endif+}++void crypton_x25519_base(uint8_t out[32], const uint8_t secret[32])+{+#ifdef CRYPTON_X25519_S2N+	uint64_t r[4], s[4];++	memcpy(s, secret, 32);+	if (use_alt())+		curve25519_x25519base_alt(r, s);+	else+		curve25519_x25519base(r, s);+	memcpy(out, r, 32);+#else+	static const uint8_t nine[32] = {9};++	crypton_curve25519_donna(out, secret, nine);+#endif+}
+ cbits/curve25519/x25519.h view
@@ -0,0 +1,16 @@+#ifndef CRYPTON_X25519_H+#define CRYPTON_X25519_H++#include <stdint.h>++/* out = secret * point, the X25519 of RFC 7748: three 32-byte little-endian+ * strings as they go over the wire. */+void crypton_x25519(uint8_t out[32], const uint8_t secret[32],+                    const uint8_t point[32]);++/* out = secret * G, which is the same thing with the base point 9 -- but+ * where the assembly is built this reads a table instead and is four to five+ * times faster, which is what a key generation costs. */+void crypton_x25519_base(uint8_t out[32], const uint8_t secret[32]);++#endif
cbits/decaf/ed448goldilocks/decaf.c view
@@ -1462,7 +1462,10 @@             uint32_t pos = __builtin_ctz((uint32_t)current), odd = (uint32_t)current >> pos;             int32_t delta = odd & mask;             if (odd & 1<<(table_bits+1)) delta -= (1<<(table_bits+1));-            current -= delta << pos;+            /* delta is negative half the time and shifting a negative+             * value left is undefined; current is unsigned, so the shift is+             * done there and means the same thing. */+            current -= (uint64_t)delta << pos;             control[position].power = pos + 16*(w-1);             control[position].addend = delta;             position--;
cbits/decaf/include/word.h view
@@ -68,7 +68,7 @@  * Expand bit 0 of the given uint8_t to a mask_t all 1 or all 0  * The input must be either 0 or 1  */-CRYPTON_DECAF_INLINE mask_t bit_to_mask(uint8_t bit) {+static CRYPTON_DECAF_INLINE mask_t bit_to_mask(uint8_t bit) { #ifdef _MSC_VER #pragma warning ( push) #pragma warning ( disable : 4146)
cbits/ed25519/ed25519.c view
@@ -12,8 +12,33 @@ #include "ed25519-randombytes.h" #include "ed25519-hash.h" #include "ed25519-crypton-exts.h"+#include "ed25519/ed25519_s2n.h"  /*+	The base point multiplied by a scalar, packed.  s2n-bignum's assembly+	where it is built -- twice the speed of the table below, and signing+	does this twice -- and ed25519-donna's own table where it is not.+*/+static void+ed25519_base_pack(ed25519_public_key out, const bignum256modm s) {+	ge25519 ALIGN(16) p;++#if defined(CRYPTON_S2N_BIGNUM) && defined(__BYTE_ORDER__) \+    && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__+	unsigned char e[32];++	contract256_modm(e, s);+	if (crypton_ed25519_base_mult(out, e)) {+		memset(e, 0, sizeof e);+		return;+	}+	memset(e, 0, sizeof e);+#endif+	ge25519_scalarmult_base_niels(&p, ge25519_niels_base_multiples, s);+	ge25519_pack(out, &p);+}++/* 	Generates a (extsk[0..31]) and aExt (extsk[32..63]) */ @@ -38,14 +63,12 @@ void ED25519_FN(ed25519_publickey) (const ed25519_secret_key sk, ed25519_public_key pk) { 	bignum256modm a;-	ge25519 ALIGN(16) A; 	hash_512bits extsk;  	/* A = aB */ 	ed25519_extsk(extsk, sk); 	expand256_modm(a, extsk, 32);-	ge25519_scalarmult_base_niels(&A, ge25519_niels_base_multiples, a);-	ge25519_pack(pk, &A);+	ed25519_base_pack(pk, a); }  @@ -53,7 +76,6 @@ ED25519_FN(ed25519_sign) (const unsigned char *m, size_t mlen, const ed25519_secret_key sk, const ed25519_public_key pk, ed25519_signature RS) { 	ed25519_hash_context ctx; 	bignum256modm r, S, a;-	ge25519 ALIGN(16) R; 	hash_512bits extsk, hashr, hram;  	ed25519_extsk(extsk, sk);@@ -66,8 +88,7 @@ 	expand256_modm(r, hashr, 64);  	/* R = rB */-	ge25519_scalarmult_base_niels(&R, ge25519_niels_base_multiples, r);-	ge25519_pack(RS, &R);+	ed25519_base_pack(RS, r);  	/* S = H(R,A,m).. */ 	ed25519_hram(hram, RS, pk, m, mlen);
+ cbits/ed25519/ed25519_s2n.c view
@@ -0,0 +1,65 @@+/*+ * Ed25519's base point multiplication through the vendored s2n-bignum.+ *+ * Signing does this twice: once for the nonce's point R, and once for the+ * public key, which crypton derives from the secret key at every signature+ * rather than trusting the one it is handed.  Both go through here, so both+ * halves of a signature move at once.+ *+ * Measured on an Apple M4, one multiplication with the encoding:+ * ed25519-donna 7.5 us against s2n-bignum 3.5.+ */+#include <string.h>++#include "ed25519/ed25519_s2n.h"++#if defined(CRYPTON_S2N_BIGNUM) && defined(__BYTE_ORDER__) \+    && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__+#define CRYPTON_ED25519_S2N 1+#include "crypton_cpu.h"++extern void edwards25519_scalarmulbase(uint64_t res[8],+                                       const uint64_t scalar[4]);+extern void edwards25519_scalarmulbase_alt(uint64_t res[8],+                                           const uint64_t scalar[4]);+extern void edwards25519_encode(uint8_t z[32], const uint64_t p[8]);++/* The same question as everywhere else in cbits/s2n: a microarchitecture one+ * on ARM that no feature bit answers, and exactly a feature bit on x86-64. */+static int use_alt(void)+{+#if defined(__aarch64__) || defined(__arm64__)+#ifdef __APPLE__+	return 1;+#else+	return 0;+#endif+#else+	return (crypton_x86_simd_features() & CRYPTON_X86_ADX) == 0;+#endif+}+#endif++int crypton_ed25519_base_mult(uint8_t out[32], const uint8_t scalar[32])+{+#ifdef CRYPTON_ED25519_S2N+	/* the assembly takes four little-endian 64-bit words, which is the+	 * same bits as the 32 little-endian bytes the scalar is kept in */+	uint64_t s[4], p[8];++	memcpy(s, scalar, 32);+	if (use_alt())+		edwards25519_scalarmulbase_alt(p, s);+	else+		edwards25519_scalarmulbase(p, s);+	edwards25519_encode(out, p);++	memset(s, 0, sizeof s);+	memset(p, 0, sizeof p);+	return 1;+#else+	(void) out;+	(void) scalar;+	return 0;+#endif+}
+ cbits/ed25519/ed25519_s2n.h view
@@ -0,0 +1,14 @@+#ifndef CRYPTON_ED25519_S2N_H+#define CRYPTON_ED25519_S2N_H++#include <stdint.h>++/* The base point multiplied by a scalar, packed into Ed25519's 32-byte+ * encoding.  The scalar is 32 little-endian bytes, already reduced.+ *+ * Returns 1 when the vendored s2n-bignum did the work and 0 when it is not+ * built, in which case the caller multiplies the base point itself.+ */+int crypton_ed25519_base_mult(uint8_t out[32], const uint8_t scalar[32]);++#endif
cbits/include32/p256/p256.h view
@@ -83,16 +83,9 @@     const crypton_p256_int* b,     crypton_p256_int* c); -// b := 1 / a % MOD-// MOD best be SECP256r1_n-void crypton_p256_modinv(-    const crypton_p256_int* MOD,-    const crypton_p256_int* a,-    crypton_p256_int* b);--// b := 1 / a % MOD+// b := 1 / a % MOD, in time that depends on a // MOD best be SECP256r1_n-// Faster than crypton_p256_modinv()+// Answers zero for an a that has no inverse, which is zero and MOD void crypton_p256_modinv_vartime(     const crypton_p256_int* MOD,     const crypton_p256_int* a,@@ -130,13 +123,6 @@ void crypton_p256_base_point_mul(const crypton_p256_int *n,                          crypton_p256_int *out_x,                          crypton_p256_int *out_y);--// {out_x,out_y} := n{in_x,in_y}-void crypton_p256_point_mul(const crypton_p256_int *n,-                    const crypton_p256_int *in_x,-                    const crypton_p256_int *in_y,-                    crypton_p256_int *out_x,-                    crypton_p256_int *out_y);  // {out_x,out_y} := n1G + n2{in_x,in_y} void crypton_p256_points_mul_vartime(
cbits/include32/p256/p256_gf.h view
@@ -76,6 +76,13 @@     0 }; static const felem kZero = {0};++/* the curve's b, in Montgomery form, for the complete addition formula */+static const felem kB = {+    0x13897bbf, 0x9cdf622, 0x43090d8, 0x2e67c4,+    0x176b5678, 0x2afdc84, 0xd196888, 0xb090e90,+    0xb8600c3+}; static const felem kP = {     0x1fffffff, 0xfffffff, 0x1fffffff, 0x3ff,     0, 0, 0x200000, 0xf000000,@@ -86,96 +93,99 @@     0, 0, 0x400000, 0xe000000,     0x1fffffff };-/* kPrecomputed contains precomputed values to aid the calculation of scalar- * multiples of the base point, G. It's actually two, equal length, tables- * concatenated.+/* kPrecomputed holds the multiples of the base point G that the comb in+ * scalar_base_mult reads.  Two tables of sixteen affine points, one after the+ * other.  *- * The first table contains (x,y) felem pairs for 16 multiples of the base- * point, G.+ * The comb takes five bits of the signed all-bits-set representation at a+ * time, from positions 52 apart, and the two tables are offset from each+ * other by 26:  *- *   Index  |  Index (binary) | Value- *       0  |           0000  | 0G (all zeros, omitted)- *       1  |           0001  | G- *       2  |           0010  | 2**64G- *       3  |           0011  | 2**64G + G- *       4  |           0100  | 2**128G- *       5  |           0101  | 2**128G + G- *       6  |           0110  | 2**128G + 2**64G- *       7  |           0111  | 2**128G + 2**64G + G- *       8  |           1000  | 2**192G- *       9  |           1001  | 2**192G + G- *      10  |           1010  | 2**192G + 2**64G- *      11  |           1011  | 2**192G + 2**64G + G- *      12  |           1100  | 2**192G + 2**128G- *      13  |           1101  | 2**192G + 2**128G + G- *      14  |           1110  | 2**192G + 2**128G + 2**64G- *      15  |           1111  | 2**192G + 2**128G + 2**64G + G+ *   first table    i, 52+i, 104+i, 156+i, 208+i+ *   second table   26+i, 78+i, 130+i, 182+i, 234+i  *- * The second table follows the same style, but the terms are 2**32G,- * 2**96G, 2**160G, 2**224G.+ * for i from 25 down to 0, which covers all 260 bits between them.  *+ * Every digit of that representation is +-1, so a block of five teeth takes+ * one of thirty-two values -- and they come in pairs that differ only by+ * sign.  So sixteen entries are enough: the top tooth is taken positive, bit+ * j of the index says that tooth j agrees with it, and where the top tooth is+ * negative the caller negates y, which costs a subtraction.  Entry zero is a+ * point like any other here, unlike the unsigned table this replaces, where+ * it stood for the infinity.+ *+ *   Index  |  Index (binary) | Value+ *       0  |           0000  | 2**208G - 2**156G - 2**104G - 2**52G - G+ *       1  |           0001  | 2**208G - 2**156G - 2**104G - 2**52G + G+ *     ...  |            ...  | ...+ *      15  |           1111  | 2**208G + 2**156G + 2**104G + 2**52G + G+ *  * This is ~2KB of data. */-static const limb kPrecomputed[NLIMBS * 2 * 15 * 2] = {-    0x11522878, 0xe730d41, 0xdb60179, 0x4afe2ff, 0x12883add, 0xcaddd88, 0x119e7edc, 0xd4a6eab, 0x3120bee,-    0x1d2aac15, 0xf25357c, 0x19e45cdd, 0x5c721d0, 0x1992c5a5, 0xa237487, 0x154ba21, 0x14b10bb, 0xae3fe3,-    0xd41a576, 0x922fc51, 0x234994f, 0x60b60d3, 0x164586ae, 0xce95f18, 0x1fe49073, 0x3fa36cc, 0x5ebcd2c,-    0xb402f2f, 0x15c70bf, 0x1561925c, 0x5a26704, 0xda91e90, 0xcdc1c7f, 0x1ea12446, 0xe1ade1e, 0xec91f22,-    0x26f7778, 0x566847e, 0xa0bec9e, 0x234f453, 0x1a31f21a, 0xd85e75c, 0x56c7109, 0xa267a00, 0xb57c050,-    0x98fb57, 0xaa837cc, 0x60c0792, 0xcfa5e19, 0x61bab9e, 0x589e39b, 0xa324c5, 0x7d6dee7, 0x2976e4b,-    0x1fc4124a, 0xa8c244b, 0x1ce86762, 0xcd61c7e, 0x1831c8e0, 0x75774e1, 0x1d96a5a9, 0x843a649, 0xc3ab0fa,-    0x6e2e7d5, 0x7673a2a, 0x178b65e8, 0x4003e9b, 0x1a1f11c2, 0x7816ea, 0xf643e11, 0x58c43df, 0xf423fc2,-    0x19633ffa, 0x891f2b2, 0x123c231c, 0x46add8c, 0x54700dd, 0x59e2b17, 0x172db40f, 0x83e277d, 0xb0dd609,-    0xfd1da12, 0x35c6e52, 0x19ede20c, 0xd19e0c0, 0x97d0f40, 0xb015b19, 0x449e3f5, 0xe10c9e, 0x33ab581,-    0x56a67ab, 0x577734d, 0x1dddc062, 0xc57b10d, 0x149b39d, 0x26a9e7b, 0xc35df9f, 0x48764cd, 0x76dbcca,-    0xca4b366, 0xe9303ab, 0x1a7480e7, 0x57e9e81, 0x1e13eb50, 0xf466cf3, 0x6f16b20, 0x4ba3173, 0xc168c33,-    0x15cb5439, 0x6a38e11, 0x73658bd, 0xb29564f, 0x3f6dc5b, 0x53b97e, 0x1322c4c0, 0x65dd7ff, 0x3a1e4f6,-    0x14e614aa, 0x9246317, 0x1bc83aca, 0xad97eed, 0xd38ce4a, 0xf82b006, 0x341f077, 0xa6add89, 0x4894acd,-    0x9f162d5, 0xf8410ef, 0x1b266a56, 0xd7f223, 0x3e0cb92, 0xe39b672, 0x6a2901a, 0x69a8556, 0x7e7c0,-    0x9b7d8d3, 0x309a80, 0x1ad05f7f, 0xc2fb5dd, 0xcbfd41d, 0x9ceb638, 0x1051825c, 0xda0cf5b, 0x812e881,-    0x6f35669, 0x6a56f2c, 0x1df8d184, 0x345820, 0x1477d477, 0x1645db1, 0xbe80c51, 0xc22be3e, 0xe35e65a,-    0x1aeb7aa0, 0xc375315, 0xf67bc99, 0x7fdd7b9, 0x191fc1be, 0x61235d, 0x2c184e9, 0x1c5a839, 0x47a1e26,-    0xb7cb456, 0x93e225d, 0x14f3c6ed, 0xccc1ac9, 0x17fe37f3, 0x4988989, 0x1a90c502, 0x2f32042, 0xa17769b,-    0xafd8c7c, 0x8191c6e, 0x1dcdb237, 0x16200c0, 0x107b32a1, 0x66c08db, 0x10d06a02, 0x3fc93, 0x5620023,-    0x16722b27, 0x68b5c59, 0x270fcfc, 0xfad0ecc, 0xe5de1c2, 0xeab466b, 0x2fc513c, 0x407f75c, 0xbaab133,-    0x9705fe9, 0xb88b8e7, 0x734c993, 0x1e1ff8f, 0x19156970, 0xabd0f00, 0x10469ea7, 0x3293ac0, 0xcdc98aa,-    0x1d843fd, 0xe14bfe8, 0x15be825f, 0x8b5212, 0xeb3fb67, 0x81cbd29, 0xbc62f16, 0x2b6fcc7, 0xf5a4e29,-    0x13560b66, 0xc0b6ac2, 0x51ae690, 0xd41e271, 0xf3e9bd4, 0x1d70aab, 0x1029f72, 0x73e1c35, 0xee70fbc,-    0xad81baf, 0x9ecc49a, 0x86c741e, 0xfe6be30, 0x176752e7, 0x23d416, 0x1f83de85, 0x27de188, 0x66f70b8,-    0x181cd51f, 0x96b6e4c, 0x188f2335, 0xa5df759, 0x17a77eb6, 0xfeb0e73, 0x154ae914, 0x2f3ec51, 0x3826b59,-    0xb91f17d, 0x1c72949, 0x1362bf0a, 0xe23fddf, 0xa5614b0, 0xf7d8f, 0x79061, 0x823d9d2, 0x8213f39,-    0x1128ae0b, 0xd095d05, 0xb85c0c2, 0x1ecb2ef, 0x24ddc84, 0xe35e901, 0x18411a4a, 0xf5ddc3d, 0x3786689,-    0x52260e8, 0x5ae3564, 0x542b10d, 0x8d93a45, 0x19952aa4, 0x996cc41, 0x1051a729, 0x4be3499, 0x52b23aa,-    0x109f307e, 0x6f5b6bb, 0x1f84e1e7, 0x77a0cfa, 0x10c4df3f, 0x25a02ea, 0xb048035, 0xe31de66, 0xc6ecaa3,-    0x28ea335, 0x2886024, 0x1372f020, 0xf55d35, 0x15e4684c, 0xf2a9e17, 0x1a4a7529, 0xcb7beb1, 0xb2a78a1,-    0x1ab21f1f, 0x6361ccf, 0x6c9179d, 0xb135627, 0x1267b974, 0x4408bad, 0x1cbff658, 0xe3d6511, 0xc7d76f,-    0x1cc7a69, 0xe7ee31b, 0x54fab4f, 0x2b914f, 0x1ad27a30, 0xcd3579e, 0xc50124c, 0x50daa90, 0xb13f72,-    0xb06aa75, 0x70f5cc6, 0x1649e5aa, 0x84a5312, 0x329043c, 0x41c4011, 0x13d32411, 0xb04a838, 0xd760d2d,-    0x1713b532, 0xbaa0c03, 0x84022ab, 0x6bcf5c1, 0x2f45379, 0x18ae070, 0x18c9e11e, 0x20bca9a, 0x66f496b,-    0x3eef294, 0x67500d2, 0xd7f613c, 0x2dbbeb, 0xb741038, 0xe04133f, 0x1582968d, 0xbe985f7, 0x1acbc1a,-    0x1a6a939f, 0x33e50f6, 0xd665ed4, 0xb4b7bd6, 0x1e5a3799, 0x6b33847, 0x17fa56ff, 0x65ef930, 0x21dc4a,-    0x2b37659, 0x450fe17, 0xb357b65, 0xdf5efac, 0x15397bef, 0x9d35a7f, 0x112ac15f, 0x624e62e, 0xa90ae2f,-    0x107eecd2, 0x1f69bbe, 0x77d6bce, 0x5741394, 0x13c684fc, 0x950c910, 0x725522b, 0xdc78583, 0x40eeabb,-    0x1fde328a, 0xbd61d96, 0xd28c387, 0x9e77d89, 0x12550c40, 0x759cb7d, 0x367ef34, 0xae2a960, 0x91b8bdc,-    0x93462a9, 0xf469ef, 0xb2e9aef, 0xd2ca771, 0x54e1f42, 0x7aaa49, 0x6316abb, 0x2413c8e, 0x5425bf9,-    0x1bed3e3a, 0xf272274, 0x1f5e7326, 0x6416517, 0xea27072, 0x9cedea7, 0x6e7633, 0x7c91952, 0xd806dce,-    0x8e2a7e1, 0xe421e1a, 0x418c9e1, 0x1dbc890, 0x1b395c36, 0xa1dc175, 0x1dc4ef73, 0x8956f34, 0xe4b5cf2,-    0x1b0d3a18, 0x3194a36, 0x6c2641f, 0xe44124c, 0xa2f4eaa, 0xa8c25ba, 0xf927ed7, 0x627b614, 0x7371cca,-    0xba16694, 0x417bc03, 0x7c0a7e3, 0x9c35c19, 0x1168a205, 0x8b6b00d, 0x10e3edc9, 0x9c19bf2, 0x5882229,-    0x1b2b4162, 0xa5cef1a, 0x1543622b, 0x9bd433e, 0x364e04d, 0x7480792, 0x5c9b5b3, 0xe85ff25, 0x408ef57,-    0x1814cfa4, 0x121b41b, 0xd248a0f, 0x3b05222, 0x39bb16a, 0xc75966d, 0xa038113, 0xa4a1769, 0x11fbc6c,-    0x917e50e, 0xeec3da8, 0x169d6eac, 0x10c1699, 0xa416153, 0xf724912, 0x15cd60b7, 0x4acbad9, 0x5efc5fa,-    0xf150ed7, 0x122b51, 0x1104b40a, 0xcb7f442, 0xfbb28ff, 0x6ac53ca, 0x196142cc, 0x7bf0fa9, 0x957651,-    0x4e0f215, 0xed439f8, 0x3f46bd5, 0x5ace82f, 0x110916b6, 0x6db078, 0xffd7d57, 0xf2ecaac, 0xca86dec,-    0x15d6b2da, 0x965ecc9, 0x1c92b4c2, 0x1f3811, 0x1cb080f5, 0x2d8b804, 0x19d1c12d, 0xf20bd46, 0x1951fa7,-    0xa3656c3, 0x523a425, 0xfcd0692, 0xd44ddc8, 0x131f0f5b, 0xaf80e4a, 0xcd9fc74, 0x99bb618, 0x2db944c,-    0xa673090, 0x1c210e1, 0x178c8d23, 0x1474383, 0x10b8743d, 0x985a55b, 0x2e74779, 0x576138, 0x9587927,-    0x133130fa, 0xbe05516, 0x9f4d619, 0xbb62570, 0x99ec591, 0xd9468fe, 0x1d07782d, 0xfc72e0b, 0x701b298,-    0x1863863b, 0x85954b8, 0x121a0c36, 0x9e7fedf, 0xf64b429, 0x9b9d71e, 0x14e2f5d8, 0xf858d3a, 0x942eea8,-    0xda5b765, 0x6edafff, 0xa9d18cc, 0xc65e4ba, 0x1c747e86, 0xe4ea915, 0x1981d7a1, 0x8395659, 0x52ed4e2,-    0x87d43b7, 0x37ab11b, 0x19d292ce, 0xf8d4692, 0x18c3053f, 0x8863e13, 0x4c146c0, 0x6bdf55a, 0x4e4457d,-    0x16152289, 0xac78ec2, 0x1a59c5a2, 0x2028b97, 0x71c2d01, 0x295851f, 0x404747b, 0x878558d, 0x7d29aa4,-    0x13d8341f, 0x8daefd7, 0x139c972d, 0x6b7ea75, 0xd4a9dde, 0xff163d8, 0x81d55d7, 0xa5bef68, 0xb7b30d8,-    0xbe73d6f, 0xaa88141, 0xd976c81, 0x7e7a9cc, 0x18beb771, 0xd773cbd, 0x13f51951, 0x9d0c177, 0x1c49a78,+static const limb kPrecomputed[NLIMBS * 2 * 16 * 2] = {+    0xe01bd76, 0xa0be8b3, 0x8494c1d, 0x609ab3d, 0x1188042f, 0x499c03d, 0x1df7cd26, 0x51b33c5, 0x1fb3bce,+    0x39cdd45, 0xdc0dd9b, 0xe3053d7, 0x1ffaf46, 0x9ac284a, 0xac051d4, 0x1c09fe1b, 0x8227cbf, 0x5bf049b,+    0xb9487d, 0x2ecb75f, 0x194825bf, 0xd70cf28, 0x14e528f3, 0x4d8670c, 0x35bbabb, 0x6b692ca, 0xd96d08,+    0x1db081dc, 0xa87ea7b, 0x190e5549, 0xa4cf420, 0x1e151385, 0xaf3d4bd, 0x4057e9f, 0x5078feb, 0x154519a,+    0xbf15dea, 0x453fa25, 0x1171c85a, 0x576c824, 0x154e7060, 0x71ede6e, 0x160467a2, 0xdea8a44, 0x81ffcb1,+    0x61a56fa, 0x76119b9, 0x110bfb9b, 0x3d527ea, 0x1997bdb4, 0xe1d1253, 0x180ce91c, 0x11950ee, 0x53d5938,+    0x694e7c9, 0xe0cf337, 0x16d8ae50, 0x202517f, 0x4d02e16, 0xd13b5fd, 0xfae97eb, 0xa1c7f60, 0x1206fe8,+    0x11b1c908, 0xf8a82f, 0x6ab17a0, 0x48058e8, 0x2d0feb, 0xfada550, 0x658edb9, 0xa17567a, 0x8daa44d,+    0x6361dc9, 0xec00c0e, 0x151a7b1c, 0xb35a683, 0x1643fe02, 0x70155c0, 0x1f131d45, 0x3998068, 0x25beef8,+    0x88138de, 0x8995ce4, 0x18565c50, 0x60f12b6, 0xc47a656, 0x4a82bd9, 0xb547a17, 0xb333474, 0xe86513a,+    0x7f8d2bc, 0x7f0f16c, 0x8cde475, 0x1ac3d5c, 0x1a832c9a, 0x6a93e7a, 0x19833281, 0xcec82db, 0x4f08cc,+    0x72c4394, 0x4686520, 0x1e845ce, 0xfb181a1, 0xf5135a5, 0xa1265d6, 0x6c63ce8, 0xe81797e, 0x5dbcd5a,+    0x2a5d603, 0x1ad4e91, 0xc86e1b3, 0x793abea, 0x1b8610a4, 0x8d5b975, 0x74dd850, 0xbbca81, 0xd7c35d8,+    0x1bab7afc, 0x4df749, 0x4acb4ea, 0xfae8c89, 0x14552ae5, 0x6dc1c20, 0x14f629f, 0x2368fe5, 0xe5a9cba,+    0x67576f7, 0x9c77c50, 0x1d63c92a, 0xbbb9ef8, 0xa7530d4, 0x963335, 0xfa09c54, 0xb6d03e3, 0xed1a022,+    0x19c59f49, 0x45c823d, 0x17a28df1, 0x80ae516, 0xe2ada82, 0x19b97fb, 0x13a9ebf, 0xf1e7606, 0xde03632,+    0x14e318b9, 0x7b57b83, 0x51a9a92, 0x3378a17, 0x1cde9289, 0x45956c2, 0x2bbab5e, 0x780b1f5, 0x8356034,+    0x75eca28, 0x6f39648, 0xf2fdbda, 0x4c65cd9, 0xb3759e7, 0x710c0b1, 0xda24432, 0x8d236aa, 0xf4c449f,+    0xaf9ba24, 0xb83ea7f, 0x1e46ecc3, 0x3f277b0, 0x6acdecd, 0x1f597d1, 0x8483d72, 0x57d29dd, 0x66d4060,+    0x167c14af, 0xc142ded, 0xc1689ca, 0x651aa52, 0x8d05768, 0x9709cfa, 0x165fd283, 0x9f6f583, 0x9833b54,+    0xd17e2d6, 0x2915c32, 0x1970ef24, 0xe8ca45b, 0x11c29e09, 0x8121f26, 0xb5afbe1, 0x10c80ec, 0x834a25c,+    0xa37f0b0, 0xd6bac16, 0xed484fc, 0x8799206, 0x13db8bb1, 0xdce615c, 0x13320329, 0x79dc25, 0x914e7af,+    0x860a414, 0xb8a9434, 0x50396ce, 0x902d3e6, 0x15c152f3, 0x3753c64, 0x970c055, 0xba296fb, 0x64bd63b,+    0x118cbb94, 0x9d4274f, 0x121cdbfe, 0xb9137cf, 0xc8bddf1, 0xa1598d0, 0x446ab41, 0x11f1df2, 0x68115f1,+    0x2f1f708, 0xee35192, 0x1dfeb3fc, 0x4e1e1a6, 0x1d9adcbb, 0x6688662, 0x63ad21b, 0x9d9a9e1, 0xb0f8b4b,+    0xcd8bc3a, 0x3577d8, 0x1ffcb97d, 0xd31e8d6, 0x1c776310, 0x95b4ef7, 0x185a82ed, 0xe40bbb0, 0xbca1ea,+    0x19462e0b, 0x2252179, 0x14f14f09, 0x565e68d, 0x7ba5f37, 0x4cd1858, 0x167941b3, 0x4d1c7a7, 0x7aef1ab,+    0x1599efe9, 0x658e78d, 0x1ad33917, 0x7e74797, 0x19152edc, 0xdf7dc18, 0xdf677c, 0x9315d96, 0x4ba8eec,+    0xc45cd82, 0x1acfa03, 0xe265b49, 0xcfa6eb, 0x89d7619, 0x7b05, 0x1ba11068, 0xe1672d3, 0x655622a,+    0x1607ca6, 0x339049, 0x5454f70, 0x10edd75, 0x1133ceb7, 0xe3eec39, 0xc263156, 0xeb6ddbe, 0x10360a7,+    0xfd5f981, 0x47dd502, 0x1e66dbbe, 0x8820b63, 0xeee91ef, 0xffde293, 0xb66325d, 0x3a5a2a, 0x6a2acbb,+    0x11e949bc, 0xf6a6d57, 0x6b2ca24, 0xad903ec, 0x1e55de0, 0xe7074ed, 0xe681934, 0xea44010, 0xaa490cc,+    0x512324a, 0x6a5eb00, 0xee5e100, 0xd60a02b, 0x1b89c993, 0x5cffb70, 0xa49030c, 0x405aee, 0xe1b27cc,+    0x2e73a15, 0x9ca8dc0, 0x781dbff, 0x9fd85e0, 0x1884e4c8, 0x40873f6, 0x32d4b69, 0xf42f753, 0xeaf4c38,+    0x2802a4c, 0x1283dac, 0x759100a, 0xcb3ba75, 0xf3203d3, 0xf89b8aa, 0x7caa59e, 0x384a60a, 0x37f69e7,+    0x1e56d569, 0x120c552, 0x181734aa, 0x4fcd9b4, 0x7918e4e, 0xcd7938a, 0x2bbd8b2, 0x19f4f97, 0xa7af533,+    0xbb69948, 0x3eff33e, 0x1f3ac118, 0x3739770, 0x58898fd, 0x623fafb, 0xa7e6d93, 0xd31a676, 0x615192d,+    0xf543d28, 0xe61ce5a, 0x10ae4b39, 0xcd5a8d7, 0x1b34c6de, 0x81997ad, 0x198e2093, 0xd9b6ff7, 0x9a5954f,+    0x16782589, 0xed3c1ab, 0x62dc4a5, 0xac12d0c, 0x8bc8b7c, 0x168ec4e, 0x177e11dc, 0x407df09, 0x4056e85,+    0x9858305, 0xfe445e9, 0x18b1230a, 0x815ee9f, 0xa852eb4, 0x89444e0, 0x1a83481, 0x479359b, 0x2192576,+    0x16d5e61b, 0xfe480c4, 0x1d60b8b7, 0x1c6e798, 0x1a01310, 0x9998572, 0x7c59f75, 0x49dda87, 0xe75e0b6,+    0x1b2b7536, 0xb267d8, 0x15443085, 0x45e5924, 0x7fb947f, 0x296915d, 0x38fc56b, 0x4bae39f, 0xb218e7c,+    0x115c3b16, 0x9d95f0c, 0xa50ac4c, 0xcce8037, 0xc3c7ba3, 0xf02773a, 0xbc2ad54, 0x26914c1, 0x19a4b8d,+    0x3f8a1a6, 0xa0b8459, 0x1f77a521, 0x7d93297, 0x1dddb4b2, 0x9e4cd1c, 0x6e28403, 0xd7ce413, 0x5575b62,+    0x12bb7dc1, 0xbdfb15e, 0xa542867, 0xe943d3e, 0x1367fbdd, 0x37b387, 0x14e4d75f, 0xb90b09d, 0xbf6ec28,+    0xa5182c8, 0x1e5b34e, 0xabe4602, 0x1c13efa, 0x8d1182, 0x1c2947a, 0x1e04e0e3, 0x6caecdb, 0x40f14e8,+    0x1b845e4a, 0xa9fb149, 0xb34f513, 0x3a0fdbb, 0xfad2335, 0x5bb9342, 0x18c5ad62, 0xc97fdc3, 0x31225c7,+    0xb28a9ee, 0x585915, 0x1e355da1, 0x26ed08e, 0xc7d06a, 0xa65f219, 0x1fdf45cd, 0xc8323ea, 0x297b9ee,+    0x1c031098, 0x9c39cf0, 0x1287d79f, 0x69a9e32, 0x10015650, 0x1c3dfc3, 0x12dcd848, 0x3155a59, 0xeff3212,+    0x1a6ecdd0, 0xd26bd07, 0x18e077ab, 0x442b477, 0x46b735f, 0x495d60c, 0x1b57a6e5, 0x76a368a, 0xf53bd50,+    0xf12c5e0, 0x80a9b4, 0x15562060, 0x4102113, 0xa144ab5, 0x5fa4e9, 0x1009f5e9, 0xe34343a, 0x26fda5a,+    0x159e06a4, 0x5fa3aad, 0x10259b5f, 0xa69947d, 0x1190417e, 0x987da3f, 0x14e1e868, 0xdcb7e1e, 0x8890f9f,+    0x14dece80, 0x94bbf5c, 0x18513a17, 0x4ca31ca, 0xc0a2713, 0xbc46074, 0x1536f6a5, 0x43991aa, 0xb9f8f1c,+    0x987ba48, 0xb0829ae, 0xd29d324, 0x6339c35, 0x18ace0a4, 0x5d53b55, 0xff829f4, 0xe882ecf, 0xc05164c,+    0x6bd6bba, 0x9dfc14f, 0x1981cab3, 0xcfebf18, 0x1ddac868, 0x94ec6d4, 0x5abd4b, 0x737fdb3, 0x18f531f,+    0xd3c2a71, 0x337178f, 0x9f7c32e, 0xd9d7fda, 0x137d191f, 0xdd0757b, 0x14b6d65, 0x179f37a, 0x67b10e1,+    0x1bfb2cfd, 0xfe4ca43, 0x1fee2930, 0x98c2aa0, 0x9826788, 0xeaf4ceb, 0x17a6be82, 0xc899ed1, 0x500fb01,+    0x10918e6f, 0x36179ed, 0xbca6643, 0x2d80942, 0xf1ef61, 0xadca21c, 0xbe5b3a7, 0xadae157, 0x12daac,+    0x1337dda8, 0x6326e5d, 0x2738e1b, 0x5cb5c54, 0x98ec8a0, 0x252647d, 0x1d5c173c, 0xbdf848d, 0x9e5217b,+    0x1d64f447, 0xb71d1a, 0xb2c2360, 0xccb6bee, 0x1245995e, 0x94a9130, 0x5b93d91, 0x76c57ff, 0xeaa91d1,+    0x3941881, 0xc2aafbd, 0x1c0540d0, 0x1a938f0, 0x1304b724, 0x8524e10, 0x1bef780f, 0xbc0ea48, 0xbe90dae,+    0x1d10d5d8, 0xca979e2, 0x10db5cc4, 0x54e2493, 0x44d38f3, 0xbcb73b, 0x12dcff4, 0xd0ab219, 0xde69db2,+    0x13594366, 0xc30e05f, 0xfc245d4, 0x8c5b52f, 0x81901c7, 0xa9d1e03, 0x11ead62e, 0xb7be89b, 0xc9c8486,+    0x132a6fa0, 0x56af9b8, 0x41cb561, 0xf74418c, 0x141c461a, 0xbc18514, 0x1d6bbb68, 0x96d43c2, 0x7108696 };
cbits/include64/p256/p256.h view
@@ -83,16 +83,9 @@     const crypton_p256_int* b,     crypton_p256_int* c); -// b := 1 / a % MOD-// MOD best be SECP256r1_n-void crypton_p256_modinv(-    const crypton_p256_int* MOD,-    const crypton_p256_int* a,-    crypton_p256_int* b);--// b := 1 / a % MOD+// b := 1 / a % MOD, in time that depends on a // MOD best be SECP256r1_n-// Faster than crypton_p256_modinv()+// Answers zero for an a that has no inverse, which is zero and MOD void crypton_p256_modinv_vartime(     const crypton_p256_int* MOD,     const crypton_p256_int* a,@@ -130,13 +123,6 @@ void crypton_p256_base_point_mul(const crypton_p256_int *n,                          crypton_p256_int *out_x,                          crypton_p256_int *out_y);--// {out_x,out_y} := n{in_x,in_y}-void crypton_p256_point_mul(const crypton_p256_int *n,-                    const crypton_p256_int *in_x,-                    const crypton_p256_int *in_y,-                    crypton_p256_int *out_x,-                    crypton_p256_int *out_y);  // {out_x,out_y} := n1G + n2{in_x,in_y} void crypton_p256_points_mul_vartime(
cbits/include64/p256/p256_gf.h view
@@ -104,102 +104,111 @@     2, 0xfc00000000000, 0x7ffffffffffff, 0xfff7fffffffff, 0x7ffff }; static const felem kZero = {0};++/* the curve's b, in Montgomery form, for the complete addition formula */+static const felem kB = {+    0x1bec453897bbf, 0x33e210c243627, 0x484bb5ab3c017, 0x41a32d11055fb,+    0x2e18030ec243a+}; static const felem kP = {     0x7ffffffffffff, 0x1fffffffffff, 0, 0x4000000000, 0x3fffffffc0000 }; static const felem k2P = {     0x7fffffffffffe, 0x3fffffffffff, 0, 0x8000000000, 0x7fffffff80000 };-/* kPrecomputed contains precomputed values to aid the calculation of scalar- * multiples of the base point, G. It's actually two, equal length, tables- * concatenated.+/* kPrecomputed holds the multiples of the base point G that the comb in+ * scalar_base_mult reads.  Two tables of sixteen affine points, one after the+ * other.  *- * The first table contains (x,y) felem pairs for 16 multiples of the base- * point, G.+ * The comb takes five bits of the signed all-bits-set representation at a+ * time, from positions 52 apart, and the two tables are offset from each+ * other by 26:  *- *   Index  |  Index (binary) | Value- *       0  |           0000  | 0G (all zeros, omitted)- *       1  |           0001  | G- *       2  |           0010  | 2**64G- *       3  |           0011  | 2**64G + G- *       4  |           0100  | 2**128G- *       5  |           0101  | 2**128G + G- *       6  |           0110  | 2**128G + 2**64G- *       7  |           0111  | 2**128G + 2**64G + G- *       8  |           1000  | 2**192G- *       9  |           1001  | 2**192G + G- *      10  |           1010  | 2**192G + 2**64G- *      11  |           1011  | 2**192G + 2**64G + G- *      12  |           1100  | 2**192G + 2**128G- *      13  |           1101  | 2**192G + 2**128G + G- *      14  |           1110  | 2**192G + 2**128G + 2**64G- *      15  |           1111  | 2**192G + 2**128G + 2**64G + G+ *   first table    i, 52+i, 104+i, 156+i, 208+i+ *   second table   26+i, 78+i, 130+i, 182+i, 234+i  *- * The second table follows the same style, but the terms are 2**32G,- * 2**96G, 2**160G, 2**224G.+ * for i from 25 down to 0, which covers all 260 bits between them.  *+ * Every digit of that representation is +-1, so a block of five teeth takes+ * one of thirty-two values -- and they come in pairs that differ only by+ * sign.  So sixteen entries are enough: the top tooth is taken positive, bit+ * j of the index says that tooth j agrees with it, and where the top tooth is+ * negative the caller negates y, which costs a subtraction.  Entry zero is a+ * point like any other here, unlike the unsigned table this replaces, where+ * it stood for the infinity.+ *+ *   Index  |  Index (binary) | Value+ *       0  |           0000  | 2**208G - 2**156G - 2**104G - 2**52G - G+ *       1  |           0001  | 2**208G - 2**156G - 2**104G - 2**52G + G+ *     ...  |            ...  | ...+ *      15  |           1111  | 2**208G + 2**156G + 2**104G + 2**52G + G+ *  * This is ~2KB of data. */-static const limb kPrecomputed[NLIMBS * 2 * 15 * 2] = {-    0x661a831522878, 0xf17fb6d805e79, 0x5889441d6ea57, 0xae33cfdb995bb, 0xc482fbb529ba,-    0x4a6af9d2aac15, 0x90e867917377c, 0x487cc962d2ae3, 0xec2a97443446e, 0x2b8ff8c52c42,-    0x45f8a2d41a576, 0xb06988d2653e4, 0x718b22c357305, 0x33fc920e79d2b, 0x17af34b0fe8db,-    0x38e17eb402f2f, 0x3382558649705, 0x47f6d48f482d1, 0x7bd42488d9b83, 0x3b247c8b86b78,-    0x4d08fc26f7778, 0x7a29a82fb2795, 0x75cd18f90d11a, 0xad8e213b0bc, 0x2d5f0142899e8,-    0x506f98098fb57, 0x2f0c98301e4aa, 0x39b30dd5cf67d, 0x9c146498ab13c, 0xa5db92df5b7b,-    0x184897fc4124a, 0xe3f73a19d8aa, 0x4e1c18e47066b, 0x27b2d4b52eaee, 0x30eac3ea10e99,-    0x4e74546e2e7d5, 0x1f4dde2d97a1d, 0x6ead0f88e1200, 0x7dec87c220f02, 0x3d08ff096310f,-    0x23e5659633ffa, 0x6ec648f08c722, 0x3172a3806ea35, 0xf6e5b681eb3c5, 0x2c3758260f89d,-    0x38dca4fd1da12, 0xf06067b78830d, 0x3194be87a068c, 0x78893c7eb602b, 0xcead60438432,-    0x6ee69a56a67ab, 0xd886f77701895, 0x67b0a4d9cee2b, 0x3586bbf3e4d53, 0x1db6f32921d93,-    0x260756ca4b366, 0x4f40e9d2039fa, 0x4f3f09f5a82bf, 0xccde2d641e8cd, 0x305a30cd2e8c5,-    0x471c235cb5439, 0xab279cd962f5a, 0x17e1fb6e2dd94, 0xfe64589800a77, 0xe8793d99775f,-    0x48c62f4e614aa, 0xbf76ef20eb2a4, 0x669c672556c, 0x24683e0eff056, 0x12252b369ab76,-    0x821de9f162d5, 0xf911ec99a95be, 0x6721f065c906b, 0x58d452035c736, 0x1f9f01a6a15,-    0x6135009b7d8d3, 0xdaeeeb417dfc0, 0x63865fea0ee17, 0x6e0a304b939d6, 0x204ba2076833d,-    0x4ade586f35669, 0x2c1077e34611a, 0x5b1a3bea3b81a, 0xf97d018a22c8b, 0x38d7996b08af8,-    0x6ea62baeb7aa0, 0xebdcbd9ef2670, 0x35dc8fe0df3fe, 0xe458309d20c24, 0x11e87898716a0,-    0x7c44bab7cb456, 0xd64d3cf1bb64, 0x189bff1bf9e66, 0xb5218a049311, 0x285dda6cbcc81,-    0x3238dcafd8c7c, 0x607736c8de0, 0xdb83d99508b1, 0x4e1a0d404cd81, 0x1588008c00ff2,-    0x16b8b36722b27, 0x876609c3f3f1a, 0x66b72ef0e17d6, 0x705f8a279d568, 0x2eaac4cd01fdd,-    0x1171ce9705fe9, 0xffc79cd3264ee, 0x700c8ab4b80f0, 0x208d3d4f57a1, 0x337262a8ca4eb,-    0x297fd01d843fd, 0xa90956fa097f8, 0x529759fdb3845, 0x1d78c5e2d0397, 0x3d6938a4adbf3,-    0x16d5853560b66, 0xf138946b9a430, 0x2ab79f4dea6a0, 0xd42053ee43ae1, 0x3b9c3ef1cf870,-    0x598934ad81baf, 0x5f1821b1d07a7, 0x416bb3a973ff3, 0x23f07bd0a047a, 0x19bdc2e09f786,-    0x56dc9981cd51f, 0xfbace23c8cd65, 0x673bd3bf5b52e, 0x46a95d229fd61, 0xe09ad64bcfb1,-    0xe5292b91f17d, 0xfeefcd8afc287, 0x58f52b0a58711, 0x4800f20c201ef, 0x2084fce608f67,-    0x12ba0b128ae0b, 0x5977ae17030b4, 0x101126ee420f6, 0xf70823495c6bd, 0xde19a27d7770,-    0x5c6ac852260e8, 0x9d22950ac4356, 0x441cca955246c, 0x660a34e5332d9, 0x14ac8ea92f8d2,-    0x6b6d7709f307e, 0x67d7e13879db, 0x2ea8626f9fbbd, 0x99609006a4b40, 0x31bb2a8f8c779,-    0x10c04828ea335, 0xae9acdcbc080a, 0x617af2342607a, 0xc7494ea53e553, 0x2ca9e2872defa,-    0x6c399fab21f1f, 0xab139b245e758, 0x3ad933dcba589, 0x4797fecb08811, 0x31f5dbf8f594,-    0x7dc6361cc7a69, 0xc8a7953ead3f9, 0x79ed693d18015, 0x418a024999a6a, 0x2c4fdc9436aa,-    0x1eb98cb06aa75, 0x2989592796a9c, 0x11194821e425, 0xe27a648228388, 0x35d834b6c12a0,-    0x541807713b532, 0x7ae0a1008aaee, 0x7017a29bcb5e, 0x6b193c23c315c, 0x19bd25ac82f2a,-    0x6a01a43eef294, 0xddf5b5fd84f19, 0x33f5ba081c016, 0xdeb052d1bc082, 0x6b2f06afa617,-    0x7ca1eda6a939f, 0xbdeb35997b50c, 0x47f2d1bccda5, 0xc2ff4adfed667, 0x87712997be4,-    0x21fc2e2b37659, 0xf7d62cd5ed951, 0x27fa9cbdf7efa, 0xba25582bf3a6b, 0x2a42b8bd89398,-    0x6d377d07eecd2, 0x9ca1df5af387, 0x1109e3427e2ba, 0xce4aa4572a19, 0x103baaef71e16,-    0x2c3b2dfde328a, 0xbec4b4a30e1ef, 0x37d92a86204f3, 0x806cfde68eb39, 0x246e2f72b8aa5,-    0x68d3de93462a9, 0x53b8acba6bbc3, 0x2492a70fa1696, 0x38c62d5760f55, 0x15096fe4904f2,-    0x4e44e9bed3e3a, 0xb28bfd79cc9bc, 0x6a77513839320, 0x480dcec6739db, 0x3601b739f2465,-    0x43c348e2a7e1, 0xe448106327879, 0x175d9cae1b0ed, 0xd3b89dee743b8, 0x392d73ca255bc,-    0x32946db0d3a18, 0x9261b09907cc, 0x5ba517a755722, 0x51f24fdaf5184, 0x1cdc732989ed8,-    0x2f7806ba16694, 0xae0c9f029f8d0, 0xd8b45102ce1, 0xca1c7db9316d6, 0x162088a67066f,-    0x39de35b2b4162, 0xa19f550d88ae9, 0x7921b27026cde, 0x94b936b66e900, 0x1023bd5fa17fc,-    0x436837814cfa4, 0x29113492283c4, 0x66d1cdd8b51d8, 0xa540702278eb2, 0x47ef1b29285d,-    0x587b50917e50e, 0xb4cda75bab3b, 0x112520b0a9886, 0x66b9ac16fee49, 0x17bf17e92b2eb,-    0x2456a2f150ed7, 0xfa214412d0280, 0x3ca7dd947fe5b, 0xa72c28598d58a, 0x255d945efc3e,-    0x2873f04e0f215, 0x74178fd1af57b, 0x788848b5b2d6, 0xb1ffafaae0db6, 0x32a1b7b3cbb2a,-    0x4bd9935d6b2da, 0x9c08f24ad30a5, 0x4e58407a80f, 0x1b3a3825a5b17, 0x6547e9fc82f5,-    0x47484aa3656c3, 0x6ee43f341a494, 0x64a98f87adea2, 0x619b3f8e95f01, 0xb6e513266ed8,-    0x421c2a673090, 0xa1c1de32348c7, 0x55b85c3a1e8a3, 0xe05ce8ef330b4, 0x2561e49c15d84,-    0x40aa2d33130fa, 0x12b827d35866f, 0xfe4cf62c8ddb, 0x2fa0ef05bb28d, 0x1c06ca63f1cb8,-    0x32a971863863b, 0xff6fc86830da1, 0x71e7b25a14cf3, 0xea9c5ebb1373a, 0x250bbaa3e1634,-    0x5b5ffeda5b765, 0xf25d2a746331b, 0x115e3a3f43632, 0x67303af43c9d5, 0x14bb538a0e559,-    0x75623687d43b7, 0xa349674a4b38d, 0x613c61829ffc6, 0x689828d8110c7, 0x139115f5af7d5,-    0xf1d856152289, 0x45cbe967168ab, 0x51f38e1680901, 0x34808e8f652b0, 0x1f4a6a921e156,-    0x35dfaf3d8341f, 0xf53ace725cb63, 0x3d86a54eef35b, 0xa103aabaffe2c, 0x2decc36296fbd,-    0x510282be73d6f, 0xd4e6365db206a, 0x4bdc5f5bb8bf3, 0xde7ea32a3aee7, 0x71269e274305,+static const limb kPrecomputed[NLIMBS * 2 * 16 * 2] = {+    0x17d166e01bd76, 0xd59ea12530768, 0x3d8c40217b04, 0x17bef9a4c9338, 0x7ecef3946ccf,+    0x1bb3639cdd45, 0xd7a338c14f5f7, 0x1d44d614250ff, 0xff813fc37580a, 0x16fc126e089f2,+    0x596ebe0b9487d, 0x6794652096fcb, 0x70ca729479eb8, 0x286b775769b0c, 0x365b421ada4b,+    0xfd4f7db081dc, 0x7a1064395526a, 0x4bdf0a89c2d26, 0xac80afd3f5e7a, 0x551466941e3f,+    0x27f44abf15dea, 0x641245c721691, 0x66eaa738302bb, 0x12c08cf44e3db, 0x207ff2c77aa29,+    0x42337261a56fa, 0x93f5442fee6dd, 0x253ccbdeda1ea, 0xbb019d239c3a2, 0x14f564e046543,+    0x19e66e694e7c9, 0x28bfdb62b9438, 0x5fd268170b101, 0x81f5d2fd7a276, 0x481bfa2871fd,+    0x71505f1b1c908, 0x2c741aac5e803, 0x55001687f5a40, 0xe8cb1db73f5b4, 0x236a913685d59,+    0x181c6361dc9, 0xd341d469ec73b, 0x5c0b21ff0159a, 0xa3e263a8ae02a, 0x96fbbe0e6601,+    0x32b9c888138de, 0x895b615971422, 0x3d9623d32b307, 0xd16a8f42e9505, 0x3a1944eacccd1,+    0x61e2d87f8d2bc, 0x1eae233791d5f, 0x67ad41964d0d6, 0x6f3066502d527, 0x13c23333b20b,+    0x50ca4072c4394, 0xc0d087a117391, 0x5d67a89ad2fd8, 0xf8d8c79d1424c, 0x176f356ba05e5,+    0x5a9d222a5d603, 0xd5f5321b86cc6, 0x175dc308523c9, 0x4e9bb0a11ab7, 0x35f0d7602ef2a,+    0x1bee93bab7afc, 0x464492b2d3a81, 0x420a2a9572fd7, 0x9429ec53edb83, 0x396a72e88da3f,+    0xef8a067576f7, 0xcf7c758f24aa7, 0x33553a986a5dd, 0x8df4138a812c6, 0x3b46808adb40f,+    0x39047b9c59f49, 0x728b5e8a37c51, 0x7fb7156d41405, 0x182753d7e3372, 0x3780d8cbc79d8,+    0x6af7074e318b9, 0xc50b946a6a49e, 0x6c2e6f494499b, 0xd457756bc8b2a, 0x20d580d1e02c7,+    0x672c9075eca28, 0x2e6cbcbf6f69b, 0xb159bacf3a63, 0xa9b448864e218, 0x3d31127e348da,+    0x7d4feaf9ba24, 0x3bd8791bb30ee, 0x7d13566f669f9, 0x750907ae43eb2, 0x19b501815f4a7,+    0x285bdb67c14af, 0xd529305a272b0, 0x4fa4682bb4328, 0xecbfa5072e13, 0x260ced527dbd6,+    0x22b864d17e2d6, 0x522de5c3bc90a, 0x7268e14f04f46, 0xb16b5f7c30243, 0x20d2897043203,+    0x57582ca37f0b0, 0xc9033b5213f35, 0x15c9edc5d8c3c, 0x966640653b9cc, 0x24539ebc1e770,+    0x152868860a414, 0x69f3140e5b3ae, 0x464ae0a979c81, 0xed2e180aa6ea7, 0x192f58eee8a5b,+    0x284e9f18cbb94, 0x9be7c8736ffa7, 0xd0645eef8dc8, 0xc888d568342b3, 0x1a0457c447c77,+    0x46a3242f1f708, 0xf0d377facff3b, 0x662ecd6e5da70, 0x84c75a436cd10, 0x2c3e2d2e766a7,+    0x6aefb0cd8bc3a, 0xf46b7ff2e5f40, 0x6f7e3bb188698, 0xc30b505db2b69, 0x2f287ab902ee,+    0x4a42f39462e0b, 0xf346d3c53c248, 0x583dd2f9bab2, 0x9ecf2836699a3, 0x1ebbc6ad3471e,+    0x31cf1b599efe9, 0xa3cbeb4ce45d9, 0x418c8a976e3f3, 0x581becef9befb, 0x12ea3bb24c576,+    0x59f406c45cd82, 0xd375b8996d246, 0x30544ebb0c867, 0x4f74220d0000f, 0x195588ab859cb,+    0x6720921607ca6, 0x6eba95153dc00, 0x439899e75b887, 0xf984c62adc7dd, 0x40d829fadb76,+    0x7baa04fd5f981, 0x5b1f99b6ef91, 0x29377748f7c41, 0xa96cc64bbffbc, 0x1a8ab2ec0e968,+    0x54daaf1e949bc, 0x81f61acb2893d, 0x4ed0f2aef056c, 0x41cd03269ce0e, 0x2a924333a9100,+    0x4bd600512324a, 0x5015bb978401a, 0x370dc4e4c9eb0, 0xb94920618b9ff, 0x386c9f301016b,+    0x151b802e73a15, 0xc2f01e076ffe7, 0x3f6c4272644fe, 0x4c65a96d2810e, 0x3abd30e3d0bdd,+    0x507b582802a4c, 0xdd3a9d6440284, 0xaa79901e9e59, 0x28f954b3df137, 0xdfda79ce1298,+    0x418aa5e56d569, 0x6cda605cd2a84, 0x38a3c8c72727e, 0x5c577b1659af2, 0x29ebd4cc67d3e,+    0x5fe67cbb69948, 0xcbb87ceb0460f, 0x2fb2c44c7e9b9, 0xd94fcdb26c47f, 0x185464b74c699,+    0x439cb4f543d28, 0xd46bc2b92ce79, 0x7add9a636f66a, 0xdf31c41270332, 0x2696553f66dbf,+    0x2783576782589, 0x968618b71297b, 0x44e45e45be560, 0x26efc23b82d1d, 0x1015ba1501f7c,+    0x488bd29858305, 0xf74fe2c48c2bf, 0x4e0542975a40a, 0x6c35069031288, 0x86495d91e4d6,+    0x4901896d5e61b, 0x73cc7582e2dff, 0x5720d009880e3, 0x1cf8b3eeb3330, 0x39d782d92776a,+    0x64cfb1b2b7536, 0x2c925510c2142, 0x15d3fdca3fa2f, 0x7c71f8ad652d2, 0x2c8639f12eb8e,+    0x32be1915c3b16, 0x401ba942b1327, 0x73a61e3dd1e67, 0x57855aa9e04e, 0x6692e349a453,+    0x1708b23f8a1a6, 0x994bfdde94868, 0x51ceeeda593ec, 0x4cdc508073c99, 0x155d6d8b5f390,+    0x3f62bd2bb7dc1, 0x1e9f2950a19ef, 0x3879b3fdeef4a, 0x769c9aebe06f6, 0x2fdbb0a2e42c2,+    0x4b669ca5182c8, 0x9f7d2af918087, 0x47a04688c10e0, 0x6fc09c1c63852, 0x103c53a1b2bb3,+    0x3f6293b845e4a, 0x7eddacd3d44ea, 0x3427d6919a9d0, 0xf18b5ac4b772, 0xc48971f25ff7,+    0x30b22ab28a9ee, 0x684778d576841, 0x219063e835137, 0xabfbe8b9b4cbe, 0xa5ee7bb20c8f,+    0x739e1c031098, 0x4f194a1f5e7e7, 0x7c3800ab2834d, 0x665b9b090387b, 0x3bfcc848c5569,+    0x4d7a0fa6ecdd0, 0x5a3be381deaf4, 0x60c235b9afa21, 0x2b6af4dca92ba, 0x3d4ef541da8da,+    0x15368f12c5e0, 0x1089d55881802, 0x4e950a255aa08, 0xea013ebd20bf4, 0x9bf696b8d0d0,+    0x74755b59e06a4, 0xca3ec0966d7d7, 0x23f8c820bf534, 0x7a9c3d0d130fb, 0x22243e7f72df8,+    0x177eb94dece80, 0x18e56144e85e5, 0x746051389a65, 0xaaa6ded4b788c, 0x2e7e3c710e646,+    0x10535c987ba48, 0xce1ab4a74c92c, 0x355c567052319, 0x3dff053e8baa7, 0x30145933a20bb,+    0x3f829e6bd6bba, 0x5f8c66072ace7, 0x6d4eed643467f, 0xcc0b57a9729d8, 0x63d4c7dcdff6,+    0x6e2f1ed3c2a71, 0xbfed27df0cb8c, 0x57b9be8c8fece, 0xe8296dacbba0e, 0x19ec43845e7cd,+    0x499487bfb2cfd, 0x15507fb8a4c3f, 0x4eb4c133c44c6, 0x46f4d7d05d5e9, 0x1403ec072267b,+    0x42f3db0918e6f, 0x4a12f29990cd, 0x21c078f7b096c, 0x5d7cb674f5b94, 0x4b6ab2b6b85,+    0x64dcbb337dda8, 0xae2a09ce386d8, 0x47d4c764502e5, 0x37ab82e784a4c, 0x279485eef7e12,+    0x6e3a35d64f447, 0xb5f72cb08d802, 0x130922ccaf665, 0xfcb727b232952, 0x3aaa4745db15f,+    0x555f7a3941881, 0x9c78701503430, 0x6109825b920d4, 0x237def01f0a49, 0x2fa436baf03a9,+    0x52f3c5d10d5d8, 0x1249c36d73132, 0x73b2269c79aa7, 0x6425b9fe81796, 0x379a76cb42ac8,+    0x61c0bf3594366, 0xda97bf0917530, 0x60340c80e3c62, 0x6e3d5ac5d53a3, 0x3272121adefa2,+    0x55f37132a6fa0, 0x20c61072d5855, 0x514a0e230d7ba, 0xbad776d17830, 0x1c421a5a5b50f };
+ cbits/include64/p256/p256_s2n.h view
@@ -0,0 +1,25 @@+/*+ * The vendored s2n-bignum assembly, behind one call that picks the variant+ * the machine wants.  See cbits/s2n/README.md for which and why.+ *+ * Only declared in the 64-bit field build: s2n-bignum is x86-64 and AArch64+ * only, and this interface is the four little-endian 64-bit words those+ * architectures give crypton_p256_int anyway.+ */+#ifndef CRYPTON_P256_S2N_H+#define CRYPTON_P256_S2N_H++#include <stdint.h>++/* res = scalar * point, all of them affine and not in Montgomery form:+ * point is x then y, four words each, and res the same.  The point at+ * infinity goes in and comes out as (0, 0). */+void crypton_s2n_p256_scalarmul(uint64_t res[8], const uint64_t scalar[4],+                                const uint64_t point[8]);++/* res = scalar * G, the same shape out.  The table it reads and the window+ * width it was built for are in cbits/p256/p256_base_table.c, which+ * cbits/p256/gen_base_table.py writes. */+void crypton_s2n_p256_scalarmulbase(uint64_t res[8], const uint64_t scalar[4]);++#endif
+ cbits/mldsa/COMMIT view
@@ -0,0 +1,2 @@+834a90d5e846ffa1e1611bd24e160bb2e9b86d35+v2.0.0
+ cbits/mldsa/LICENSE view
@@ -0,0 +1,305 @@+mldsa-native is a fork of the public domain Dilithium reference implementation,+available on https://github.com/pq-crystals/dilithium.++All new files and all files derived from the Dilithium reference+implementation are made available under the Apache-2.0 license OR+the ISC license OR the MIT license. These licenses are+reproduced at the bottom of this file.++Files outside the library itself may carry different terms. Every file+states its own SPDX-License-Identifier, which determines the terms that+apply to it. In particular:++The code in test/notrandombytes/*, and its copies in+examples/*/test_only_rng/* and scripts/notrandombytes, is derived from+https://cr.yp.to/papers.html#surf and licensed under+LicenseRef-PD-hp OR CC0-1.0 OR 0BSD OR MIT-0 OR MIT.+It is only used for testing purposes.++The benchmarking code in test/hal/* carries the+MIT license. It is only used for testing purposes.++The tiny_sha3 code in+examples/bring_your_own_fips202/custom_fips202/tiny_sha3/* and+examples/custom_backend/mldsa_native/src/fips202/native/custom/src/*+carries the MIT license. It is only used to demonstrate custom+FIPS-202 implementations.++The proofs in proofs/* are in part derived from Amazon Web Services+verification infrastructure and licensed under+Apache-2.0 OR ISC OR MIT-0, MIT-0, or MIT-0 AND Apache-2.0. The IACR+document class proofs/isabelle/neon_ntt/document/iacrtrans.cls is+licensed under CC0-1.0. None of this is part of the library.++Documentation is licensed under CC-BY-4.0.++```+Copyright (c) The mldsa-native project authors+Copyright (c) The mlkem-native project authors+Copyright (c) 2020 Dougall Johnson+Copyright (c) 2022 Arm Limited+SPDX-License-Identifier: MIT++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in+all copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.+```++ISC license for mldsa-native content+------------------------------------++Copyright (c) The mldsa-native project authors++Permission to use, copy, modify, and/or distribute this software for any purpose+with or without fee is hereby granted, provided that the above copyright notice+and this permission notice appear in all copies.++THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH+REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND+FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,+INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS+OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER+TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF+THIS SOFTWARE.+++MIT license for mldsa-native content+------------------------------------++Copyright (c) The mldsa-native project authors++Permission is hereby granted, free of charge, to any person obtaining a copy of+this software and associated documentation files (the “Software”), to deal in+the Software without restriction, including without limitation the rights to+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of+the Software, and to permit persons to whom the Software is furnished to do so,+subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.++Apache-2.0 license for mldsa-native content+-------------------------------------------++                                 Apache License+                           Version 2.0, January 2004+                        http://www.apache.org/licenses/++   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION++   1. Definitions.++      "License" shall mean the terms and conditions for use, reproduction,+      and distribution as defined by Sections 1 through 9 of this document.++      "Licensor" shall mean the copyright owner or entity authorized by+      the copyright owner that is granting the License.++      "Legal Entity" shall mean the union of the acting entity and all+      other entities that control, are controlled by, or are under common+      control with that entity. For the purposes of this definition,+      "control" means (i) the power, direct or indirect, to cause the+      direction or management of such entity, whether by contract or+      otherwise, or (ii) ownership of fifty percent (50%) or more of the+      outstanding shares, or (iii) beneficial ownership of such entity.++      "You" (or "Your") shall mean an individual or Legal Entity+      exercising permissions granted by this License.++      "Source" form shall mean the preferred form for making modifications,+      including but not limited to software source code, documentation+      source, and configuration files.++      "Object" form shall mean any form resulting from mechanical+      transformation or translation of a Source form, including but+      not limited to compiled object code, generated documentation,+      and conversions to other media types.++      "Work" shall mean the work of authorship, whether in Source or+      Object form, made available under the License, as indicated by a+      copyright notice that is included in or attached to the work+      (an example is provided in the Appendix below).++      "Derivative Works" shall mean any work, whether in Source or Object+      form, that is based on (or derived from) the Work and for which the+      editorial revisions, annotations, elaborations, or other modifications+      represent, as a whole, an original work of authorship. For the purposes+      of this License, Derivative Works shall not include works that remain+      separable from, or merely link (or bind by name) to the interfaces of,+      the Work and Derivative Works thereof.++      "Contribution" shall mean any work of authorship, including+      the original version of the Work and any modifications or additions+      to that Work or Derivative Works thereof, that is intentionally+      submitted to Licensor for inclusion in the Work by the copyright owner+      or by an individual or Legal Entity authorized to submit on behalf of+      the copyright owner. For the purposes of this definition, "submitted"+      means any form of electronic, verbal, or written communication sent+      to the Licensor or its representatives, including but not limited to+      communication on electronic mailing lists, source code control systems,+      and issue tracking systems that are managed by, or on behalf of, the+      Licensor for the purpose of discussing and improving the Work, but+      excluding communication that is conspicuously marked or otherwise+      designated in writing by the copyright owner as "Not a Contribution."++      "Contributor" shall mean Licensor and any individual or Legal Entity+      on behalf of whom a Contribution has been received by Licensor and+      subsequently incorporated within the Work.++   2. Grant of Copyright License. Subject to the terms and conditions of+      this License, each Contributor hereby grants to You a perpetual,+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable+      copyright license to reproduce, prepare Derivative Works of,+      publicly display, publicly perform, sublicense, and distribute the+      Work and such Derivative Works in Source or Object form.++   3. Grant of Patent License. Subject to the terms and conditions of+      this License, each Contributor hereby grants to You a perpetual,+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable+      (except as stated in this section) patent license to make, have made,+      use, offer to sell, sell, import, and otherwise transfer the Work,+      where such license applies only to those patent claims licensable+      by such Contributor that are necessarily infringed by their+      Contribution(s) alone or by combination of their Contribution(s)+      with the Work to which such Contribution(s) was submitted. If You+      institute patent litigation against any entity (including a+      cross-claim or counterclaim in a lawsuit) alleging that the Work+      or a Contribution incorporated within the Work constitutes direct+      or contributory patent infringement, then any patent licenses+      granted to You under this License for that Work shall terminate+      as of the date such litigation is filed.++   4. Redistribution. You may reproduce and distribute copies of the+      Work or Derivative Works thereof in any medium, with or without+      modifications, and in Source or Object form, provided that You+      meet the following conditions:++      (a) You must give any other recipients of the Work or+          Derivative Works a copy of this License; and++      (b) You must cause any modified files to carry prominent notices+          stating that You changed the files; and++      (c) You must retain, in the Source form of any Derivative Works+          that You distribute, all copyright, patent, trademark, and+          attribution notices from the Source form of the Work,+          excluding those notices that do not pertain to any part of+          the Derivative Works; and++      (d) If the Work includes a "NOTICE" text file as part of its+          distribution, then any Derivative Works that You distribute must+          include a readable copy of the attribution notices contained+          within such NOTICE file, excluding those notices that do not+          pertain to any part of the Derivative Works, in at least one+          of the following places: within a NOTICE text file distributed+          as part of the Derivative Works; within the Source form or+          documentation, if provided along with the Derivative Works; or,+          within a display generated by the Derivative Works, if and+          wherever such third-party notices normally appear. The contents+          of the NOTICE file are for informational purposes only and+          do not modify the License. You may add Your own attribution+          notices within Derivative Works that You distribute, alongside+          or as an addendum to the NOTICE text from the Work, provided+          that such additional attribution notices cannot be construed+          as modifying the License.++      You may add Your own copyright statement to Your modifications and+      may provide additional or different license terms and conditions+      for use, reproduction, or distribution of Your modifications, or+      for any such Derivative Works as a whole, provided Your use,+      reproduction, and distribution of the Work otherwise complies with+      the conditions stated in this License.++   5. Submission of Contributions. Unless You explicitly state otherwise,+      any Contribution intentionally submitted for inclusion in the Work+      by You to the Licensor shall be under the terms and conditions of+      this License, without any additional terms or conditions.+      Notwithstanding the above, nothing herein shall supersede or modify+      the terms of any separate license agreement you may have executed+      with Licensor regarding such Contributions.++   6. Trademarks. This License does not grant permission to use the trade+      names, trademarks, service marks, or product names of the Licensor,+      except as required for reasonable and customary use in describing the+      origin of the Work and reproducing the content of the NOTICE file.++   7. Disclaimer of Warranty. Unless required by applicable law or+      agreed to in writing, Licensor provides the Work (and each+      Contributor provides its Contributions) on an "AS IS" BASIS,+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or+      implied, including, without limitation, any warranties or conditions+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A+      PARTICULAR PURPOSE. You are solely responsible for determining the+      appropriateness of using or redistributing the Work and assume any+      risks associated with Your exercise of permissions under this License.++   8. Limitation of Liability. In no event and under no legal theory,+      whether in tort (including negligence), contract, or otherwise,+      unless required by applicable law (such as deliberate and grossly+      negligent acts) or agreed to in writing, shall any Contributor be+      liable to You for damages, including any direct, indirect, special,+      incidental, or consequential damages of any character arising as a+      result of this License or out of the use or inability to use the+      Work (including but not limited to damages for loss of goodwill,+      work stoppage, computer failure or malfunction, or any and all+      other commercial damages or losses), even if such Contributor+      has been advised of the possibility of such damages.++   9. Accepting Warranty or Additional Liability. While redistributing+      the Work or Derivative Works thereof, You may choose to offer,+      and charge a fee for, acceptance of support, warranty, indemnity,+      or other liability obligations and/or rights consistent with this+      License. However, in accepting such obligations, You may act only+      on Your own behalf and on Your sole responsibility, not on behalf+      of any other Contributor, and only if You agree to indemnify,+      defend, and hold each Contributor harmless for any liability+      incurred by, or claims asserted against, such Contributor by reason+      of your accepting any such warranty or additional liability.++   END OF TERMS AND CONDITIONS++   APPENDIX: How to apply the Apache License to your work.++      To apply the Apache License to your work, attach the following+      boilerplate notice, with the fields enclosed by brackets "[]"+      replaced with your own identifying information. (Don't include+      the brackets!)  The text should be enclosed in the appropriate+      comment syntax for the file format. We also recommend that a+      file or class name and description of purpose be included on the+      same "printed page" as the copyright notice for easier+      identification within third-party archives.++   Copyright (c) The mldsa-native project authors++   Licensed under the Apache License, Version 2.0 (the "License");+   you may not use this file except in compliance with the License.+   You may obtain a copy of the License at++       http://www.apache.org/licenses/LICENSE-2.0++   Unless required by applicable law or agreed to in writing, software+   distributed under the License is distributed on an "AS IS" BASIS,+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+   See the License for the specific language governing permissions and+   limitations under the License.
+ cbits/mldsa/README.md view
@@ -0,0 +1,60 @@+# mldsa-native++ML-DSA (FIPS 204) from the PQ Code Package, vendored here and built for all+three parameter sets.++## Where it comes from++<https://github.com/pq-code-package/mldsa-native>.  `COMMIT` holds the+revision this tree is at; `import.sh` puts it there and is how the tree is+refreshed.  Nothing here is edited by hand -- every choice crypton makes is+made in `crypton_mldsa.h`, `crypton_mldsa.c` and `crypton.cabal`, so that a+re-import is a straight overwrite.++## Licence++`Apache-2.0 OR ISC OR MIT`, the same three-way form as `cbits/s2n` and by+some of the same authors.  crypton takes it under ISC, which is already in+the package's `license:` field.  `LICENSE` is the upstream file and is+listed in `license-files:`.++## What is taken and what is not++The whole of `mldsa/`, less the backends for architectures crypton does not+build for: the 32-bit+`src/fips202/native/armv81m`.  (Unlike mlkem-native, this one ships only the+AArch64 and x86-64 arithmetic backends, so there is nothing else to drop.)  Every reference to those is behind an+`MLD_SYS_` guard that cannot be true on the architectures crypton does+build for, so dropping them changes no build and keeps a few dozen files of+unreachable assembly out of the release tarball.  To take one back, delete+its line from `import.sh` and add its directory to `extra-source-files:`.++## How it is built++Upstream builds for one parameter set at a time.  `crypton_mldsa.c`+includes the amalgamation once per set -- the level-independent half kept by+exactly one of them -- which is how a single crypton offers ML-DSA-44, 65+and 87.  `crypton_mldsa_asm.S` does the same for the assembly, which is+level-independent and so is included once.++This is the shape `cbits/aes/armv8.c` already uses for the three AES key+sizes, and it has the same hazard: **cabal does not know that the wrapper+depends on the tree it includes.**  After changing anything under+`cbits/mldsa`, touch `crypton_mldsa.c`, or the build keeps the object it+already has and the change is not tested.++The hand-written backends are selected by `CRYPTON_MLDSA_NATIVE_BACKEND`,+which `crypton.cabal` defines on x86-64 and AArch64 other than Windows --+the same exclusion, and for the same reasons, as `cbits/s2n`.  Everywhere+else the portable C is built, which is the same code and passes the same+tests.++There is no randomised API: `MLD_CONFIG_NO_RANDOMIZED_API` is set, no+`randombytes()` is needed, and randomness is drawn in Haskell through+`MonadRandom` as it is for every other key crypton generates.++The symbols are `crypton_mldsa44_*`, `crypton_mldsa65_*` and+`crypton_mldsa87_*` rather than upstream's defaults, so that an+application linking another copy of mldsa-native -- through some other+library, or its own -- does not present the linker with two sets of+functions answering to one set of names.
+ cbits/mldsa/crypton_mldsa.c view
@@ -0,0 +1,31 @@+/*+ * All three ML-DSA parameter sets in one translation unit.+ *+ * mldsa-native is built for one parameter set at a time; a build wanting+ * several includes the amalgamation once per set, with the level-independent+ * half kept by exactly one of them.  This is the same shape as+ * cbits/aes/armv8.c, which includes cbits/aes/armv8_impl.c three times for+ * the three AES key sizes.+ *+ * NOTE, as there: cabal does not know that this file depends on the tree it+ * includes.  After changing anything under cbits/mldsa, touch this file, or+ * the build will quietly keep the object it already has.+ */+#include "crypton_mldsa.h"++#define MLD_CONFIG_MULTILEVEL_WITH_SHARED 1+#define MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS+#define MLD_CONFIG_PARAMETER_SET 44+#include "mldsa_native.c"+#undef MLD_CONFIG_MULTILEVEL_WITH_SHARED+#undef MLD_CONFIG_PARAMETER_SET++#define MLD_CONFIG_MULTILEVEL_NO_SHARED+#define MLD_CONFIG_PARAMETER_SET 65+#include "mldsa_native.c"+#undef MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS+#undef MLD_CONFIG_PARAMETER_SET++#define MLD_CONFIG_PARAMETER_SET 87+#include "mldsa_native.c"+#undef MLD_CONFIG_PARAMETER_SET
+ cbits/mldsa/crypton_mldsa.h view
@@ -0,0 +1,79 @@+/*+ * What crypton asks of the vendored mldsa-native, in one place.  The tree+ * under cbits/mldsa is upstream's and is overwritten by import.sh, so every+ * choice crypton makes is made here instead of by editing it.+ *+ * Included first by both crypton_mldsa.c and crypton_mldsa_asm.S, so it must+ * hold nothing the assembler cannot read: directives, and C behind+ * __ASSEMBLER__.+ */+#ifndef CRYPTON_MLDSA_H+#define CRYPTON_MLDSA_H++/*+ * The symbols are crypton's own, not the default PQCP_MLDSA_NATIVE_*.  An+ * application is free to link another copy of mldsa-native -- through some+ * other library, or its own -- and two copies answering to one set of names+ * is a problem the linker resolves silently and in nobody's favour.  With+ * MLD_CONFIG_MULTILEVEL_BUILD the level is appended, so the entry points+ * are crypton_mldsa44_*, crypton_mldsa65_* and crypton_mldsa87_*.+ */+#define MLD_CONFIG_NAMESPACE_PREFIX crypton_mldsa+#define MLD_CONFIG_MULTILEVEL_BUILD++/*+ * No randomised API, so no randombytes() to provide.  Randomness is drawn+ * in Haskell through MonadRandom, the way every other key in crypton is+ * generated, and the deterministic entry points are what the FFI calls.+ * That also keeps the C free of any opinion about where entropy comes from.+ */+#define MLD_CONFIG_NO_RANDOMIZED_API++/*+ * The hand-written backends, where crypton.cabal says the architecture has+ * them.  Without this the portable C is built, which is correct everywhere+ * and is what every other architecture gets.+ */+#ifdef CRYPTON_MLDSA_NATIVE_BACKEND+#define MLD_CONFIG_USE_NATIVE_BACKEND_ARITH+#define MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202++/*+ * Which of the backend's instructions this processor actually has.+ *+ * Without this both packages decide at compile time: the backend is built+ * because the architecture has the instructions in principle, and is then+ * called with no check.  Upstream says so plainly -- if the machine that+ * runs the binary may differ from the one that built it, this function+ * MUST be provided -- and crypton is a library whose builds are copied+ * between machines and shipped by distributions.  Without it a binary+ * built on a host with AVX2 dies with SIGILL on one without (#316).+ *+ * The dispatch sites are upstream's; each falls back to the portable C+ * when this answers no.  The answers come from crypton's own detection,+ * the same that AES-NI, PCLMUL and the s2n-bignum variants use.+ */+#define MLD_CONFIG_CUSTOM_CAPABILITY_FUNC+#ifndef __ASSEMBLER__+#include "src/sys.h"+#include "crypton_cpu.h"++static MLD_INLINE int mld_sys_check_capability(mld_sys_cap cap)+{+#if defined(__x86_64__)+    if (cap == MLD_SYS_CAP_X86_64_AVX2)+        return (crypton_x86_simd_features() & CRYPTON_X86_AVX2) != 0;+#elif defined(__aarch64__)+    /* NEON is not optional on AArch64; the Keccak instructions are. */+    if (cap == MLD_SYS_CAP_AARCH64_NEON)+        return 1;+    if (cap == MLD_SYS_CAP_AARCH64_SHA3)+        return (crypton_arm_features() & CRYPTON_ARM_SHA3) != 0;+#endif+    (void)cap;+    return 0;+}+#endif /* !__ASSEMBLER__ */+#endif++#endif /* CRYPTON_MLDSA_H */
+ cbits/mldsa/crypton_mldsa_asm.S view
@@ -0,0 +1,14 @@+/*+ * The assembly half, which is level-independent: it is included once, with+ * the shared directives kept, and covers all three parameter sets.  See the+ * comment at the top of mldsa_native_asm.S.+ *+ * Built only where crypton.cabal defines CRYPTON_MLDSA_NATIVE_BACKEND; on+ * every other architecture this file is not in asm-sources at all.+ */+#include "crypton_mldsa.h"++#define MLD_CONFIG_MULTILEVEL_WITH_SHARED 1+#define MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS+#define MLD_CONFIG_PARAMETER_SET 65+#include "mldsa_native_asm.S"
+ cbits/mldsa/import.sh view
@@ -0,0 +1,40 @@+#!/bin/sh+# Re-import the vendored parts of the PQ Code Package's mldsa-native.+#+# The files are kept unmodified.  Everything crypton decides -- which+# parameter sets exist, what the symbols are called, that there is no+# randomised API -- is decided in cbits/mldsa/crypton_mldsa.c and in+# crypton.cabal, not by editing anything here.  Run this from cbits/mldsa:+#+#     ./import.sh [tag-or-commit]+#+# and commit the result together with the COMMIT line it writes, so that the+# tree always says which upstream revision it holds.+set -eu++REPO=https://github.com/pq-code-package/mldsa-native+REV=${1:-v2.0.0}+HERE=$(cd "$(dirname "$0")" && pwd)+TMP=$(mktemp -d)+trap 'rm -rf "$TMP"' EXIT++git clone -q "$REPO" "$TMP/u"+git -C "$TMP/u" checkout -q "$REV"++rm -rf "$HERE/src"+cp "$TMP/u/mldsa/mldsa_native.c" "$HERE/"+cp "$TMP/u/mldsa/mldsa_native.h" "$HERE/"+cp "$TMP/u/mldsa/mldsa_native_asm.S" "$HERE/"+cp "$TMP/u/mldsa/mldsa_native_config.h" "$HERE/"+cp -R "$TMP/u/mldsa/src" "$HERE/src"+cp "$TMP/u/LICENSE" "$HERE/LICENSE"++# The 32-bit Armv8.1-M Keccak, for an architecture crypton does not build+# for.  Every reference to it is behind an MLD_SYS_ guard that cannot be true+# on the ones it does, so dropping it changes no build and keeps unreachable+# assembly out of the release tarball.+rm -rf "$HERE/src/fips202/native/armv81m"++git -C "$TMP/u" rev-parse HEAD > "$HERE/COMMIT"+git -C "$TMP/u" describe --tags --exact-match 2>/dev/null >> "$HERE/COMMIT" || true+echo "imported $(head -1 "$HERE/COMMIT")"
+ cbits/mldsa/mldsa_native.c view
@@ -0,0 +1,803 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mldsa-native repository.+ *          Do not modify it directly.+ */++/******************************************************************************+ *+ * Single compilation unit (SCU) for fixed-level build of mldsa-native+ *+ * This compilation unit bundles together all source files for a build+ * of mldsa-native for a fixed security level (MLDSA-44/65/87).+ *+ * # API+ *+ * The API exposed by this file is described in mldsa_native.h.+ *+ * # Multi-level build+ *+ * If you want an SCU build of mldsa-native with support for multiple security+ * levels, you need to include this file multiple times, and set+ * MLD_CONFIG_MULTILEVEL_WITH_SHARED and MLD_CONFIG_MULTILEVEL_NO_SHARED+ * appropriately. This is exemplified in examples/monolithic_build_multilevel+ * and examples/monolithic_build_multilevel_native.+ *+ * # Configuration+ *+ * The following options from the mldsa-native configuration are relevant:+ *+ * - MLD_CONFIG_FIPS202_CUSTOM_HEADER+ *   Set this option if you use a custom FIPS202 implementation.+ *+ * - MLD_CONFIG_USE_NATIVE_BACKEND_ARITH+ *   Set this option if you want to include the native arithmetic backends+ *   in your build.+ *+ * - MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202+ *   Set this option if you want to include the native FIPS202 backends+ *   in your build.+ *+ * - MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS+ *   Set this option if you want to keep the directives defined in+ *   level-independent headers. This is needed for a multi-level build.+ */++/* If parts of the mldsa-native source tree are not used,+ * consider reducing this header via `unifdef`.+ *+ * Example:+ * ```bash+ * unifdef -UMLD_CONFIG_USE_NATIVE_BACKEND_ARITH mldsa_native.c+ * ```+ */++#include "src/common.h"++#include "src/ct.c"+#include "src/debug.c"+#include "src/packing.c"+#include "src/poly.c"+#include "src/poly_kl.c"+#include "src/polyvec.c"+#include "src/polyvec_lazy.c"+#include "src/sign.c"++#if !defined(MLD_CONFIG_FIPS202_CUSTOM_HEADER)+#include "src/fips202/fips202.c"+#include "src/fips202/fips202x4.c"+#include "src/fips202/keccakf1600.c"+#endif++#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_ARITH)+#if defined(MLD_SYS_AARCH64)+#include "src/native/aarch64/src/aarch64_zetas.c"+#include "src/native/aarch64/src/polyz_unpack_table.c"+#include "src/native/aarch64/src/rej_uniform_eta_table.c"+#include "src/native/aarch64/src/rej_uniform_table.c"+#endif /* MLD_SYS_AARCH64 */+#if defined(MLD_SYS_X86_64)+#include "src/native/x86_64/src/consts.c"+#include "src/native/x86_64/src/rej_uniform_table.c"+#endif+#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_ARITH */++#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202)+#if defined(MLD_SYS_AARCH64)+#include "src/fips202/native/aarch64/src/keccakf1600_round_constants.c"+#endif+#if defined(MLD_SYS_X86_64)+#include "src/fips202/native/x86_64/src/keccakf1600_constants.c"+#endif+#if defined(MLD_SYS_ARMV81M_MVE)+#include "src/fips202/native/armv81m/src/keccak_f1600_x4_mve.c"+#include "src/fips202/native/armv81m/src/keccakf1600_round_constants.c"+#endif+#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 */++/* Macro #undef's+ *+ * The following undefines macros from headers+ * included by the source files imported above.+ *+ * This is to allow building and linking multiple builds+ * of mldsa-native for varying parameter sets through concatenation+ * of this file, as if the files had been compiled separately.+ * If this is not relevant to you, you may remove the following.+ */++/*+ * Undefine macros from MLD_CONFIG_PARAMETER_SET-specific files+ */+/* mldsa/mldsa_native.h */+#undef MLDSA44_BYTES+#undef MLDSA44_CRHBYTES+#undef MLDSA44_PUBLICKEYBYTES+#undef MLDSA44_RNDBYTES+#undef MLDSA44_SECRETKEYBYTES+#undef MLDSA44_SEEDBYTES+#undef MLDSA44_TRBYTES+#undef MLDSA65_BYTES+#undef MLDSA65_CRHBYTES+#undef MLDSA65_PUBLICKEYBYTES+#undef MLDSA65_RNDBYTES+#undef MLDSA65_SECRETKEYBYTES+#undef MLDSA65_SEEDBYTES+#undef MLDSA65_TRBYTES+#undef MLDSA87_BYTES+#undef MLDSA87_CRHBYTES+#undef MLDSA87_PUBLICKEYBYTES+#undef MLDSA87_RNDBYTES+#undef MLDSA87_SECRETKEYBYTES+#undef MLDSA87_SEEDBYTES+#undef MLDSA87_TRBYTES+#undef MLDSA_BYTES+#undef MLDSA_BYTES_+#undef MLDSA_CRHBYTES+#undef MLDSA_PUBLICKEYBYTES+#undef MLDSA_PUBLICKEYBYTES_+#undef MLDSA_RNDBYTES+#undef MLDSA_SECRETKEYBYTES+#undef MLDSA_SECRETKEYBYTES_+#undef MLDSA_SEEDBYTES+#undef MLDSA_TRBYTES+#undef MLD_API_CONCAT+#undef MLD_API_CONCAT_+#undef MLD_API_CONCAT_UNDERSCORE+#undef MLD_API_MUST_CHECK_RETURN_VALUE+#undef MLD_API_NAMESPACE+#undef MLD_API_NAMESPACE_PREFIX+#undef MLD_API_QUALIFIER+#undef MLD_DOMAIN_SEPARATION_MAX_BYTES+#undef MLD_ERR_FAIL+#undef MLD_ERR_INVALID_ARG+#undef MLD_ERR_INVALID_KEY+#undef MLD_ERR_INVALID_SIGNATURE+#undef MLD_ERR_OUT_OF_MEMORY+#undef MLD_ERR_PCT_FAIL+#undef MLD_ERR_RNG_FAIL+#undef MLD_ERR_SIGNING_PAUSED+#undef MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED+#undef MLD_H+#undef MLD_MAX3_+#undef MLD_MAX4_+#undef MLD_PREHASH_NONE+#undef MLD_PREHASH_SHA2_224+#undef MLD_PREHASH_SHA2_256+#undef MLD_PREHASH_SHA2_384+#undef MLD_PREHASH_SHA2_512+#undef MLD_PREHASH_SHA2_512_224+#undef MLD_PREHASH_SHA2_512_256+#undef MLD_PREHASH_SHA3_224+#undef MLD_PREHASH_SHA3_256+#undef MLD_PREHASH_SHA3_384+#undef MLD_PREHASH_SHA3_512+#undef MLD_PREHASH_SHAKE_128+#undef MLD_PREHASH_SHAKE_256+#undef MLD_TOTAL_ALLOC_44+#undef MLD_TOTAL_ALLOC_44_KEYPAIR+#undef MLD_TOTAL_ALLOC_44_KEYPAIR_NO_PCT+#undef MLD_TOTAL_ALLOC_44_KEYPAIR_PCT+#undef MLD_TOTAL_ALLOC_44_PK_FROM_SK+#undef MLD_TOTAL_ALLOC_44_SIGN+#undef MLD_TOTAL_ALLOC_44_VERIFY+#undef MLD_TOTAL_ALLOC_65+#undef MLD_TOTAL_ALLOC_65_KEYPAIR+#undef MLD_TOTAL_ALLOC_65_KEYPAIR_NO_PCT+#undef MLD_TOTAL_ALLOC_65_KEYPAIR_PCT+#undef MLD_TOTAL_ALLOC_65_PK_FROM_SK+#undef MLD_TOTAL_ALLOC_65_SIGN+#undef MLD_TOTAL_ALLOC_65_VERIFY+#undef MLD_TOTAL_ALLOC_87+#undef MLD_TOTAL_ALLOC_87_KEYPAIR+#undef MLD_TOTAL_ALLOC_87_KEYPAIR_NO_PCT+#undef MLD_TOTAL_ALLOC_87_KEYPAIR_PCT+#undef MLD_TOTAL_ALLOC_87_PK_FROM_SK+#undef MLD_TOTAL_ALLOC_87_SIGN+#undef MLD_TOTAL_ALLOC_87_VERIFY+/* mldsa/src/common.h */+#undef MLD_ADD_PARAM_SET+#undef MLD_ALLOC+#undef MLD_APPLY+#undef MLD_ASM_FN_SIZE+#undef MLD_ASM_FN_SYMBOL+#undef MLD_ASM_NAMESPACE+#undef MLD_BUILD_INTERNAL+#undef MLD_COMMON_H+#undef MLD_CONCAT+#undef MLD_CONCAT_+#undef MLD_EMPTY_CU+#undef MLD_ERR_FAIL+#undef MLD_ERR_INVALID_ARG+#undef MLD_ERR_INVALID_KEY+#undef MLD_ERR_INVALID_SIGNATURE+#undef MLD_ERR_OUT_OF_MEMORY+#undef MLD_ERR_PCT_FAIL+#undef MLD_ERR_RNG_FAIL+#undef MLD_ERR_SIGNING_PAUSED+#undef MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED+#undef MLD_EXTERNAL_API+#undef MLD_FIPS202X4_HEADER_FILE+#undef MLD_FIPS202_HEADER_FILE+#undef MLD_FREE+#undef MLD_INTERNAL_API+#undef MLD_INTERNAL_DATA_DECLARATION+#undef MLD_INTERNAL_DATA_DEFINITION+#undef MLD_MULTILEVEL_BUILD+#undef MLD_NAMESPACE+#undef MLD_NAMESPACE_KL+#undef MLD_NAMESPACE_PREFIX+#undef MLD_NAMESPACE_PREFIX_KL+#undef mld_memcpy+#undef mld_memset+/* mldsa/src/packing.h */+#undef MLD_PACKING_H+#undef mld_pack_sig_c+#undef mld_pack_sig_h+#undef mld_pack_sig_z+#undef mld_pack_sk_rho_key_tr_s2+#undef mld_pack_sk_s1+#undef mld_sig_unpack_hints+#undef mld_unpack_pk_t1+#undef mld_unpack_sk+/* mldsa/src/params.h */+#undef MLDSA_BETA+#undef MLDSA_CRHBYTES+#undef MLDSA_CRYPTO_BYTES+#undef MLDSA_CRYPTO_PUBLICKEYBYTES+#undef MLDSA_CRYPTO_SECRETKEYBYTES+#undef MLDSA_CTILDEBYTES+#undef MLDSA_D+#undef MLDSA_ETA+#undef MLDSA_GAMMA1+#undef MLDSA_GAMMA2+#undef MLDSA_GAMMA2_32+#undef MLDSA_GAMMA2_88+#undef MLDSA_K+#undef MLDSA_L+#undef MLDSA_N+#undef MLDSA_OMEGA+#undef MLDSA_PK_END+#undef MLDSA_PK_RHO_BYTES+#undef MLDSA_PK_RHO_OFFSET+#undef MLDSA_PK_T1_BYTES+#undef MLDSA_PK_T1_OFFSET+#undef MLDSA_POLYETA_PACKEDBYTES+#undef MLDSA_POLYT0_PACKEDBYTES+#undef MLDSA_POLYT1_PACKEDBYTES+#undef MLDSA_POLYVECH_PACKEDBYTES+#undef MLDSA_POLYW1_PACKEDBYTES+#undef MLDSA_POLYW1_PACKEDBYTES_32+#undef MLDSA_POLYW1_PACKEDBYTES_88+#undef MLDSA_POLYZ_PACKEDBYTES+#undef MLDSA_Q+#undef MLDSA_Q_HALF+#undef MLDSA_RNDBYTES+#undef MLDSA_SEEDBYTES+#undef MLDSA_SIG_C_BYTES+#undef MLDSA_SIG_C_OFFSET+#undef MLDSA_SIG_END+#undef MLDSA_SIG_H_BYTES+#undef MLDSA_SIG_H_OFFSET+#undef MLDSA_SIG_Z_BYTES+#undef MLDSA_SIG_Z_OFFSET+#undef MLDSA_SK_END+#undef MLDSA_SK_KEY_BYTES+#undef MLDSA_SK_KEY_OFFSET+#undef MLDSA_SK_RHO_BYTES+#undef MLDSA_SK_RHO_OFFSET+#undef MLDSA_SK_S1_BYTES+#undef MLDSA_SK_S1_OFFSET+#undef MLDSA_SK_S2_BYTES+#undef MLDSA_SK_S2_OFFSET+#undef MLDSA_SK_T0_BYTES+#undef MLDSA_SK_T0_OFFSET+#undef MLDSA_SK_TR_BYTES+#undef MLDSA_SK_TR_OFFSET+#undef MLDSA_TAU+#undef MLDSA_TRBYTES+#undef MLD_MAX_KAPPA+#undef MLD_PARAMS_H+/* mldsa/src/poly_kl.h */+#undef MLD_POLYETA_UNPACK_LOWER_BOUND+#undef MLD_POLY_KL_H+#undef mld_poly_challenge+#undef mld_poly_decompose+#undef mld_poly_uniform_eta+#undef mld_poly_uniform_eta_4x+#undef mld_poly_uniform_gamma1+#undef mld_poly_uniform_gamma1_4x+#undef mld_poly_use_hint+#undef mld_polyeta_pack+#undef mld_polyeta_unpack+#undef mld_polyw1_pack+#undef mld_polyz_pack+#undef mld_polyz_unpack+/* mldsa/src/polyvec.h */+#undef MLD_POLYVEC_H+#undef mld_polyveck+#undef mld_polyveck_caddq+#undef mld_polyveck_chknorm+#undef mld_polyveck_decompose+#undef mld_polyveck_invntt_tomont+#undef mld_polyveck_ntt+#undef mld_polyveck_pack_eta+#undef mld_polyveck_pack_w1+#undef mld_polyveck_reduce+#undef mld_polyveck_unpack_eta+#undef mld_polyvecl+#undef mld_polyvecl_chknorm+#undef mld_polyvecl_ntt+#undef mld_polyvecl_pack_eta+#undef mld_polyvecl_pointwise_acc_montgomery+#undef mld_polyvecl_uniform_gamma1+#undef mld_polyvecl_unpack_eta+#undef mld_polyvecl_unpack_z+/* mldsa/src/polyvec_lazy.h */+#undef MLD_POLYVEC_LAZY_H+#undef mld_poly_permute_bitrev_to_custom_optional+#undef mld_polymat+#undef mld_polymat_eager+#undef mld_polymat_lazy+#undef mld_polyvec_matrix_expand+#undef mld_polyvec_matrix_expand_eager+#undef mld_polyvec_matrix_expand_lazy+#undef mld_polyvec_matrix_pointwise_montgomery+#undef mld_polyvec_matrix_pointwise_montgomery_row+#undef mld_polyvec_matrix_pointwise_montgomery_row_eager+#undef mld_polyvec_matrix_pointwise_montgomery_row_lazy+#undef mld_polyvec_matrix_pointwise_montgomery_yvec+#undef mld_polyvec_matrix_pointwise_montgomery_yvec_eager+#undef mld_polyvec_matrix_pointwise_montgomery_yvec_lazy+#undef mld_sk_s1hat+#undef mld_sk_s1hat_eager+#undef mld_sk_s1hat_get_poly+#undef mld_sk_s1hat_get_poly_eager+#undef mld_sk_s1hat_get_poly_lazy+#undef mld_sk_s1hat_lazy+#undef mld_sk_s2hat+#undef mld_sk_s2hat_eager+#undef mld_sk_s2hat_get_poly+#undef mld_sk_s2hat_get_poly_eager+#undef mld_sk_s2hat_get_poly_lazy+#undef mld_sk_s2hat_lazy+#undef mld_sk_t0hat+#undef mld_sk_t0hat_eager+#undef mld_sk_t0hat_get_poly+#undef mld_sk_t0hat_get_poly_eager+#undef mld_sk_t0hat_get_poly_lazy+#undef mld_sk_t0hat_lazy+#undef mld_unpack_sk_s1hat+#undef mld_unpack_sk_s1hat_eager+#undef mld_unpack_sk_s1hat_lazy+#undef mld_unpack_sk_s2hat+#undef mld_unpack_sk_s2hat_eager+#undef mld_unpack_sk_s2hat_lazy+#undef mld_unpack_sk_t0hat+#undef mld_unpack_sk_t0hat_eager+#undef mld_unpack_sk_t0hat_lazy+#undef mld_yvec+#undef mld_yvec_eager+#undef mld_yvec_get_poly+#undef mld_yvec_get_poly_eager+#undef mld_yvec_get_poly_lazy+#undef mld_yvec_init+#undef mld_yvec_init_eager+#undef mld_yvec_init_lazy+#undef mld_yvec_lazy+/* mldsa/src/rounding.h */+#undef MLD_2_POW_D+#undef MLD_ROUNDING_H+#undef mld_decompose+#undef mld_make_hint+#undef mld_power2round+#undef mld_use_hint+/* mldsa/src/sign.h */+#undef MLD_DOMAIN_SEPARATION_MAX_BYTES+#undef MLD_PREHASH_NONE+#undef MLD_PREHASH_SHA2_224+#undef MLD_PREHASH_SHA2_256+#undef MLD_PREHASH_SHA2_384+#undef MLD_PREHASH_SHA2_512+#undef MLD_PREHASH_SHA2_512_224+#undef MLD_PREHASH_SHA2_512_256+#undef MLD_PREHASH_SHA3_224+#undef MLD_PREHASH_SHA3_256+#undef MLD_PREHASH_SHA3_384+#undef MLD_PREHASH_SHA3_512+#undef MLD_PREHASH_SHAKE_128+#undef MLD_PREHASH_SHAKE_256+#undef MLD_SIGN_H+#undef mld_prepare_domain_separation_prefix+#undef mld_sign_keypair+#undef mld_sign_keypair_internal+#undef mld_sign_pk_from_sk+#undef mld_sign_signature+#undef mld_sign_signature_extmu+#undef mld_sign_signature_internal+#undef mld_sign_signature_pre_hash_internal+#undef mld_sign_signature_pre_hash_shake256+#undef mld_sign_verify+#undef mld_sign_verify_extmu+#undef mld_sign_verify_internal+#undef mld_sign_verify_pre_hash_internal+#undef mld_sign_verify_pre_hash_shake256++#if !defined(MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS)+/*+ * Undefine macros from MLD_CONFIG_PARAMETER_SET-generic files+ */+/* mldsa/src/context.h */+#undef MLD_CONTEXT_H+#undef MLD_CONTEXT_PARAMETERS_0+#undef MLD_CONTEXT_PARAMETERS_1+#undef MLD_CONTEXT_PARAMETERS_2+#undef MLD_CONTEXT_PARAMETERS_3+#undef MLD_CONTEXT_PARAMETERS_4+#undef MLD_CONTEXT_PARAMETERS_5+#undef MLD_CONTEXT_PARAMETERS_6+#undef MLD_CONTEXT_PARAMETERS_7+#undef MLD_CONTEXT_PARAMETERS_8+#undef MLD_CONTEXT_PARAMETERS_9+#undef MLD_CONTEXT_UNUSED+#undef mld_sign_attempt+#undef mld_sign_finish+#undef mld_sign_resume+/* mldsa/src/ct.h */+#undef MLD_CT_H+#undef MLD_USE_ASM_VALUE_BARRIER+#undef mld_ct_opt_blocker_u64+/* mldsa/src/debug.h */+#undef MLD_DEBUG_H+#undef mld_assert+#undef mld_assert_abs_bound+#undef mld_assert_abs_bound_2d+#undef mld_assert_bound+#undef mld_assert_bound_2d+#undef mld_debug_check_assert+#undef mld_debug_check_bounds+/* mldsa/src/poly.h */+#undef MLD_FQMUL_BOUND+#undef MLD_INTT_BOUND+#undef MLD_NTT_BOUND+#undef MLD_POLY_H+#undef mld_poly_add+#undef mld_poly_caddq+#undef mld_poly_chknorm+#undef mld_poly_invntt_tomont+#undef mld_poly_ntt+#undef mld_poly_pointwise_montgomery+#undef mld_poly_power2round+#undef mld_poly_reduce+#undef mld_poly_shiftl+#undef mld_poly_sub+#undef mld_poly_uniform+#undef mld_poly_uniform_4x+#undef mld_polyt0_pack+#undef mld_polyt0_unpack+#undef mld_polyt1_pack+#undef mld_polyt1_unpack+#undef mld_polyw1_pack_32+#undef mld_polyw1_pack_88+/* mldsa/src/randombytes.h */+#undef MLD_RANDOMBYTES_H+/* mldsa/src/reduce.h */+#undef MLD_MONT+#undef MLD_REDUCE32_DOMAIN_MAX+#undef MLD_REDUCE32_RANGE_MAX+#undef MLD_REDUCE_H+/* mldsa/src/symmetric.h */+#undef MLD_STREAM128_BLOCKBYTES+#undef MLD_STREAM256_BLOCKBYTES+#undef MLD_SYMMETRIC_H+#undef mld_xof128_absorb_once+#undef mld_xof128_ctx+#undef mld_xof128_init+#undef mld_xof128_release+#undef mld_xof128_squeezeblocks+#undef mld_xof128_x4_absorb+#undef mld_xof128_x4_ctx+#undef mld_xof128_x4_init+#undef mld_xof128_x4_release+#undef mld_xof128_x4_squeezeblocks+#undef mld_xof256_absorb_once+#undef mld_xof256_ctx+#undef mld_xof256_init+#undef mld_xof256_release+#undef mld_xof256_squeezeblocks+#undef mld_xof256_x4_absorb+#undef mld_xof256_x4_ctx+#undef mld_xof256_x4_init+#undef mld_xof256_x4_release+#undef mld_xof256_x4_squeezeblocks+/* mldsa/src/sys.h */+#undef MLD_ALIGN+#undef MLD_ALIGN_UP+#undef MLD_ALWAYS_INLINE+#undef MLD_CET_ENDBR+#undef MLD_CT_TESTING_DECLASSIFY+#undef MLD_CT_TESTING_SECRET+#undef MLD_DEFAULT_ALIGN+#undef MLD_HAVE_INLINE_ASM+#undef MLD_INLINE+#undef MLD_MUST_CHECK_RETURN_VALUE+#undef MLD_NOINLINE+#undef MLD_RESTRICT+#undef MLD_STATIC_TESTABLE+#undef MLD_SYSV_ABI+#undef MLD_SYSV_ABI_SUPPORTED+#undef MLD_SYS_AARCH64+#undef MLD_SYS_AARCH64_EB+#undef MLD_SYS_AARCH64_NEON+#undef MLD_SYS_APPLE+#undef MLD_SYS_ARMV81M_MVE+#undef MLD_SYS_BIG_ENDIAN+#undef MLD_SYS_H+#undef MLD_SYS_LINUX+#undef MLD_SYS_LITTLE_ENDIAN+#undef MLD_SYS_PPC64LE+#undef MLD_SYS_RISCV32+#undef MLD_SYS_RISCV64+#undef MLD_SYS_RISCV64_RVV+#undef MLD_SYS_WINDOWS+#undef MLD_SYS_X86_64+#undef MLD_SYS_X86_64_AVX2+/* mldsa/src/cbmc.h */+#undef MLD_CBMC_H+#undef __contract__+#undef __loop__++#if !defined(MLD_CONFIG_FIPS202_CUSTOM_HEADER)+/*+ * Undefine macros from FIPS-202 files+ */+/* mldsa/src/fips202/fips202.h */+#undef MLD_FIPS202_FIPS202_H+#undef MLD_KECCAK_LANES+#undef SHA3_256_HASHBYTES+#undef SHA3_256_RATE+#undef SHA3_512_HASHBYTES+#undef SHA3_512_RATE+#undef SHAKE128_RATE+#undef SHAKE256_RATE+#undef mld_shake128_absorb+#undef mld_shake128_finalize+#undef mld_shake128_init+#undef mld_shake128_release+#undef mld_shake128_squeeze+#undef mld_shake256+#undef mld_shake256_absorb+#undef mld_shake256_finalize+#undef mld_shake256_init+#undef mld_shake256_release+#undef mld_shake256_squeeze+/* mldsa/src/fips202/fips202x4.h */+#undef MLD_FIPS202_FIPS202X4_H+#undef mld_shake128x4_absorb_once+#undef mld_shake128x4_init+#undef mld_shake128x4_release+#undef mld_shake128x4_squeezeblocks+#undef mld_shake256x4_absorb_once+#undef mld_shake256x4_init+#undef mld_shake256x4_release+#undef mld_shake256x4_squeezeblocks+/* mldsa/src/fips202/keccakf1600.h */+#undef MLD_FIPS202_KECCAKF1600_H+#undef MLD_KECCAK_LANES+#undef MLD_KECCAK_WAY+#undef mld_keccakf1600_extract_bytes+#undef mld_keccakf1600_permute+#undef mld_keccakf1600_xor_bytes+#undef mld_keccakf1600x4_extract_bytes+#undef mld_keccakf1600x4_permute+#undef mld_keccakf1600x4_xor_bytes+#endif /* !MLD_CONFIG_FIPS202_CUSTOM_HEADER */++#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202)+/* mldsa/src/fips202/native/api.h */+#undef MLD_FIPS202_NATIVE_API_H+#undef MLD_NATIVE_FUNC_FALLBACK+#undef MLD_NATIVE_FUNC_SUCCESS+/* mldsa/src/fips202/native/auto.h */+#undef MLD_FIPS202_NATIVE_AUTO_H+#if defined(MLD_SYS_AARCH64)+/*+ * Undefine macros from native code (FIPS202, AArch64)+ */+/* mldsa/src/fips202/native/aarch64/auto.h */+#undef MLD_FIPS202_NATIVE_AARCH64_AUTO_H+/* mldsa/src/fips202/native/aarch64/src/fips202_native_aarch64.h */+#undef MLD_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H+#undef mld_keccak_f1600_x1_scalar_aarch64_asm+#undef mld_keccak_f1600_x1_v84a_aarch64_asm+#undef mld_keccak_f1600_x2_v84a_aarch64_asm+#undef mld_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm+#undef mld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm+#undef mld_keccakf1600_round_constants+/* mldsa/src/fips202/native/aarch64/x1_scalar.h */+#undef MLD_FIPS202_AARCH64_NEED_X1_SCALAR+#undef MLD_FIPS202_NATIVE_AARCH64_X1_SCALAR_H+#undef MLD_USE_NATIVE_FIPS202_X1+/* mldsa/src/fips202/native/aarch64/x1_v84a.h */+#undef MLD_FIPS202_AARCH64_NEED_X1_V84A+#undef MLD_FIPS202_NATIVE_AARCH64_X1_V84A_H+#undef MLD_USE_NATIVE_FIPS202_X1+/* mldsa/src/fips202/native/aarch64/x2_v84a.h */+#undef MLD_FIPS202_AARCH64_NEED_X2_V84A+#undef MLD_FIPS202_NATIVE_AARCH64_X2_V84A_H+#undef MLD_USE_NATIVE_FIPS202_X4+/* mldsa/src/fips202/native/aarch64/x4_v8a_scalar.h */+#undef MLD_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID+#undef MLD_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H+#undef MLD_USE_NATIVE_FIPS202_X4+/* mldsa/src/fips202/native/aarch64/x4_v8a_v84a_scalar.h */+#undef MLD_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID+#undef MLD_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H+#undef MLD_USE_NATIVE_FIPS202_X4+#endif /* MLD_SYS_AARCH64 */+#if defined(MLD_SYS_X86_64)+/*+ * Undefine macros from native code (FIPS202, x86_64)+ */+/* mldsa/src/fips202/native/x86_64/keccak_f1600_x4_avx2.h */+#undef MLD_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H+#undef MLD_FIPS202_X86_64_NEED_X4_AVX2+#undef MLD_USE_NATIVE_FIPS202_X4+/* mldsa/src/fips202/native/x86_64/src/fips202_native_x86_64.h */+#undef MLD_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H+#undef mld_keccak_f1600_x4_avx2_asm+#undef mld_keccak_rho56+#undef mld_keccak_rho8+#undef mld_keccakf1600_round_constants+#endif /* MLD_SYS_X86_64 */+#if defined(MLD_SYS_ARMV81M_MVE)+/*+ * Undefine macros from native code (FIPS202, Armv8.1-M)+ */+/* mldsa/src/fips202/native/armv81m/mve.h */+#undef MLD_FIPS202_ARMV81M_NEED_X4+#undef MLD_FIPS202_NATIVE_ARMV81M+#undef MLD_FIPS202_NATIVE_ARMV81M_MVE_H+#undef MLD_USE_NATIVE_FIPS202_X4+#undef MLD_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES+#undef MLD_USE_NATIVE_FIPS202_X4_XOR_BYTES+#undef mld_keccak_f1600_x4_native_impl+/* mldsa/src/fips202/native/armv81m/src/fips202_native_armv81m.h */+#undef MLD_FIPS202_NATIVE_ARMV81M_SRC_FIPS202_NATIVE_ARMV81M_H+#undef mld_keccak_f1600_x4_mve_asm+#undef mld_keccak_f1600_x4_state_extract_bytes_asm+#undef mld_keccak_f1600_x4_state_xor_bytes_asm+#undef mld_keccakf1600_round_constants+#endif /* MLD_SYS_ARMV81M_MVE */+#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 */+#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_ARITH)+/* mldsa/src/native/api.h */+#undef MLD_FQMUL_BOUND+#undef MLD_INTT_BOUND+#undef MLD_NATIVE_API_H+#undef MLD_NATIVE_FUNC_FALLBACK+#undef MLD_NATIVE_FUNC_SUCCESS+#undef MLD_NTT_BOUND+#undef MLD_REDUCE32_RANGE_MAX+/* mldsa/src/native/meta.h */+#undef MLD_NATIVE_META_H+#if defined(MLD_SYS_AARCH64)+/*+ * Undefine macros from native code (Arith, AArch64)+ */+/* mldsa/src/native/aarch64/meta.h */+#undef MLD_ARITH_BACKEND_AARCH64+#undef MLD_NATIVE_AARCH64_META_H+#undef MLD_USE_NATIVE_INTT+#undef MLD_USE_NATIVE_NTT+#undef MLD_USE_NATIVE_POINTWISE_MONTGOMERY+#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4+#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5+#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7+#undef MLD_USE_NATIVE_POLYZ_UNPACK_17+#undef MLD_USE_NATIVE_POLYZ_UNPACK_19+#undef MLD_USE_NATIVE_POLY_CADDQ+#undef MLD_USE_NATIVE_POLY_CHKNORM+#undef MLD_USE_NATIVE_POLY_DECOMPOSE_32+#undef MLD_USE_NATIVE_POLY_DECOMPOSE_88+#undef MLD_USE_NATIVE_POLY_USE_HINT_32+#undef MLD_USE_NATIVE_POLY_USE_HINT_88+#undef MLD_USE_NATIVE_REJ_UNIFORM+#undef MLD_USE_NATIVE_REJ_UNIFORM_ETA2+#undef MLD_USE_NATIVE_REJ_UNIFORM_ETA4+/* mldsa/src/native/aarch64/src/arith_native_aarch64.h */+#undef MLD_AARCH64_REJ_UNIFORM_ETA2_BUFLEN+#undef MLD_AARCH64_REJ_UNIFORM_ETA4_BUFLEN+#undef MLD_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H+#undef mld_aarch64_intt_zetas_layer123456+#undef mld_aarch64_intt_zetas_layer78+#undef mld_aarch64_ntt_zetas_layer123456+#undef mld_aarch64_ntt_zetas_layer78+#undef mld_intt_aarch64_asm+#undef mld_ntt_aarch64_asm+#undef mld_poly_caddq_aarch64_asm+#undef mld_poly_chknorm_aarch64_asm+#undef mld_poly_decompose_32_aarch64_asm+#undef mld_poly_decompose_88_aarch64_asm+#undef mld_poly_pointwise_montgomery_aarch64_asm+#undef mld_poly_use_hint_32_aarch64_asm+#undef mld_poly_use_hint_88_aarch64_asm+#undef mld_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm+#undef mld_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm+#undef mld_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm+#undef mld_polyz_unpack_17_aarch64_asm+#undef mld_polyz_unpack_17_indices+#undef mld_polyz_unpack_19_aarch64_asm+#undef mld_polyz_unpack_19_indices+#undef mld_rej_uniform_aarch64_asm+#undef mld_rej_uniform_eta2_aarch64_asm+#undef mld_rej_uniform_eta4_aarch64_asm+#undef mld_rej_uniform_eta_table+#undef mld_rej_uniform_table+#endif /* MLD_SYS_AARCH64 */+#if defined(MLD_SYS_X86_64)+/*+ * Undefine macros from native code (Arith, X86_64)+ */+/* mldsa/src/native/x86_64/meta.h */+#undef MLD_ARITH_BACKEND_X86_64_DEFAULT+#undef MLD_NATIVE_X86_64_META_H+#undef MLD_USE_NATIVE_INTT+#undef MLD_USE_NATIVE_NTT+#undef MLD_USE_NATIVE_NTT_CUSTOM_ORDER+#undef MLD_USE_NATIVE_POINTWISE_MONTGOMERY+#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4+#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5+#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7+#undef MLD_USE_NATIVE_POLYZ_UNPACK_17+#undef MLD_USE_NATIVE_POLYZ_UNPACK_19+#undef MLD_USE_NATIVE_POLY_CADDQ+#undef MLD_USE_NATIVE_POLY_CHKNORM+#undef MLD_USE_NATIVE_POLY_DECOMPOSE_32+#undef MLD_USE_NATIVE_POLY_DECOMPOSE_88+#undef MLD_USE_NATIVE_POLY_USE_HINT_32+#undef MLD_USE_NATIVE_POLY_USE_HINT_88+#undef MLD_USE_NATIVE_REJ_UNIFORM+#undef MLD_USE_NATIVE_REJ_UNIFORM_ETA2+#undef MLD_USE_NATIVE_REJ_UNIFORM_ETA4+/* mldsa/src/native/x86_64/src/arith_native_x86_64.h */+#undef MLD_AVX2_REJ_UNIFORM_BUFLEN+#undef MLD_AVX2_REJ_UNIFORM_ETA2_BUFLEN+#undef MLD_AVX2_REJ_UNIFORM_ETA4_BUFLEN+#undef MLD_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H+#undef mld_invntt_avx2_asm+#undef mld_ntt_avx2_asm+#undef mld_nttunpack_avx2_asm+#undef mld_pointwise_acc_l4_avx2_asm+#undef mld_pointwise_acc_l5_avx2_asm+#undef mld_pointwise_acc_l7_avx2_asm+#undef mld_pointwise_avx2_asm+#undef mld_poly_caddq_avx2_asm+#undef mld_poly_chknorm_avx2_asm+#undef mld_poly_decompose_32_avx2_asm+#undef mld_poly_decompose_88_avx2_asm+#undef mld_poly_use_hint_32_avx2_asm+#undef mld_poly_use_hint_88_avx2_asm+#undef mld_polyz_unpack_17_avx2_asm+#undef mld_polyz_unpack_19_avx2_asm+#undef mld_rej_uniform_avx2_asm+#undef mld_rej_uniform_eta2_avx2_asm+#undef mld_rej_uniform_eta4_avx2_asm+#undef mld_rej_uniform_table+/* mldsa/src/native/x86_64/src/consts.h */+#undef MLD_AVX2_BACKEND_DATA_OFFSET_8XDIV+#undef MLD_AVX2_BACKEND_DATA_OFFSET_8XDIV_QINV+#undef MLD_AVX2_BACKEND_DATA_OFFSET_8XQ+#undef MLD_AVX2_BACKEND_DATA_OFFSET_8XQINV+#undef MLD_AVX2_BACKEND_DATA_OFFSET_ZETAS+#undef MLD_AVX2_BACKEND_DATA_OFFSET_ZETAS_QINV+#undef MLD_NATIVE_X86_64_SRC_CONSTS_H+#undef mld_qdata+#endif /* MLD_SYS_X86_64 */+#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_ARITH */+#endif /* !MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS */
+ cbits/mldsa/mldsa_native.h view
@@ -0,0 +1,956 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ */++#ifndef MLD_H+#define MLD_H++/*+ * Public API for mldsa-native+ *+ * This header defines the public API of a single build of mldsa-native.+ *+ * Make sure the configuration file is in the include path+ * (this is "mldsa_native_config.h" by default, or MLD_CONFIG_FILE if defined).+ *+ * # API conventions+ *+ * Conventions shared by all functions below (return values, pointer validity,+ * output buffers on error) are documented in API-CONVENTIONS.md.+ *+ * # Multi-level builds+ *+ * This header specifies a build of mldsa-native for a fixed security level.+ * If you need multiple security levels, leave the security level unspecified+ * in the configuration file and include this header multiple times, setting+ * MLD_CONFIG_PARAMETER_SET accordingly for each, and #undef'ing the MLD_H+ * guard to allow multiple inclusions.+ */++/******************************* Key sizes ************************************/++/* Sizes of cryptographic material, per parameter set */+/* See mldsa/src/params.h for the arithmetic expressions giving rise to these */+/* check-magic: off */+#define MLDSA44_SECRETKEYBYTES 2560+#define MLDSA44_PUBLICKEYBYTES 1312+#define MLDSA44_BYTES 2420++#define MLDSA65_SECRETKEYBYTES 4032+#define MLDSA65_PUBLICKEYBYTES 1952+#define MLDSA65_BYTES 3309++#define MLDSA87_SECRETKEYBYTES 4896+#define MLDSA87_PUBLICKEYBYTES 2592+#define MLDSA87_BYTES 4627+/* check-magic: on */++/* Size of seed and randomness in bytes (level-independent) */+#define MLDSA_SEEDBYTES 32+#define MLDSA44_SEEDBYTES MLDSA_SEEDBYTES+#define MLDSA65_SEEDBYTES MLDSA_SEEDBYTES+#define MLDSA87_SEEDBYTES MLDSA_SEEDBYTES++/* Size of CRH output in bytes (level-independent) */+#define MLDSA_CRHBYTES 64+#define MLDSA44_CRHBYTES MLDSA_CRHBYTES+#define MLDSA65_CRHBYTES MLDSA_CRHBYTES+#define MLDSA87_CRHBYTES MLDSA_CRHBYTES++/* Size of TR in bytes (level-independent)+ *+ * TR = SHAKE256(pk, 64) is the hash of the public key. Callers of the+ * external-mu API (signature_extmu / verify_extmu) that compute the message+ * representative mu = SHAKE256(TR || M', MLDSA_CRHBYTES) themselves -- e.g. to+ * sign or verify a message that is too large or streamed to hold in memory --+ * need this constant to size the TR buffer. */+#define MLDSA_TRBYTES 64+#define MLDSA44_TRBYTES MLDSA_TRBYTES+#define MLDSA65_TRBYTES MLDSA_TRBYTES+#define MLDSA87_TRBYTES MLDSA_TRBYTES++/* Size of randomness for signing in bytes (level-independent) */+#define MLDSA_RNDBYTES 32+#define MLDSA44_RNDBYTES MLDSA_RNDBYTES+#define MLDSA65_RNDBYTES MLDSA_RNDBYTES+#define MLDSA87_RNDBYTES MLDSA_RNDBYTES++/* Sizes of cryptographic material, as a function of LVL=44,65,87 */+#define MLDSA_SECRETKEYBYTES_(LVL) MLDSA##LVL##_SECRETKEYBYTES+#define MLDSA_PUBLICKEYBYTES_(LVL) MLDSA##LVL##_PUBLICKEYBYTES+#define MLDSA_BYTES_(LVL) MLDSA##LVL##_BYTES+#define MLDSA_SECRETKEYBYTES(LVL) MLDSA_SECRETKEYBYTES_(LVL)+#define MLDSA_PUBLICKEYBYTES(LVL) MLDSA_PUBLICKEYBYTES_(LVL)+#define MLDSA_BYTES(LVL) MLDSA_BYTES_(LVL)++/****************************** Error codes ***********************************/++/* Generic failure condition, reserved for failures not covered by a more+ * specific error code. */+#define MLD_ERR_FAIL (-1)+/* An allocation failed. This can only happen if MLD_CONFIG_CUSTOM_ALLOC_FREE+ * is defined and the provided MLD_CUSTOM_ALLOC can fail. */+#define MLD_ERR_OUT_OF_MEMORY (-2)+/* An RNG failure occurred. Might be due to insufficient entropy or+ * system misconfiguration. */+#define MLD_ERR_RNG_FAIL (-3)+/* The signing rejection-sampling loop exceeded+ * MLD_CONFIG_MAX_SIGNING_ATTEMPTS iterations without producing a valid+ * signature. With a FIPS 204 Appendix C compliant bound (>= 821) this+ * has probability < 2^-256. */+#define MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED (-4)+/* Signing was paused before completing, at the request of a caller-provided+ * MLD_CONFIG_SIGN_HOOK_ATTEMPT hook (see mldsa_native_config.h). The caller+ * resumes by re-invoking signing with the same inputs; the attempt hook,+ * together with MLD_CONFIG_SIGN_HOOK_RESUME, decides where to continue. */+#define MLD_ERR_SIGNING_PAUSED (-5)+/* Signature verification failed: the signature is not valid for the given+ * message and public key. Returned by the verification API. */+#define MLD_ERR_INVALID_SIGNATURE (-6)+/* Secret key validation failed: the secret key is malformed or internally+ * inconsistent. Returned by pk_from_sk. */+#define MLD_ERR_INVALID_KEY (-7)+/* The Pairwise Consistency Test failed. Only possible when+ * MLD_CONFIG_KEYGEN_PCT is enabled; signals that the freshly generated key+ * pair failed its sign/verify self-test. */+#define MLD_ERR_PCT_FAIL (-8)+/* An argument was invalid, e.g. an unsupported pre-hash algorithm or a context+ * string longer than 255 bytes. */+#define MLD_ERR_INVALID_ARG (-9)++/********************* Namespacing and Qualifiers *****************************/++#define MLD_API_CONCAT_(x, y) x##y+#define MLD_API_CONCAT(x, y) MLD_API_CONCAT_(x, y)+#define MLD_API_CONCAT_UNDERSCORE(x, y) MLD_API_CONCAT(MLD_API_CONCAT(x, _), y)++/* You need to make sure the config file is in the include path. */+#if defined(MLD_CONFIG_FILE)+#include MLD_CONFIG_FILE+#else+#include "mldsa_native_config.h"+#endif++/* Namespace prefix for the public API symbols. For multi-level builds, the+ * parameter set is appended to disambiguate the security levels. */+#if defined(MLD_CONFIG_MULTILEVEL_BUILD)+#define MLD_API_NAMESPACE_PREFIX \+  MLD_API_CONCAT(MLD_CONFIG_NAMESPACE_PREFIX, MLD_CONFIG_PARAMETER_SET)+#else+#define MLD_API_NAMESPACE_PREFIX MLD_CONFIG_NAMESPACE_PREFIX+#endif++#define MLD_API_NAMESPACE(sym) \+  MLD_API_CONCAT_UNDERSCORE(MLD_API_NAMESPACE_PREFIX, sym)++#if defined(__GNUC__) || defined(__clang__)+#define MLD_API_MUST_CHECK_RETURN_VALUE __attribute__((warn_unused_result))+#else+#define MLD_API_MUST_CHECK_RETURN_VALUE+#endif++#if defined(MLD_CONFIG_EXTERNAL_API_QUALIFIER)+#define MLD_API_QUALIFIER MLD_CONFIG_EXTERNAL_API_QUALIFIER+#else+#define MLD_API_QUALIFIER+#endif++/* Hash algorithm constants for domain separation */+#define MLD_PREHASH_NONE 0+#define MLD_PREHASH_SHA2_224 1+#define MLD_PREHASH_SHA2_256 2+#define MLD_PREHASH_SHA2_384 3+#define MLD_PREHASH_SHA2_512 4+#define MLD_PREHASH_SHA2_512_224 5+#define MLD_PREHASH_SHA2_512_256 6+#define MLD_PREHASH_SHA3_224 7+#define MLD_PREHASH_SHA3_256 8+#define MLD_PREHASH_SHA3_384 9+#define MLD_PREHASH_SHA3_512 10+#define MLD_PREHASH_SHAKE_128 11+#define MLD_PREHASH_SHAKE_256 12++/* Maximum formatted domain separation message length */+#define MLD_DOMAIN_SEPARATION_MAX_BYTES (2 + 255 + 11 + 64)++/****************************** Function API **********************************/++#if !defined(MLD_CONFIG_CONSTANTS_ONLY)++#include <stddef.h>+#include <stdint.h>++#ifdef __cplusplus+extern "C"+{+#endif++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+/**+ * Generate a public-private key pair from a seed.+ *+ * When MLD_CONFIG_KEYGEN_PCT is set, performs a Pairwise Consistency Test+ * (PCT) as required by FIPS 140-3 IG.+ *+ * @warning The seed must be generated by a cryptographically secure random+ *          number generator.+ *+ * @spec{Implements @[FIPS204, Algorithm 6, ML-DSA.KeyGen_internal].}+ *+ * @param[out] pk      Output public key.+ * @param[out] sk      Output private key.+ * @param[in]  seed    Input random seed.+ * @param      context Application context. Only present when+ *                     MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                               Success.+ * @retval MLD_ERR_OUT_OF_MEMORY           MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                         used and an allocation via+ *                                         MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_RNG_FAIL                Random number generation failed+ *                                         during the PCT. Only possible when+ *                                         MLD_CONFIG_KEYGEN_PCT is enabled.+ * @retval MLD_ERR_SIGNING_PAUSED          The PCT's signing step was paused by+ *                                         a MLD_CONFIG_SIGN_HOOK_ATTEMPT hook.+ *                                         This should currently never happen:+ *                                         signing hooks require+ *                                         MLD_CONFIG_NO_RANDOMIZED_API, which+ *                                         is incompatible with+ *                                         MLD_CONFIG_KEYGEN_PCT, so the two+ *                                         cannot be enabled simultaneously.+ * @retval MLD_ERR_PCT_FAIL                MLD_CONFIG_KEYGEN_PCT is enabled and+ *                                         the PCT check failed.+ */+MLD_API_QUALIFIER+MLD_API_MUST_CHECK_RETURN_VALUE+int MLD_API_NAMESPACE(keypair_internal)(+    uint8_t pk[MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET)],+    uint8_t sk[MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET)],+    const uint8_t seed[MLDSA_SEEDBYTES]+#ifdef MLD_CONFIG_CONTEXT_PARAMETER+    ,+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);++#if !defined(MLD_CONFIG_CORE_API_ONLY)+#if !defined(MLD_CONFIG_NO_RANDOMIZED_API)+/**+ * Generate a public-private key pair.+ *+ * When MLD_CONFIG_KEYGEN_PCT is set, performs a Pairwise Consistency Test+ * (PCT) as required by FIPS 140-3 IG.+ *+ * @spec{Implements @[FIPS204, Algorithm 1, ML-DSA.KeyGen].}+ *+ * @param[out] pk      Output public key.+ * @param[out] sk      Output private key.+ * @param      context Application context. Only present when+ *                     MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                               Success.+ * @retval MLD_ERR_OUT_OF_MEMORY           MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                         used and an allocation via+ *                                         MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_RNG_FAIL                Random number generation failed.+ * @retval MLD_ERR_SIGNING_PAUSED          The PCT's signing step was paused by+ *                                         a MLD_CONFIG_SIGN_HOOK_ATTEMPT hook.+ *                                         This should currently never happen:+ *                                         signing hooks require+ *                                         MLD_CONFIG_NO_RANDOMIZED_API, which+ *                                         is incompatible with+ *                                         MLD_CONFIG_KEYGEN_PCT, so the two+ *                                         cannot be enabled simultaneously.+ * @retval MLD_ERR_PCT_FAIL                MLD_CONFIG_KEYGEN_PCT is enabled and+ *                                         the PCT check failed.+ */+MLD_API_QUALIFIER+MLD_API_MUST_CHECK_RETURN_VALUE+int MLD_API_NAMESPACE(keypair)(+    uint8_t pk[MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET)],+    uint8_t sk[MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET)]+#ifdef MLD_CONFIG_CONTEXT_PARAMETER+    ,+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);+#endif /* !MLD_CONFIG_NO_RANDOMIZED_API */+#endif /* !MLD_CONFIG_CORE_API_ONLY */+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+/**+ * Compute signature using a caller-supplied random seed and prefix.+ *+ * On error (non-zero return value), the signature buffer sig is zeroized.+ *+ * @spec{Implements @[FIPS204, Algorithm 7, ML-DSA.Sign_internal].}+ *+ * @warning This function does not perform secret key validation.+ *          Callers importing serialized keys can use pk_from_sk+ *          to validate them before signing.+ *+ * @param[out] sig        Pointer to buffer to hold the generated signature of+ *                        MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes.+ * @param[in]  m          Pointer to message to be signed (when+ *                        externalmu == 0), or to a precomputed+ *                        message representative mu (when externalmu != 0).+ * @param      mlen       Length of m. Must equal MLDSA_CRHBYTES when+ *                        externalmu != 0.+ * @param[in]  pre        Pointer to prefix string. Ignored when+ *                        externalmu != 0.+ * @param      prelen     Length of prefix string. Ignored when+ *                        externalmu != 0.+ * @param[in]  rnd        Random seed.+ * @param[in]  sk         Bit-packed secret key; assumed to be valid.+ * @param      externalmu 0: m/mlen is the raw message; mu = H(tr, pre, m) is+ *                        computed internally.+ *                        non-zero: m points to a precomputed mu of+ *                        MLDSA_CRHBYTES bytes; pre/prelen unused.+ * @param      context    Application context. Only present when+ *                        MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                        MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                               Success.+ * @retval MLD_ERR_OUT_OF_MEMORY           MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                         used and an allocation via+ *                                         MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded+ *                                         MLD_CONFIG_MAX_SIGNING_ATTEMPTS+ *                                         iterations.+ * @retval MLD_ERR_SIGNING_PAUSED          A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook+ *                                         paused signing; re-invoke to resume.+ */+MLD_API_QUALIFIER+MLD_API_MUST_CHECK_RETURN_VALUE+int MLD_API_NAMESPACE(signature_internal)(+    uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], const uint8_t *m,+    size_t mlen, const uint8_t *pre, size_t prelen,+    const uint8_t rnd[MLDSA_RNDBYTES],+    const uint8_t sk[MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET)],+    int externalmu+#ifdef MLD_CONFIG_CONTEXT_PARAMETER+    ,+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);++#if !defined(MLD_CONFIG_CORE_API_ONLY)+#if !defined(MLD_CONFIG_NO_RANDOMIZED_API)+/**+ * Compute signature. This function implements the randomized variant of+ * ML-DSA. If you require the deterministic variant, use+ * signature_internal directly.+ *+ * @spec{Implements @[FIPS204, Algorithm 2, ML-DSA.Sign].}+ *+ * @warning This function does not perform secret key validation.+ *          Callers importing serialized keys can use pk_from_sk+ *          to validate them before signing.+ *+ * @param[out] sig     Pointer to buffer to hold the generated signature of+ *                     MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes.+ * @param[in]  m       Pointer to message to be signed. May be NULL if+ *                     mlen == 0.+ * @param      mlen    Length of message.+ * @param[in]  ctx     Pointer to context string. May be NULL if ctxlen == 0.+ * @param      ctxlen  Length of context string. Should be <= 255.+ * @param[in]  sk      Bit-packed secret key; assumed to be valid.+ * @param      context Application context. Only present when+ *                     MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                               Success.+ * @retval MLD_ERR_OUT_OF_MEMORY           MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                         used and an allocation via+ *                                         MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_RNG_FAIL                Random number generation failed.+ * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded+ *                                         MLD_CONFIG_MAX_SIGNING_ATTEMPTS+ *                                         iterations.+ * @retval MLD_ERR_SIGNING_PAUSED          A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook+ *                                         paused signing; re-invoke to resume.+ * @retval MLD_ERR_INVALID_ARG             The context string exceeded 255+ *                                         bytes.+ */+MLD_API_QUALIFIER+MLD_API_MUST_CHECK_RETURN_VALUE+int MLD_API_NAMESPACE(signature)(+    uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], const uint8_t *m,+    size_t mlen, const uint8_t *ctx, size_t ctxlen,+    const uint8_t sk[MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET)]+#ifdef MLD_CONFIG_CONTEXT_PARAMETER+    ,+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);++/**+ * Compute signature in "external mu" mode: the caller has already computed+ * the message representative mu = SHAKE256(tr || M', 64), where+ * tr = SHAKE256(pk, 64) and M' is the FIPS 204 formatted message (e.g.+ * 0x00 || ctxlen || ctx || msg for pure ML-DSA). This is useful when the+ * message is large or streamed and cannot be held in memory.+ *+ * @spec{Implements @[FIPS204, Algorithm 2, ML-DSA.Sign external mu variant].}+ *+ * @warning This function does not perform secret key validation.+ *          Callers importing serialized keys can use pk_from_sk+ *          to validate them before signing.+ *+ * @param[out] sig     Pointer to buffer to hold the generated signature of+ *                     MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes.+ * @param[in]  mu      Precomputed message representative.+ * @param[in]  sk      Bit-packed secret key; assumed to be valid.+ * @param      context Application context. Only present when+ *                     MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                               Success.+ * @retval MLD_ERR_OUT_OF_MEMORY           MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                         used and an allocation via+ *                                         MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_RNG_FAIL                Random number generation failed.+ * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded+ *                                         MLD_CONFIG_MAX_SIGNING_ATTEMPTS+ *                                         iterations.+ * @retval MLD_ERR_SIGNING_PAUSED          A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook+ *                                         paused signing; re-invoke to resume.+ */+MLD_API_QUALIFIER+MLD_API_MUST_CHECK_RETURN_VALUE+int MLD_API_NAMESPACE(signature_extmu)(+    uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)],+    const uint8_t mu[MLDSA_CRHBYTES],+    const uint8_t sk[MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET)]+#ifdef MLD_CONFIG_CONTEXT_PARAMETER+    ,+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);++#endif /* !MLD_CONFIG_NO_RANDOMIZED_API */+#endif /* !MLD_CONFIG_CORE_API_ONLY */+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+/**+ * Verify signature. Internal API.+ *+ * @spec{Implements @[FIPS204, Algorithm 8, ML-DSA.Verify_internal].}+ *+ * @param[in] sig        Pointer to input signature of+ *                       MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes.+ * @param[in] m          Pointer to message (when externalmu == 0), or to a+ *                       precomputed message representative mu (when+ *                       externalmu != 0).+ * @param     mlen       Length of m. Must equal MLDSA_CRHBYTES when+ *                       externalmu != 0.+ * @param[in] pre        Pointer to prefix string. Ignored when externalmu != 0.+ * @param     prelen     Length of prefix string. Ignored when externalmu != 0.+ * @param[in] pk         Bit-packed public key.+ * @param     externalmu 0: m/mlen is the raw message; mu = H(H(pk), pre, m) is+ *                       computed internally.+ *                       non-zero: m points to a precomputed mu of+ *                       MLDSA_CRHBYTES bytes; pre/prelen unused.+ * @param     context    Application context. Only present when+ *                       MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                       MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                         Success.+ * @retval MLD_ERR_OUT_OF_MEMORY     MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                   used and an allocation via+ *                                   MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed.+ */+MLD_API_QUALIFIER+MLD_API_MUST_CHECK_RETURN_VALUE+int MLD_API_NAMESPACE(verify_internal)(+    const uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], const uint8_t *m,+    size_t mlen, const uint8_t *pre, size_t prelen,+    const uint8_t pk[MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET)],+    int externalmu+#ifdef MLD_CONFIG_CONTEXT_PARAMETER+    ,+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);++#if !defined(MLD_CONFIG_CORE_API_ONLY)+/**+ * Verify signature.+ *+ * @spec{Implements @[FIPS204, Algorithm 3, ML-DSA.Verify].}+ *+ * @param[in] sig     Pointer to input signature of+ *                    MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes.+ * @param[in] m       Pointer to message. May be NULL if mlen == 0.+ * @param     mlen    Length of message.+ * @param[in] ctx     Pointer to context string. May be NULL if ctxlen == 0.+ * @param     ctxlen  Length of context string.+ * @param[in] pk      Bit-packed public key.+ * @param     context Application context. Only present when+ *                    MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                    MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                         Success.+ * @retval MLD_ERR_OUT_OF_MEMORY     MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                   used and an allocation via+ *                                   MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed.+ * @retval MLD_ERR_INVALID_ARG       The context string exceeded 255 bytes.+ */+MLD_API_QUALIFIER+MLD_API_MUST_CHECK_RETURN_VALUE+int MLD_API_NAMESPACE(verify)(+    const uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], const uint8_t *m,+    size_t mlen, const uint8_t *ctx, size_t ctxlen,+    const uint8_t pk[MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET)]+#ifdef MLD_CONFIG_CONTEXT_PARAMETER+    ,+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);++/**+ * Verify signature in "external mu" mode: the caller has already computed+ * the message representative mu = SHAKE256(tr || M', 64), where+ * tr = SHAKE256(pk, 64) and M' is the FIPS 204 formatted message (e.g.+ * 0x00 || ctxlen || ctx || msg for pure ML-DSA). The same mu must have+ * been used at signing time.+ *+ * @spec{Implements @[FIPS204, Algorithm 3, ML-DSA.Verify external mu variant].}+ *+ * @param[in] sig     Pointer to input signature of+ *                    MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes.+ * @param[in] mu      Precomputed message representative.+ * @param[in] pk      Bit-packed public key.+ * @param     context Application context. Only present when+ *                    MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                    MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                         Success.+ * @retval MLD_ERR_OUT_OF_MEMORY     MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                   used and an allocation via+ *                                   MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed.+ */+MLD_API_QUALIFIER+MLD_API_MUST_CHECK_RETURN_VALUE+int MLD_API_NAMESPACE(verify_extmu)(+    const uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)],+    const uint8_t mu[MLDSA_CRHBYTES],+    const uint8_t pk[MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET)]+#ifdef MLD_CONFIG_CONTEXT_PARAMETER+    ,+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);++#endif /* !MLD_CONFIG_CORE_API_ONLY */+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_CORE_API_ONLY)+#if !defined(MLD_CONFIG_NO_SIGN_API)+/**+ * Compute signature with pre-hashed message.+ *+ * @spec{Implements @[FIPS204, Algorithm 4, HashML-DSA.Sign].}+ *+ * Supported hash algorithm constants:+ *   MLD_PREHASH_SHA2_224, MLD_PREHASH_SHA2_256, MLD_PREHASH_SHA2_384,+ *   MLD_PREHASH_SHA2_512, MLD_PREHASH_SHA2_512_224, MLD_PREHASH_SHA2_512_256,+ *   MLD_PREHASH_SHA3_224, MLD_PREHASH_SHA3_256, MLD_PREHASH_SHA3_384,+ *   MLD_PREHASH_SHA3_512, MLD_PREHASH_SHAKE_128, MLD_PREHASH_SHAKE_256.+ *+ * MLD_PREHASH_NONE is rejected by this API.+ *+ * @warning This is an unstable API that may change in the future. If you need+ * a stable API use signature_pre_hash_shake256.+ *+ * @warning This function does not perform secret key validation.+ *          Callers importing serialized keys can use pk_from_sk+ *          to validate them before signing.+ *+ * @param[out] sig     Pointer to buffer to hold the generated signature of+ *                     MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes.+ * @param[in]  ph      Pointer to pre-hashed message.+ * @param      phlen   Length of pre-hashed message. Must match the output+ *                     length of hashalg (the digest size for SHA-2/SHA-3,+ *                     32 bytes for MLD_PREHASH_SHAKE_128, 64 bytes for+ *                     MLD_PREHASH_SHAKE_256).+ * @param[in]  ctx     Pointer to context string. May be NULL if ctxlen == 0.+ * @param      ctxlen  Length of context string.+ * @param[in]  rnd     Random seed.+ * @param[in]  sk      Bit-packed secret key; assumed to be valid.+ * @param      hashalg Hash algorithm constant (one of MLD_PREHASH_*).+ * @param      context Application context. Only present when+ *                     MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                               Success.+ * @retval MLD_ERR_OUT_OF_MEMORY           MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                         used and an allocation via+ *                                         MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded+ *                                         MLD_CONFIG_MAX_SIGNING_ATTEMPTS+ *                                         iterations.+ * @retval MLD_ERR_SIGNING_PAUSED          A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook+ *                                         paused signing; re-invoke to resume.+ * @retval MLD_ERR_INVALID_ARG             The pre-hash algorithm was+ *                                         MLD_PREHASH_NONE or unsupported,+ *                                         phlen did not match the output+ *                                         length of hashalg, or the context+ *                                         string exceeded 255 bytes.+ */+MLD_API_QUALIFIER+MLD_API_MUST_CHECK_RETURN_VALUE+int MLD_API_NAMESPACE(signature_pre_hash_internal)(+    uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], const uint8_t *ph,+    size_t phlen, const uint8_t *ctx, size_t ctxlen,+    const uint8_t rnd[MLDSA_RNDBYTES],+    const uint8_t sk[MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET)],+    int hashalg+#ifdef MLD_CONFIG_CONTEXT_PARAMETER+    ,+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+/**+ * Verifies signature with pre-hashed message.+ *+ * @spec{Implements @[FIPS204, Algorithm 5, HashML-DSA.Verify].}+ *+ * Supported hash algorithm constants:+ *   MLD_PREHASH_SHA2_224, MLD_PREHASH_SHA2_256, MLD_PREHASH_SHA2_384,+ *   MLD_PREHASH_SHA2_512, MLD_PREHASH_SHA2_512_224, MLD_PREHASH_SHA2_512_256,+ *   MLD_PREHASH_SHA3_224, MLD_PREHASH_SHA3_256, MLD_PREHASH_SHA3_384,+ *   MLD_PREHASH_SHA3_512, MLD_PREHASH_SHAKE_128, MLD_PREHASH_SHAKE_256.+ *+ * MLD_PREHASH_NONE is rejected by this API.+ *+ * @warning This is an unstable API that may change in the future. If you need+ * a stable API use verify_pre_hash_shake256.+ *+ * @param[in] sig     Pointer to input signature of+ *                    MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes.+ * @param[in] ph      Pointer to pre-hashed message.+ * @param     phlen   Length of pre-hashed message. Must match the output+ *                    length of hashalg (the digest size for SHA-2/SHA-3,+ *                    32 bytes for MLD_PREHASH_SHAKE_128, 64 bytes for+ *                    MLD_PREHASH_SHAKE_256).+ * @param[in] ctx     Pointer to context string. May be NULL if ctxlen == 0.+ * @param     ctxlen  Length of context string.+ * @param[in] pk      Bit-packed public key.+ * @param     hashalg Hash algorithm constant (one of MLD_PREHASH_*).+ * @param     context Application context. Only present when+ *                    MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                    MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                         Success.+ * @retval MLD_ERR_OUT_OF_MEMORY     MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                   used and an allocation via+ *                                   MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed.+ * @retval MLD_ERR_INVALID_ARG       The pre-hash algorithm was+ *                                   MLD_PREHASH_NONE or unsupported, phlen+ *                                   did not match the output length of+ *                                   hashalg, or the context string exceeded+ *                                   255 bytes.+ */+MLD_API_QUALIFIER+MLD_API_MUST_CHECK_RETURN_VALUE+int MLD_API_NAMESPACE(verify_pre_hash_internal)(+    const uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], const uint8_t *ph,+    size_t phlen, const uint8_t *ctx, size_t ctxlen,+    const uint8_t pk[MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET)],+    int hashalg+#ifdef MLD_CONFIG_CONTEXT_PARAMETER+    ,+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+/**+ * Compute signature with pre-hashed message using SHAKE256. This function+ * computes the SHAKE256 hash of the message internally.+ *+ * @spec{Implements @[FIPS204, Algorithm 4, HashML-DSA.Sign] with SHAKE256 as+ * the pre-hash.}+ *+ * @warning This function does not perform secret key validation.+ *          Callers importing serialized keys can use pk_from_sk+ *          to validate them before signing.+ *+ * @param[out] sig     Pointer to buffer to hold the generated signature of+ *                     MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes.+ * @param[in]  m       Pointer to message to be hashed and signed. May be+ *                     NULL if mlen == 0.+ * @param      mlen    Length of message.+ * @param[in]  ctx     Pointer to context string. May be NULL if ctxlen == 0.+ * @param      ctxlen  Length of context string.+ * @param[in]  rnd     Random seed.+ * @param[in]  sk      Bit-packed secret key; assumed to be valid.+ * @param      context Application context. Only present when+ *                     MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                               Success.+ * @retval MLD_ERR_OUT_OF_MEMORY           MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                         used and an allocation via+ *                                         MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded+ *                                         MLD_CONFIG_MAX_SIGNING_ATTEMPTS+ *                                         iterations.+ * @retval MLD_ERR_SIGNING_PAUSED          A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook+ *                                         paused signing; re-invoke to resume.+ * @retval MLD_ERR_INVALID_ARG             The context string exceeded 255+ *                                         bytes.+ */+MLD_API_QUALIFIER+MLD_API_MUST_CHECK_RETURN_VALUE+int MLD_API_NAMESPACE(signature_pre_hash_shake256)(+    uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], const uint8_t *m,+    size_t mlen, const uint8_t *ctx, size_t ctxlen,+    const uint8_t rnd[MLDSA_RNDBYTES],+    const uint8_t sk[MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET)]+#ifdef MLD_CONFIG_CONTEXT_PARAMETER+    ,+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+/**+ * Verify signature with pre-hashed message using SHAKE256. This function+ * computes the SHAKE256 hash of the message internally.+ *+ * @spec{Implements @[FIPS204, Algorithm 5, HashML-DSA.Verify] with SHAKE256 as+ * the pre-hash.}+ *+ * @param[in] sig     Pointer to input signature of+ *                    MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes.+ * @param[in] m       Pointer to message to be hashed and verified. May be+ *                    NULL if mlen == 0.+ * @param     mlen    Length of message.+ * @param[in] ctx     Pointer to context string. May be NULL if ctxlen == 0.+ * @param     ctxlen  Length of context string.+ * @param[in] pk      Bit-packed public key.+ * @param     context Application context. Only present when+ *                    MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                    MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                         Success.+ * @retval MLD_ERR_OUT_OF_MEMORY     MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                   used and an allocation via+ *                                   MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed.+ * @retval MLD_ERR_INVALID_ARG       The context string exceeded 255 bytes.+ */+MLD_API_QUALIFIER+MLD_API_MUST_CHECK_RETURN_VALUE+int MLD_API_NAMESPACE(verify_pre_hash_shake256)(+    const uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], const uint8_t *m,+    size_t mlen, const uint8_t *ctx, size_t ctxlen,+    const uint8_t pk[MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET)]+#ifdef MLD_CONFIG_CONTEXT_PARAMETER+    ,+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);+#endif /* !MLD_CONFIG_NO_VERIFY_API */+#endif /* !MLD_CONFIG_CORE_API_ONLY */++#if !defined(MLD_CONFIG_CORE_API_ONLY)+/**+ * Prepare domain separation prefix for ML-DSA signing.+ *+ * For pure ML-DSA (hashalg == MLD_PREHASH_NONE):+ *   Format: 0x00 || ctxlen (1 byte) || ctx.+ *+ * For HashML-DSA (hashalg != MLD_PREHASH_NONE):+ *   Format: 0x01 || ctxlen (1 byte) || ctx || oid (11 bytes) || ph.+ *+ * This function is useful for building incremental signing APIs.+ *+ * @spec{For HashML-DSA (hashalg != MLD_PREHASH_NONE), implements+ * @[FIPS204, Algorithm 4, line 23]. For Pure ML-DSA+ * (hashalg == MLD_PREHASH_NONE), implements+ * ```+ *    M' <- BytesToBits(IntegerToBytes(0, 1)+ *           || IntegerToBytes(|ctx|, 1)+ *           || ctx+ * ```+ * which is part of @[FIPS204, Algorithm 2, ML-DSA.Sign, line 10] and+ * @[FIPS204, Algorithm 3, ML-DSA.Verify, line 5].}+ *+ * @param[out] prefix  Output domain separation prefix buffer.+ * @param[in]  ph      Pointer to pre-hashed message (ignored for pure+ *                     ML-DSA).+ * @param      phlen   Length of pre-hashed message; must match the output+ *                     length of hashalg (ignored for pure ML-DSA).+ * @param[in]  ctx     Pointer to context string. May be NULL if ctxlen == 0.+ * @param      ctxlen  Length of context string.+ * @param      hashalg Hash algorithm constant (MLD_PREHASH_NONE for pure+ *                     ML-DSA, or MLD_PREHASH_* for HashML-DSA).+ *+ * @return The total length of the formatted prefix, or 0 on error.+ *         Errors are:+ *         - The context string exceeded 255 bytes.+ *         - For HashML-DSA: hashalg was unsupported, ph was NULL, or phlen+ *           did not match the output length of hashalg.+ */+MLD_API_QUALIFIER+MLD_API_MUST_CHECK_RETURN_VALUE+size_t MLD_API_NAMESPACE(prepare_domain_separation_prefix)(+    uint8_t prefix[MLD_DOMAIN_SEPARATION_MAX_BYTES], const uint8_t *ph,+    size_t phlen, const uint8_t *ctx, size_t ctxlen, int hashalg);++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+/**+ * Perform basic validity checks on secret key, and derive public key.+ *+ * Referring to the decoding of the secret key `sk=(rho, K, tr, s1, s2, t0)`+ * (cf. @[FIPS204, Algorithm 25, skDecode]), the following checks are+ * performed:+ *   - Check that s1 and s2 have coefficients in [-MLDSA_ETA, MLDSA_ETA].+ *   - Check that t0 and tr stored in sk match recomputed values.+ *+ * @note This function leaks whether the secret key is valid or invalid+ * through its return value and timing.+ *+ * @param[out] pk      Output public key.+ * @param[in]  sk      Input secret key.+ * @param      context Application context. Only present when+ *                     MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                    Success.+ * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was used and an+ *                               allocation via MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_INVALID_KEY   Secret key validation failed.+ */+MLD_API_QUALIFIER+MLD_API_MUST_CHECK_RETURN_VALUE+int MLD_API_NAMESPACE(pk_from_sk)(+    uint8_t pk[MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET)],+    const uint8_t sk[MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET)]+#ifdef MLD_CONFIG_CONTEXT_PARAMETER+    ,+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */+#endif /* !MLD_CONFIG_CORE_API_ONLY */++#ifdef __cplusplus+}+#endif++#undef MLD_API_NAMESPACE_PREFIX++#endif /* !MLD_CONFIG_CONSTANTS_ONLY */++/***************************** Memory Usage **********************************/++/*+ * By default mldsa-native performs all memory allocations on the stack.+ * Alternatively, mldsa-native supports custom allocation of large structures+ * through the `MLD_CONFIG_CUSTOM_ALLOC_FREE` configuration option.+ * See mldsa_native_config.h for details.+ *+ * `MLD_TOTAL_ALLOC_{44,65,87}_{KEYPAIR,SIGN,VERIFY}` indicates the maximum+ * (accumulative) allocation via MLD_ALLOC for each parameter set and operation.+ * Note that some stack allocation remains even+ * when using custom allocators, so these values are lower than total stack+ * usage with the default stack-only allocation.+ *+ * These constants may be used to implement custom allocations using a+ * fixed-sized buffer and a simple allocator (e.g., bump allocator).+ */+/* check-magic: off */+#if !defined(MLD_CONFIG_REDUCE_RAM)+#define MLD_TOTAL_ALLOC_44_KEYPAIR_NO_PCT 26912+#define MLD_TOTAL_ALLOC_44_KEYPAIR_PCT 48480+#define MLD_TOTAL_ALLOC_44_PK_FROM_SK 28480+#define MLD_TOTAL_ALLOC_44_SIGN 44704+#define MLD_TOTAL_ALLOC_44_VERIFY 24448+#define MLD_TOTAL_ALLOC_65_KEYPAIR_NO_PCT 44320+#define MLD_TOTAL_ALLOC_65_KEYPAIR_PCT 74624+#define MLD_TOTAL_ALLOC_65_PK_FROM_SK 46720+#define MLD_TOTAL_ALLOC_65_SIGN 69312+#define MLD_TOTAL_ALLOC_65_VERIFY 39872+#define MLD_TOTAL_ALLOC_87_KEYPAIR_NO_PCT 75040+#define MLD_TOTAL_ALLOC_87_KEYPAIR_PCT 115488+#define MLD_TOTAL_ALLOC_87_PK_FROM_SK 78272+#define MLD_TOTAL_ALLOC_87_SIGN 108224+#define MLD_TOTAL_ALLOC_87_VERIFY 68800+#else /* !MLD_CONFIG_REDUCE_RAM */+#define MLD_TOTAL_ALLOC_44_KEYPAIR_NO_PCT 11584+#define MLD_TOTAL_ALLOC_44_KEYPAIR_PCT 16896+#define MLD_TOTAL_ALLOC_44_PK_FROM_SK 13152+#define MLD_TOTAL_ALLOC_44_SIGN 13120+#define MLD_TOTAL_ALLOC_44_VERIFY 9120+#define MLD_TOTAL_ALLOC_65_KEYPAIR_NO_PCT 14656+#define MLD_TOTAL_ALLOC_65_KEYPAIR_PCT 22560+#define MLD_TOTAL_ALLOC_65_PK_FROM_SK 17056+#define MLD_TOTAL_ALLOC_65_SIGN 17248+#define MLD_TOTAL_ALLOC_65_VERIFY 10208+#define MLD_TOTAL_ALLOC_87_KEYPAIR_NO_PCT 18752+#define MLD_TOTAL_ALLOC_87_KEYPAIR_PCT 28608+#define MLD_TOTAL_ALLOC_87_PK_FROM_SK 21984+#define MLD_TOTAL_ALLOC_87_SIGN 21344+#define MLD_TOTAL_ALLOC_87_VERIFY 12512+#endif /* MLD_CONFIG_REDUCE_RAM */+/* check-magic: on */++/*+ * MLD_TOTAL_ALLOC_*_KEYPAIR adapts based on MLD_CONFIG_KEYGEN_PCT.+ */+#if defined(MLD_CONFIG_KEYGEN_PCT)+#define MLD_TOTAL_ALLOC_44_KEYPAIR MLD_TOTAL_ALLOC_44_KEYPAIR_PCT+#define MLD_TOTAL_ALLOC_65_KEYPAIR MLD_TOTAL_ALLOC_65_KEYPAIR_PCT+#define MLD_TOTAL_ALLOC_87_KEYPAIR MLD_TOTAL_ALLOC_87_KEYPAIR_PCT+#else+#define MLD_TOTAL_ALLOC_44_KEYPAIR MLD_TOTAL_ALLOC_44_KEYPAIR_NO_PCT+#define MLD_TOTAL_ALLOC_65_KEYPAIR MLD_TOTAL_ALLOC_65_KEYPAIR_NO_PCT+#define MLD_TOTAL_ALLOC_87_KEYPAIR MLD_TOTAL_ALLOC_87_KEYPAIR_NO_PCT+#endif++#define MLD_MAX3_(a, b, c) \+  ((a) > (b) ? ((a) > (c) ? (a) : (c)) : ((b) > (c) ? (b) : (c)))+#define MLD_MAX4_(a, b, c, d) MLD_MAX3_((a), (b), MLD_MAX3_((c), (d), (d)))++/*+ * `MLD_TOTAL_ALLOC_{44,65,87}` is the maximum across standard API operations+ * (keygen, sign, verify) for each parameter set.+ */+#define MLD_TOTAL_ALLOC_44                                             \+  MLD_MAX4_(MLD_TOTAL_ALLOC_44_KEYPAIR, MLD_TOTAL_ALLOC_44_PK_FROM_SK, \+            MLD_TOTAL_ALLOC_44_SIGN, MLD_TOTAL_ALLOC_44_VERIFY)+#define MLD_TOTAL_ALLOC_65                                             \+  MLD_MAX4_(MLD_TOTAL_ALLOC_65_KEYPAIR, MLD_TOTAL_ALLOC_65_PK_FROM_SK, \+            MLD_TOTAL_ALLOC_65_SIGN, MLD_TOTAL_ALLOC_65_VERIFY)+#define MLD_TOTAL_ALLOC_87                                             \+  MLD_MAX4_(MLD_TOTAL_ALLOC_87_KEYPAIR, MLD_TOTAL_ALLOC_87_PK_FROM_SK, \+            MLD_TOTAL_ALLOC_87_SIGN, MLD_TOTAL_ALLOC_87_VERIFY)++#endif /* !MLD_H */
+ cbits/mldsa/mldsa_native_asm.S view
@@ -0,0 +1,830 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mldsa-native repository.+ *          Do not modify it directly.+ */++/******************************************************************************+ *+ * Single assembly unit for fixed-level build of mldsa-native+ *+ * This assembly unit bundles together all assembly files for a build+ * of mldsa-native for a fixed security level (MLDSA-44/65/87).+ *+ * # Multi-level build+ *+ * If you want an SCU build of mldsa-native with support for multiple security+ * levels, you should include this file once with+ * MLD_CONFIG_MULTILEVEL_WITH_SHARED set.+ *+ * (You could also follow the same pattern as for mldsa_native.c+ *  and include it for every level, setting MLD_CONFIG_MULTILEVEL_NO_SHARED+ *  for all but one. For builds with MLD_CONFIG_MULTILEVEL_NO_SHARED, this+ *  file will then be ignored.)+ *+ * # Configuration+ *+ * The following options from the mldsa-native configuration are relevant:+ *+ * - MLD_CONFIG_FIPS202_CUSTOM_HEADER+ *   Set this option if you use a custom FIPS202 implementation.+ *+ * - MLD_CONFIG_USE_NATIVE_BACKEND_ARITH+ *   Set this option if you want to include the native arithmetic backends+ *   in your build.+ *+ * - MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202+ *   Set this option if you want to include the native FIPS202 backends+ *   in your build.+ *+ * - MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS+ *   Set this option if you want to keep the directives defined in+ *   level-independent headers. This is needed for a multi-level build.+ */++/* If parts of the mldsa-native source tree are not used,+ * consider reducing this header via `unifdef`.+ *+ * Example:+ * ```bash+ * unifdef -UMLD_CONFIG_USE_NATIVE_BACKEND_ARITH mldsa_native_asm.S+ * ```+ */++#include "src/common.h"++#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_ARITH)+#if defined(MLD_SYS_AARCH64)+#include "src/native/aarch64/src/mldsa_intt_aarch64_asm.S"+#include "src/native/aarch64/src/mldsa_ntt_aarch64_asm.S"+#include "src/native/aarch64/src/mldsa_pointwise_montgomery_aarch64_asm.S"+#include "src/native/aarch64/src/mldsa_poly_caddq_aarch64_asm.S"+#include "src/native/aarch64/src/mldsa_poly_chknorm_aarch64_asm.S"+#include "src/native/aarch64/src/mldsa_poly_decompose_32_aarch64_asm.S"+#include "src/native/aarch64/src/mldsa_poly_decompose_88_aarch64_asm.S"+#include "src/native/aarch64/src/mldsa_poly_use_hint_32_aarch64_asm.S"+#include "src/native/aarch64/src/mldsa_poly_use_hint_88_aarch64_asm.S"+#include "src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm.S"+#include "src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm.S"+#include "src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm.S"+#include "src/native/aarch64/src/mldsa_polyz_unpack_17_aarch64_asm.S"+#include "src/native/aarch64/src/mldsa_polyz_unpack_19_aarch64_asm.S"+#include "src/native/aarch64/src/mldsa_rej_uniform_aarch64_asm.S"+#include "src/native/aarch64/src/mldsa_rej_uniform_eta2_aarch64_asm.S"+#include "src/native/aarch64/src/mldsa_rej_uniform_eta4_aarch64_asm.S"+#endif /* MLD_SYS_AARCH64 */+#if defined(MLD_SYS_X86_64)+#include "src/native/x86_64/src/mldsa_intt_avx2_asm.S"+#include "src/native/x86_64/src/mldsa_ntt_avx2_asm.S"+#include "src/native/x86_64/src/mldsa_nttunpack_avx2_asm.S"+#include "src/native/x86_64/src/mldsa_pointwise_acc_l4_avx2_asm.S"+#include "src/native/x86_64/src/mldsa_pointwise_acc_l5_avx2_asm.S"+#include "src/native/x86_64/src/mldsa_pointwise_acc_l7_avx2_asm.S"+#include "src/native/x86_64/src/mldsa_pointwise_avx2_asm.S"+#include "src/native/x86_64/src/mldsa_poly_caddq_avx2_asm.S"+#include "src/native/x86_64/src/mldsa_poly_chknorm_avx2_asm.S"+#include "src/native/x86_64/src/mldsa_poly_decompose_32_avx2_asm.S"+#include "src/native/x86_64/src/mldsa_poly_decompose_88_avx2_asm.S"+#include "src/native/x86_64/src/mldsa_poly_use_hint_32_avx2_asm.S"+#include "src/native/x86_64/src/mldsa_poly_use_hint_88_avx2_asm.S"+#include "src/native/x86_64/src/mldsa_polyz_unpack_17_avx2_asm.S"+#include "src/native/x86_64/src/mldsa_polyz_unpack_19_avx2_asm.S"+#include "src/native/x86_64/src/mldsa_rej_uniform_avx2_asm.S"+#include "src/native/x86_64/src/mldsa_rej_uniform_eta2_avx2_asm.S"+#include "src/native/x86_64/src/mldsa_rej_uniform_eta4_avx2_asm.S"+#endif /* MLD_SYS_X86_64 */+#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_ARITH */++#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202)+#if defined(MLD_SYS_AARCH64)+#include "src/fips202/native/aarch64/src/keccak_f1600_x1_scalar_aarch64_asm.S"+#include "src/fips202/native/aarch64/src/keccak_f1600_x1_v84a_aarch64_asm.S"+#include "src/fips202/native/aarch64/src/keccak_f1600_x2_v84a_aarch64_asm.S"+#include "src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.S"+#include "src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.S"+#endif /* MLD_SYS_AARCH64 */+#if defined(MLD_SYS_X86_64)+#include "src/fips202/native/x86_64/src/keccak_f1600_x4_avx2_asm.S"+#endif+#if defined(MLD_SYS_ARMV81M_MVE)+#include "src/fips202/native/armv81m/src/keccak_f1600_x4_mve.S"+#include "src/fips202/native/armv81m/src/keccak_f1600_x4_state_extract_bytes_mve.S"+#include "src/fips202/native/armv81m/src/keccak_f1600_x4_state_xor_bytes_mve.S"+#endif+#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 */+++/* Macro #undef's+ *+ * The following undefines macros from headers+ * included by the source files imported above.+ *+ * This is to allow building and linking multiple builds+ * of mldsa-native for varying parameter sets through concatenation+ * of this file, as if the files had been compiled separately.+ * If this is not relevant to you, you may remove the following.+ *+ * NOTE: This is not needed for the assembly SCU since, at present,+ * there is no need to include it multiple times.+ * We keep it for uniformity with mldsa_native.c only.+ *+ * NOTE: To avoid having to distinguish between which headers are included+ * from the assembly files, we #undef the same set of directives+ * as in mldsa_native.c+ */++/*+ * Undefine macros from MLD_CONFIG_PARAMETER_SET-specific files+ */+/* mldsa/mldsa_native.h */+#undef MLDSA44_BYTES+#undef MLDSA44_CRHBYTES+#undef MLDSA44_PUBLICKEYBYTES+#undef MLDSA44_RNDBYTES+#undef MLDSA44_SECRETKEYBYTES+#undef MLDSA44_SEEDBYTES+#undef MLDSA44_TRBYTES+#undef MLDSA65_BYTES+#undef MLDSA65_CRHBYTES+#undef MLDSA65_PUBLICKEYBYTES+#undef MLDSA65_RNDBYTES+#undef MLDSA65_SECRETKEYBYTES+#undef MLDSA65_SEEDBYTES+#undef MLDSA65_TRBYTES+#undef MLDSA87_BYTES+#undef MLDSA87_CRHBYTES+#undef MLDSA87_PUBLICKEYBYTES+#undef MLDSA87_RNDBYTES+#undef MLDSA87_SECRETKEYBYTES+#undef MLDSA87_SEEDBYTES+#undef MLDSA87_TRBYTES+#undef MLDSA_BYTES+#undef MLDSA_BYTES_+#undef MLDSA_CRHBYTES+#undef MLDSA_PUBLICKEYBYTES+#undef MLDSA_PUBLICKEYBYTES_+#undef MLDSA_RNDBYTES+#undef MLDSA_SECRETKEYBYTES+#undef MLDSA_SECRETKEYBYTES_+#undef MLDSA_SEEDBYTES+#undef MLDSA_TRBYTES+#undef MLD_API_CONCAT+#undef MLD_API_CONCAT_+#undef MLD_API_CONCAT_UNDERSCORE+#undef MLD_API_MUST_CHECK_RETURN_VALUE+#undef MLD_API_NAMESPACE+#undef MLD_API_NAMESPACE_PREFIX+#undef MLD_API_QUALIFIER+#undef MLD_DOMAIN_SEPARATION_MAX_BYTES+#undef MLD_ERR_FAIL+#undef MLD_ERR_INVALID_ARG+#undef MLD_ERR_INVALID_KEY+#undef MLD_ERR_INVALID_SIGNATURE+#undef MLD_ERR_OUT_OF_MEMORY+#undef MLD_ERR_PCT_FAIL+#undef MLD_ERR_RNG_FAIL+#undef MLD_ERR_SIGNING_PAUSED+#undef MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED+#undef MLD_H+#undef MLD_MAX3_+#undef MLD_MAX4_+#undef MLD_PREHASH_NONE+#undef MLD_PREHASH_SHA2_224+#undef MLD_PREHASH_SHA2_256+#undef MLD_PREHASH_SHA2_384+#undef MLD_PREHASH_SHA2_512+#undef MLD_PREHASH_SHA2_512_224+#undef MLD_PREHASH_SHA2_512_256+#undef MLD_PREHASH_SHA3_224+#undef MLD_PREHASH_SHA3_256+#undef MLD_PREHASH_SHA3_384+#undef MLD_PREHASH_SHA3_512+#undef MLD_PREHASH_SHAKE_128+#undef MLD_PREHASH_SHAKE_256+#undef MLD_TOTAL_ALLOC_44+#undef MLD_TOTAL_ALLOC_44_KEYPAIR+#undef MLD_TOTAL_ALLOC_44_KEYPAIR_NO_PCT+#undef MLD_TOTAL_ALLOC_44_KEYPAIR_PCT+#undef MLD_TOTAL_ALLOC_44_PK_FROM_SK+#undef MLD_TOTAL_ALLOC_44_SIGN+#undef MLD_TOTAL_ALLOC_44_VERIFY+#undef MLD_TOTAL_ALLOC_65+#undef MLD_TOTAL_ALLOC_65_KEYPAIR+#undef MLD_TOTAL_ALLOC_65_KEYPAIR_NO_PCT+#undef MLD_TOTAL_ALLOC_65_KEYPAIR_PCT+#undef MLD_TOTAL_ALLOC_65_PK_FROM_SK+#undef MLD_TOTAL_ALLOC_65_SIGN+#undef MLD_TOTAL_ALLOC_65_VERIFY+#undef MLD_TOTAL_ALLOC_87+#undef MLD_TOTAL_ALLOC_87_KEYPAIR+#undef MLD_TOTAL_ALLOC_87_KEYPAIR_NO_PCT+#undef MLD_TOTAL_ALLOC_87_KEYPAIR_PCT+#undef MLD_TOTAL_ALLOC_87_PK_FROM_SK+#undef MLD_TOTAL_ALLOC_87_SIGN+#undef MLD_TOTAL_ALLOC_87_VERIFY+/* mldsa/src/common.h */+#undef MLD_ADD_PARAM_SET+#undef MLD_ALLOC+#undef MLD_APPLY+#undef MLD_ASM_FN_SIZE+#undef MLD_ASM_FN_SYMBOL+#undef MLD_ASM_NAMESPACE+#undef MLD_BUILD_INTERNAL+#undef MLD_COMMON_H+#undef MLD_CONCAT+#undef MLD_CONCAT_+#undef MLD_EMPTY_CU+#undef MLD_ERR_FAIL+#undef MLD_ERR_INVALID_ARG+#undef MLD_ERR_INVALID_KEY+#undef MLD_ERR_INVALID_SIGNATURE+#undef MLD_ERR_OUT_OF_MEMORY+#undef MLD_ERR_PCT_FAIL+#undef MLD_ERR_RNG_FAIL+#undef MLD_ERR_SIGNING_PAUSED+#undef MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED+#undef MLD_EXTERNAL_API+#undef MLD_FIPS202X4_HEADER_FILE+#undef MLD_FIPS202_HEADER_FILE+#undef MLD_FREE+#undef MLD_INTERNAL_API+#undef MLD_INTERNAL_DATA_DECLARATION+#undef MLD_INTERNAL_DATA_DEFINITION+#undef MLD_MULTILEVEL_BUILD+#undef MLD_NAMESPACE+#undef MLD_NAMESPACE_KL+#undef MLD_NAMESPACE_PREFIX+#undef MLD_NAMESPACE_PREFIX_KL+#undef mld_memcpy+#undef mld_memset+/* mldsa/src/packing.h */+#undef MLD_PACKING_H+#undef mld_pack_sig_c+#undef mld_pack_sig_h+#undef mld_pack_sig_z+#undef mld_pack_sk_rho_key_tr_s2+#undef mld_pack_sk_s1+#undef mld_sig_unpack_hints+#undef mld_unpack_pk_t1+#undef mld_unpack_sk+/* mldsa/src/params.h */+#undef MLDSA_BETA+#undef MLDSA_CRHBYTES+#undef MLDSA_CRYPTO_BYTES+#undef MLDSA_CRYPTO_PUBLICKEYBYTES+#undef MLDSA_CRYPTO_SECRETKEYBYTES+#undef MLDSA_CTILDEBYTES+#undef MLDSA_D+#undef MLDSA_ETA+#undef MLDSA_GAMMA1+#undef MLDSA_GAMMA2+#undef MLDSA_GAMMA2_32+#undef MLDSA_GAMMA2_88+#undef MLDSA_K+#undef MLDSA_L+#undef MLDSA_N+#undef MLDSA_OMEGA+#undef MLDSA_PK_END+#undef MLDSA_PK_RHO_BYTES+#undef MLDSA_PK_RHO_OFFSET+#undef MLDSA_PK_T1_BYTES+#undef MLDSA_PK_T1_OFFSET+#undef MLDSA_POLYETA_PACKEDBYTES+#undef MLDSA_POLYT0_PACKEDBYTES+#undef MLDSA_POLYT1_PACKEDBYTES+#undef MLDSA_POLYVECH_PACKEDBYTES+#undef MLDSA_POLYW1_PACKEDBYTES+#undef MLDSA_POLYW1_PACKEDBYTES_32+#undef MLDSA_POLYW1_PACKEDBYTES_88+#undef MLDSA_POLYZ_PACKEDBYTES+#undef MLDSA_Q+#undef MLDSA_Q_HALF+#undef MLDSA_RNDBYTES+#undef MLDSA_SEEDBYTES+#undef MLDSA_SIG_C_BYTES+#undef MLDSA_SIG_C_OFFSET+#undef MLDSA_SIG_END+#undef MLDSA_SIG_H_BYTES+#undef MLDSA_SIG_H_OFFSET+#undef MLDSA_SIG_Z_BYTES+#undef MLDSA_SIG_Z_OFFSET+#undef MLDSA_SK_END+#undef MLDSA_SK_KEY_BYTES+#undef MLDSA_SK_KEY_OFFSET+#undef MLDSA_SK_RHO_BYTES+#undef MLDSA_SK_RHO_OFFSET+#undef MLDSA_SK_S1_BYTES+#undef MLDSA_SK_S1_OFFSET+#undef MLDSA_SK_S2_BYTES+#undef MLDSA_SK_S2_OFFSET+#undef MLDSA_SK_T0_BYTES+#undef MLDSA_SK_T0_OFFSET+#undef MLDSA_SK_TR_BYTES+#undef MLDSA_SK_TR_OFFSET+#undef MLDSA_TAU+#undef MLDSA_TRBYTES+#undef MLD_MAX_KAPPA+#undef MLD_PARAMS_H+/* mldsa/src/poly_kl.h */+#undef MLD_POLYETA_UNPACK_LOWER_BOUND+#undef MLD_POLY_KL_H+#undef mld_poly_challenge+#undef mld_poly_decompose+#undef mld_poly_uniform_eta+#undef mld_poly_uniform_eta_4x+#undef mld_poly_uniform_gamma1+#undef mld_poly_uniform_gamma1_4x+#undef mld_poly_use_hint+#undef mld_polyeta_pack+#undef mld_polyeta_unpack+#undef mld_polyw1_pack+#undef mld_polyz_pack+#undef mld_polyz_unpack+/* mldsa/src/polyvec.h */+#undef MLD_POLYVEC_H+#undef mld_polyveck+#undef mld_polyveck_caddq+#undef mld_polyveck_chknorm+#undef mld_polyveck_decompose+#undef mld_polyveck_invntt_tomont+#undef mld_polyveck_ntt+#undef mld_polyveck_pack_eta+#undef mld_polyveck_pack_w1+#undef mld_polyveck_reduce+#undef mld_polyveck_unpack_eta+#undef mld_polyvecl+#undef mld_polyvecl_chknorm+#undef mld_polyvecl_ntt+#undef mld_polyvecl_pack_eta+#undef mld_polyvecl_pointwise_acc_montgomery+#undef mld_polyvecl_uniform_gamma1+#undef mld_polyvecl_unpack_eta+#undef mld_polyvecl_unpack_z+/* mldsa/src/polyvec_lazy.h */+#undef MLD_POLYVEC_LAZY_H+#undef mld_poly_permute_bitrev_to_custom_optional+#undef mld_polymat+#undef mld_polymat_eager+#undef mld_polymat_lazy+#undef mld_polyvec_matrix_expand+#undef mld_polyvec_matrix_expand_eager+#undef mld_polyvec_matrix_expand_lazy+#undef mld_polyvec_matrix_pointwise_montgomery+#undef mld_polyvec_matrix_pointwise_montgomery_row+#undef mld_polyvec_matrix_pointwise_montgomery_row_eager+#undef mld_polyvec_matrix_pointwise_montgomery_row_lazy+#undef mld_polyvec_matrix_pointwise_montgomery_yvec+#undef mld_polyvec_matrix_pointwise_montgomery_yvec_eager+#undef mld_polyvec_matrix_pointwise_montgomery_yvec_lazy+#undef mld_sk_s1hat+#undef mld_sk_s1hat_eager+#undef mld_sk_s1hat_get_poly+#undef mld_sk_s1hat_get_poly_eager+#undef mld_sk_s1hat_get_poly_lazy+#undef mld_sk_s1hat_lazy+#undef mld_sk_s2hat+#undef mld_sk_s2hat_eager+#undef mld_sk_s2hat_get_poly+#undef mld_sk_s2hat_get_poly_eager+#undef mld_sk_s2hat_get_poly_lazy+#undef mld_sk_s2hat_lazy+#undef mld_sk_t0hat+#undef mld_sk_t0hat_eager+#undef mld_sk_t0hat_get_poly+#undef mld_sk_t0hat_get_poly_eager+#undef mld_sk_t0hat_get_poly_lazy+#undef mld_sk_t0hat_lazy+#undef mld_unpack_sk_s1hat+#undef mld_unpack_sk_s1hat_eager+#undef mld_unpack_sk_s1hat_lazy+#undef mld_unpack_sk_s2hat+#undef mld_unpack_sk_s2hat_eager+#undef mld_unpack_sk_s2hat_lazy+#undef mld_unpack_sk_t0hat+#undef mld_unpack_sk_t0hat_eager+#undef mld_unpack_sk_t0hat_lazy+#undef mld_yvec+#undef mld_yvec_eager+#undef mld_yvec_get_poly+#undef mld_yvec_get_poly_eager+#undef mld_yvec_get_poly_lazy+#undef mld_yvec_init+#undef mld_yvec_init_eager+#undef mld_yvec_init_lazy+#undef mld_yvec_lazy+/* mldsa/src/rounding.h */+#undef MLD_2_POW_D+#undef MLD_ROUNDING_H+#undef mld_decompose+#undef mld_make_hint+#undef mld_power2round+#undef mld_use_hint+/* mldsa/src/sign.h */+#undef MLD_DOMAIN_SEPARATION_MAX_BYTES+#undef MLD_PREHASH_NONE+#undef MLD_PREHASH_SHA2_224+#undef MLD_PREHASH_SHA2_256+#undef MLD_PREHASH_SHA2_384+#undef MLD_PREHASH_SHA2_512+#undef MLD_PREHASH_SHA2_512_224+#undef MLD_PREHASH_SHA2_512_256+#undef MLD_PREHASH_SHA3_224+#undef MLD_PREHASH_SHA3_256+#undef MLD_PREHASH_SHA3_384+#undef MLD_PREHASH_SHA3_512+#undef MLD_PREHASH_SHAKE_128+#undef MLD_PREHASH_SHAKE_256+#undef MLD_SIGN_H+#undef mld_prepare_domain_separation_prefix+#undef mld_sign_keypair+#undef mld_sign_keypair_internal+#undef mld_sign_pk_from_sk+#undef mld_sign_signature+#undef mld_sign_signature_extmu+#undef mld_sign_signature_internal+#undef mld_sign_signature_pre_hash_internal+#undef mld_sign_signature_pre_hash_shake256+#undef mld_sign_verify+#undef mld_sign_verify_extmu+#undef mld_sign_verify_internal+#undef mld_sign_verify_pre_hash_internal+#undef mld_sign_verify_pre_hash_shake256++#if !defined(MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS)+/*+ * Undefine macros from MLD_CONFIG_PARAMETER_SET-generic files+ */+/* mldsa/src/context.h */+#undef MLD_CONTEXT_H+#undef MLD_CONTEXT_PARAMETERS_0+#undef MLD_CONTEXT_PARAMETERS_1+#undef MLD_CONTEXT_PARAMETERS_2+#undef MLD_CONTEXT_PARAMETERS_3+#undef MLD_CONTEXT_PARAMETERS_4+#undef MLD_CONTEXT_PARAMETERS_5+#undef MLD_CONTEXT_PARAMETERS_6+#undef MLD_CONTEXT_PARAMETERS_7+#undef MLD_CONTEXT_PARAMETERS_8+#undef MLD_CONTEXT_PARAMETERS_9+#undef MLD_CONTEXT_UNUSED+#undef mld_sign_attempt+#undef mld_sign_finish+#undef mld_sign_resume+/* mldsa/src/ct.h */+#undef MLD_CT_H+#undef MLD_USE_ASM_VALUE_BARRIER+#undef mld_ct_opt_blocker_u64+/* mldsa/src/debug.h */+#undef MLD_DEBUG_H+#undef mld_assert+#undef mld_assert_abs_bound+#undef mld_assert_abs_bound_2d+#undef mld_assert_bound+#undef mld_assert_bound_2d+#undef mld_debug_check_assert+#undef mld_debug_check_bounds+/* mldsa/src/poly.h */+#undef MLD_FQMUL_BOUND+#undef MLD_INTT_BOUND+#undef MLD_NTT_BOUND+#undef MLD_POLY_H+#undef mld_poly_add+#undef mld_poly_caddq+#undef mld_poly_chknorm+#undef mld_poly_invntt_tomont+#undef mld_poly_ntt+#undef mld_poly_pointwise_montgomery+#undef mld_poly_power2round+#undef mld_poly_reduce+#undef mld_poly_shiftl+#undef mld_poly_sub+#undef mld_poly_uniform+#undef mld_poly_uniform_4x+#undef mld_polyt0_pack+#undef mld_polyt0_unpack+#undef mld_polyt1_pack+#undef mld_polyt1_unpack+#undef mld_polyw1_pack_32+#undef mld_polyw1_pack_88+/* mldsa/src/randombytes.h */+#undef MLD_RANDOMBYTES_H+/* mldsa/src/reduce.h */+#undef MLD_MONT+#undef MLD_REDUCE32_DOMAIN_MAX+#undef MLD_REDUCE32_RANGE_MAX+#undef MLD_REDUCE_H+/* mldsa/src/symmetric.h */+#undef MLD_STREAM128_BLOCKBYTES+#undef MLD_STREAM256_BLOCKBYTES+#undef MLD_SYMMETRIC_H+#undef mld_xof128_absorb_once+#undef mld_xof128_ctx+#undef mld_xof128_init+#undef mld_xof128_release+#undef mld_xof128_squeezeblocks+#undef mld_xof128_x4_absorb+#undef mld_xof128_x4_ctx+#undef mld_xof128_x4_init+#undef mld_xof128_x4_release+#undef mld_xof128_x4_squeezeblocks+#undef mld_xof256_absorb_once+#undef mld_xof256_ctx+#undef mld_xof256_init+#undef mld_xof256_release+#undef mld_xof256_squeezeblocks+#undef mld_xof256_x4_absorb+#undef mld_xof256_x4_ctx+#undef mld_xof256_x4_init+#undef mld_xof256_x4_release+#undef mld_xof256_x4_squeezeblocks+/* mldsa/src/sys.h */+#undef MLD_ALIGN+#undef MLD_ALIGN_UP+#undef MLD_ALWAYS_INLINE+#undef MLD_CET_ENDBR+#undef MLD_CT_TESTING_DECLASSIFY+#undef MLD_CT_TESTING_SECRET+#undef MLD_DEFAULT_ALIGN+#undef MLD_HAVE_INLINE_ASM+#undef MLD_INLINE+#undef MLD_MUST_CHECK_RETURN_VALUE+#undef MLD_NOINLINE+#undef MLD_RESTRICT+#undef MLD_STATIC_TESTABLE+#undef MLD_SYSV_ABI+#undef MLD_SYSV_ABI_SUPPORTED+#undef MLD_SYS_AARCH64+#undef MLD_SYS_AARCH64_EB+#undef MLD_SYS_AARCH64_NEON+#undef MLD_SYS_APPLE+#undef MLD_SYS_ARMV81M_MVE+#undef MLD_SYS_BIG_ENDIAN+#undef MLD_SYS_H+#undef MLD_SYS_LINUX+#undef MLD_SYS_LITTLE_ENDIAN+#undef MLD_SYS_PPC64LE+#undef MLD_SYS_RISCV32+#undef MLD_SYS_RISCV64+#undef MLD_SYS_RISCV64_RVV+#undef MLD_SYS_WINDOWS+#undef MLD_SYS_X86_64+#undef MLD_SYS_X86_64_AVX2+/* mldsa/src/cbmc.h */+#undef MLD_CBMC_H+#undef __contract__+#undef __loop__++#if !defined(MLD_CONFIG_FIPS202_CUSTOM_HEADER)+/*+ * Undefine macros from FIPS-202 files+ */+/* mldsa/src/fips202/fips202.h */+#undef MLD_FIPS202_FIPS202_H+#undef MLD_KECCAK_LANES+#undef SHA3_256_HASHBYTES+#undef SHA3_256_RATE+#undef SHA3_512_HASHBYTES+#undef SHA3_512_RATE+#undef SHAKE128_RATE+#undef SHAKE256_RATE+#undef mld_shake128_absorb+#undef mld_shake128_finalize+#undef mld_shake128_init+#undef mld_shake128_release+#undef mld_shake128_squeeze+#undef mld_shake256+#undef mld_shake256_absorb+#undef mld_shake256_finalize+#undef mld_shake256_init+#undef mld_shake256_release+#undef mld_shake256_squeeze+/* mldsa/src/fips202/fips202x4.h */+#undef MLD_FIPS202_FIPS202X4_H+#undef mld_shake128x4_absorb_once+#undef mld_shake128x4_init+#undef mld_shake128x4_release+#undef mld_shake128x4_squeezeblocks+#undef mld_shake256x4_absorb_once+#undef mld_shake256x4_init+#undef mld_shake256x4_release+#undef mld_shake256x4_squeezeblocks+/* mldsa/src/fips202/keccakf1600.h */+#undef MLD_FIPS202_KECCAKF1600_H+#undef MLD_KECCAK_LANES+#undef MLD_KECCAK_WAY+#undef mld_keccakf1600_extract_bytes+#undef mld_keccakf1600_permute+#undef mld_keccakf1600_xor_bytes+#undef mld_keccakf1600x4_extract_bytes+#undef mld_keccakf1600x4_permute+#undef mld_keccakf1600x4_xor_bytes+#endif /* !MLD_CONFIG_FIPS202_CUSTOM_HEADER */++#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202)+/* mldsa/src/fips202/native/api.h */+#undef MLD_FIPS202_NATIVE_API_H+#undef MLD_NATIVE_FUNC_FALLBACK+#undef MLD_NATIVE_FUNC_SUCCESS+/* mldsa/src/fips202/native/auto.h */+#undef MLD_FIPS202_NATIVE_AUTO_H+#if defined(MLD_SYS_AARCH64)+/*+ * Undefine macros from native code (FIPS202, AArch64)+ */+/* mldsa/src/fips202/native/aarch64/auto.h */+#undef MLD_FIPS202_NATIVE_AARCH64_AUTO_H+/* mldsa/src/fips202/native/aarch64/src/fips202_native_aarch64.h */+#undef MLD_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H+#undef mld_keccak_f1600_x1_scalar_aarch64_asm+#undef mld_keccak_f1600_x1_v84a_aarch64_asm+#undef mld_keccak_f1600_x2_v84a_aarch64_asm+#undef mld_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm+#undef mld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm+#undef mld_keccakf1600_round_constants+/* mldsa/src/fips202/native/aarch64/x1_scalar.h */+#undef MLD_FIPS202_AARCH64_NEED_X1_SCALAR+#undef MLD_FIPS202_NATIVE_AARCH64_X1_SCALAR_H+#undef MLD_USE_NATIVE_FIPS202_X1+/* mldsa/src/fips202/native/aarch64/x1_v84a.h */+#undef MLD_FIPS202_AARCH64_NEED_X1_V84A+#undef MLD_FIPS202_NATIVE_AARCH64_X1_V84A_H+#undef MLD_USE_NATIVE_FIPS202_X1+/* mldsa/src/fips202/native/aarch64/x2_v84a.h */+#undef MLD_FIPS202_AARCH64_NEED_X2_V84A+#undef MLD_FIPS202_NATIVE_AARCH64_X2_V84A_H+#undef MLD_USE_NATIVE_FIPS202_X4+/* mldsa/src/fips202/native/aarch64/x4_v8a_scalar.h */+#undef MLD_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID+#undef MLD_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H+#undef MLD_USE_NATIVE_FIPS202_X4+/* mldsa/src/fips202/native/aarch64/x4_v8a_v84a_scalar.h */+#undef MLD_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID+#undef MLD_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H+#undef MLD_USE_NATIVE_FIPS202_X4+#endif /* MLD_SYS_AARCH64 */+#if defined(MLD_SYS_X86_64)+/*+ * Undefine macros from native code (FIPS202, x86_64)+ */+/* mldsa/src/fips202/native/x86_64/keccak_f1600_x4_avx2.h */+#undef MLD_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H+#undef MLD_FIPS202_X86_64_NEED_X4_AVX2+#undef MLD_USE_NATIVE_FIPS202_X4+/* mldsa/src/fips202/native/x86_64/src/fips202_native_x86_64.h */+#undef MLD_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H+#undef mld_keccak_f1600_x4_avx2_asm+#undef mld_keccak_rho56+#undef mld_keccak_rho8+#undef mld_keccakf1600_round_constants+#endif /* MLD_SYS_X86_64 */+#if defined(MLD_SYS_ARMV81M_MVE)+/*+ * Undefine macros from native code (FIPS202, Armv8.1-M)+ */+/* mldsa/src/fips202/native/armv81m/mve.h */+#undef MLD_FIPS202_ARMV81M_NEED_X4+#undef MLD_FIPS202_NATIVE_ARMV81M+#undef MLD_FIPS202_NATIVE_ARMV81M_MVE_H+#undef MLD_USE_NATIVE_FIPS202_X4+#undef MLD_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES+#undef MLD_USE_NATIVE_FIPS202_X4_XOR_BYTES+#undef mld_keccak_f1600_x4_native_impl+/* mldsa/src/fips202/native/armv81m/src/fips202_native_armv81m.h */+#undef MLD_FIPS202_NATIVE_ARMV81M_SRC_FIPS202_NATIVE_ARMV81M_H+#undef mld_keccak_f1600_x4_mve_asm+#undef mld_keccak_f1600_x4_state_extract_bytes_asm+#undef mld_keccak_f1600_x4_state_xor_bytes_asm+#undef mld_keccakf1600_round_constants+#endif /* MLD_SYS_ARMV81M_MVE */+#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 */+#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_ARITH)+/* mldsa/src/native/api.h */+#undef MLD_FQMUL_BOUND+#undef MLD_INTT_BOUND+#undef MLD_NATIVE_API_H+#undef MLD_NATIVE_FUNC_FALLBACK+#undef MLD_NATIVE_FUNC_SUCCESS+#undef MLD_NTT_BOUND+#undef MLD_REDUCE32_RANGE_MAX+/* mldsa/src/native/meta.h */+#undef MLD_NATIVE_META_H+#if defined(MLD_SYS_AARCH64)+/*+ * Undefine macros from native code (Arith, AArch64)+ */+/* mldsa/src/native/aarch64/meta.h */+#undef MLD_ARITH_BACKEND_AARCH64+#undef MLD_NATIVE_AARCH64_META_H+#undef MLD_USE_NATIVE_INTT+#undef MLD_USE_NATIVE_NTT+#undef MLD_USE_NATIVE_POINTWISE_MONTGOMERY+#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4+#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5+#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7+#undef MLD_USE_NATIVE_POLYZ_UNPACK_17+#undef MLD_USE_NATIVE_POLYZ_UNPACK_19+#undef MLD_USE_NATIVE_POLY_CADDQ+#undef MLD_USE_NATIVE_POLY_CHKNORM+#undef MLD_USE_NATIVE_POLY_DECOMPOSE_32+#undef MLD_USE_NATIVE_POLY_DECOMPOSE_88+#undef MLD_USE_NATIVE_POLY_USE_HINT_32+#undef MLD_USE_NATIVE_POLY_USE_HINT_88+#undef MLD_USE_NATIVE_REJ_UNIFORM+#undef MLD_USE_NATIVE_REJ_UNIFORM_ETA2+#undef MLD_USE_NATIVE_REJ_UNIFORM_ETA4+/* mldsa/src/native/aarch64/src/arith_native_aarch64.h */+#undef MLD_AARCH64_REJ_UNIFORM_ETA2_BUFLEN+#undef MLD_AARCH64_REJ_UNIFORM_ETA4_BUFLEN+#undef MLD_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H+#undef mld_aarch64_intt_zetas_layer123456+#undef mld_aarch64_intt_zetas_layer78+#undef mld_aarch64_ntt_zetas_layer123456+#undef mld_aarch64_ntt_zetas_layer78+#undef mld_intt_aarch64_asm+#undef mld_ntt_aarch64_asm+#undef mld_poly_caddq_aarch64_asm+#undef mld_poly_chknorm_aarch64_asm+#undef mld_poly_decompose_32_aarch64_asm+#undef mld_poly_decompose_88_aarch64_asm+#undef mld_poly_pointwise_montgomery_aarch64_asm+#undef mld_poly_use_hint_32_aarch64_asm+#undef mld_poly_use_hint_88_aarch64_asm+#undef mld_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm+#undef mld_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm+#undef mld_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm+#undef mld_polyz_unpack_17_aarch64_asm+#undef mld_polyz_unpack_17_indices+#undef mld_polyz_unpack_19_aarch64_asm+#undef mld_polyz_unpack_19_indices+#undef mld_rej_uniform_aarch64_asm+#undef mld_rej_uniform_eta2_aarch64_asm+#undef mld_rej_uniform_eta4_aarch64_asm+#undef mld_rej_uniform_eta_table+#undef mld_rej_uniform_table+#endif /* MLD_SYS_AARCH64 */+#if defined(MLD_SYS_X86_64)+/*+ * Undefine macros from native code (Arith, X86_64)+ */+/* mldsa/src/native/x86_64/meta.h */+#undef MLD_ARITH_BACKEND_X86_64_DEFAULT+#undef MLD_NATIVE_X86_64_META_H+#undef MLD_USE_NATIVE_INTT+#undef MLD_USE_NATIVE_NTT+#undef MLD_USE_NATIVE_NTT_CUSTOM_ORDER+#undef MLD_USE_NATIVE_POINTWISE_MONTGOMERY+#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4+#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5+#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7+#undef MLD_USE_NATIVE_POLYZ_UNPACK_17+#undef MLD_USE_NATIVE_POLYZ_UNPACK_19+#undef MLD_USE_NATIVE_POLY_CADDQ+#undef MLD_USE_NATIVE_POLY_CHKNORM+#undef MLD_USE_NATIVE_POLY_DECOMPOSE_32+#undef MLD_USE_NATIVE_POLY_DECOMPOSE_88+#undef MLD_USE_NATIVE_POLY_USE_HINT_32+#undef MLD_USE_NATIVE_POLY_USE_HINT_88+#undef MLD_USE_NATIVE_REJ_UNIFORM+#undef MLD_USE_NATIVE_REJ_UNIFORM_ETA2+#undef MLD_USE_NATIVE_REJ_UNIFORM_ETA4+/* mldsa/src/native/x86_64/src/arith_native_x86_64.h */+#undef MLD_AVX2_REJ_UNIFORM_BUFLEN+#undef MLD_AVX2_REJ_UNIFORM_ETA2_BUFLEN+#undef MLD_AVX2_REJ_UNIFORM_ETA4_BUFLEN+#undef MLD_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H+#undef mld_invntt_avx2_asm+#undef mld_ntt_avx2_asm+#undef mld_nttunpack_avx2_asm+#undef mld_pointwise_acc_l4_avx2_asm+#undef mld_pointwise_acc_l5_avx2_asm+#undef mld_pointwise_acc_l7_avx2_asm+#undef mld_pointwise_avx2_asm+#undef mld_poly_caddq_avx2_asm+#undef mld_poly_chknorm_avx2_asm+#undef mld_poly_decompose_32_avx2_asm+#undef mld_poly_decompose_88_avx2_asm+#undef mld_poly_use_hint_32_avx2_asm+#undef mld_poly_use_hint_88_avx2_asm+#undef mld_polyz_unpack_17_avx2_asm+#undef mld_polyz_unpack_19_avx2_asm+#undef mld_rej_uniform_avx2_asm+#undef mld_rej_uniform_eta2_avx2_asm+#undef mld_rej_uniform_eta4_avx2_asm+#undef mld_rej_uniform_table+/* mldsa/src/native/x86_64/src/consts.h */+#undef MLD_AVX2_BACKEND_DATA_OFFSET_8XDIV+#undef MLD_AVX2_BACKEND_DATA_OFFSET_8XDIV_QINV+#undef MLD_AVX2_BACKEND_DATA_OFFSET_8XQ+#undef MLD_AVX2_BACKEND_DATA_OFFSET_8XQINV+#undef MLD_AVX2_BACKEND_DATA_OFFSET_ZETAS+#undef MLD_AVX2_BACKEND_DATA_OFFSET_ZETAS_QINV+#undef MLD_NATIVE_X86_64_SRC_CONSTS_H+#undef mld_qdata+#endif /* MLD_SYS_X86_64 */+#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_ARITH */+#endif /* !MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS */
+ cbits/mldsa/mldsa_native_config.h view
@@ -0,0 +1,855 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS140_3_IG]+ *   Implementation Guidance for FIPS 140-3 and the Cryptographic Module+ *   Validation Program+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/projects/cryptographic-module-validation-program/fips-140-3-ig-announcements+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ *+ * - [FIPS204_UPDATES]+ *   FIPS 204 Potential Updates (Errata)+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/files/pubs/fips/204/final/docs/fips-204-potential-updates.xlsx+ */++#ifndef MLD_CONFIG_H+#define MLD_CONFIG_H++/**+ * MLD_CONFIG_PARAMETER_SET+ *+ * Specifies the parameter set for ML-DSA+ * - MLD_CONFIG_PARAMETER_SET=44 corresponds to ML-DSA-44+ * - MLD_CONFIG_PARAMETER_SET=65 corresponds to ML-DSA-65+ * - MLD_CONFIG_PARAMETER_SET=87 corresponds to ML-DSA-87+ *+ * If you want to support multiple parameter sets, build the+ * library multiple times and set MLD_CONFIG_MULTILEVEL_BUILD.+ * See MLD_CONFIG_MULTILEVEL_BUILD for how to do this while+ * minimizing code duplication.+ *+ * This can also be set using CFLAGS.+ */+#ifndef MLD_CONFIG_PARAMETER_SET+#define MLD_CONFIG_PARAMETER_SET \+  44 /* Change this for different security strengths */+#endif++/**+ * MLD_CONFIG_FILE+ *+ * If defined, this is a header that will be included instead+ * of the default configuration file mldsa/mldsa_native_config.h.+ *+ * When you need to build mldsa-native in multiple configurations,+ * using varying MLD_CONFIG_FILE can be more convenient+ * than configuring everything through CFLAGS.+ *+ * To use, MLD_CONFIG_FILE _must_ be defined prior+ * to the inclusion of any mldsa-native headers. For example,+ * it can be set by passing `-DMLD_CONFIG_FILE="..."`+ * on the command line.+ */+/* #define MLD_CONFIG_FILE "mldsa_native_config.h" */++/**+ * MLD_CONFIG_NAMESPACE_PREFIX+ *+ * The prefix to use to namespace global symbols from mldsa/.+ *+ * In a multi-level build, level-dependent symbols will+ * additionally be prefixed with the parameter set (44/65/87).+ *+ * This can also be set using CFLAGS.+ */+#if !defined(MLD_CONFIG_NAMESPACE_PREFIX)+#define MLD_CONFIG_NAMESPACE_PREFIX MLD_DEFAULT_NAMESPACE_PREFIX+#endif++/**+ * MLD_CONFIG_MULTILEVEL_BUILD+ *+ * Set this if the build is part of a multi-level build supporting+ * multiple parameter sets.+ *+ * If you need only a single parameter set, keep this unset.+ *+ * To build mldsa-native with support for all parameter sets,+ * build it three times -- once per parameter set -- and set the+ * option MLD_CONFIG_MULTILEVEL_WITH_SHARED for exactly one of+ * them, and MLD_CONFIG_MULTILEVEL_NO_SHARED for the others.+ * MLD_CONFIG_MULTILEVEL_BUILD should be set for all of them.+ *+ * See examples/multilevel_build for an example.+ *+ * This can also be set using CFLAGS.+ */+/* #define MLD_CONFIG_MULTILEVEL_BUILD */++/**+ * MLD_CONFIG_EXTERNAL_API_QUALIFIER+ *+ * If set, this option provides an additional function+ * qualifier to be added to declarations of mldsa-native's+ * public API.+ *+ * The primary use case for this option are single-CU builds+ * where the public API exposed by mldsa-native is wrapped by+ * another API in the consuming application. In this case,+ * even mldsa-native's public API can be marked `static`.+ */+/* #define MLD_CONFIG_EXTERNAL_API_QUALIFIER */++/**+ * MLD_CONFIG_NO_KEYPAIR_API+ *+ * By default, mldsa-native includes support for generating key+ * pairs. If you don't need this, set MLD_CONFIG_NO_KEYPAIR_API+ * to exclude keypair, keypair_internal,+ * pk_from_sk, and all internal APIs only needed by+ * those functions.+ */+/* #define MLD_CONFIG_NO_KEYPAIR_API */++/**+ * MLD_CONFIG_NO_SIGN_API+ *+ * By default, mldsa-native includes support for creating+ * signatures. If you don't need this, set MLD_CONFIG_NO_SIGN_API+ * to exclude signature,+ * signature_extmu, signature_internal,+ * signature_pre_hash_internal,+ * signature_pre_hash_shake256, and all internal APIs+ * only needed by those functions.+ */+/* #define MLD_CONFIG_NO_SIGN_API */++/**+ * MLD_CONFIG_NO_VERIFY_API+ *+ * By default, mldsa-native includes support for verifying+ * signatures. If you don't need this, set+ * MLD_CONFIG_NO_VERIFY_API to exclude verify,+ * verify_extmu, verify_internal,+ * verify_pre_hash_internal,+ * verify_pre_hash_shake256, and all internal APIs+ * only needed by those functions.+ */+/* #define MLD_CONFIG_NO_VERIFY_API */++/**+ * MLD_CONFIG_CORE_API_ONLY+ *+ * Set this to remove all public APIs except+ * keypair_internal, signature_internal,+ * and verify_internal.+ */+/* #define MLD_CONFIG_CORE_API_ONLY */++/**+ * MLD_CONFIG_NO_RANDOMIZED_API+ *+ * If this option is set, mldsa-native will be built without the+ * randomized API functions (keypair,+ * signature, and signature_extmu).+ * This allows users to build mldsa-native without providing a+ * randombytes() implementation if they only need the+ * internal deterministic API+ * (keypair_internal, signature_internal).+ *+ * @note This option is incompatible with MLD_CONFIG_KEYGEN_PCT+ * as the current PCT implementation requires+ * signature().+ */+/* #define MLD_CONFIG_NO_RANDOMIZED_API */++/**+ * MLD_CONFIG_CONSTANTS_ONLY+ *+ * If you only need the size constants (MLDSA_PUBLICKEYBYTES, etc.)+ * but no function declarations, set MLD_CONFIG_CONSTANTS_ONLY.+ *+ * This only affects the public header mldsa_native.h, not+ * the implementation.+ */+/* #define MLD_CONFIG_CONSTANTS_ONLY */+/******************************************************************************+ *+ * Build-only configuration options+ *+ * The remaining configurations are build-options only.+ * They do not affect the API described in mldsa_native.h.+ *+ *****************************************************************************/+#if defined(MLD_BUILD_INTERNAL)++/**+ * MLD_CONFIG_MULTILEVEL_WITH_SHARED+ *+ * This is for multi-level builds of mldsa-native only. If you+ * need only a single parameter set, keep this unset.+ *+ * If this is set, all MLD_CONFIG_PARAMETER_SET-independent+ * code will be included in the build, including code needed only+ * for other parameter sets.+ *+ * Example: mld_polyw1_pack_88 is only needed for+ * MLD_CONFIG_PARAMETER_SET == 44. Yet, if this option is set for a+ * build with MLD_CONFIG_PARAMETER_SET == 65/87, it would be included.+ *+ * To build mldsa-native with support for all parameter sets,+ * build it three times -- once per parameter set -- and set the+ * option MLD_CONFIG_MULTILEVEL_WITH_SHARED for exactly one of+ * them, and MLD_CONFIG_MULTILEVEL_NO_SHARED for the others.+ * MLD_CONFIG_MULTILEVEL_BUILD should be set for all of them.+ *+ * See examples/multilevel_build for an example.+ *+ * This can also be set using CFLAGS.+ */+/* #define MLD_CONFIG_MULTILEVEL_WITH_SHARED */++/**+ * MLD_CONFIG_MULTILEVEL_NO_SHARED+ *+ * This is for multi-level builds of mldsa-native only. If you+ * need only a single parameter set, keep this unset.+ *+ * If this is set, no MLD_CONFIG_PARAMETER_SET-independent code+ * will be included in the build.+ *+ * To build mldsa-native with support for all parameter sets,+ * build it three times -- once per parameter set -- and set the+ * option MLD_CONFIG_MULTILEVEL_WITH_SHARED for exactly one of+ * them, and MLD_CONFIG_MULTILEVEL_NO_SHARED for the others.+ * MLD_CONFIG_MULTILEVEL_BUILD should be set for all of them.+ *+ * See examples/multilevel_build for an example.+ *+ * This can also be set using CFLAGS.+ */+/* #define MLD_CONFIG_MULTILEVEL_NO_SHARED */++/**+ * MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS+ *+ * This is only relevant for single compilation unit (SCU)+ * builds of mldsa-native. In this case, it determines whether+ * directives defined in parameter-set-independent headers should+ * be #undef'ined or not at the end of the SCU file. This is+ * needed in multilevel builds.+ *+ * See examples/multilevel_build_native for an example.+ *+ * This can also be set using CFLAGS.+ */+/* #define MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS */++/**+ * MLD_CONFIG_USE_NATIVE_BACKEND_ARITH+ *+ * Determines whether a native arithmetic backend should be used.+ *+ * The arithmetic backend covers performance-critical functions+ * such as the number-theoretic transform (NTT).+ *+ * If this option is unset, the C backend will be used.+ *+ * If this option is set, the arithmetic backend to be used is+ * determined by MLD_CONFIG_ARITH_BACKEND_FILE: If the latter is+ * unset, the default backend for your target architecture+ * will be used. If set, it must be the name of a backend metadata+ * file.+ *+ * This can also be set using CFLAGS.+ */+#if !defined(MLD_CONFIG_USE_NATIVE_BACKEND_ARITH)+/* #define MLD_CONFIG_USE_NATIVE_BACKEND_ARITH */+#endif++/**+ * MLD_CONFIG_ARITH_BACKEND_FILE+ *+ * The arithmetic backend to use.+ *+ * If MLD_CONFIG_USE_NATIVE_BACKEND_ARITH is unset, this option+ * is ignored.+ *+ * If MLD_CONFIG_USE_NATIVE_BACKEND_ARITH is set, this option must+ * either be undefined or the filename of an arithmetic backend.+ * If unset, the default backend will be used.+ *+ * This can be set using CFLAGS.+ */+#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_ARITH) && \+    !defined(MLD_CONFIG_ARITH_BACKEND_FILE)+#define MLD_CONFIG_ARITH_BACKEND_FILE "native/meta.h"+#endif++/**+ * MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202+ *+ * Determines whether a native FIPS202 backend should be used.+ *+ * The FIPS202 backend covers 1x/2x/4x-fold Keccak-f1600, which is+ * the performance bottleneck of SHA3 and SHAKE.+ *+ * If this option is unset, the C backend will be used.+ *+ * If this option is set, the FIPS202 backend to be used is+ * determined by MLD_CONFIG_FIPS202_BACKEND_FILE: If the latter is+ * unset, the default backend for your target architecture+ * will be used. If set, it must be the name of a backend metadata+ * file.+ *+ * This can also be set using CFLAGS.+ */+#if !defined(MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202)+/* #define MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 */+#endif++/**+ * MLD_CONFIG_FIPS202_BACKEND_FILE+ *+ * The FIPS-202 backend to use.+ *+ * If MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 is set, this option+ * must either be undefined or the filename of a FIPS202 backend.+ * If unset, the default backend will be used.+ *+ * This can be set using CFLAGS.+ */+#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202) && \+    !defined(MLD_CONFIG_FIPS202_BACKEND_FILE)+#define MLD_CONFIG_FIPS202_BACKEND_FILE "fips202/native/auto.h"+#endif++/**+ * MLD_CONFIG_FIPS202_CUSTOM_HEADER+ *+ * Custom header to use for FIPS-202+ *+ * This should only be set if you intend to use a custom+ * FIPS-202 implementation, different from the one shipped+ * with mldsa-native.+ *+ * If set, it must be the name of a file serving as the+ * replacement for mldsa/src/fips202/fips202.h, and exposing+ * the same API (see FIPS202.md).+ */+/* #define MLD_CONFIG_FIPS202_CUSTOM_HEADER "SOME_FILE.h" */++/**+ * MLD_CONFIG_FIPS202X4_CUSTOM_HEADER+ *+ * Custom header to use for FIPS-202-X4+ *+ * This should only be set if you intend to use a custom+ * FIPS-202 implementation, different from the one shipped+ * with mldsa-native.+ *+ * If set, it must be the name of a file serving as the+ * replacement for mldsa/src/fips202/fips202x4.h, and exposing+ * the same API (see FIPS202.md).+ */+/* #define MLD_CONFIG_FIPS202X4_CUSTOM_HEADER "SOME_FILE.h" */++/**+ * MLD_CONFIG_CUSTOM_ZEROIZE+ *+ * In compliance with @[FIPS204, Section 3.6.3], mldsa-native zeroizes+ * intermediate buffers before returning from function calls. By default,+ * those buffers are allocated from the stack; if MLD_CONFIG_CUSTOM_ALLOC_FREE+ * is set, they are (mostly -- few exceptions remain at present) allocated from+ * the configured custom allocator.+ *+ * mldsa-native also zeroizes caller-owned output buffers as needed to uphold+ * the API convention that outputs be either unmodified or zeroized upon+ * failure.+ *+ * Set this option and define `mld_zeroize` if you want to use a custom+ * method to zeroize intermediate and output buffers.+ *+ * The default implementation uses SecureZeroMemory on Windows and a+ * memset + compiler barrier otherwise. If neither of those is available on+ * the target platform, compilation will fail, and you will need to use+ * MLD_CONFIG_CUSTOM_ZEROIZE to provide a custom implementation of+ * `mld_zeroize()`.+ *+ * @warning+ *   The zeroization conducted by mldsa-native reduces the likelihood of data+ *   leaking on the stack or custom allocators, but it does not eliminate it.+ *   For example, the C standard makes no guarantee about where a compiler+ *   allocates local structures and whether/where it makes copies of them.+ *   Also, in addition to entire structures, there may also be potentially+ *   exploitable leakage of individual values on the stack. If you need+ *   bullet-proof zeroization of the stack, you need to consider additional+ *   measures instead of what this feature provides. In this case, you can+ *   set mld_zeroize to a no-op. Note that in this case you are also responsible+ *   for zeroizing output buffers upon failure.+ */+/* #define MLD_CONFIG_CUSTOM_ZEROIZE+   #if !defined(__ASSEMBLER__)+   #include <stdint.h>+   #include "src/src.h"+   static MLD_INLINE void mld_zeroize(void *ptr, size_t len)+   {+       ... your implementation ...+   }+   #endif+*/++/**+ * MLD_CONFIG_CUSTOM_RANDOMBYTES+ *+ * mldsa-native does not provide a secure randombytes+ * implementation. Such an implementation has to be provided by+ * the consumer.+ *+ * If this option is not set, mldsa-native expects a function+ * int randombytes(uint8_t *out, size_t outlen).+ *+ * Set this option and define `mld_randombytes` if you want to+ * use a custom method to sample randombytes with a different name+ * or signature.+ */+/* #define MLD_CONFIG_CUSTOM_RANDOMBYTES+   #if !defined(__ASSEMBLER__)+   #include <stdint.h>+   #include "src/src.h"+   static MLD_INLINE int mld_randombytes(uint8_t *ptr, size_t len)+   {+       ... your implementation ...+       return 0;+   }+   #endif+*/++/**+ * MLD_CONFIG_CUSTOM_CAPABILITY_FUNC+ *+ * mldsa-native backends may rely on specific hardware features.+ * Those backends will only be included in an mldsa-native build+ * if support for the respective features is enabled at+ * compile-time. However, when building for a heterogeneous set+ * of CPUs to run the resulting binary/library on, feature+ * detection at _runtime_ is needed to decide whether a backend+ * can be used or not.+ *+ * Set this option and define `mld_sys_check_capability` if you+ * want to use a custom method to dispatch between implementations.+ *+ * Return value 1 indicates that a capability is supported.+ * Return value 0 indicates that a capability is not supported.+ *+ * If this option is not set, mldsa-native uses compile-time+ * feature detection only to decide which backend to use.+ *+ * If you compile mldsa-native on a system with different+ * capabilities than the system that the resulting binary/library+ * will be run on, you must use this option.+ */+/* #define MLD_CONFIG_CUSTOM_CAPABILITY_FUNC+   static MLD_INLINE int mld_sys_check_capability(mld_sys_cap cap)+   {+       ... your implementation ...+   }+*/++/**+ * MLD_CONFIG_CUSTOM_ALLOC_FREE+ *+ * Set this option and define `MLD_CUSTOM_ALLOC` and+ * `MLD_CUSTOM_FREE` if you want to use custom allocation for+ * large local structures or buffers.+ *+ * By default, all buffers/structures are allocated on the stack.+ * If this option is set, most of them will be allocated via+ * MLD_CUSTOM_ALLOC.+ *+ * Parameters to MLD_CUSTOM_ALLOC:+ * - T* v: Target pointer to declare.+ * - T: Type of structure to be allocated+ * - N: Number of elements to be allocated.+ *+ * Parameters to MLD_CUSTOM_FREE:+ * - T* v: Target pointer to free. May be NULL.+ * - T: Type of structure to be freed.+ * - N: Number of elements to be freed.+ *+ * @warning This option is experimental. Its scope, configuration and+ *          function/macro signatures may change at any time. We expect a+ *          stable API in a future version.+ *+ * @note Even if this option is set, some allocations further down+ * the call stack will still be made from the stack. Those will+ * likely be added to the scope of this option in the future.+ *+ * @note MLD_CUSTOM_ALLOC need not guarantee a successful+ * allocation nor include error handling. Upon failure, the+ * target pointer should simply be set to NULL. The calling+ * code will handle this case and invoke MLD_CUSTOM_FREE.+ */+/* #define MLD_CONFIG_CUSTOM_ALLOC_FREE+   #if !defined(__ASSEMBLER__)+   #include <stdlib.h>+   #define MLD_CUSTOM_ALLOC(v, T, N)                              \+     T* (v) = (T *)aligned_alloc(MLD_DEFAULT_ALIGN,               \+                                 MLD_ALIGN_UP(sizeof(T) * (N)))+   #define MLD_CUSTOM_FREE(v, T, N) free(v)+   #endif+*/++/**+ * MLD_CONFIG_CUSTOM_MEMCPY+ *+ * Set this option and define `mld_memcpy` if you want to+ * use a custom method to copy memory instead of the standard+ * library memcpy function.+ *+ * The custom implementation must have the same signature and+ * behavior as the standard memcpy function:+ * void *mld_memcpy(void *dest, const void *src, size_t n)+ */+/* #define MLD_CONFIG_CUSTOM_MEMCPY+   #if !defined(__ASSEMBLER__)+   #include <stdint.h>+   #include "src/src.h"+   static MLD_INLINE void *mld_memcpy(void *dest, const void *src, size_t n)+   {+       ... your implementation ...+   }+   #endif+*/++/**+ * MLD_CONFIG_CUSTOM_MEMSET+ *+ * Set this option and define `mld_memset` if you want to+ * use a custom method to set memory instead of the standard+ * library memset function.+ *+ * The custom implementation must have the same signature and+ * behavior as the standard memset function:+ * void *mld_memset(void *s, int c, size_t n)+ */+/* #define MLD_CONFIG_CUSTOM_MEMSET+   #if !defined(__ASSEMBLER__)+   #include <stdint.h>+   #include "src/src.h"+   static MLD_INLINE void *mld_memset(void *s, int c, size_t n)+   {+       ... your implementation ...+   }+   #endif+*/++/**+ * MLD_CONFIG_INTERNAL_API_QUALIFIER+ *+ * If set, this option provides an additional qualifier+ * to be added to declarations of internal API functions and data.+ *+ * The primary use case for this option are single-CU builds,+ * in which case this option can be set to `static`.+ */+/* #define MLD_CONFIG_INTERNAL_API_QUALIFIER */++/**+ * MLD_CONFIG_CT_TESTING_ENABLED+ *+ * If set, mldsa-native annotates data as secret / public using+ * valgrind's annotations VALGRIND_MAKE_MEM_UNDEFINED and+ * VALGRIND_MAKE_MEM_DEFINED, enabling various checks for secret-+ * dependent control flow of variable time execution (depending+ * on the exact version of valgrind installed).+ */+/* #define MLD_CONFIG_CT_TESTING_ENABLED */++/**+ * MLD_CONFIG_NO_ASM+ *+ * If this option is set, mldsa-native will be built without+ * use of native code or inline assembly.+ *+ * By default, inline assembly is used to implement value barriers.+ * Without inline assembly, mldsa-native will use a global volatile+ * 'opt blocker' instead; see ct.h.+ *+ * Inline assembly is also used to implement a secure zeroization+ * function on non-Windows platforms. If this option is set and+ * the target platform is not Windows, you MUST set+ * MLD_CONFIG_CUSTOM_ZEROIZE and provide a custom zeroization+ * function.+ *+ * If this option is set, MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 and+ * MLD_CONFIG_USE_NATIVE_BACKEND_ARITH will be ignored, and no+ * native backends will be used.+ */+/* #define MLD_CONFIG_NO_ASM */++/**+ * MLD_CONFIG_NO_ASM_VALUE_BARRIER+ *+ * If this option is set, mldsa-native will be built without+ * use of native code or inline assembly for value barriers.+ *+ * By default, inline assembly (if available) is used to implement+ * value barriers.+ * Without inline assembly, mldsa-native will use a global volatile+ * 'opt blocker' instead; see ct.h.+ */+/* #define MLD_CONFIG_NO_ASM_VALUE_BARRIER */++/**+ * MLD_CONFIG_KEYGEN_PCT+ *+ * Compliance with @[FIPS140_3_IG, p.87] requires a+ * Pairwise Consistency Test (PCT) to be carried out on a freshly+ * generated keypair before it can be exported.+ *+ * Set this option if such a check should be implemented.+ * In this case, keypair_internal and+ * keypair will return MLD_ERR_PCT_FAIL if the+ * PCT failed.+ *+ * @note This feature will drastically lower the performance of+ * key generation.+ *+ * @note This option is incompatible with MLD_CONFIG_NO_SIGN_API+ * and MLD_CONFIG_NO_VERIFY_API as the current PCT implementation+ * requires signature() and verify().+ */+/* #define MLD_CONFIG_KEYGEN_PCT */++/**+ * MLD_CONFIG_KEYGEN_PCT_BREAKAGE_TEST+ *+ * If this option is set, the user must provide a runtime+ * function `static inline int mld_break_pct() { ... }` to+ * indicate whether the PCT should be made fail.+ *+ * This option only has an effect if MLD_CONFIG_KEYGEN_PCT is set.+ */+/* #define MLD_CONFIG_KEYGEN_PCT_BREAKAGE_TEST+   #if !defined(__ASSEMBLER__)+   #include "src/src.h"+   static MLD_INLINE int mld_break_pct(void)+   {+       ... return 0/1 depending on whether PCT should be broken ...+   }+   #endif+*/++/**+ * MLD_CONFIG_MAX_SIGNING_ATTEMPTS+ *+ * Upper bound on the number of rejection-sampling iterations+ * performed by ML-DSA signing (@[FIPS204, Algorithm 7]).+ *+ * If a valid signature is not produced within this many+ * attempts, signing returns MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED.+ * This is useful in timing-sensitive environments that+ * require a deterministic worst-case bound on signing time.+ *+ * For FIPS 204 compliance, this value MUST be at least 821,+ * cf. @[FIPS204, Appendix C] and @[FIPS204_UPDATES], which is+ * chosen so that the signing failure rate is < 2^{-256}.+ *+ * Default: Largest possible value before internal counters+ * would overflow. This is larger than the FIPS204 bound.+ *+ * In particular, in the default configuration, the signing+ * failure rate is < 2^{-256}.+ */+/* #define MLD_CONFIG_MAX_SIGNING_ATTEMPTS 821 */++/**+ * MLD_CONFIG_SERIAL_FIPS202_ONLY+ *+ * Set this to use a FIPS202 implementation with global state+ * that supports only one active Keccak computation at a time+ * (e.g. some hardware accelerators).+ *+ * If this option is set, ML-DSA will use FIPS202 operations+ * serially, ensuring that only one SHAKE context is active+ * at any given time.+ *+ * This allows offloading Keccak computations to a hardware+ * accelerator that holds only a single Keccak state locally,+ * rather than requiring support for multiple concurrent+ * Keccak states.+ *+ * @note Depending on the target CPU, this may reduce+ * performance when using software FIPS202 implementations.+ * Only enable this when you have to.+ */+/* #define MLD_CONFIG_SERIAL_FIPS202_ONLY */++/**+ * MLD_CONFIG_CONTEXT_PARAMETER+ *+ * Set this to add a caller-supplied context parameter to the public API+ * functions, which is then forwarded unchanged to the custom callbacks+ * (allocation, and signing hooks below).+ *+ * When this option is set, every public API function gains a trailing+ * parameter+ *+ *   MLD_CONFIG_CONTEXT_PARAMETER_TYPE context+ *+ * as its last argument; its type is configured via+ * MLD_CONFIG_CONTEXT_PARAMETER_TYPE (see below). mldsa-native treats this+ * value as opaque: it never dereferences it and only passes it on to the+ * configurable hook macros. It is meant to carry per-caller state -- e.g. a+ * pointer to a memory pool for the allocation hooks, or the resume state for+ * the signing hooks -- into those hooks.+ *+ * When this option is unset (the default), no extra parameter is added and+ * the hook macros never receive a context argument.+ *+ * The hooks that receive the context are the allocation hooks (see+ * MLD_CONFIG_CUSTOM_ALLOC_FREE) and the signing hooks (see+ * MLD_CONFIG_SIGN_HOOK_RESUME / _ATTEMPT / _FINISH); each is documented with+ * its own option below.+ */+/* #define MLD_CONFIG_CONTEXT_PARAMETER */++/**+ * MLD_CONFIG_CONTEXT_PARAMETER_TYPE+ *+ * Set this to define the type of the context parameter added by+ * MLD_CONFIG_CONTEXT_PARAMETER. It can be any C type usable as a function+ * parameter, e.g. `void *` or a pointer to a caller-defined struct such as+ * `struct my_ctx *`.+ *+ * This option must be defined if and only if MLD_CONFIG_CONTEXT_PARAMETER is+ * defined; defining one without the other is a compile-time error.+ */+/* #define MLD_CONFIG_CONTEXT_PARAMETER_TYPE void* */++/**+ * Signing hooks: MLD_CONFIG_SIGN_HOOK_RESUME / _ATTEMPT / _FINISH+ *+ * Three optional, independent hooks into the ML-DSA signing rejection-sampling+ * loop. Each is enabled by defining the matching option, in which case the+ * integration must provide the corresponding function. If a hook needs+ * per-operation state, enable MLD_CONFIG_CONTEXT_PARAMETER; the context is then+ * appended as the last argument.+ *+ * @warning This feature is experimental. Its scope, configuration and+ *          function signatures may change at any time, including after v2.+ *+ * Enabling any of the hooks requires MLD_CONFIG_NO_RANDOMIZED_API (restricting+ * the public API to deterministic operations). This is because the restartable+ * signing as enabled by the signing hooks only produces the uninterrupted+ * signature when the randomness is fixed across calls. A logging-only use+ * (attempt always returns 0; resume/finish merely observe) would be safe with+ * the randomized API too, but for now the requirement is imposed uniformly on+ * all three hooks.+ *+ * Note: Randomized signing is a shim wrapper around deterministic signing, and+ * all helper functions you need to build it are exposed publicly. Thus, if you+ * need a restartable, randomized signing operation, you can build your own by+ * replicating the logic and adding the RNG seed to the restart context. In this+ * case, please also consider letting the mldsa-native maintainers know of your+ * need for randomized, restartable signing, so the feature can be appropriately+ * prioritized.+ *+ * - MLD_CONFIG_SIGN_HOOK_ATTEMPT: int mld_sign_hook_attempt(attempt[, ctxt])+ *   Called before each attempt. Returns 0 to proceed, or non-zero to pause:+ *   signing then returns MLD_ERR_SIGNING_PAUSED with `attempt` as the resume+ *   point (needs MLD_CONFIG_SIGN_HOOK_RESUME to resume; otherwise just aborts).+ *   Always returning 0 makes it a logging/benchmarking hook.+ *+ * - MLD_CONFIG_SIGN_HOOK_RESUME: uint16_t mld_sign_hook_resume([ctxt])+ *   Returns the attempt to resume from (0 for a fresh operation), i.e. the one+ *   recorded when a previous call paused.+ *+ * - MLD_CONFIG_SIGN_HOOK_FINISH: void mld_sign_hook_finish(attempt[, ctxt])+ *   Called on success with the succeeding attempt. Observe-only.+ *+ * When an option is unset, the hook is a no-op (resume to 0, attempt proceeds),+ * i.e. ordinary one-shot signing.+ *+ * Independent of MLD_CONFIG_MAX_SIGNING_ATTEMPTS, which is a static upper bound+ * on the number of signing attempts.+ *+ * See test/src/test_sign_hook.c for a worked example using all three.+ */+/* #define MLD_CONFIG_SIGN_HOOK_RESUME+   #define MLD_CONFIG_SIGN_HOOK_ATTEMPT+   #define MLD_CONFIG_SIGN_HOOK_FINISH+   #if !defined(__ASSEMBLER__)+   #include <stdint.h>+   #include "src/sys.h"+   static MLD_INLINE uint16_t mld_sign_hook_resume(void)+   {+       ... return the attempt to resume from ...+   }+   static MLD_INLINE int mld_sign_hook_attempt(uint16_t attempt)+   {+       ... return non-zero to pause here; for resume, store attempt ...+       return 0;+   }+   static MLD_INLINE void mld_sign_hook_finish(uint16_t attempt)+   {+       ... mark the operation complete (attempt = successful attempt) ...+   }+   #endif+*/++/**+ * MLD_CONFIG_REDUCE_RAM+ *+ * Set this to reduce RAM usage. This trades memory for performance.+ *+ * For expected memory usage, see the MLD_TOTAL_ALLOC_* constants defined in+ * mldsa_native.h.+ *+ * This option is useful for embedded systems with tight RAM constraints but+ * relaxed performance requirements.+ *+ */+/* #define MLD_CONFIG_REDUCE_RAM */++/*************************  Config internals  ********************************/++#endif /* MLD_BUILD_INTERNAL */++/* Default namespace+ *+ * Don't change this. If you need a different namespace, re-define+ * MLD_CONFIG_NAMESPACE_PREFIX above instead, and remove the following.+ *+ * The default MLDSA namespace is+ *+ *   PQCP_MLDSA_NATIVE_MLDSA<LEVEL>_+ *+ * e.g., PQCP_MLDSA_NATIVE_MLDSA44_+ */++#if defined(MLD_CONFIG_MULTILEVEL_BUILD)+/* In a multi-level build the parameter set is appended by the namespacing+ * machinery, so the default prefix must not embed it. */+#define MLD_DEFAULT_NAMESPACE_PREFIX PQCP_MLDSA_NATIVE_MLDSA+#elif MLD_CONFIG_PARAMETER_SET == 44+#define MLD_DEFAULT_NAMESPACE_PREFIX PQCP_MLDSA_NATIVE_MLDSA44+#elif MLD_CONFIG_PARAMETER_SET == 65+#define MLD_DEFAULT_NAMESPACE_PREFIX PQCP_MLDSA_NATIVE_MLDSA65+#elif MLD_CONFIG_PARAMETER_SET == 87+#define MLD_DEFAULT_NAMESPACE_PREFIX PQCP_MLDSA_NATIVE_MLDSA87+#endif++#endif /* !MLD_CONFIG_H */
+ cbits/mldsa/src/cbmc.h view
@@ -0,0 +1,233 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLD_CBMC_H+#define MLD_CBMC_H++/***************************************************+ * Basic replacements for __CPROVER_XXX contracts+ ***************************************************/+/*+ * The `__contract__` / `__loop__` annotation macros use a+ * leading-double-underscore spelling in line with other CBMC macros.+ * clang-tidy flags these as reserved identifiers; we suppress the diagnostic+ * at each definition site (NOLINT) rather than disabling the check globally,+ * so it stays active for the rest of the tree.+ */+#ifndef CBMC++/* clang-format off */+#define __contract__(x) /* NOLINT(bugprone-reserved-identifier,cert-dcl37-c,cert-dcl51-cpp) */+#define __loop__(x) /* NOLINT(bugprone-reserved-identifier,cert-dcl37-c,cert-dcl51-cpp) */+/* clang-format on */+#define cassert(x)++#else /* !CBMC */+++/* clang-format off */+#define __contract__(x) x /* NOLINT(bugprone-reserved-identifier,cert-dcl37-c,cert-dcl51-cpp) */+#define __loop__(x) x /* NOLINT(bugprone-reserved-identifier,cert-dcl37-c,cert-dcl51-cpp) */+/* clang-format on */++/* Conditionally expand to __VA_ARGS__ depending on MLD_CONFIG_REDUCE_RAM. */+#if defined(MLD_CONFIG_REDUCE_RAM)+#define MLD_IF_REDUCE_RAM(...) __VA_ARGS__+#define MLD_IF_NOT_REDUCE_RAM(...)+#else+#define MLD_IF_REDUCE_RAM(...)+#define MLD_IF_NOT_REDUCE_RAM(...) __VA_ARGS__+#endif++/* https://diffblue.github.io/cbmc/contracts-assigns.html */+#define assigns(...) __CPROVER_assigns(__VA_ARGS__)++/* https://diffblue.github.io/cbmc/contracts-requires-ensures.html */+#define requires(...) __CPROVER_requires(__VA_ARGS__)+#define ensures(...) __CPROVER_ensures(__VA_ARGS__)+/* https://diffblue.github.io/cbmc/contracts-loops.html */+#define invariant(...) __CPROVER_loop_invariant(__VA_ARGS__)+#define decreases(...) __CPROVER_decreases(__VA_ARGS__)+/* cassert to avoid confusion with in-built assert */+#define cassert(x) __CPROVER_assert(x, "cbmc assertion failed")+#define assume(...) __CPROVER_assume(__VA_ARGS__)++/***************************************************+ * Macros for "expression" forms that may appear+ * _inside_ top-level contracts.+ ***************************************************/++/*+ * function return value - useful inside ensures+ * https://diffblue.github.io/cbmc/contracts-functions.html+ */+#define return_value (__CPROVER_return_value)++/*+ * assigns l-value targets+ * https://diffblue.github.io/cbmc/contracts-assigns.html+ */+#define object_whole(...) __CPROVER_object_whole(__VA_ARGS__)+#define memory_slice(...) __CPROVER_object_upto(__VA_ARGS__)+#define same_object(...) __CPROVER_same_object(__VA_ARGS__)++/*+ * Pointer-related predicates+ * https://diffblue.github.io/cbmc/contracts-memory-predicates.html+ */+#define memory_no_alias(...) __CPROVER_is_fresh(__VA_ARGS__)+#define readable(...) __CPROVER_r_ok(__VA_ARGS__)+#define writeable(...) __CPROVER_w_ok(__VA_ARGS__)++/* Maximum supported buffer size+ *+ * Larger buffers may be supported, but due to internal modeling constraints+ * in CBMC, the proofs of memory- and type-safety won't be able to run.+ *+ * If you find yourself in need for a buffer size larger than this,+ * please contact the maintainers, so we can prioritize work to relax+ * this somewhat artificial bound.+ */+#define MLD_MAX_BUFFER_SIZE (SIZE_MAX >> 12)+++/*+ * History variables+ * https://diffblue.github.io/cbmc/contracts-history-variables.html+ */+#define old(...) __CPROVER_old(__VA_ARGS__)+#define loop_entry(...) __CPROVER_loop_entry(__VA_ARGS__)++/*+ * Quantifiers+ * Note that the range on qvar is _exclusive_ between qvar_lb .. qvar_ub+ * https://diffblue.github.io/cbmc/contracts-quantifiers.html+ *+ * The quantified variable is declared as uint32_t, so these macros+ * quantify only over indices in [0, UINT32_MAX). Bounds larger than+ * UINT32_MAX (4 GiB) are NOT supported: the explicit (uint32_t) casts+ * on the bounds will trigger CBMC's conversion check if a wider bound+ * (e.g. a size_t > UINT32_MAX) is passed.+ *+ * Quantifying over size_t (64-bit) was found to blow up SMT proof+ * times, so we deliberately keep the index width at 32 bits. Callers+ * dealing with size_t-typed buffers must add an explicit+ *   requires(len <= UINT32_MAX)+ * precondition.+ */++/*+ * Prevent clang-format from corrupting CBMC's special ==> operator+ */+/* clang-format off */+#define forall(qvar, qvar_lb, qvar_ub, predicate)                              \+  __CPROVER_forall                                                             \+  {                                                                            \+    uint32_t qvar;                                                             \+    ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==> (predicate) \+  }++#define exists(qvar, qvar_lb, qvar_ub, predicate)                              \+  __CPROVER_exists                                                             \+  {                                                                            \+    uint32_t qvar;                                                             \+    ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) && (predicate) \+  }+/* clang-format on */++/***************************************************+ * Convenience macros for common contract patterns+ ***************************************************/+/*+ * Prevent clang-format from corrupting CBMC's special ==> operator+ */+/* clang-format off */+#define CBMC_CONCAT_(left, right) left##right+#define CBMC_CONCAT(left, right) CBMC_CONCAT_(left, right)++#define array_bound_core(qvar, qvar_lb, qvar_ub, array_var,            \+                         value_lb, value_ub)                           \+  __CPROVER_forall                                                     \+  {                                                                    \+    uint32_t qvar;                                                     \+    ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==> \+        (((int)(value_lb) <= ((array_var)[(qvar)])) &&                 \+         (((array_var)[(qvar)]) < (int)(value_ub)))                    \+  }++#define array_bound(array_var, qvar_lb, qvar_ub, value_lb, value_ub) \+  array_bound_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), (qvar_lb),   \+      (qvar_ub), (array_var), (value_lb), (value_ub))++#define array_unchanged_core(qvar, qvar_lb, qvar_ub, array_var)                   \+  __CPROVER_forall                                                                \+  {                                                                               \+    uint32_t qvar;                                                                \+    ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==>         \+    ((array_var)[(qvar)]) == (old(* (int32_t (*)[(qvar_ub)])(array_var)))[(qvar)] \+  }++#define array_unchanged(array_var, N) \+    array_unchanged_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), 0, (N), (array_var))++#define array_unchanged_u64_core(qvar, qvar_lb, qvar_ub, array_var)                \+  __CPROVER_forall                                                                 \+  {                                                                                \+    uint32_t qvar;                                                                 \+    ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==>          \+    ((array_var)[(qvar)]) == (old(* (uint64_t (*)[(qvar_ub)])(array_var)))[(qvar)] \+  }++#define array_unchanged_u64(array_var, N) \+    array_unchanged_u64_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), 0, (N), (array_var))++#define array_unchanged_u8_core(qvar, qvar_lb, qvar_ub, array_var)                \+  __CPROVER_forall                                                                \+  {                                                                               \+    uint32_t qvar;                                                                \+    ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==>         \+    ((array_var)[(qvar)]) == (old(* (uint8_t (*)[(qvar_ub)])(array_var)))[(qvar)] \+  }++#define array_unchanged_u8(array_var, N) \+    array_unchanged_u8_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), 0, (N), (array_var))++#define array_zeroized_u8_core(qvar, qvar_lb, qvar_ub, array_var)              \+  __CPROVER_forall                                                            \+  {                                                                           \+    uint32_t qvar;                                                            \+    ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==>     \+    ((array_var)[(qvar)]) == 0                                                \+  }++#define array_zeroized_u8(array_var, N) \+    array_zeroized_u8_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), 0, (N), (array_var))+/* clang-format on */++/*+ * Output-buffer discipline on failure, as documented in API-CONVENTIONS.md:+ * when a function fails, each caller-owned output buffer is left either+ * fully unchanged or fully zeroized -- never holding partially computed or+ * stale data that could be mistaken for a valid result.+ *+ * Note the disjunction is over the buffer as a whole: it is not enough for+ * each byte to be individually either unchanged or zero.+ */+#define array_unchanged_or_zeroized_u8(array_var, N) \+  (array_unchanged_u8((array_var), (N)) || array_zeroized_u8((array_var), (N)))++/* Wrapper around array_bound operating on absolute values.+ *+ * The absolute value bound `k` is exclusive.+ *+ * Note that since the lower bound in array_bound is inclusive, we have to+ * raise it by 1 here.+ */+#define array_abs_bound(arr, lb, ub, k) \+  array_bound((arr), (lb), (ub), -((int)(k)) + 1, (k))++#endif /* CBMC */++#endif /* !MLD_CBMC_H */
+ cbits/mldsa/src/common.h view
@@ -0,0 +1,301 @@+/*+ * Copyright (c) The mldsa-native project authors+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLD_COMMON_H+#define MLD_COMMON_H++#ifndef __ASSEMBLER__+#include <stdint.h>+#endif+++#define MLD_BUILD_INTERNAL++#if defined(MLD_CONFIG_FILE)+#include MLD_CONFIG_FILE+#else+#include "mldsa_native_config.h"+#endif++#include "params.h"+#include "sys.h"++/* Internal and public API have external linkage by default, but+ * this can be overwritten by the user, e.g. for single-CU builds. */+#if !defined(MLD_CONFIG_INTERNAL_API_QUALIFIER)+#define MLD_INTERNAL_API+#define MLD_INTERNAL_DATA_DECLARATION extern+#define MLD_INTERNAL_DATA_DEFINITION+#else+#define MLD_INTERNAL_API MLD_CONFIG_INTERNAL_API_QUALIFIER+#define MLD_INTERNAL_DATA_DECLARATION MLD_CONFIG_INTERNAL_API_QUALIFIER+#define MLD_INTERNAL_DATA_DEFINITION MLD_CONFIG_INTERNAL_API_QUALIFIER+#endif++#if !defined(MLD_CONFIG_EXTERNAL_API_QUALIFIER)+#define MLD_EXTERNAL_API+#else+#define MLD_EXTERNAL_API MLD_CONFIG_EXTERNAL_API_QUALIFIER+#endif++#if defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) || \+    defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED)+#define MLD_MULTILEVEL_BUILD+#endif++#define MLD_CONCAT_(x1, x2) x1##x2+#define MLD_CONCAT(x1, x2) MLD_CONCAT_(x1, x2)++#if defined(MLD_MULTILEVEL_BUILD)+#define MLD_ADD_PARAM_SET(s) MLD_CONCAT(s, MLD_CONFIG_PARAMETER_SET)+#else+#define MLD_ADD_PARAM_SET(s) s+#endif++#define MLD_NAMESPACE_PREFIX MLD_CONCAT(MLD_CONFIG_NAMESPACE_PREFIX, _)+#define MLD_NAMESPACE_PREFIX_KL \+  MLD_CONCAT(MLD_ADD_PARAM_SET(MLD_CONFIG_NAMESPACE_PREFIX), _)++/* Functions are prefixed by MLD_CONFIG_NAMESPACE_PREFIX.+ *+ * If multiple parameter sets are used, functions depending on the parameter+ * set are additionally prefixed with 44/65/87. See mldsa_native_config.h.+ *+ * Example: If MLD_CONFIG_NAMESPACE_PREFIX is PQCP_MLDSA_NATIVE, then+ * MLD_NAMESPACE_KL(keypair) becomes PQCP_MLDSA_NATIVE44_keypair/+ * PQCP_MLDSA_NATIVE65_keypair/PQCP_MLDSA_NATIVE87_keypair.+ */+#define MLD_NAMESPACE(s) MLD_CONCAT(MLD_NAMESPACE_PREFIX, s)+#define MLD_NAMESPACE_KL(s) MLD_CONCAT(MLD_NAMESPACE_PREFIX_KL, s)++/* On Apple platforms, we need to emit leading underscore+ * in front of assembly symbols. We thus introduce a separate+ * namespace wrapper for ASM symbols. */+#if !defined(__APPLE__)+#define MLD_ASM_NAMESPACE(sym) MLD_NAMESPACE(sym)+#else+#define MLD_ASM_NAMESPACE(sym) MLD_CONCAT(_, MLD_NAMESPACE(sym))+#endif++/*+ * On X86_64 if control-flow protections (CET) are enabled (through+ * -fcf-protection=), we add an endbr64 instruction at every global function+ * label.  See sys.h for more details+ */+#if defined(MLD_SYS_X86_64)+#define MLD_ASM_FN_SYMBOL(sym) MLD_ASM_NAMESPACE(sym) : MLD_CET_ENDBR+#elif defined(MLD_SYS_ARMV81M_MVE)+/* clang-format off */+#define MLD_ASM_FN_SYMBOL(sym) \+  .type MLD_ASM_NAMESPACE(sym), %function; \+  MLD_ASM_NAMESPACE(sym) :+/* clang-format on */+#else /* !MLD_SYS_X86_64 && MLD_SYS_ARMV81M_MVE */+#define MLD_ASM_FN_SYMBOL(sym) MLD_ASM_NAMESPACE(sym) :+#endif /* !MLD_SYS_X86_64 && !MLD_SYS_ARMV81M_MVE */++/*+ * Output the size of an assembly function.+ */+#if defined(__ELF__)+#define MLD_ASM_FN_SIZE(sym) \+  .size MLD_ASM_NAMESPACE(sym), .- MLD_ASM_NAMESPACE(sym)+#else+#define MLD_ASM_FN_SIZE(sym)+#endif++/* We aim to simplify the user's life by supporting builds where+ * all source files are included, even those that are not needed.+ * Those files are appropriately guarded and will be empty when unneeded.+ * The following is to avoid compilers complaining about this. */+#define MLD_EMPTY_CU(s) extern int MLD_NAMESPACE_KL(empty_cu_##s);++/* MLD_CONFIG_NO_ASM takes precedence over MLD_USE_NATIVE_XXX */+#if defined(MLD_CONFIG_NO_ASM)+#undef MLD_CONFIG_USE_NATIVE_BACKEND_ARITH+#undef MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202+#endif++#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_ARITH) && \+    !defined(MLD_CONFIG_ARITH_BACKEND_FILE)+#error Bad configuration: MLD_CONFIG_USE_NATIVE_BACKEND_ARITH is set, but MLD_CONFIG_ARITH_BACKEND_FILE is not.+#endif++#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202) && \+    !defined(MLD_CONFIG_FIPS202_BACKEND_FILE)+#error Bad configuration: MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 is set, but MLD_CONFIG_FIPS202_BACKEND_FILE is not.+#endif++#if defined(MLD_CONFIG_NO_RANDOMIZED_API) && defined(MLD_CONFIG_KEYGEN_PCT)+#error Bad configuration: MLD_CONFIG_NO_RANDOMIZED_API is incompatible with MLD_CONFIG_KEYGEN_PCT as the current PCT implementation requires signature()+#endif++#if defined(MLD_CONFIG_NO_SIGN_API) && defined(MLD_CONFIG_KEYGEN_PCT)+#error Bad configuration: MLD_CONFIG_NO_SIGN_API is incompatible with MLD_CONFIG_KEYGEN_PCT as the current PCT implementation requires signature()+#endif++#if defined(MLD_CONFIG_NO_VERIFY_API) && defined(MLD_CONFIG_KEYGEN_PCT)+#error Bad configuration: MLD_CONFIG_NO_VERIFY_API is incompatible with MLD_CONFIG_KEYGEN_PCT as the current PCT implementation requires verify()+#endif++#if defined(MLD_CONFIG_CORE_API_ONLY) && defined(MLD_CONFIG_KEYGEN_PCT)+#error Bad configuration: MLD_CONFIG_CORE_API_ONLY is incompatible with MLD_CONFIG_KEYGEN_PCT as the current PCT implementation requires signature() and verify()+#endif++#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_ARITH)+#include MLD_CONFIG_ARITH_BACKEND_FILE+/* Include to enforce consistency of API and implementation,+ * and conduct sanity checks on the backend.+ *+ * Keep this _after_ the inclusion of the backend; otherwise,+ * the sanity checks won't have an effect. */+#if defined(MLD_CHECK_APIS) && !defined(__ASSEMBLER__)+#include "native/api.h"+#endif+#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_ARITH */++#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202)+#include MLD_CONFIG_FIPS202_BACKEND_FILE+/* Include to enforce consistency of API and implementation,+ * and conduct sanity checks on the backend.+ *+ * Keep this _after_ the inclusion of the backend; otherwise,+ * the sanity checks won't have an effect. */+#if defined(MLD_CHECK_APIS) && !defined(__ASSEMBLER__)+#include "fips202/native/api.h"+#endif+#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 */++#if !defined(MLD_CONFIG_FIPS202_CUSTOM_HEADER)+#define MLD_FIPS202_HEADER_FILE "fips202/fips202.h"+#else+#define MLD_FIPS202_HEADER_FILE MLD_CONFIG_FIPS202_CUSTOM_HEADER+#endif++#if !defined(MLD_CONFIG_FIPS202X4_CUSTOM_HEADER)+#define MLD_FIPS202X4_HEADER_FILE "fips202/fips202x4.h"+#else+#define MLD_FIPS202X4_HEADER_FILE MLD_CONFIG_FIPS202X4_CUSTOM_HEADER+#endif++/* Standard library function replacements */+#if !defined(__ASSEMBLER__)+#if !defined(MLD_CONFIG_CUSTOM_MEMCPY)+#include <string.h>+#define mld_memcpy memcpy+#endif++#if !defined(MLD_CONFIG_CUSTOM_MEMSET)+#include <string.h>+#define mld_memset memset+#endif++/* Allocation macros for large local structures+ *+ * MLD_ALLOC(v, T, N) declares T *v and attempts to point it to an T[N]+ * MLD_FREE(v, T, N) zeroizes and frees the allocation+ *+ * Default implementation uses stack allocation.+ * Can be overridden by setting the config option MLD_CONFIG_CUSTOM_ALLOC_FREE+ * and defining MLD_CUSTOM_ALLOC and MLD_CUSTOM_FREE.+ */+#if defined(MLD_CONFIG_CUSTOM_ALLOC_FREE) != \+    (defined(MLD_CUSTOM_ALLOC) && defined(MLD_CUSTOM_FREE))+#error Bad configuration: MLD_CONFIG_CUSTOM_ALLOC_FREE must be set together with MLD_CUSTOM_ALLOC and MLD_CUSTOM_FREE+#endif++/* Context-parameter machinery (MLD_CONTEXT_PARAMETERS_n and related config+ * checks). Kept in a separate, level-generic header for readability; included+ * here so it is available to the allocation macros below and to all consumers+ * of common.h. */+#include "context.h"++#if !defined(MLD_CONFIG_CUSTOM_ALLOC_FREE)+/* Default: stack allocation */++/* This is a declaration macro, not an expression macro: T is a type and v is+ * a declarator, neither of which can be wrapped in parentheses. The+ * bugprone-macro-parentheses diagnostic is therefore a false positive here. */+#define MLD_ALLOC(v, T, N, context) \+  MLD_ALIGN T mld_alloc_##v[N];     \+  T *v = mld_alloc_##v /* NOLINT(bugprone-macro-parentheses) */++/* The MLD_FREE macro body references mld_zeroize(), which is declared in+ * ct.h. We deliberately do NOT include ct.h here: doing so would create a+ * circular dependency (ct.h includes common.h), and common.h itself never+ * calls mld_zeroize() -- only the macro expansion does. Each translation+ * unit that uses MLD_FREE therefore includes ct.h directly. */+#define MLD_FREE(v, T, N, context)                     \+  do                                                   \+  {                                                    \+    MLD_CONTEXT_UNUSED(context);                       \+    mld_zeroize(mld_alloc_##v, sizeof(mld_alloc_##v)); \+    (v) = NULL;                                        \+  } while (0)++#else /* !MLD_CONFIG_CUSTOM_ALLOC_FREE */++/* Custom allocation */++/*+ * The indirection here is necessary to use MLD_CONTEXT_PARAMETERS_3 here.+ */+#define MLD_APPLY(f, args) f args++#define MLD_ALLOC(v, T, N, context) \+  MLD_APPLY(MLD_CUSTOM_ALLOC, MLD_CONTEXT_PARAMETERS_3(v, T, N, context))++#define MLD_FREE(v, T, N, context)                                            \+  do                                                                          \+  {                                                                           \+    if (v != NULL)                                                            \+    {                                                                         \+      mld_zeroize(v, sizeof(T) * (N));                                        \+      MLD_APPLY(MLD_CUSTOM_FREE, MLD_CONTEXT_PARAMETERS_3(v, T, N, context)); \+      v = NULL;                                                               \+    }                                                                         \+  } while (0)++#endif /* MLD_CONFIG_CUSTOM_ALLOC_FREE */++/****************************** Error codes ***********************************/++/* Generic failure condition, reserved for failures not covered by a more+ * specific error code. */+#define MLD_ERR_FAIL (-1)+/* An allocation failed. This can only happen if MLD_CONFIG_CUSTOM_ALLOC_FREE+ * is defined and the provided MLD_CUSTOM_ALLOC can fail. */+#define MLD_ERR_OUT_OF_MEMORY (-2)+/* An RNG failure occurred. Might be due to insufficient entropy or+ * system misconfiguration. */+#define MLD_ERR_RNG_FAIL (-3)+/* The signing rejection-sampling loop exceeded+ * MLD_CONFIG_MAX_SIGNING_ATTEMPTS iterations without producing a valid+ * signature. With a FIPS 204 Appendix C compliant bound (>= 821) this+ * has probability < 2^-256. */+#define MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED (-4)+/* Signing was paused before completing, at the request of a caller-provided+ * MLD_CONFIG_SIGN_HOOK_ATTEMPT hook (see mldsa_native_config.h). The caller+ * resumes by re-invoking signing with the same inputs; the attempt hook,+ * together with MLD_CONFIG_SIGN_HOOK_RESUME, decides where to continue. */+#define MLD_ERR_SIGNING_PAUSED (-5)+/* Signature verification failed: the signature is not valid for the given+ * message and public key. Returned by the verification API. */+#define MLD_ERR_INVALID_SIGNATURE (-6)+/* Secret key validation failed: the secret key is malformed or internally+ * inconsistent. Returned by pk_from_sk. */+#define MLD_ERR_INVALID_KEY (-7)+/* The Pairwise Consistency Test failed. Only possible when+ * MLD_CONFIG_KEYGEN_PCT is enabled; signals that the freshly generated key+ * pair failed its sign/verify self-test. */+#define MLD_ERR_PCT_FAIL (-8)+/* An argument was invalid, e.g. an unsupported pre-hash algorithm or a context+ * string longer than 255 bytes. */+#define MLD_ERR_INVALID_ARG (-9)+++#endif /* !__ASSEMBLER__ */++#endif /* !MLD_COMMON_H */
+ cbits/mldsa/src/context.h view
@@ -0,0 +1,152 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLD_CONTEXT_H+#define MLD_CONTEXT_H++/* This header is included by common.h once the configuration has been pulled+ * in; it is not meant to be included directly. */+#if !defined(__ASSEMBLER__)++#include <stdint.h>+#include "cbmc.h"+#include "sys.h"++/*+ * If the integration wants to provide a context parameter for use in+ * platform-specific hooks, then it should define this parameter.+ *+ * The MLD_CONTEXT_PARAMETERS_n macros are intended to be used with macros+ * defining the function names and expand to either pass or discard the context+ * argument as required by the current build.  If there is no context parameter+ * requested then these are removed from the prototypes and from all calls.+ */+#ifdef MLD_CONFIG_CONTEXT_PARAMETER+#define MLD_CONTEXT_PARAMETERS_0(context) (context)+#define MLD_CONTEXT_PARAMETERS_1(arg0, context) (arg0, context)+#define MLD_CONTEXT_PARAMETERS_2(arg0, arg1, context) (arg0, arg1, context)+#define MLD_CONTEXT_PARAMETERS_3(arg0, arg1, arg2, context) \+  (arg0, arg1, arg2, context)+#define MLD_CONTEXT_PARAMETERS_4(arg0, arg1, arg2, arg3, context) \+  (arg0, arg1, arg2, arg3, context)+#define MLD_CONTEXT_PARAMETERS_5(arg0, arg1, arg2, arg3, arg4, context) \+  (arg0, arg1, arg2, arg3, arg4, context)+#define MLD_CONTEXT_PARAMETERS_6(arg0, arg1, arg2, arg3, arg4, arg5, context) \+  (arg0, arg1, arg2, arg3, arg4, arg5, context)+#define MLD_CONTEXT_PARAMETERS_7(arg0, arg1, arg2, arg3, arg4, arg5, arg6, \+                                 context)                                  \+  (arg0, arg1, arg2, arg3, arg4, arg5, arg6, context)+#define MLD_CONTEXT_PARAMETERS_8(arg0, arg1, arg2, arg3, arg4, arg5, arg6, \+                                 arg7, context)                            \+  (arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7, context)+#define MLD_CONTEXT_PARAMETERS_9(arg0, arg1, arg2, arg3, arg4, arg5, arg6, \+                                 arg7, arg8, context)                      \+  (arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, context)+#else /* MLD_CONFIG_CONTEXT_PARAMETER */+#define MLD_CONTEXT_PARAMETERS_0(context) ()+#define MLD_CONTEXT_PARAMETERS_1(arg0, context) (arg0)+#define MLD_CONTEXT_PARAMETERS_2(arg0, arg1, context) (arg0, arg1)+#define MLD_CONTEXT_PARAMETERS_3(arg0, arg1, arg2, context) (arg0, arg1, arg2)+#define MLD_CONTEXT_PARAMETERS_4(arg0, arg1, arg2, arg3, context) \+  (arg0, arg1, arg2, arg3)+#define MLD_CONTEXT_PARAMETERS_5(arg0, arg1, arg2, arg3, arg4, context) \+  (arg0, arg1, arg2, arg3, arg4)+#define MLD_CONTEXT_PARAMETERS_6(arg0, arg1, arg2, arg3, arg4, arg5, context) \+  (arg0, arg1, arg2, arg3, arg4, arg5)+#define MLD_CONTEXT_PARAMETERS_7(arg0, arg1, arg2, arg3, arg4, arg5, arg6, \+                                 context)                                  \+  (arg0, arg1, arg2, arg3, arg4, arg5, arg6)+#define MLD_CONTEXT_PARAMETERS_8(arg0, arg1, arg2, arg3, arg4, arg5, arg6, \+                                 arg7, context)                            \+  (arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7)+#define MLD_CONTEXT_PARAMETERS_9(arg0, arg1, arg2, arg3, arg4, arg5, arg6, \+                                 arg7, arg8, context)                      \+  (arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8)+#endif /* !MLD_CONFIG_CONTEXT_PARAMETER */++/* Consume a context parameter carried only for the integration's benefit,+ * avoiding -Wunused-parameter; expands to nothing when no context is+ * configured. */+#if defined(MLD_CONFIG_CONTEXT_PARAMETER)+#define MLD_CONTEXT_UNUSED(context) ((void)(context))+#else+#define MLD_CONTEXT_UNUSED(context) ((void)0)+#endif++#if defined(MLD_CONFIG_CONTEXT_PARAMETER_TYPE) != \+    defined(MLD_CONFIG_CONTEXT_PARAMETER)+#error MLD_CONFIG_CONTEXT_PARAMETER_TYPE must be defined if and only if MLD_CONFIG_CONTEXT_PARAMETER is defined+#endif++/* The signing hooks tie into the rejection-sampling loop. A pausing attempt+ * hook only reproduces the uninterrupted signature if the randomness is fixed+ * across calls, and thus requires the deterministic API.+ * For now we impose that requirement on all three hooks uniformly: enabling any+ * of them requires MLD_CONFIG_NO_RANDOMIZED_API. This also rules out+ * MLD_CONFIG_KEYGEN_PCT (whose PCT needs the randomized signature(), see+ * common.h).+ *+ * A logging-only use (attempt always returns 0; resume/finish merely observe)+ * would be safe with the randomized API too, but the restriction is applied+ * uniformly for now. */+#if (defined(MLD_CONFIG_SIGN_HOOK_RESUME) ||  \+     defined(MLD_CONFIG_SIGN_HOOK_ATTEMPT) || \+     defined(MLD_CONFIG_SIGN_HOOK_FINISH)) && \+    !defined(MLD_CONFIG_NO_RANDOMIZED_API)+#error Signing hooks (MLD_CONFIG_SIGN_HOOK_RESUME / _ATTEMPT / _FINISH) require MLD_CONFIG_NO_RANDOMIZED_API+#endif /* (MLD_CONFIG_SIGN_HOOK_RESUME || MLD_CONFIG_SIGN_HOOK_ATTEMPT || \+          MLD_CONFIG_SIGN_HOOK_FINISH) && !MLD_CONFIG_NO_RANDOMIZED_API */++/* Signing hooks (MLD_CONFIG_SIGN_HOOK_RESUME / _ATTEMPT / _FINISH; documented+ * in mldsa_native_config.h). The following macros route the call sites to+ * mld_sign_hook_*, appending or dropping the context argument; each unset hook+ * uses the dummy below. */+#define mld_sign_resume mld_sign_hook_resume MLD_CONTEXT_PARAMETERS_0+#define mld_sign_attempt mld_sign_hook_attempt MLD_CONTEXT_PARAMETERS_1+#define mld_sign_finish mld_sign_hook_finish MLD_CONTEXT_PARAMETERS_1++/* We don't use mld_sign_resume here because MLD_CONTEXT_PARAMETERS_0 is+ * unsuitable for function declarations: it misses `void` as the placeholder+ * argument. */+#if !defined(MLD_CONFIG_SIGN_HOOK_RESUME)+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE uint16_t mld_sign_hook_resume(+#if defined(MLD_CONFIG_CONTEXT_PARAMETER)+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context+#else+    void+#endif+)+__contract__(assigns() ensures(1))+{+  MLD_CONTEXT_UNUSED(context);+  return 0;+}+#endif /* !MLD_CONFIG_SIGN_HOOK_RESUME */++#if !defined(MLD_CONFIG_SIGN_HOOK_ATTEMPT)+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_sign_attempt(+    uint16_t attempt, MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(assigns() ensures(1))+{+  ((void)attempt);+  MLD_CONTEXT_UNUSED(context);+  return 0;+}+#endif /* !MLD_CONFIG_SIGN_HOOK_ATTEMPT */++#if !defined(MLD_CONFIG_SIGN_HOOK_FINISH)+static MLD_INLINE void mld_sign_finish(+    uint16_t attempt, MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(assigns() ensures(1))+{+  ((void)attempt);+  MLD_CONTEXT_UNUSED(context);+}+#endif /* !MLD_CONFIG_SIGN_HOOK_FINISH */++#endif /* !__ASSEMBLER__ */++#endif /* !MLD_CONTEXT_H */
+ cbits/mldsa/src/ct.c view
@@ -0,0 +1,21 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#include "ct.h"++#if !defined(MLD_USE_ASM_VALUE_BARRIER) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)+/*+ * Masking value used in constant-time functions from+ * ct.h to block the compiler's range analysis and+ * thereby reduce the risk of compiler-introduced branches.+ */+volatile uint64_t mld_ct_opt_blocker_u64 = 0;++#else /* !MLD_USE_ASM_VALUE_BARRIER && !MLD_CONFIG_MULTILEVEL_NO_SHARED */++MLD_EMPTY_CU(ct)++#endif /* !(!MLD_USE_ASM_VALUE_BARRIER && !MLD_CONFIG_MULTILEVEL_NO_SHARED) */
+ cbits/mldsa/src/ct.h view
@@ -0,0 +1,373 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ *+ * - [libmceliece]+ *   libmceliece implementation of Classic McEliece+ *   Bernstein, Chou+ *   https://lib.mceliece.org/+ *+ * - [optblocker]+ *   PQC forum post on opt-blockers using volatile globals+ *   Daniel J. Bernstein+ *   https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/hqbtIGFKIpU/m/H14H0wOlBgAJ+ */++#ifndef MLD_CT_H+#define MLD_CT_H++#include "cbmc.h"+#include "common.h"++/* Constant-time comparisons and conditional operations++   We reduce the risk for compilation into variable-time code+   through the use of 'value barriers'.++   Functionally, a value barrier is a no-op. To the compiler, however,+   it constitutes an arbitrary modification of its input, and therefore+   harden's value propagation and range analysis.++   We consider two approaches to implement a value barrier:+   - An empty inline asm block which marks the target value as clobbered.+   - XOR'ing with the value of a volatile global that's set to 0;+     see @[optblocker] for a discussion of this idea, and+     @[libmceliece, inttypes/crypto_intN.h] for an implementation.++   The first approach is cheap because it only prevents the compiler+   from reasoning about the value of the variable past the barrier,+   but does not directly generate additional instructions.++   The second approach generates redundant loads and XOR operations+   and therefore comes at a higher runtime cost. However, it appears+   more robust towards optimization, as compilers should never drop+   a volatile load.++   We use the empty-ASM value barrier for GCC and clang, and fall+   back to the global volatile barrier otherwise.++   The global value barrier can be forced by setting+   MLD_CONFIG_NO_ASM_VALUE_BARRIER.++*/++#if defined(MLD_HAVE_INLINE_ASM) && !defined(MLD_CONFIG_NO_ASM_VALUE_BARRIER)+#define MLD_USE_ASM_VALUE_BARRIER+#endif+++#if !defined(MLD_USE_ASM_VALUE_BARRIER)+/*+ * Declaration of global volatile that the global value barrier+ * is loading from and masking with.+ */+#define mld_ct_opt_blocker_u64 MLD_NAMESPACE(ct_opt_blocker_u64)+extern volatile uint64_t mld_ct_opt_blocker_u64;+++/* Helper functions for obtaining global masks of various sizes */++/* This contract is not proved but treated as an axiom.+ *+ * Its validity relies on the assumption that the global opt-blocker+ * constant mld_ct_opt_blocker_u64 is not modified.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE uint64_t mld_ct_get_optblocker_u64(void)+__contract__(ensures(return_value == 0)) { return mld_ct_opt_blocker_u64; }++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int64_t mld_ct_get_optblocker_i64(void)+__contract__(ensures(return_value == 0)) { return (int64_t)mld_ct_get_optblocker_u64(); }++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE uint32_t mld_ct_get_optblocker_u32(void)+__contract__(ensures(return_value == 0)) { return (uint32_t)mld_ct_get_optblocker_u64(); }++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE uint8_t mld_ct_get_optblocker_u8(void)+__contract__(ensures(return_value == 0)) { return (uint8_t)mld_ct_get_optblocker_u64(); }++/* Opt-blocker based implementation of value barriers */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int64_t mld_value_barrier_i64(int64_t b)+__contract__(ensures(return_value == b)) { return (b ^ mld_ct_get_optblocker_i64()); }++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE uint32_t mld_value_barrier_u32(uint32_t b)+__contract__(ensures(return_value == b)) { return (b ^ mld_ct_get_optblocker_u32()); }++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE uint8_t mld_value_barrier_u8(uint8_t b)+__contract__(ensures(return_value == b)) { return (b ^ mld_ct_get_optblocker_u8()); }+++#else  /* !MLD_USE_ASM_VALUE_BARRIER */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int64_t mld_value_barrier_i64(int64_t b)+__contract__(ensures(return_value == b))+{+  __asm__ volatile("" : "+r"(b));+  return b;+}++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE uint32_t mld_value_barrier_u32(uint32_t b)+__contract__(ensures(return_value == b))+{+  __asm__ volatile("" : "+r"(b));+  return b;+}++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE uint8_t mld_value_barrier_u8(uint8_t b)+__contract__(ensures(return_value == b))+{+  __asm__ volatile("" : "+r"(b));+  return b;+}+#endif /* MLD_USE_ASM_VALUE_BARRIER */++#ifdef CBMC+#pragma CPROVER check push+#pragma CPROVER check disable "conversion"+#endif++/**+ * Cast uint32 value to int32.+ *+ * @param x Input value.+ *+ * @return For uint32_t x, the unique y in int32_t so that x == y mod 2^32.+ *         Concretely:+ *         - x <  2^31: returns x+ *         - x >= 2^31: returns x - 2^32+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_ALWAYS_INLINE int32_t mld_cast_uint32_to_int32(uint32_t x)+{+  /*+   * PORTABILITY: This relies on uint32_t -> int32_t+   * being implemented as the inverse of int32_t -> uint32_t,+   * which is implementation-defined (C99 6.3.1.3 (3))+   * CBMC (correctly) fails to prove this conversion is OK,+   * so we have to suppress that check here+   */+  return (int32_t)x;+}++#ifdef CBMC+#pragma CPROVER check pop+#endif+++/**+ * Cast int64 value to uint32 as per C standard.+ *+ * @param x Input value.+ *+ * @return For int64_t x, the unique y in uint32_t so that x == y mod 2^32.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_ALWAYS_INLINE uint32_t mld_cast_int64_to_uint32(int64_t x)+{+  return (uint32_t)(x & (int64_t)UINT32_MAX);+}++/**+ * Cast int32 value to uint32 as per C standard.+ *+ * @param x Input value.+ *+ * @return For int32_t x, the unique y in uint32_t so that x == y mod 2^32.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_ALWAYS_INLINE uint32_t mld_cast_int32_to_uint32(int32_t x)+{+  return mld_cast_int64_to_uint32((int64_t)x);+}++/**+ * Functionally equivalent to cond ? a : b, but implemented with guards against+ * compiler-introduced branches.+ *+ * @param a    First alternative.+ * @param b    Second alternative.+ * @param cond Condition variable.+ *+ * @return a if cond is 0xFFFFFFFF, b if cond is 0.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int32_t mld_ct_sel_int32(int32_t a, int32_t b, uint32_t cond)+__contract__(+  requires(cond == 0x0 || cond == 0xFFFFFFFF)+  ensures(return_value == (cond ? a : b))+)+{+  uint32_t au = mld_cast_int32_to_uint32(a);+  uint32_t bu = mld_cast_int32_to_uint32(b);+  uint32_t res = bu ^ (mld_value_barrier_u32(cond) & (au ^ bu));+  return mld_cast_uint32_to_int32(res);+}++/**+ * Return 0 if input is zero, and -1 otherwise.+ *+ * @param x Value to be converted into a mask.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE uint32_t mld_ct_cmask_nonzero_u32(uint32_t x)+__contract__(ensures(return_value == ((x == 0) ? 0 : 0xFFFFFFFF)))+{+  int64_t tmp = mld_value_barrier_i64(-((int64_t)x));+  /*+   * PORTABILITY: Right-shift on a signed integer is+   * implementation-defined for negative left argument.+   * Here, we assume it's sign-preserving "arithmetic" shift right.+   * See (C99 6.5.7 (5))+   */+  tmp >>= 32;+  return mld_cast_int64_to_uint32(tmp);+}++/**+ * Return 0 if input is zero, and -1 otherwise.+ *+ * @param x Value to be converted into a mask.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE uint8_t mld_ct_cmask_nonzero_u8(uint8_t x)+__contract__(ensures(return_value == ((x == 0) ? 0 : 0xFF)))+{+  uint32_t mask = mld_ct_cmask_nonzero_u32((uint32_t)x);+  return (uint8_t)(mask & 0xFF);+}++/**+ * Return 0 if input is non-negative, and -1 otherwise.+ *+ * @param x Value to be converted into a mask.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE uint32_t mld_ct_cmask_neg_i32(int32_t x)+__contract__(+  ensures(return_value == ((x < 0) ? 0xFFFFFFFF : 0))+)+{+  int64_t tmp = mld_value_barrier_i64((int64_t)x);+  /*+   * PORTABILITY: Right-shift on a signed integer is+   * implementation-defined for negative left argument.+   * Here, we assume it's sign-preserving "arithmetic" shift right.+   * See (C99 6.5.7 (5))+   */+  tmp >>= 31;+  return mld_cast_int64_to_uint32(tmp);+}++/**+ * Return -x if x<0, x otherwise.+ *+ * @param x Input value.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int32_t mld_ct_abs_i32(int32_t x)+__contract__(+  requires(x >= -INT32_MAX)+  ensures(return_value == ((x < 0) ? -x : x))+)+{+  return mld_ct_sel_int32(-x, x, mld_ct_cmask_neg_i32(x));+}++/**+ * Compare two arrays for equality in constant time.+ *+ * @param[in] a   Pointer to first byte array.+ * @param[in] b   Pointer to second byte array.+ * @param     len Length of the byte arrays, upper-bounded to UINT16_MAX to+ *                control proof complexity only.+ *+ * @return 0 if the byte arrays are equal, 0xFF otherwise.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE uint8_t mld_ct_memcmp(const uint8_t *a, const uint8_t *b,+                                        const size_t len)+__contract__(+  requires(len <= UINT16_MAX)+  requires(memory_no_alias(a, len))+  requires(memory_no_alias(b, len))+  ensures((return_value == 0) || (return_value == 0xFF))+  ensures((return_value == 0) == forall(i, 0, len, (a[i] == b[i]))))+{+  uint8_t r = 0, s = 0;+  unsigned i;++  for (i = 0; i < len; i++)+  __loop__(+    invariant(i <= len)+    invariant((r == 0) == (forall(k, 0, i, (a[k] == b[k]))))+    decreases(len - i))+  {+    r |= a[i] ^ b[i];+    /* s is useless, but prevents the loop from being aborted once r=0xff. */+    s ^= a[i] ^ b[i];+  }++  /*+   * - Convert r into a mask; this may not be necessary, but is an additional+   *   safeguard+   *   towards leaking information about a and b.+   * - XOR twice with s, separated by a value barrier, to prevent the compile+   *   from dropping the s computation in the loop.+   */+  return (mld_value_barrier_u8(mld_ct_cmask_nonzero_u8(r) ^ s) ^ s);+}++/**+ * Force-zeroize a buffer.+ *+ * @[FIPS204, Section 3.6.3] Destruction of intermediate values.+ *+ * @param[out] ptr Pointer to buffer to be zeroed.+ * @param      len Amount of bytes to be zeroed.+ */+#if !defined(MLD_CONFIG_CUSTOM_ZEROIZE)+#if defined(MLD_SYS_WINDOWS)+#include <windows.h>+#elif !defined(MLD_HAVE_INLINE_ASM)+#error No plausibly-secure implementation of mld_zeroize available. Please provide your own using MLD_CONFIG_CUSTOM_ZEROIZE.+#endif++static MLD_INLINE void mld_zeroize(void *ptr, size_t len)+__contract__(+  requires(len <= UINT32_MAX)+  requires(memory_no_alias(ptr, len))+  assigns(memory_slice(ptr, len))+  ensures(array_zeroized_u8((uint8_t *)ptr, len)))+{+#if defined(MLD_SYS_WINDOWS)+  SecureZeroMemory(ptr, len);+#else+  mld_memset(ptr, 0, len);+  /* This follows OpenSSL and seems sufficient to prevent the compiler+   * from optimizing away the memset.+   *+   * If there was a reliable way to detect availability of memset_s(),+   * that would be preferred. */+  __asm__ __volatile__("" : : "r"(ptr) : "memory");+#endif /* !MLD_SYS_WINDOWS */+}+#endif /* !MLD_CONFIG_CUSTOM_ZEROIZE */++#endif /* !MLD_CT_H */
+ cbits/mldsa/src/debug.c view
@@ -0,0 +1,75 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* NOTE: You can remove this file unless you compile with MLDSA_DEBUG. */++#include "common.h"++#if !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++#if defined(MLDSA_DEBUG)++#include <inttypes.h>+#include <stdio.h>+#include <stdlib.h>+#include "debug.h"++#define MLD_DEBUG_ERROR_HEADER "[ERROR:%s:%04d] "++MLD_INTERNAL_API+void mld_debug_check_assert(const char *file, int line, const int val)+{+  if (val == 0)+  {+    fprintf(stderr, MLD_DEBUG_ERROR_HEADER "Assertion failed (value %d)\n",+            file, line, val);+    exit(1);+  }+}++MLD_INTERNAL_API+void mld_debug_check_bounds(const char *file, int line, const int32_t *ptr,+                            unsigned len, int64_t lower_bound_exclusive,+                            int64_t upper_bound_exclusive)+{+  int err = 0;+  unsigned i;+  for (i = 0; i < len; i++)+  {+    int32_t val = ptr[i];+    if (!(val > lower_bound_exclusive && val < upper_bound_exclusive))+    {+      fprintf(stderr,+              MLD_DEBUG_ERROR_HEADER+              "Bounds assertion failed: Index %u, value %d out of bounds "+              "(%" PRId64 ",%" PRId64 ")\n",+              file, line, i, (int)val, lower_bound_exclusive,+              upper_bound_exclusive);+      err = 1;+    }+  }++  if (err == 1)+  {+    exit(1);+  }+}++#else /* MLDSA_DEBUG */++MLD_EMPTY_CU(debug)++#endif /* !MLDSA_DEBUG */++#else /* !MLD_CONFIG_MULTILEVEL_NO_SHARED */++MLD_EMPTY_CU(debug)++#endif /* MLD_CONFIG_MULTILEVEL_NO_SHARED */++/* To facilitate single-compilation-unit (SCU) builds, undefine all macros.+ * Don't modify by hand -- this is auto-generated by scripts/autogen. */+#undef MLD_DEBUG_ERROR_HEADER
+ cbits/mldsa/src/debug.h view
@@ -0,0 +1,125 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLD_DEBUG_H+#define MLD_DEBUG_H+#include "common.h"++#if defined(MLDSA_DEBUG)++/**+ * Check debug assertion.+ *+ * Prints an error message to stderr and calls exit(1) if not.+ *+ * @param file Filename.+ * @param line Line number.+ * @param val  Value asserted to be non-zero.+ */+#define mld_debug_check_assert MLD_NAMESPACE(mldsa_debug_assert)+MLD_INTERNAL_API+void mld_debug_check_assert(const char *file, int line, const int val);++/**+ * Check whether values in an array of int32_t are within specified bounds.+ *+ * Prints an error message to stderr and calls exit(1) if not.+ *+ * @param     file                  Filename.+ * @param     line                  Line number.+ * @param[in] ptr                   Base of array to be checked.+ * @param     len                   Number of int32_t in ptr.+ * @param     lower_bound_exclusive Exclusive lower bound.+ * @param     upper_bound_exclusive Exclusive upper bound.+ */+#define mld_debug_check_bounds MLD_NAMESPACE(mldsa_debug_check_bounds)+MLD_INTERNAL_API+void mld_debug_check_bounds(const char *file, int line, const int32_t *ptr,+                            unsigned len, int64_t lower_bound_exclusive,+                            int64_t upper_bound_exclusive);++/* Check assertion, calling exit() upon failure+ *+ * val: Value that's asserted to be non-zero+ */+#define mld_assert(val) mld_debug_check_assert(__FILE__, __LINE__, (val))++/* Check bounds in array of int32_t's+ * ptr: Base of int32_t array; will be explicitly cast to int32_t*,+ *      so you may pass a byte-compatible type such as mld_poly or mld_polyvec.+ * len: Number of int32_t in array+ * value_lb: Inclusive lower value bound+ * value_ub: Exclusive upper value bound */+#define mld_assert_bound(ptr, len, value_lb, value_ub)                      \+  mld_debug_check_bounds(__FILE__, __LINE__, (const int32_t *)(ptr), (len), \+                         ((int64_t)(value_lb)) - 1, (value_ub))++/* Check absolute bounds in array of int32_t's+ * ptr: Base of array, expression of type int32_t*+ * len: Number of int32_t in array+ * value_abs_bd: Exclusive absolute upper bound */+#define mld_assert_abs_bound(ptr, len, value_abs_bd)               \+  mld_assert_bound((ptr), (len), (-((int64_t)(value_abs_bd)) + 1), \+                   (value_abs_bd))++/* Version of bounds assertions for 2-dimensional arrays */+#define mld_assert_bound_2d(ptr, len0, len1, value_lb, value_ub) \+  mld_assert_bound((ptr), ((len0) * (len1)), (value_lb), (value_ub))++#define mld_assert_abs_bound_2d(ptr, len0, len1, value_abs_bd) \+  mld_assert_abs_bound((ptr), ((len0) * (len1)), (value_abs_bd))++/* When running CBMC, convert debug assertions into proof obligations */+#elif defined(CBMC)+#include "cbmc.h"++#define mld_assert(val) cassert(val)++#define mld_assert_bound(ptr, len, value_lb, value_ub) \+  cassert(array_bound(((int32_t *)(ptr)), 0, (len), (value_lb), (value_ub)))++#define mld_assert_abs_bound(ptr, len, value_abs_bd) \+  cassert(array_abs_bound(((int32_t *)(ptr)), 0, (len), (value_abs_bd)))++/* Because of https://github.com/diffblue/cbmc/issues/8570, we can't+ * just use a single flattened array_bound(...) here. */+#define mld_assert_bound_2d(ptr, M, N, value_lb, value_ub)              \+  cassert(forall(kN, 0, (M),                                            \+                 array_bound(&((int32_t (*)[(N)])(ptr))[kN][0], 0, (N), \+                             (value_lb), (value_ub))))++#define mld_assert_abs_bound_2d(ptr, M, N, value_abs_bd)                    \+  cassert(forall(kN, 0, (M),                                                \+                 array_abs_bound(&((int32_t (*)[(N)])(ptr))[kN][0], 0, (N), \+                                 (value_abs_bd))))++#else /* !MLDSA_DEBUG && CBMC */++#define mld_assert(val) \+  do                    \+  {                     \+  } while (0)+#define mld_assert_bound(ptr, len, value_lb, value_ub) \+  do                                                   \+  {                                                    \+  } while (0)+#define mld_assert_abs_bound(ptr, len, value_abs_bd) \+  do                                                 \+  {                                                  \+  } while (0)++#define mld_assert_bound_2d(ptr, len0, len1, value_lb, value_ub) \+  do                                                             \+  {                                                              \+  } while (0)++#define mld_assert_abs_bound_2d(ptr, len0, len1, value_abs_bd) \+  do                                                           \+  {                                                            \+  } while (0)+++#endif /* !MLDSA_DEBUG && !CBMC */+#endif /* !MLD_DEBUG_H */
+ cbits/mldsa/src/fips202/fips202.c view
@@ -0,0 +1,270 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ *+ * - [mupq]+ *   Common files for pqm4, pqm3, pqriscv+ *   Kannwischer, Petri, Rijneveld, Schwabe, Stoffelen+ *   https://github.com/mupq/mupq+ *+ * - [supercop]+ *   SUPERCOP benchmarking framework+ *   Daniel J. Bernstein+ *   http://bench.cr.yp.to/supercop.html+ *+ * - [tweetfips]+ *   'tweetfips202' FIPS202 implementation+ *   Van Assche, Bernstein, Schwabe+ *   https://keccak.team/2015/tweetfips202.html+ */++/* Based on the CC0 implementation from @[mupq] and the public domain+ * implementation @[supercop, crypto_hash/keccakc512/simple/]+ * by Ronny Van Keer, and the public domain @[tweetfips] implementation. */++#include <stddef.h>++#include "../common.h"+#include "../ct.h"+#include "fips202.h"+#include "keccakf1600.h"+#if !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++/**+ * Initializes the Keccak state.+ *+ * @param[out] s Pointer to Keccak state.+ */+static void keccak_init(uint64_t s[MLD_KECCAK_LANES])+__contract__(+  requires(memory_no_alias(s, sizeof(uint64_t) * MLD_KECCAK_LANES))+  assigns(memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES))+)+{+  mld_memset(s, 0, sizeof(uint64_t) * MLD_KECCAK_LANES);+}++/**+ * Absorb step of Keccak; incremental.+ *+ * @param[in,out] s     Pointer to Keccak state.+ * @param         pos   Position in current block to be absorbed.+ * @param         r     Rate in bytes (e.g., 168 for SHAKE128).+ * @param[in]     in    Pointer to input to be absorbed into s.+ * @param         inlen Length of input in bytes.+ *+ * @return New position pos in current block.+ */+static unsigned int keccak_absorb(uint64_t s[MLD_KECCAK_LANES],+                                  unsigned int pos, unsigned int r,+                                  const uint8_t *in, size_t inlen)+__contract__(+  requires(inlen <= MLD_MAX_BUFFER_SIZE)+  requires(r > 0)+  requires(r < sizeof(uint64_t) * MLD_KECCAK_LANES)+  requires(pos <= r)+  requires(memory_no_alias(s, sizeof(uint64_t) * MLD_KECCAK_LANES))+  requires(memory_no_alias(in, inlen))+  assigns(memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES))+  ensures(return_value < r))+{+  while (inlen >= r - pos)+  __loop__(+    assigns(pos, in, inlen,+      memory_slice(s, sizeof(uint64_t) *  MLD_KECCAK_LANES))+    invariant(inlen <= loop_entry(inlen))+    invariant(pos <= r)+    invariant(in == loop_entry(in) + (loop_entry(inlen) - inlen))+    decreases(inlen + pos))+  {+    mld_keccakf1600_xor_bytes(s, in, pos, r - pos);+    inlen -= r - pos;+    in += r - pos;+    mld_keccakf1600_permute(s);+    pos = 0;+  }+  /* Safety: At this point, inlen < r, so the truncation to unsigned is safe. */+  mld_keccakf1600_xor_bytes(s, in, pos, (unsigned)inlen);++  /* Safety: At this point, inlen < r and pos <= r so the truncation to unsigned+   * is safe. */+  return (unsigned)(pos + inlen);+}++/**+ * Finalize absorb step.+ *+ * @param[in,out] s   Pointer to Keccak state.+ * @param         pos Position in current block to be absorbed.+ * @param         r   Rate in bytes (e.g., 168 for SHAKE128).+ * @param         p   Domain separation byte.+ */+static void keccak_finalize(uint64_t s[MLD_KECCAK_LANES], unsigned int pos,+                            unsigned int r, uint8_t p)+__contract__(+  requires(pos <= r && r < sizeof(uint64_t) * MLD_KECCAK_LANES)+  requires((r / 8) >= 1)+  requires(memory_no_alias(s, sizeof(uint64_t) * MLD_KECCAK_LANES))+  assigns(memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES))+)+{+  uint8_t b = 0x80;+  mld_keccakf1600_xor_bytes(s, &p, pos, 1);+  mld_keccakf1600_xor_bytes(s, &b, r - 1, 1);+}++/**+ * Squeeze step of Keccak. Squeezes arbitrarily many bytes. Modifies the+ * state. Can be called multiple times to keep squeezing, i.e., is+ * incremental.+ *+ * @param[out]    out    Pointer to output data.+ * @param         outlen Number of bytes to be squeezed (written to out).+ * @param[in,out] s      Pointer to input/output Keccak state.+ * @param         pos    Number of bytes in current block already squeezed.+ * @param         r      Rate in bytes (e.g., 168 for SHAKE128).+ *+ * @return New position pos in current block.+ */+static unsigned int keccak_squeeze(uint8_t *out, size_t outlen,+                                   uint64_t s[MLD_KECCAK_LANES],+                                   unsigned int pos, unsigned int r)+__contract__(+  requires((r == SHAKE128_RATE && pos <= SHAKE128_RATE) ||+           (r == SHAKE256_RATE && pos <= SHAKE256_RATE) ||+           (r == SHA3_512_RATE && pos <= SHA3_512_RATE))+  requires(outlen <= 8 * r /* somewhat arbitrary bound */)+  requires(memory_no_alias(s, sizeof(uint64_t) * MLD_KECCAK_LANES))+  requires(memory_no_alias(out, outlen))+  assigns(memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES))+  assigns(memory_slice(out, outlen))+  ensures(return_value <= r))+{+  unsigned int i;+  size_t out_offset = 0;++  /* Reference: This code is re-factored from the reference implementation+   * to facilitate proof with CBMC and to improve readability.+   *+   * Take a mutable copy of outlen to count down the number of bytes+   * still to squeeze. The initial value of outlen is needed for the CBMC+   * assigns() clauses. */+  size_t bytes_to_go = outlen;++  while (bytes_to_go > 0)+  __loop__(+    assigns(i, bytes_to_go, pos, out_offset, memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES), memory_slice(out, outlen))+    invariant(bytes_to_go <= outlen)+    invariant(out_offset == outlen - bytes_to_go)+    invariant(pos <= r)+    decreases(bytes_to_go)+  )+  {+    if (pos == r)+    {+      mld_keccakf1600_permute(s);+      pos = 0;+    }+    /* Safety: If bytes_to_go < r - pos, truncation to unsigned is safe. */+    i = bytes_to_go < r - pos ? (unsigned)bytes_to_go : r - pos;+    mld_keccakf1600_extract_bytes(s, out + out_offset, pos, i);+    bytes_to_go -= i;+    pos += i;+    out_offset += i;+  }++  return pos;+}++MLD_INTERNAL_API+void mld_shake128_init(mld_shake128ctx *state)+{+  keccak_init(state->s);+  state->pos = 0;+}++MLD_INTERNAL_API+void mld_shake128_absorb(mld_shake128ctx *state, const uint8_t *in,+                         size_t inlen)+{+  state->pos = keccak_absorb(state->s, state->pos, SHAKE128_RATE, in, inlen);+}++MLD_INTERNAL_API+void mld_shake128_finalize(mld_shake128ctx *state)+{+  keccak_finalize(state->s, state->pos, SHAKE128_RATE, 0x1F);+  state->pos = SHAKE128_RATE;+}++MLD_INTERNAL_API+void mld_shake128_squeeze(uint8_t *out, size_t outlen, mld_shake128ctx *state)+{+  state->pos = keccak_squeeze(out, outlen, state->s, state->pos, SHAKE128_RATE);+}++MLD_INTERNAL_API+void mld_shake128_release(mld_shake128ctx *state)+{+  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(state, sizeof(mld_shake128ctx));+}++MLD_INTERNAL_API+void mld_shake256_init(mld_shake256ctx *state)+{+  keccak_init(state->s);+  state->pos = 0;+}++MLD_INTERNAL_API+void mld_shake256_absorb(mld_shake256ctx *state, const uint8_t *in,+                         size_t inlen)+{+  state->pos = keccak_absorb(state->s, state->pos, SHAKE256_RATE, in, inlen);+}++MLD_INTERNAL_API+void mld_shake256_finalize(mld_shake256ctx *state)+{+  keccak_finalize(state->s, state->pos, SHAKE256_RATE, 0x1F);+  state->pos = SHAKE256_RATE;+}++MLD_INTERNAL_API+void mld_shake256_squeeze(uint8_t *out, size_t outlen, mld_shake256ctx *state)+{+  state->pos = keccak_squeeze(out, outlen, state->s, state->pos, SHAKE256_RATE);+}++MLD_INTERNAL_API+void mld_shake256_release(mld_shake256ctx *state)+{+  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(state, sizeof(mld_shake256ctx));+}++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_CORE_API_ONLY)+MLD_INTERNAL_API+void mld_shake256(uint8_t *out, size_t outlen, const uint8_t *in, size_t inlen)+{+  mld_shake256ctx state;++  mld_shake256_init(&state);+  mld_shake256_absorb(&state, in, inlen);+  mld_shake256_finalize(&state);+  mld_shake256_squeeze(out, outlen, &state);+  mld_shake256_release(&state);+}+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_CORE_API_ONLY */++#endif /* !MLD_CONFIG_MULTILEVEL_NO_SHARED */
+ cbits/mldsa/src/fips202/fips202.h view
@@ -0,0 +1,224 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLD_FIPS202_FIPS202_H+#define MLD_FIPS202_FIPS202_H++#include <stddef.h>+#include "../cbmc.h"+#include "../common.h"++#define SHAKE128_RATE 168+#define SHAKE256_RATE 136+#define SHA3_256_RATE 136+#define SHA3_512_RATE 72+#define MLD_KECCAK_LANES 25+#define SHA3_256_HASHBYTES 32+#define SHA3_512_HASHBYTES 64+++/** Context for the incremental SHAKE128 XOF. */+typedef struct+{+  uint64_t s[MLD_KECCAK_LANES]; /**< Keccak state. */+  unsigned int pos; /**< Byte position within the current Keccak block. */+} mld_shake128ctx;++/** Context for the incremental SHAKE256 XOF. */+typedef struct+{+  uint64_t s[MLD_KECCAK_LANES]; /**< Keccak state. */+  unsigned int pos; /**< Byte position within the current Keccak block. */+} mld_shake256ctx;++#define mld_shake128_init MLD_NAMESPACE(shake128_init)+/**+ * Initializes state for use as SHAKE128 XOF.+ *+ * @param[out] state Pointer to (uninitialized) state.+ */+MLD_INTERNAL_API+void mld_shake128_init(mld_shake128ctx *state)+__contract__(+  requires(memory_no_alias(state, sizeof(mld_shake128ctx)))+  assigns(memory_slice(state, sizeof(mld_shake128ctx)))+  ensures(state->pos == 0)+);++#define mld_shake128_absorb MLD_NAMESPACE(shake128_absorb)+/**+ * Absorb step of the SHAKE128 XOF. Absorbs arbitrarily many bytes. Can be+ * called multiple times to absorb multiple chunks of data.+ *+ * @param[in,out] state Pointer to (initialized) output state.+ * @param[in]     in    Pointer to input to be absorbed into s.+ * @param         inlen Length of input in bytes.+ */+MLD_INTERNAL_API+void mld_shake128_absorb(mld_shake128ctx *state, const uint8_t *in,+                         size_t inlen)+__contract__(+  requires(inlen <= MLD_MAX_BUFFER_SIZE)+  requires(memory_no_alias(state, sizeof(mld_shake128ctx)))+  requires(memory_no_alias(in, inlen))+  requires(state->pos <= SHAKE128_RATE)+  assigns(memory_slice(state, sizeof(mld_shake128ctx)))+  ensures(state->pos <= SHAKE128_RATE)+);++#define mld_shake128_finalize MLD_NAMESPACE(shake128_finalize)+/**+ * Concludes the absorb phase of the SHAKE128 XOF.+ *+ * @param[in,out] state Pointer to state.+ */+MLD_INTERNAL_API+void mld_shake128_finalize(mld_shake128ctx *state)+__contract__(+  requires(memory_no_alias(state, sizeof(mld_shake128ctx)))+  requires(state->pos <= SHAKE128_RATE)+  assigns(memory_slice(state, sizeof(mld_shake128ctx)))+  ensures(state->pos <= SHAKE128_RATE)+);++#define mld_shake128_squeeze MLD_NAMESPACE(shake128_squeeze)+/**+ * Squeeze step of SHAKE128 XOF. Squeezes arbitrarily many bytes. Can be+ * called multiple times to keep squeezing.+ *+ * @param[out]    out    Pointer to output blocks.+ * @param         outlen Number of bytes to be squeezed (written to output).+ * @param[in,out] state  Pointer to input/output state.+ */+MLD_INTERNAL_API+void mld_shake128_squeeze(uint8_t *out, size_t outlen, mld_shake128ctx *state)+__contract__(+  requires(outlen <= 8 * SHAKE128_RATE /* somewhat arbitrary bound */)+  requires(memory_no_alias(state, sizeof(mld_shake128ctx)))+  requires(memory_no_alias(out, outlen))+  requires(state->pos <= SHAKE128_RATE)+  assigns(memory_slice(state, sizeof(mld_shake128ctx)))+  assigns(memory_slice(out, outlen))+  ensures(state->pos <= SHAKE128_RATE)+);++#define mld_shake128_release MLD_NAMESPACE(shake128_release)+/**+ * Release and securely zero the SHAKE128 state.+ *+ * @param[in,out] state Pointer to state.+ */+MLD_INTERNAL_API+void mld_shake128_release(mld_shake128ctx *state)+__contract__(+  requires(memory_no_alias(state, sizeof(mld_shake128ctx)))+  assigns(memory_slice(state, sizeof(mld_shake128ctx)))+);++#define mld_shake256_init MLD_NAMESPACE(shake256_init)+/**+ * Initializes state for use as SHAKE256 XOF.+ *+ * @param[out] state Pointer to (uninitialized) state.+ */+MLD_INTERNAL_API+void mld_shake256_init(mld_shake256ctx *state)+__contract__(+  requires(memory_no_alias(state, sizeof(mld_shake256ctx)))+  assigns(memory_slice(state, sizeof(mld_shake256ctx)))+  ensures(state->pos == 0)+);++#define mld_shake256_absorb MLD_NAMESPACE(shake256_absorb)+/**+ * Absorb step of the SHAKE256 XOF. Absorbs arbitrarily many bytes. Can be+ * called multiple times to absorb multiple chunks of data.+ *+ * @param[in,out] state Pointer to (initialized) output state.+ * @param[in]     in    Pointer to input to be absorbed into s.+ * @param         inlen Length of input in bytes.+ */+MLD_INTERNAL_API+void mld_shake256_absorb(mld_shake256ctx *state, const uint8_t *in,+                         size_t inlen)+__contract__(+  requires(inlen <= MLD_MAX_BUFFER_SIZE)+  requires(memory_no_alias(state, sizeof(mld_shake256ctx)))+  requires(memory_no_alias(in, inlen))+  requires(state->pos <= SHAKE256_RATE)+  assigns(memory_slice(state, sizeof(mld_shake256ctx)))+  ensures(state->pos <= SHAKE256_RATE)+);++#define mld_shake256_finalize MLD_NAMESPACE(shake256_finalize)+/**+ * Concludes the absorb phase of the SHAKE256 XOF.+ *+ * @param[in,out] state Pointer to state.+ */+MLD_INTERNAL_API+void mld_shake256_finalize(mld_shake256ctx *state)+__contract__(+  requires(memory_no_alias(state, sizeof(mld_shake256ctx)))+  requires(state->pos <= SHAKE256_RATE)+  assigns(memory_slice(state, sizeof(mld_shake256ctx)))+  ensures(state->pos <= SHAKE256_RATE)+);++#define mld_shake256_squeeze MLD_NAMESPACE(shake256_squeeze)+/**+ * Squeeze step of SHAKE256 XOF. Squeezes arbitrarily many bytes. Can be+ * called multiple times to keep squeezing.+ *+ * @param[out]    out    Pointer to output blocks.+ * @param         outlen Number of bytes to be squeezed (written to output).+ * @param[in,out] state  Pointer to input/output state.+ */+MLD_INTERNAL_API+void mld_shake256_squeeze(uint8_t *out, size_t outlen, mld_shake256ctx *state)+__contract__(+  requires(outlen <= 8 * SHAKE256_RATE /* somewhat arbitrary bound */)+  requires(memory_no_alias(state, sizeof(mld_shake256ctx)))+  requires(memory_no_alias(out, outlen))+  requires(state->pos <= SHAKE256_RATE)+  assigns(memory_slice(state, sizeof(mld_shake256ctx)))+  assigns(memory_slice(out, outlen))+  ensures(state->pos <= SHAKE256_RATE)+);++#define mld_shake256_release MLD_NAMESPACE(shake256_release)+/**+ * Release and securely zero the SHAKE256 state.+ *+ * @param[in,out] state Pointer to state.+ */+MLD_INTERNAL_API+void mld_shake256_release(mld_shake256ctx *state)+__contract__(+  requires(memory_no_alias(state, sizeof(mld_shake256ctx)))+  assigns(memory_slice(state, sizeof(mld_shake256ctx)))+);++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_CORE_API_ONLY)+#define mld_shake256 MLD_NAMESPACE(shake256)+/**+ * SHAKE256 XOF with non-incremental API.+ *+ * @param[out] out    Pointer to output.+ * @param      outlen Requested output length in bytes.+ * @param[in]  in     Pointer to input.+ * @param      inlen  Length of input in bytes.+ */+MLD_INTERNAL_API+void mld_shake256(uint8_t *out, size_t outlen, const uint8_t *in, size_t inlen)+__contract__(+  requires(inlen <= MLD_MAX_BUFFER_SIZE)+  requires(outlen <= 8 * SHAKE256_RATE /* somewhat arbitrary bound */)+  requires(memory_no_alias(in, inlen))+  requires(memory_no_alias(out, outlen))+  assigns(memory_slice(out, outlen))+);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_CORE_API_ONLY */++#endif /* !MLD_FIPS202_FIPS202_H */
+ cbits/mldsa/src/fips202/fips202x4.c view
@@ -0,0 +1,187 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ */++#include "../common.h"+#if !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \+    !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY)++#include "../ct.h"+#include "fips202.h"+#include "fips202x4.h"+#include "keccakf1600.h"++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_REDUCE_RAM) || \+    defined(MLD_UNIT_TEST)+static void mld_keccak_absorb_once_x4(uint64_t *s, unsigned r,+                                      const uint8_t *in0, const uint8_t *in1,+                                      const uint8_t *in2, const uint8_t *in3,+                                      size_t inlen, uint8_t p)+__contract__(+  requires(inlen <= MLD_MAX_BUFFER_SIZE)+  requires(memory_no_alias(s, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY))+  requires(r > 0)+  requires(r <= sizeof(uint64_t) * MLD_KECCAK_LANES)+  requires(memory_no_alias(in0, inlen))+  requires(memory_no_alias(in1, inlen))+  requires(memory_no_alias(in2, inlen))+  requires(memory_no_alias(in3, inlen))+  assigns(memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY)))+{+  while (inlen >= r)+  __loop__(+    assigns(inlen, in0, in1, in2, in3, memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY))+    invariant(inlen <= loop_entry(inlen))+    invariant(in0 == loop_entry(in0) + (loop_entry(inlen) - inlen))+    invariant(in1 == loop_entry(in1) + (loop_entry(inlen) - inlen))+    invariant(in2 == loop_entry(in2) + (loop_entry(inlen) - inlen))+    invariant(in3 == loop_entry(in3) + (loop_entry(inlen) - inlen))+    decreases(inlen))+  {+    mld_keccakf1600x4_xor_bytes(s, in0, in1, in2, in3, 0, r);+    mld_keccakf1600x4_permute(s);++    in0 += r;+    in1 += r;+    in2 += r;+    in3 += r;+    inlen -= r;+  }++  /* Safety: At this point, inlen < r, so the truncations to unsigned are safe+   * below. */+  if (inlen > 0)+  {+    mld_keccakf1600x4_xor_bytes(s, in0, in1, in2, in3, 0, (unsigned)inlen);+  }++  if (inlen == r - 1)+  {+    p |= 128;+    mld_keccakf1600x4_xor_bytes(s, &p, &p, &p, &p, (unsigned)inlen, 1);+  }+  else+  {+    mld_keccakf1600x4_xor_bytes(s, &p, &p, &p, &p, (unsigned)inlen, 1);+    p = 128;+    mld_keccakf1600x4_xor_bytes(s, &p, &p, &p, &p, r - 1, 1);+  }+}++static void mld_keccak_squeezeblocks_x4(uint8_t *out0, uint8_t *out1,+                                        uint8_t *out2, uint8_t *out3,+                                        size_t nblocks, uint64_t *s, unsigned r)+__contract__(+    requires(r <= sizeof(uint64_t) * MLD_KECCAK_LANES)+    requires(nblocks <= 8 /* somewhat arbitrary bound */)+    requires(memory_no_alias(s, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY))+    requires(memory_no_alias(out0, nblocks * r))+    requires(memory_no_alias(out1, nblocks * r))+    requires(memory_no_alias(out2, nblocks * r))+    requires(memory_no_alias(out3, nblocks * r))+    assigns(memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY))+    assigns(memory_slice(out0, nblocks * r))+    assigns(memory_slice(out1, nblocks * r))+    assigns(memory_slice(out2, nblocks * r))+    assigns(memory_slice(out3, nblocks * r)))+{+  size_t current_offset = 0;+  while (nblocks > 0)+  __loop__(+    assigns(nblocks, current_offset,+            memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY),+            memory_slice(out0, nblocks * r),+            memory_slice(out1, nblocks * r),+            memory_slice(out2, nblocks * r),+            memory_slice(out3, nblocks * r))+    invariant(nblocks <= loop_entry(nblocks))+    invariant(current_offset == (loop_entry(nblocks) - nblocks) * r)+    decreases(nblocks))+  {+    mld_keccakf1600x4_permute(s);+    mld_keccakf1600x4_extract_bytes(+        s, &out0[current_offset], &out1[current_offset], &out2[current_offset],+        &out3[current_offset], 0, r);+    current_offset += r;+    nblocks--;+  }+}+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_REDUCE_RAM || \+          MLD_UNIT_TEST */++#if !defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+MLD_INTERNAL_API+void mld_shake128x4_absorb_once(mld_shake128x4ctx *state, const uint8_t *in0,+                                const uint8_t *in1, const uint8_t *in2,+                                const uint8_t *in3, size_t inlen)+{+  mld_memset(state, 0, sizeof(mld_shake128x4ctx));+  mld_keccak_absorb_once_x4(state->ctx, SHAKE128_RATE, in0, in1, in2, in3,+                            inlen, 0x1F);+}++MLD_INTERNAL_API+void mld_shake128x4_squeezeblocks(uint8_t *out0, uint8_t *out1, uint8_t *out2,+                                  uint8_t *out3, size_t nblocks,+                                  mld_shake128x4ctx *state)+{+  mld_keccak_squeezeblocks_x4(out0, out1, out2, out3, nblocks, state->ctx,+                              SHAKE128_RATE);+}++MLD_INTERNAL_API+void mld_shake128x4_init(mld_shake128x4ctx *state) { (void)state; }+MLD_INTERNAL_API+void mld_shake128x4_release(mld_shake128x4ctx *state)+{+  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(state, sizeof(mld_shake128x4ctx));+}+#endif /* !MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) ||                                  \+    (!defined(MLD_CONFIG_NO_SIGN_API) &&                                    \+     (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)))+MLD_INTERNAL_API+void mld_shake256x4_absorb_once(mld_shake256x4ctx *state, const uint8_t *in0,+                                const uint8_t *in1, const uint8_t *in2,+                                const uint8_t *in3, size_t inlen)+{+  mld_memset(state, 0, sizeof(mld_shake256x4ctx));+  mld_keccak_absorb_once_x4(state->ctx, SHAKE256_RATE, in0, in1, in2, in3,+                            inlen, 0x1F);+}++MLD_INTERNAL_API+void mld_shake256x4_squeezeblocks(uint8_t *out0, uint8_t *out1, uint8_t *out2,+                                  uint8_t *out3, size_t nblocks,+                                  mld_shake256x4ctx *state)+{+  mld_keccak_squeezeblocks_x4(out0, out1, out2, out3, nblocks, state->ctx,+                              SHAKE256_RATE);+}++MLD_INTERNAL_API+void mld_shake256x4_init(mld_shake256x4ctx *state) { (void)state; }+MLD_INTERNAL_API+void mld_shake256x4_release(mld_shake256x4ctx *state)+{+  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(state, sizeof(mld_shake256x4ctx));+}+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || (!MLD_CONFIG_NO_SIGN_API && \+          (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST)) */++#endif /* !MLD_CONFIG_MULTILEVEL_NO_SHARED && !MLD_CONFIG_SERIAL_FIPS202_ONLY \+        */
+ cbits/mldsa/src/fips202/fips202x4.h view
@@ -0,0 +1,125 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLD_FIPS202_FIPS202X4_H+#define MLD_FIPS202_FIPS202X4_H++#include "../common.h"++#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY)++#include <stddef.h>++#include "../cbmc.h"+#include "fips202.h"+#include "keccakf1600.h"++/** Context for the non-incremental 4-way SHAKE128 API. */+typedef struct+{+  uint64_t ctx[MLD_KECCAK_LANES *+               MLD_KECCAK_WAY]; /**< 4-way Keccak state, stored sequentially. */+} mld_shake128x4ctx;++/** Context for the 4-way batched SHAKE256 XOF. */+typedef struct+{+  uint64_t ctx[MLD_KECCAK_LANES *+               MLD_KECCAK_WAY]; /**< Interleaved 4-way Keccak state. */+} mld_shake256x4ctx;++#if !defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+#define mld_shake128x4_absorb_once MLD_NAMESPACE(shake128x4_absorb_once)+MLD_INTERNAL_API+void mld_shake128x4_absorb_once(mld_shake128x4ctx *state, const uint8_t *in0,+                                const uint8_t *in1, const uint8_t *in2,+                                const uint8_t *in3, size_t inlen)+__contract__(+  requires(inlen <= MLD_MAX_BUFFER_SIZE)+  requires(memory_no_alias(state, sizeof(mld_shake128x4ctx)))+  requires(memory_no_alias(in0, inlen))+  requires(memory_no_alias(in1, inlen))+  requires(memory_no_alias(in2, inlen))+  requires(memory_no_alias(in3, inlen))+  assigns(memory_slice(state, sizeof(mld_shake128x4ctx)))+);++#define mld_shake128x4_squeezeblocks MLD_NAMESPACE(shake128x4_squeezeblocks)+MLD_INTERNAL_API+void mld_shake128x4_squeezeblocks(uint8_t *out0, uint8_t *out1, uint8_t *out2,+                                  uint8_t *out3, size_t nblocks,+                                  mld_shake128x4ctx *state)+__contract__(+  requires(nblocks <= 8 /* somewhat arbitrary bound */)+  requires(memory_no_alias(state, sizeof(mld_shake128x4ctx)))+  requires(memory_no_alias(out0, nblocks * SHAKE128_RATE))+  requires(memory_no_alias(out1, nblocks * SHAKE128_RATE))+  requires(memory_no_alias(out2, nblocks * SHAKE128_RATE))+  requires(memory_no_alias(out3, nblocks * SHAKE128_RATE))+  assigns(memory_slice(out0, nblocks * SHAKE128_RATE),+    memory_slice(out1, nblocks * SHAKE128_RATE),+    memory_slice(out2, nblocks * SHAKE128_RATE),+    memory_slice(out3, nblocks * SHAKE128_RATE),+    memory_slice(state, sizeof(mld_shake128x4ctx)))+);++#define mld_shake128x4_init MLD_NAMESPACE(shake128x4_init)+MLD_INTERNAL_API+void mld_shake128x4_init(mld_shake128x4ctx *state);++#define mld_shake128x4_release MLD_NAMESPACE(shake128x4_release)+MLD_INTERNAL_API+void mld_shake128x4_release(mld_shake128x4ctx *state);+#endif /* !MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || \+    (!defined(MLD_CONFIG_NO_SIGN_API) &&   \+     (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)))+#define mld_shake256x4_absorb_once MLD_NAMESPACE(shake256x4_absorb_once)+MLD_INTERNAL_API+void mld_shake256x4_absorb_once(mld_shake256x4ctx *state, const uint8_t *in0,+                                const uint8_t *in1, const uint8_t *in2,+                                const uint8_t *in3, size_t inlen)+__contract__(+  requires(inlen <= MLD_MAX_BUFFER_SIZE)+  requires(memory_no_alias(state, sizeof(mld_shake256x4ctx)))+  requires(memory_no_alias(in0, inlen))+  requires(memory_no_alias(in1, inlen))+  requires(memory_no_alias(in2, inlen))+  requires(memory_no_alias(in3, inlen))+  assigns(memory_slice(state, sizeof(mld_shake256x4ctx)))+);++#define mld_shake256x4_squeezeblocks MLD_NAMESPACE(shake256x4_squeezeblocks)+MLD_INTERNAL_API+void mld_shake256x4_squeezeblocks(uint8_t *out0, uint8_t *out1, uint8_t *out2,+                                  uint8_t *out3, size_t nblocks,+                                  mld_shake256x4ctx *state)+__contract__(+  requires(nblocks <= 8 /* somewhat arbitrary bound */)+  requires(memory_no_alias(state, sizeof(mld_shake256x4ctx)))+  requires(memory_no_alias(out0, nblocks * SHAKE256_RATE))+  requires(memory_no_alias(out1, nblocks * SHAKE256_RATE))+  requires(memory_no_alias(out2, nblocks * SHAKE256_RATE))+  requires(memory_no_alias(out3, nblocks * SHAKE256_RATE))+  assigns(memory_slice(out0, nblocks * SHAKE256_RATE),+    memory_slice(out1, nblocks * SHAKE256_RATE),+    memory_slice(out2, nblocks * SHAKE256_RATE),+    memory_slice(out3, nblocks * SHAKE256_RATE),+    memory_slice(state, sizeof(mld_shake256x4ctx)))+);++#define mld_shake256x4_init MLD_NAMESPACE(shake256x4_init)+MLD_INTERNAL_API+void mld_shake256x4_init(mld_shake256x4ctx *state);++#define mld_shake256x4_release MLD_NAMESPACE(shake256x4_release)+MLD_INTERNAL_API+void mld_shake256x4_release(mld_shake256x4ctx *state);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || (!MLD_CONFIG_NO_SIGN_API && \+          (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST)) */++#endif /* !MLD_CONFIG_SERIAL_FIPS202_ONLY */+#endif /* !MLD_FIPS202_FIPS202X4_H */
+ cbits/mldsa/src/fips202/keccakf1600.c view
@@ -0,0 +1,510 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [mupq]+ *   Common files for pqm4, pqm3, pqriscv+ *   Kannwischer, Petri, Rijneveld, Schwabe, Stoffelen+ *   https://github.com/mupq/mupq+ *+ * - [supercop]+ *   SUPERCOP benchmarking framework+ *   Daniel J. Bernstein+ *   http://bench.cr.yp.to/supercop.html+ *+ * - [tweetfips]+ *   'tweetfips202' FIPS202 implementation+ *   Van Assche, Bernstein, Schwabe+ *   https://keccak.team/2015/tweetfips202.html+ */++/* Based on the CC0 implementation from @[mupq] and the public domain+ * implementation @[supercop, crypto_hash/keccakc512/simple/]+ * by Ronny Van Keer, and the public domain @[tweetfips] implementation. */++#include "keccakf1600.h"+#if !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++#define MLD_KECCAK_NROUNDS 24+#define MLD_KECCAK_ROL(a, offset) (((a) << (offset)) ^ ((a) >> (64 - (offset))))++MLD_INTERNAL_API+void mld_keccakf1600_extract_bytes(uint64_t *state, unsigned char *data,+                                   unsigned offset, unsigned length)+{+  unsigned i;+#if defined(MLD_SYS_LITTLE_ENDIAN)+  uint8_t *state_ptr = (uint8_t *)state + offset;+  for (i = 0; i < length; i++)+  __loop__(invariant(i <= length)+           decreases(length - i))+  {+    data[i] = state_ptr[i];+  }+#else  /* MLD_SYS_LITTLE_ENDIAN */+  /* Portable version */+  for (i = 0; i < length; i++)+  __loop__(invariant(i <= length)+           decreases(length - i))+  {+    data[i] = (state[(offset + i) >> 3] >> (8 * ((offset + i) & 0x07))) & 0xFF;+  }+#endif /* !MLD_SYS_LITTLE_ENDIAN */+}++MLD_INTERNAL_API+void mld_keccakf1600_xor_bytes(uint64_t *state, const unsigned char *data,+                               unsigned offset, unsigned length)+{+  unsigned i;+#if defined(MLD_SYS_LITTLE_ENDIAN)+  uint8_t *state_ptr = (uint8_t *)state + offset;+  for (i = 0; i < length; i++)+  __loop__(invariant(i <= length)+           decreases(length - i))+  {+    state_ptr[i] ^= data[i];+  }+#else  /* MLD_SYS_LITTLE_ENDIAN */+  /* Portable version */+  for (i = 0; i < length; i++)+  __loop__(invariant(i <= length)+           decreases(length - i))+  {+    state[(offset + i) >> 3] ^= (uint64_t)data[i]+                                << (8 * ((offset + i) & 0x07));+  }+#endif /* !MLD_SYS_LITTLE_ENDIAN */+}++#if (!defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_REDUCE_RAM) || \+     defined(MLD_UNIT_TEST)) &&                                                \+    !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY)+static void mld_keccakf1600x4_extract_bytes_c(uint64_t *state,+                                              unsigned char *data0,+                                              unsigned char *data1,+                                              unsigned char *data2,+                                              unsigned char *data3,+                                              unsigned offset, unsigned length)+__contract__(+    requires(0 <= offset && offset <= MLD_KECCAK_LANES * sizeof(uint64_t) &&+         0 <= length && length <= MLD_KECCAK_LANES * sizeof(uint64_t) - offset)+    requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY))+    requires(memory_no_alias(data0, length))+    requires(memory_no_alias(data1, length))+    requires(memory_no_alias(data2, length))+    requires(memory_no_alias(data3, length))+    assigns(memory_slice(data0, length))+    assigns(memory_slice(data1, length))+    assigns(memory_slice(data2, length))+    assigns(memory_slice(data3, length))+)+{+  mld_keccakf1600_extract_bytes(state + MLD_KECCAK_LANES * 0, data0, offset,+                                length);+  mld_keccakf1600_extract_bytes(state + MLD_KECCAK_LANES * 1, data1, offset,+                                length);+  mld_keccakf1600_extract_bytes(state + MLD_KECCAK_LANES * 2, data2, offset,+                                length);+  mld_keccakf1600_extract_bytes(state + MLD_KECCAK_LANES * 3, data3, offset,+                                length);+}++MLD_INTERNAL_API+void mld_keccakf1600x4_extract_bytes(uint64_t *state, unsigned char *data0,+                                     unsigned char *data1, unsigned char *data2,+                                     unsigned char *data3, unsigned offset,+                                     unsigned length)+{+#if defined(MLD_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES)+  if (mld_keccakf1600_extract_bytes_x4_native(state, data0, data1, data2, data3,+                                              offset, length) ==+      MLD_NATIVE_FUNC_SUCCESS)+  {+    return;+  }+#endif /* MLD_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES */+  mld_keccakf1600x4_extract_bytes_c(state, data0, data1, data2, data3, offset,+                                    length);+}++static void mld_keccakf1600x4_xor_bytes_c(uint64_t *state,+                                          const unsigned char *data0,+                                          const unsigned char *data1,+                                          const unsigned char *data2,+                                          const unsigned char *data3,+                                          unsigned offset, unsigned length)+__contract__(+    requires(0 <= offset && offset <= MLD_KECCAK_LANES * sizeof(uint64_t) &&+         0 <= length && length <= MLD_KECCAK_LANES * sizeof(uint64_t) - offset)+    requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY))+    requires(memory_no_alias(data0, length))+    /* Case 1: all input buffers are distinct; Case 2: All input buffers are the same */+    requires((data0 == data1 &&+              data0 == data2 &&+              data0 == data3) ||+         (memory_no_alias(data1, length) &&+              memory_no_alias(data2, length) &&+              memory_no_alias(data3, length)))+    assigns(memory_slice(state, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY))+)+{+  mld_keccakf1600_xor_bytes(state + MLD_KECCAK_LANES * 0, data0, offset,+                            length);+  mld_keccakf1600_xor_bytes(state + MLD_KECCAK_LANES * 1, data1, offset,+                            length);+  mld_keccakf1600_xor_bytes(state + MLD_KECCAK_LANES * 2, data2, offset,+                            length);+  mld_keccakf1600_xor_bytes(state + MLD_KECCAK_LANES * 3, data3, offset,+                            length);+}++MLD_INTERNAL_API+void mld_keccakf1600x4_xor_bytes(uint64_t *state, const unsigned char *data0,+                                 const unsigned char *data1,+                                 const unsigned char *data2,+                                 const unsigned char *data3, unsigned offset,+                                 unsigned length)+{+#if defined(MLD_USE_NATIVE_FIPS202_X4_XOR_BYTES)+  if (mld_keccakf1600_xor_bytes_x4_native(state, data0, data1, data2, data3,+                                          offset,+                                          length) == MLD_NATIVE_FUNC_SUCCESS)+  {+    return;+  }+#endif /* MLD_USE_NATIVE_FIPS202_X4_XOR_BYTES */+  mld_keccakf1600x4_xor_bytes_c(state, data0, data1, data2, data3, offset,+                                length);+}++MLD_INTERNAL_API+void mld_keccakf1600x4_permute(uint64_t *state)+{+#if defined(MLD_USE_NATIVE_FIPS202_X4)+  if (mld_keccak_f1600_x4_native(state) == MLD_NATIVE_FUNC_SUCCESS)+  {+    return;+  }+#endif /* MLD_USE_NATIVE_FIPS202_X4 */+  mld_keccakf1600_permute(state + MLD_KECCAK_LANES * 0);+  mld_keccakf1600_permute(state + MLD_KECCAK_LANES * 1);+  mld_keccakf1600_permute(state + MLD_KECCAK_LANES * 2);+  mld_keccakf1600_permute(state + MLD_KECCAK_LANES * 3);+}+#endif /* (!MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_REDUCE_RAM || \+          MLD_UNIT_TEST) && !MLD_CONFIG_SERIAL_FIPS202_ONLY */++static const uint64_t mld_KeccakF_RoundConstants[MLD_KECCAK_NROUNDS] = {+    (uint64_t)0x0000000000000001ULL, (uint64_t)0x0000000000008082ULL,+    (uint64_t)0x800000000000808aULL, (uint64_t)0x8000000080008000ULL,+    (uint64_t)0x000000000000808bULL, (uint64_t)0x0000000080000001ULL,+    (uint64_t)0x8000000080008081ULL, (uint64_t)0x8000000000008009ULL,+    (uint64_t)0x000000000000008aULL, (uint64_t)0x0000000000000088ULL,+    (uint64_t)0x0000000080008009ULL, (uint64_t)0x000000008000000aULL,+    (uint64_t)0x000000008000808bULL, (uint64_t)0x800000000000008bULL,+    (uint64_t)0x8000000000008089ULL, (uint64_t)0x8000000000008003ULL,+    (uint64_t)0x8000000000008002ULL, (uint64_t)0x8000000000000080ULL,+    (uint64_t)0x000000000000800aULL, (uint64_t)0x800000008000000aULL,+    (uint64_t)0x8000000080008081ULL, (uint64_t)0x8000000000008080ULL,+    (uint64_t)0x0000000080000001ULL, (uint64_t)0x8000000080008008ULL};++MLD_STATIC_TESTABLE+void mld_keccakf1600_permute_c(uint64_t *state)+__contract__(+    requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES))+    assigns(memory_slice(state, sizeof(uint64_t) * MLD_KECCAK_LANES))+)+{+  unsigned round;++  uint64_t Aba, Abe, Abi, Abo, Abu;+  uint64_t Aga, Age, Agi, Ago, Agu;+  uint64_t Aka, Ake, Aki, Ako, Aku;+  uint64_t Ama, Ame, Ami, Amo, Amu;+  uint64_t Asa, Ase, Asi, Aso, Asu;+  uint64_t BCa, BCe, BCi, BCo, BCu;+  uint64_t Da, De, Di, Do, Du;+  uint64_t Eba, Ebe, Ebi, Ebo, Ebu;+  uint64_t Ega, Ege, Egi, Ego, Egu;+  uint64_t Eka, Eke, Eki, Eko, Eku;+  uint64_t Ema, Eme, Emi, Emo, Emu;+  uint64_t Esa, Ese, Esi, Eso, Esu;++  /* copyFromState(A, state) */+  Aba = state[0];+  Abe = state[1];+  Abi = state[2];+  Abo = state[3];+  Abu = state[4];+  Aga = state[5];+  Age = state[6];+  Agi = state[7];+  Ago = state[8];+  Agu = state[9];+  Aka = state[10];+  Ake = state[11];+  Aki = state[12];+  Ako = state[13];+  Aku = state[14];+  Ama = state[15];+  Ame = state[16];+  Ami = state[17];+  Amo = state[18];+  Amu = state[19];+  Asa = state[20];+  Ase = state[21];+  Asi = state[22];+  Aso = state[23];+  Asu = state[24];++  for (round = 0; round < MLD_KECCAK_NROUNDS; round += 2)+  __loop__(invariant(round <= MLD_KECCAK_NROUNDS && round % 2 == 0)+           decreases(MLD_KECCAK_NROUNDS - round))+  {+    /* prepareTheta */+    BCa = Aba ^ Aga ^ Aka ^ Ama ^ Asa;+    BCe = Abe ^ Age ^ Ake ^ Ame ^ Ase;+    BCi = Abi ^ Agi ^ Aki ^ Ami ^ Asi;+    BCo = Abo ^ Ago ^ Ako ^ Amo ^ Aso;+    BCu = Abu ^ Agu ^ Aku ^ Amu ^ Asu;++    /* thetaRhoPiChiIotaPrepareTheta(round, A, E) */+    Da = BCu ^ MLD_KECCAK_ROL(BCe, 1);+    De = BCa ^ MLD_KECCAK_ROL(BCi, 1);+    Di = BCe ^ MLD_KECCAK_ROL(BCo, 1);+    Do = BCi ^ MLD_KECCAK_ROL(BCu, 1);+    Du = BCo ^ MLD_KECCAK_ROL(BCa, 1);++    Aba ^= Da;+    BCa = Aba;+    Age ^= De;+    BCe = MLD_KECCAK_ROL(Age, 44);+    Aki ^= Di;+    BCi = MLD_KECCAK_ROL(Aki, 43);+    Amo ^= Do;+    BCo = MLD_KECCAK_ROL(Amo, 21);+    Asu ^= Du;+    BCu = MLD_KECCAK_ROL(Asu, 14);+    Eba = BCa ^ ((~BCe) & BCi);+    Eba ^= (uint64_t)mld_KeccakF_RoundConstants[round];+    Ebe = BCe ^ ((~BCi) & BCo);+    Ebi = BCi ^ ((~BCo) & BCu);+    Ebo = BCo ^ ((~BCu) & BCa);+    Ebu = BCu ^ ((~BCa) & BCe);++    Abo ^= Do;+    BCa = MLD_KECCAK_ROL(Abo, 28);+    Agu ^= Du;+    BCe = MLD_KECCAK_ROL(Agu, 20);+    Aka ^= Da;+    BCi = MLD_KECCAK_ROL(Aka, 3);+    Ame ^= De;+    BCo = MLD_KECCAK_ROL(Ame, 45);+    Asi ^= Di;+    BCu = MLD_KECCAK_ROL(Asi, 61);+    Ega = BCa ^ ((~BCe) & BCi);+    Ege = BCe ^ ((~BCi) & BCo);+    Egi = BCi ^ ((~BCo) & BCu);+    Ego = BCo ^ ((~BCu) & BCa);+    Egu = BCu ^ ((~BCa) & BCe);++    Abe ^= De;+    BCa = MLD_KECCAK_ROL(Abe, 1);+    Agi ^= Di;+    BCe = MLD_KECCAK_ROL(Agi, 6);+    Ako ^= Do;+    BCi = MLD_KECCAK_ROL(Ako, 25);+    Amu ^= Du;+    BCo = MLD_KECCAK_ROL(Amu, 8);+    Asa ^= Da;+    BCu = MLD_KECCAK_ROL(Asa, 18);+    Eka = BCa ^ ((~BCe) & BCi);+    Eke = BCe ^ ((~BCi) & BCo);+    Eki = BCi ^ ((~BCo) & BCu);+    Eko = BCo ^ ((~BCu) & BCa);+    Eku = BCu ^ ((~BCa) & BCe);++    Abu ^= Du;+    BCa = MLD_KECCAK_ROL(Abu, 27);+    Aga ^= Da;+    BCe = MLD_KECCAK_ROL(Aga, 36);+    Ake ^= De;+    BCi = MLD_KECCAK_ROL(Ake, 10);+    Ami ^= Di;+    BCo = MLD_KECCAK_ROL(Ami, 15);+    Aso ^= Do;+    BCu = MLD_KECCAK_ROL(Aso, 56);+    Ema = BCa ^ ((~BCe) & BCi);+    Eme = BCe ^ ((~BCi) & BCo);+    Emi = BCi ^ ((~BCo) & BCu);+    Emo = BCo ^ ((~BCu) & BCa);+    Emu = BCu ^ ((~BCa) & BCe);++    Abi ^= Di;+    BCa = MLD_KECCAK_ROL(Abi, 62);+    Ago ^= Do;+    BCe = MLD_KECCAK_ROL(Ago, 55);+    Aku ^= Du;+    BCi = MLD_KECCAK_ROL(Aku, 39);+    Ama ^= Da;+    BCo = MLD_KECCAK_ROL(Ama, 41);+    Ase ^= De;+    BCu = MLD_KECCAK_ROL(Ase, 2);+    Esa = BCa ^ ((~BCe) & BCi);+    Ese = BCe ^ ((~BCi) & BCo);+    Esi = BCi ^ ((~BCo) & BCu);+    Eso = BCo ^ ((~BCu) & BCa);+    Esu = BCu ^ ((~BCa) & BCe);++    /* prepareTheta */+    BCa = Eba ^ Ega ^ Eka ^ Ema ^ Esa;+    BCe = Ebe ^ Ege ^ Eke ^ Eme ^ Ese;+    BCi = Ebi ^ Egi ^ Eki ^ Emi ^ Esi;+    BCo = Ebo ^ Ego ^ Eko ^ Emo ^ Eso;+    BCu = Ebu ^ Egu ^ Eku ^ Emu ^ Esu;++    /* thetaRhoPiChiIotaPrepareTheta(round+1, E, A) */+    Da = BCu ^ MLD_KECCAK_ROL(BCe, 1);+    De = BCa ^ MLD_KECCAK_ROL(BCi, 1);+    Di = BCe ^ MLD_KECCAK_ROL(BCo, 1);+    Do = BCi ^ MLD_KECCAK_ROL(BCu, 1);+    Du = BCo ^ MLD_KECCAK_ROL(BCa, 1);++    Eba ^= Da;+    BCa = Eba;+    Ege ^= De;+    BCe = MLD_KECCAK_ROL(Ege, 44);+    Eki ^= Di;+    BCi = MLD_KECCAK_ROL(Eki, 43);+    Emo ^= Do;+    BCo = MLD_KECCAK_ROL(Emo, 21);+    Esu ^= Du;+    BCu = MLD_KECCAK_ROL(Esu, 14);+    Aba = BCa ^ ((~BCe) & BCi);+    Aba ^= (uint64_t)mld_KeccakF_RoundConstants[round + 1];+    Abe = BCe ^ ((~BCi) & BCo);+    Abi = BCi ^ ((~BCo) & BCu);+    Abo = BCo ^ ((~BCu) & BCa);+    Abu = BCu ^ ((~BCa) & BCe);++    Ebo ^= Do;+    BCa = MLD_KECCAK_ROL(Ebo, 28);+    Egu ^= Du;+    BCe = MLD_KECCAK_ROL(Egu, 20);+    Eka ^= Da;+    BCi = MLD_KECCAK_ROL(Eka, 3);+    Eme ^= De;+    BCo = MLD_KECCAK_ROL(Eme, 45);+    Esi ^= Di;+    BCu = MLD_KECCAK_ROL(Esi, 61);+    Aga = BCa ^ ((~BCe) & BCi);+    Age = BCe ^ ((~BCi) & BCo);+    Agi = BCi ^ ((~BCo) & BCu);+    Ago = BCo ^ ((~BCu) & BCa);+    Agu = BCu ^ ((~BCa) & BCe);++    Ebe ^= De;+    BCa = MLD_KECCAK_ROL(Ebe, 1);+    Egi ^= Di;+    BCe = MLD_KECCAK_ROL(Egi, 6);+    Eko ^= Do;+    BCi = MLD_KECCAK_ROL(Eko, 25);+    Emu ^= Du;+    BCo = MLD_KECCAK_ROL(Emu, 8);+    Esa ^= Da;+    BCu = MLD_KECCAK_ROL(Esa, 18);+    Aka = BCa ^ ((~BCe) & BCi);+    Ake = BCe ^ ((~BCi) & BCo);+    Aki = BCi ^ ((~BCo) & BCu);+    Ako = BCo ^ ((~BCu) & BCa);+    Aku = BCu ^ ((~BCa) & BCe);++    Ebu ^= Du;+    BCa = MLD_KECCAK_ROL(Ebu, 27);+    Ega ^= Da;+    BCe = MLD_KECCAK_ROL(Ega, 36);+    Eke ^= De;+    BCi = MLD_KECCAK_ROL(Eke, 10);+    Emi ^= Di;+    BCo = MLD_KECCAK_ROL(Emi, 15);+    Eso ^= Do;+    BCu = MLD_KECCAK_ROL(Eso, 56);+    Ama = BCa ^ ((~BCe) & BCi);+    Ame = BCe ^ ((~BCi) & BCo);+    Ami = BCi ^ ((~BCo) & BCu);+    Amo = BCo ^ ((~BCu) & BCa);+    Amu = BCu ^ ((~BCa) & BCe);++    Ebi ^= Di;+    BCa = MLD_KECCAK_ROL(Ebi, 62);+    Ego ^= Do;+    BCe = MLD_KECCAK_ROL(Ego, 55);+    Eku ^= Du;+    BCi = MLD_KECCAK_ROL(Eku, 39);+    Ema ^= Da;+    BCo = MLD_KECCAK_ROL(Ema, 41);+    Ese ^= De;+    BCu = MLD_KECCAK_ROL(Ese, 2);+    Asa = BCa ^ ((~BCe) & BCi);+    Ase = BCe ^ ((~BCi) & BCo);+    Asi = BCi ^ ((~BCo) & BCu);+    Aso = BCo ^ ((~BCu) & BCa);+    Asu = BCu ^ ((~BCa) & BCe);+  }++  /* copyToState(state, A) */+  state[0] = Aba;+  state[1] = Abe;+  state[2] = Abi;+  state[3] = Abo;+  state[4] = Abu;+  state[5] = Aga;+  state[6] = Age;+  state[7] = Agi;+  state[8] = Ago;+  state[9] = Agu;+  state[10] = Aka;+  state[11] = Ake;+  state[12] = Aki;+  state[13] = Ako;+  state[14] = Aku;+  state[15] = Ama;+  state[16] = Ame;+  state[17] = Ami;+  state[18] = Amo;+  state[19] = Amu;+  state[20] = Asa;+  state[21] = Ase;+  state[22] = Asi;+  state[23] = Aso;+  state[24] = Asu;+}++MLD_INTERNAL_API+void mld_keccakf1600_permute(uint64_t *state)+{+#if defined(MLD_USE_NATIVE_FIPS202_X1)+  if (mld_keccak_f1600_x1_native(state) == MLD_NATIVE_FUNC_SUCCESS)+  {+    return;+  }+#endif /* MLD_USE_NATIVE_FIPS202_X1 */+  mld_keccakf1600_permute_c(state);+}++#else /* !MLD_CONFIG_MULTILEVEL_NO_SHARED */++MLD_EMPTY_CU(keccakf1600)++#endif /* MLD_CONFIG_MULTILEVEL_NO_SHARED */++/* To facilitate single-compilation-unit (SCU) builds, undefine all macros.+ * Don't modify by hand -- this is auto-generated by scripts/autogen. */+#undef MLD_KECCAK_NROUNDS+#undef MLD_KECCAK_ROL
+ cbits/mldsa/src/fips202/keccakf1600.h view
@@ -0,0 +1,110 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLD_FIPS202_KECCAKF1600_H+#define MLD_FIPS202_KECCAKF1600_H+#include "../cbmc.h"+#include "../common.h"+#include "fips202.h"++#define MLD_KECCAK_LANES 25+#define MLD_KECCAK_WAY 4++/*+ * WARNING:+ * The contents of this structure, including the placement+ * and interleaving of Keccak lanes, are IMPLEMENTATION-DEFINED.+ * The struct is only exposed here to allow its construction on the stack.+ */++#define mld_keccakf1600_extract_bytes MLD_NAMESPACE(keccakf1600_extract_bytes)+MLD_INTERNAL_API+void mld_keccakf1600_extract_bytes(uint64_t *state, unsigned char *data,+                                   unsigned offset, unsigned length)+__contract__(+    requires(0 <= offset && offset <= MLD_KECCAK_LANES * sizeof(uint64_t) &&+         0 <= length && length <= MLD_KECCAK_LANES * sizeof(uint64_t) - offset)+    requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES))+    requires(memory_no_alias(data, length))+    assigns(memory_slice(data, length))+);++#define mld_keccakf1600_xor_bytes MLD_NAMESPACE(keccakf1600_xor_bytes)+MLD_INTERNAL_API+void mld_keccakf1600_xor_bytes(uint64_t *state, const unsigned char *data,+                               unsigned offset, unsigned length)+__contract__(+    requires(0 <= offset && offset <= MLD_KECCAK_LANES * sizeof(uint64_t) &&+         0 <= length && length <= MLD_KECCAK_LANES * sizeof(uint64_t) - offset)+    requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES))+    requires(memory_no_alias(data, length))+    assigns(memory_slice(state, sizeof(uint64_t) * MLD_KECCAK_LANES))+);++#if (!defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_REDUCE_RAM) || \+     defined(MLD_UNIT_TEST)) &&                                                \+    !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY)+#define mld_keccakf1600x4_extract_bytes \+  MLD_NAMESPACE(keccakf1600x4_extract_bytes)+MLD_INTERNAL_API+void mld_keccakf1600x4_extract_bytes(uint64_t *state, unsigned char *data0,+                                     unsigned char *data1, unsigned char *data2,+                                     unsigned char *data3, unsigned offset,+                                     unsigned length)+__contract__(+    requires(0 <= offset && offset <= MLD_KECCAK_LANES * sizeof(uint64_t) &&+         0 <= length && length <= MLD_KECCAK_LANES * sizeof(uint64_t) - offset)+    requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY))+    requires(memory_no_alias(data0, length))+    requires(memory_no_alias(data1, length))+    requires(memory_no_alias(data2, length))+    requires(memory_no_alias(data3, length))+    assigns(memory_slice(data0, length))+    assigns(memory_slice(data1, length))+    assigns(memory_slice(data2, length))+    assigns(memory_slice(data3, length))+);++#define mld_keccakf1600x4_xor_bytes MLD_NAMESPACE(keccakf1600x4_xor_bytes)+MLD_INTERNAL_API+void mld_keccakf1600x4_xor_bytes(uint64_t *state, const unsigned char *data0,+                                 const unsigned char *data1,+                                 const unsigned char *data2,+                                 const unsigned char *data3, unsigned offset,+                                 unsigned length)+__contract__(+    requires(0 <= offset && offset <= MLD_KECCAK_LANES * sizeof(uint64_t) &&+         0 <= length && length <= MLD_KECCAK_LANES * sizeof(uint64_t) - offset)+    requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY))+    requires(memory_no_alias(data0, length))+    /* Case 1: all input buffers are distinct; Case 2: All input buffers are the same */+    requires((data0 == data1 &&+              data0 == data2 &&+              data0 == data3) ||+         (memory_no_alias(data1, length) &&+              memory_no_alias(data2, length) &&+              memory_no_alias(data3, length)))+    assigns(memory_slice(state, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY))+);++#define mld_keccakf1600x4_permute MLD_NAMESPACE(keccakf1600x4_permute)+MLD_INTERNAL_API+void mld_keccakf1600x4_permute(uint64_t *state)+__contract__(+    requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY))+    assigns(memory_slice(state, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY))+);+#endif /* (!MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_REDUCE_RAM || \+          MLD_UNIT_TEST) && !MLD_CONFIG_SERIAL_FIPS202_ONLY */++#define mld_keccakf1600_permute MLD_NAMESPACE(keccakf1600_permute)+MLD_INTERNAL_API+void mld_keccakf1600_permute(uint64_t *state)+__contract__(+    requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES))+    assigns(memory_slice(state, sizeof(uint64_t) * MLD_KECCAK_LANES))+);++#endif /* !MLD_FIPS202_KECCAKF1600_H */
+ cbits/mldsa/src/fips202/native/aarch64/auto.h view
@@ -0,0 +1,85 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [HYBRID]+ *   Hybrid scalar/vector implementations of Keccak and SPHINCS+ on AArch64+ *   Becker, Kannwischer+ *   https://eprint.iacr.org/2022/1243+ */++#ifndef MLD_FIPS202_NATIVE_AARCH64_AUTO_H+#define MLD_FIPS202_NATIVE_AARCH64_AUTO_H+/* Default FIPS202 assembly profile for AArch64 systems */++/*+ * Default logic to decide which implementation to use.+ *+ */++/*+ * Keccak-f1600+ *+ * - On Arm-based Apple CPUs, or CPUs with MLD_SYS_AARCH64_FAST_SHA3 set,+ *   we pick a pure Neon implementation.+ * - Otherwise, unless MLD_SYS_AARCH64_SLOW_BARREL_SHIFTER is set,+ *   we use lazy-rotation scalar assembly from @[HYBRID].+ * - Otherwise, if MLD_SYS_AARCH64_SLOW_BARREL_SHIFTER is set, we+ *   fall back to the standard C implementation.+ */+#if defined(__ARM_FEATURE_SHA3) && \+    (defined(__APPLE__) || defined(MLD_SYS_AARCH64_FAST_SHA3))+#include "x1_v84a.h"+#elif !defined(MLD_SYS_AARCH64_SLOW_BARREL_SHIFTER)+#include "x1_scalar.h"+#endif++#if (!defined(MLD_CONFIG_NO_KEYPAIR_API) ||                                  \+     !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_REDUCE_RAM)) && \+    !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY)+/* Batched, SIMD-based Keccak-f1600 implementations. */+#if defined(MLD_SYS_AARCH64_NEON)++/*+ * Keccak-f1600x2/x4+ *+ * The optimal implementation is highly CPU-specific; see @[HYBRID].+ *+ * For now, if v8.4-A is not implemented, we fall back to Keccak-f1600.+ * If v8.4-A is implemented and we are on an Apple CPU (or a CPU with+ * MLD_SYS_AARCH64_FAST_SHA3 set), we use a plain Neon-based implementation.+ * If v8.4-A is implemented and we are on neither, we use a+ * scalar/Neon/Neon hybrid.+ * The reason for this distinction is that Apple CPUs (and other CPUs flagged+ * via MLD_SYS_AARCH64_FAST_SHA3) appear to implement the SHA3 instructions on+ * all SIMD units, while Arm CPUs prior to Cortex-X4 don't, and ordinary Neon+ * instructions are still needed.+ */+#if defined(__ARM_FEATURE_SHA3)+/*+ * For Apple-M cores (and other cores flagged via MLD_SYS_AARCH64_FAST_SHA3),+ * we use a plain implementation leveraging SHA3 instructions only.+ */+#if defined(__APPLE__) || defined(MLD_SYS_AARCH64_FAST_SHA3)+#include "x2_v84a.h"+#else+#include "x4_v8a_v84a_scalar.h"+#endif++#else /* __ARM_FEATURE_SHA3 */++#include "x4_v8a_scalar.h"++#endif /* !__ARM_FEATURE_SHA3 */++#endif /* MLD_SYS_AARCH64_NEON */++#endif /* (!MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_SIGN_API || \+          !MLD_CONFIG_REDUCE_RAM) && !MLD_CONFIG_SERIAL_FIPS202_ONLY */++#endif /* !MLD_FIPS202_NATIVE_AARCH64_AUTO_H */
+ cbits/mldsa/src/fips202/native/aarch64/src/fips202_native_aarch64.h view
@@ -0,0 +1,69 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLD_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H+#define MLD_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H+++#include "../../../../cbmc.h"+#include "../../../../common.h"+++#define mld_keccakf1600_round_constants \+  MLD_NAMESPACE(keccakf1600_round_constants)+MLD_INTERNAL_DATA_DECLARATION const uint64_t+    mld_keccakf1600_round_constants[24];++#define mld_keccak_f1600_x1_scalar_aarch64_asm \+  MLD_NAMESPACE(keccak_f1600_x1_scalar_aarch64_asm)+void mld_keccak_f1600_x1_scalar_aarch64_asm(uint64_t state[25],+                                            const uint64_t rc[24])+__contract__(+  requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 1))+  requires(rc == mld_keccakf1600_round_constants)+  assigns(memory_slice(state, sizeof(uint64_t) * 25 * 1))+);++#define mld_keccak_f1600_x1_v84a_aarch64_asm \+  MLD_NAMESPACE(keccak_f1600_x1_v84a_aarch64_asm)+void mld_keccak_f1600_x1_v84a_aarch64_asm(uint64_t state[25],+                                          const uint64_t rc[24])+__contract__(+  requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 1))+  requires(rc == mld_keccakf1600_round_constants)+  assigns(memory_slice(state, sizeof(uint64_t) * 25 * 1))+);++#define mld_keccak_f1600_x2_v84a_aarch64_asm \+  MLD_NAMESPACE(keccak_f1600_x2_v84a_aarch64_asm)+void mld_keccak_f1600_x2_v84a_aarch64_asm(uint64_t state[50],+                                          const uint64_t rc[24])+__contract__(+  requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 2))+  requires(rc == mld_keccakf1600_round_constants)+  assigns(memory_slice(state, sizeof(uint64_t) * 25 * 2))+);++#define mld_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm \+  MLD_NAMESPACE(keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm)+void mld_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm(uint64_t state[100],+                                                       const uint64_t rc[24])+__contract__(+  requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4))+  requires(rc == mld_keccakf1600_round_constants)+  assigns(memory_slice(state, sizeof(uint64_t) * 25 * 4))+);++#define mld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm \+  MLD_NAMESPACE(keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm)+void mld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm(+    uint64_t state[100], const uint64_t rc[24])+__contract__(+  requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4))+  requires(rc == mld_keccakf1600_round_constants)+  assigns(memory_slice(state, sizeof(uint64_t) * 25 * 4))+);++#endif /* !MLD_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H */
+ cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x1_scalar_aarch64_asm.S view
@@ -0,0 +1,378 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * Copyright (c) 2021-2022 Arm Limited+ * Copyright (c) 2022 Matthias Kannwischer+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++// Author: Hanno Becker <hanno.becker@arm.com>+// Author: Matthias Kannwischer <matthias@kannwischer.eu>++/*yaml+  Name: keccak_f1600_x1_scalar_aarch64_asm+  Description: AArch64 scalar implementation of Keccak-f[1600] permutation for single state+  Signature: void mld_keccak_f1600_x1_scalar_aarch64_asm(uint64_t state[25], const uint64_t rc[24])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    x0:+      type: buffer+      size_bytes: 200+      permissions: read/write+      c_parameter: uint64_t state[25]+      description: Keccak state (25 x uint64_t)+    x1:+      type: buffer+      size_bytes: 192+      permissions: read-only+      c_parameter: uint64_t const *rc+      description: Round constants (24 x uint64_t)+  Stack:+    bytes: 128+    description: register preservation and temporary storage+*/++#include "../../../../common.h"+#if defined(MLD_FIPS202_AARCH64_NEED_X1_SCALAR) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/fips202/aarch64/src/keccak_f1600_x1_scalar_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(keccak_f1600_x1_scalar_aarch64_asm)+MLD_ASM_FN_SYMBOL(keccak_f1600_x1_scalar_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0x80+        .cfi_adjust_cfa_offset 0x80+        stp x19, x20, [sp, #0x20]+        .cfi_rel_offset x19, 0x20+        .cfi_rel_offset x20, 0x28+        stp x21, x22, [sp, #0x30]+        .cfi_rel_offset x21, 0x30+        .cfi_rel_offset x22, 0x38+        stp x23, x24, [sp, #0x40]+        .cfi_rel_offset x23, 0x40+        .cfi_rel_offset x24, 0x48+        stp x25, x26, [sp, #0x50]+        .cfi_rel_offset x25, 0x50+        .cfi_rel_offset x26, 0x58+        stp x27, x28, [sp, #0x60]+        .cfi_rel_offset x27, 0x60+        .cfi_rel_offset x28, 0x68+        stp x29, x30, [sp, #0x70]+        .cfi_rel_offset x29, 0x70+        .cfi_rel_offset x30, 0x78++Lmld_keccak_f1600_x1_scalar_initial:+        mov x26, x1+        str x1, [sp, #0x8]+        ldp x1, x6, [x0]+        ldp x11, x16, [x0, #0x10]+        ldp x21, x2, [x0, #0x20]+        ldp x7, x12, [x0, #0x30]+        ldp x17, x22, [x0, #0x40]+        ldp x3, x8, [x0, #0x50]+        ldp x13, x28, [x0, #0x60]+        ldp x23, x4, [x0, #0x70]+        ldp x9, x14, [x0, #0x80]+        ldp x19, x24, [x0, #0x90]+        ldp x5, x10, [x0, #0xa0]+        ldp x15, x20, [x0, #0xb0]+        ldr x25, [x0, #0xc0]+        str x0, [sp]+        eor x30, x24, x25+        eor x27, x9, x10+        eor x0, x30, x21+        eor x26, x27, x6+        eor x27, x26, x7+        eor x29, x0, x22+        eor x26, x29, x23+        eor x29, x4, x5+        eor x30, x29, x1+        eor x0, x27, x8+        eor x29, x30, x2+        eor x30, x19, x20+        eor x30, x30, x16+        eor x27, x26, x0, ror #63+        eor x4, x4, x27+        eor x30, x30, x17+        eor x30, x30, x28+        eor x29, x29, x3+        eor x0, x0, x30, ror #63+        eor x30, x30, x29, ror #63+        eor x22, x22, x30+        eor x23, x23, x30+        str x23, [sp, #0x18]+        eor x23, x14, x15+        eor x14, x14, x0+        eor x23, x23, x11+        eor x15, x15, x0+        eor x1, x1, x27+        eor x23, x23, x12+        eor x23, x23, x13+        eor x11, x11, x0+        eor x29, x29, x23, ror #63+        eor x23, x23, x26, ror #63+        eor x26, x13, x0+        eor x13, x28, x23+        eor x28, x24, x30+        eor x24, x16, x23+        eor x16, x21, x30+        eor x21, x25, x30+        eor x30, x19, x23+        eor x19, x20, x23+        eor x20, x17, x23+        eor x17, x12, x0+        eor x0, x2, x27+        eor x2, x6, x29+        eor x6, x8, x29+        bic x8, x28, x13, ror #47+        eor x12, x3, x27+        bic x3, x13, x17, ror #19+        eor x5, x5, x27+        ldr x27, [sp, #0x18]+        bic x25, x17, x2, ror #5+        eor x9, x9, x29+        eor x23, x25, x5, ror #52+        eor x3, x3, x2, ror #24+        eor x8, x8, x17, ror #2+        eor x17, x10, x29+        bic x25, x12, x22, ror #47+        eor x29, x7, x29+        bic x10, x4, x27, ror #2+        bic x7, x5, x28, ror #10+        eor x10, x10, x20, ror #50+        eor x13, x7, x13, ror #57+        bic x7, x2, x5, ror #47+        eor x2, x25, x24, ror #39+        bic x25, x20, x11, ror #57+        bic x5, x17, x4, ror #25+        eor x25, x25, x17, ror #53+        bic x17, x11, x17, ror #60+        eor x28, x7, x28, ror #57+        bic x7, x9, x12, ror #42+        eor x7, x7, x22, ror #25+        bic x22, x22, x24, ror #56+        bic x24, x24, x15, ror #31+        eor x22, x22, x15, ror #23+        bic x20, x27, x20, ror #48+        bic x15, x15, x9, ror #16+        eor x12, x15, x12, ror #58+        eor x15, x5, x27, ror #27+        eor x5, x20, x11, ror #41+        ldr x11, [sp, #0x8]+        eor x20, x17, x4, ror #21+        eor x17, x24, x9, ror #47+        mov x24, #0x1               // =1+        bic x9, x0, x16, ror #9+        str x24, [sp, #0x10]+        bic x24, x29, x1, ror #44+        bic x27, x1, x21, ror #50+        bic x4, x26, x29, ror #63+        eor x1, x1, x4, ror #21+        ldr x11, [x11]+        bic x4, x21, x30, ror #57+        eor x21, x24, x21, ror #30+        eor x24, x9, x19, ror #44+        bic x9, x14, x6, ror #5+        eor x9, x9, x0, ror #43+        bic x0, x6, x0, ror #38+        eor x1, x1, x11+        eor x11, x4, x26, ror #35+        eor x4, x0, x16, ror #47+        bic x0, x16, x19, ror #35+        eor x16, x27, x30, ror #43+        bic x27, x30, x26, ror #42+        bic x26, x19, x14, ror #41+        eor x19, x0, x14, ror #12+        eor x14, x26, x6, ror #46+        eor x6, x27, x29, ror #41++Lmld_keccak_f1600_x1_scalar_loop:+        eor x0, x15, x11, ror #52+        eor x0, x0, x13, ror #48+        eor x26, x8, x9, ror #57+        eor x27, x0, x14, ror #10+        eor x29, x16, x28, ror #63+        eor x26, x26, x6, ror #51+        eor x30, x23, x22, ror #50+        eor x0, x26, x10, ror #31+        eor x29, x29, x19, ror #37+        eor x27, x27, x12, ror #5+        eor x30, x30, x24, ror #34+        eor x0, x0, x7, ror #27+        eor x26, x30, x21, ror #26+        eor x26, x26, x25, ror #15+        ror x30, x27, #0x3e+        eor x30, x30, x26, ror #57+        ror x26, x26, #0x3a+        eor x16, x30, x16+        eor x28, x30, x28, ror #63+        str x28, [sp, #0x18]+        eor x29, x29, x17, ror #36+        eor x28, x1, x2, ror #61+        eor x19, x30, x19, ror #37+        eor x29, x29, x20, ror #2+        eor x28, x28, x4, ror #54+        eor x26, x26, x0, ror #55+        eor x28, x28, x3, ror #39+        eor x28, x28, x5, ror #25+        ror x0, x0, #0x38+        eor x0, x0, x29, ror #63+        eor x27, x28, x27, ror #61+        eor x13, x0, x13, ror #46+        eor x28, x29, x28, ror #63+        eor x29, x30, x20, ror #2+        eor x20, x26, x3, ror #39+        eor x11, x0, x11, ror #50+        eor x25, x28, x25, ror #9+        eor x3, x28, x21, ror #20+        eor x21, x26, x1+        eor x9, x27, x9, ror #49+        eor x24, x28, x24, ror #28+        eor x1, x30, x17, ror #36+        eor x14, x0, x14, ror #8+        eor x22, x28, x22, ror #44+        eor x8, x27, x8, ror #56+        eor x17, x27, x7, ror #19+        eor x15, x0, x15, ror #62+        bic x7, x20, x22, ror #47+        eor x4, x26, x4, ror #54+        eor x0, x0, x12, ror #3+        eor x28, x28, x23, ror #58+        eor x23, x26, x2, ror #61+        eor x26, x26, x5, ror #25+        eor x2, x7, x16, ror #39+        bic x7, x9, x20, ror #42+        bic x30, x15, x9, ror #16+        eor x7, x7, x22, ror #25+        eor x12, x30, x20, ror #58+        bic x20, x22, x16, ror #56+        eor x30, x27, x6, ror #43+        eor x22, x20, x15, ror #23+        bic x6, x19, x13, ror #42+        eor x6, x6, x17, ror #41+        bic x5, x13, x17, ror #63+        eor x5, x21, x5, ror #21+        bic x17, x17, x21, ror #44+        eor x27, x27, x10, ror #23+        bic x21, x21, x25, ror #50+        bic x20, x27, x4, ror #25+        bic x10, x16, x15, ror #31+        eor x16, x21, x19, ror #43+        eor x21, x17, x25, ror #30+        bic x19, x25, x19, ror #57+        ldr x25, [sp, #0x10]+        eor x17, x10, x9, ror #47+        ldr x9, [sp, #0x8]+        eor x15, x20, x28, ror #27+        bic x20, x4, x28, ror #2+        eor x10, x20, x1, ror #50+        bic x20, x11, x27, ror #60+        eor x20, x20, x4, ror #21+        bic x4, x28, x1, ror #48+        bic x1, x1, x11, ror #57+        ldr x28, [x9, x25, lsl #3]+        ldr x9, [sp, #0x18]+        add x25, x25, #0x1+        str x25, [sp, #0x10]+        cmp x25, #0x17+        eor x25, x1, x27, ror #53+        bic x27, x30, x26, ror #47+        eor x1, x5, x28+        eor x5, x4, x11, ror #41+        eor x11, x19, x13, ror #35+        bic x13, x26, x24, ror #10+        eor x28, x27, x24, ror #57+        bic x27, x24, x9, ror #47+        bic x19, x23, x3, ror #9+        bic x4, x29, x14, ror #41+        eor x24, x19, x29, ror #44+        bic x29, x3, x29, ror #35+        eor x13, x13, x9, ror #57+        eor x19, x29, x14, ror #12+        bic x29, x9, x0, ror #19+        bic x14, x14, x8, ror #5+        eor x9, x14, x23, ror #43+        eor x14, x4, x8, ror #46+        bic x23, x8, x23, ror #38+        eor x8, x27, x0, ror #2+        eor x4, x23, x3, ror #47+        bic x3, x0, x30, ror #5+        eor x23, x3, x26, ror #52+        eor x3, x29, x30, ror #24+        b.le Lmld_keccak_f1600_x1_scalar_loop+        ror x6, x6, #0x2b+        ror x11, x11, #0x32+        ror x21, x21, #0x14+        ror x2, x2, #0x3d+        ror x7, x7, #0x13+        ror x12, x12, #0x3+        ror x17, x17, #0x24+        ror x22, x22, #0x2c+        ror x3, x3, #0x27+        ror x8, x8, #0x38+        ror x13, x13, #0x2e+        ror x28, x28, #0x3f+        ror x23, x23, #0x3a+        ror x4, x4, #0x36+        ror x9, x9, #0x31+        ror x14, x14, #0x8+        ror x19, x19, #0x25+        ror x24, x24, #0x1c+        ror x5, x5, #0x19+        ror x10, x10, #0x17+        ror x15, x15, #0x3e+        ror x20, x20, #0x2+        ror x25, x25, #0x9+        ldr x0, [sp]+        stp x1, x6, [x0]+        stp x11, x16, [x0, #0x10]+        stp x21, x2, [x0, #0x20]+        stp x7, x12, [x0, #0x30]+        stp x17, x22, [x0, #0x40]+        stp x3, x8, [x0, #0x50]+        stp x13, x28, [x0, #0x60]+        stp x23, x4, [x0, #0x70]+        stp x9, x14, [x0, #0x80]+        stp x19, x24, [x0, #0x90]+        stp x5, x10, [x0, #0xa0]+        stp x15, x20, [x0, #0xb0]+        str x25, [x0, #0xc0]+        ldp x19, x20, [sp, #0x20]+        .cfi_restore x19+        .cfi_restore x20+        ldp x21, x22, [sp, #0x30]+        .cfi_restore x21+        .cfi_restore x22+        ldp x23, x24, [sp, #0x40]+        .cfi_restore x23+        .cfi_restore x24+        ldp x25, x26, [sp, #0x50]+        .cfi_restore x25+        .cfi_restore x26+        ldp x27, x28, [sp, #0x60]+        .cfi_restore x27+        .cfi_restore x28+        ldp x29, x30, [sp, #0x70]+        .cfi_restore x29+        .cfi_restore x30+        add sp, sp, #0x80+        .cfi_adjust_cfa_offset -0x80+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(keccak_f1600_x1_scalar_aarch64_asm)++#endif /* MLD_FIPS202_AARCH64_NEED_X1_SCALAR && \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x1_v84a_aarch64_asm.S view
@@ -0,0 +1,207 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * Copyright (c) 2021-2022 Arm Limited+ * Copyright (c) 2022 Matthias Kannwischer+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [HYBRID]+ *   Hybrid scalar/vector implementations of Keccak and SPHINCS+ on AArch64+ *   Becker, Kannwischer+ *   https://eprint.iacr.org/2022/1243+ */++/*yaml+  Name: keccak_f1600_x1_v84a_aarch64_asm+  Description: AArch64 ARMv8.4-A implementation of Keccak-f[1600] permutation for single state+  Signature: void mld_keccak_f1600_x1_v84a_aarch64_asm(uint64_t state[25], const uint64_t rc[24])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON, SHA3]+    x0:+      type: buffer+      size_bytes: 200+      permissions: read/write+      c_parameter: uint64_t state[25]+      description: Keccak state (25 x uint64_t)+    x1:+      type: buffer+      size_bytes: 192+      permissions: read-only+      c_parameter: uint64_t const *rc+      description: Round constants (24 x uint64_t)+  Stack:+    bytes: 64+    description: register preservation+*/++//+// Author: Hanno Becker <hanno.becker@arm.com>+// Author: Matthias Kannwischer <matthias@kannwischer.eu>+//+// This implementation is essentially from the paper @[HYBRID].+// The only difference is interleaving/deinterleaving of Keccak state+// during load and store, so that the caller need not do this.+//++#include "../../../../common.h"+#if defined(MLD_FIPS202_AARCH64_NEED_X1_V84A) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++#if defined(__ARM_FEATURE_SHA3)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/fips202/aarch64/src/keccak_f1600_x1_v84a_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(keccak_f1600_x1_v84a_aarch64_asm)+MLD_ASM_FN_SYMBOL(keccak_f1600_x1_v84a_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0x40+        .cfi_adjust_cfa_offset 0x40+        stp d8, d9, [sp]+        .cfi_rel_offset d8, 0x0+        .cfi_rel_offset d9, 0x8+        stp d10, d11, [sp, #0x10]+        .cfi_rel_offset d10, 0x10+        .cfi_rel_offset d11, 0x18+        stp d12, d13, [sp, #0x20]+        .cfi_rel_offset d12, 0x20+        .cfi_rel_offset d13, 0x28+        stp d14, d15, [sp, #0x30]+        .cfi_rel_offset d14, 0x30+        .cfi_rel_offset d15, 0x38+        ldp d0, d1, [x0]+        ldp d2, d3, [x0, #0x10]+        ldp d4, d5, [x0, #0x20]+        ldp d6, d7, [x0, #0x30]+        ldp d8, d9, [x0, #0x40]+        ldp d10, d11, [x0, #0x50]+        ldp d12, d13, [x0, #0x60]+        ldp d14, d15, [x0, #0x70]+        ldp d16, d17, [x0, #0x80]+        ldp d18, d19, [x0, #0x90]+        ldp d20, d21, [x0, #0xa0]+        ldp d22, d23, [x0, #0xb0]+        ldr d24, [x0, #0xc0]+        mov x2, #0x18               // =24++Lmld_keccak_f1600_x1_v84a_loop:+        eor3 v30.16b, v0.16b, v5.16b, v10.16b+        eor3 v29.16b, v1.16b, v6.16b, v11.16b+        eor3 v28.16b, v2.16b, v7.16b, v12.16b+        eor3 v27.16b, v3.16b, v8.16b, v13.16b+        eor3 v26.16b, v4.16b, v9.16b, v14.16b+        eor3 v30.16b, v30.16b, v15.16b, v20.16b+        eor3 v29.16b, v29.16b, v16.16b, v21.16b+        eor3 v28.16b, v28.16b, v17.16b, v22.16b+        eor3 v27.16b, v27.16b, v18.16b, v23.16b+        eor3 v26.16b, v26.16b, v19.16b, v24.16b+        rax1 v25.2d, v30.2d, v28.2d+        rax1 v28.2d, v28.2d, v26.2d+        rax1 v26.2d, v26.2d, v29.2d+        rax1 v29.2d, v29.2d, v27.2d+        rax1 v27.2d, v27.2d, v30.2d+        eor v30.16b, v0.16b, v26.16b+        xar v0.2d, v2.2d, v29.2d, #0x2+        xar v2.2d, v12.2d, v29.2d, #0x15+        xar v12.2d, v13.2d, v28.2d, #0x27+        xar v13.2d, v19.2d, v27.2d, #0x38+        xar v19.2d, v23.2d, v28.2d, #0x8+        xar v23.2d, v15.2d, v26.2d, #0x17+        xar v15.2d, v1.2d, v25.2d, #0x3f+        xar v1.2d, v8.2d, v28.2d, #0x9+        xar v8.2d, v16.2d, v25.2d, #0x13+        xar v16.2d, v7.2d, v29.2d, #0x3a+        xar v7.2d, v10.2d, v26.2d, #0x3d+        xar v10.2d, v3.2d, v28.2d, #0x24+        xar v3.2d, v18.2d, v28.2d, #0x2b+        xar v18.2d, v17.2d, v29.2d, #0x31+        xar v17.2d, v11.2d, v25.2d, #0x36+        xar v11.2d, v9.2d, v27.2d, #0x2c+        xar v9.2d, v22.2d, v29.2d, #0x3+        xar v22.2d, v14.2d, v27.2d, #0x19+        xar v14.2d, v20.2d, v26.2d, #0x2e+        xar v20.2d, v4.2d, v27.2d, #0x25+        xar v4.2d, v24.2d, v27.2d, #0x32+        xar v24.2d, v21.2d, v25.2d, #0x3e+        xar v21.2d, v5.2d, v26.2d, #0x1c+        xar v27.2d, v6.2d, v25.2d, #0x14+        ld1r { v31.2d }, [x1], #8+        bcax v5.16b, v10.16b, v7.16b, v11.16b+        bcax v6.16b, v11.16b, v8.16b, v7.16b+        bcax v7.16b, v7.16b, v9.16b, v8.16b+        bcax v8.16b, v8.16b, v10.16b, v9.16b+        bcax v9.16b, v9.16b, v11.16b, v10.16b+        bcax v10.16b, v15.16b, v12.16b, v16.16b+        bcax v11.16b, v16.16b, v13.16b, v12.16b+        bcax v12.16b, v12.16b, v14.16b, v13.16b+        bcax v13.16b, v13.16b, v15.16b, v14.16b+        bcax v14.16b, v14.16b, v16.16b, v15.16b+        bcax v15.16b, v20.16b, v17.16b, v21.16b+        bcax v16.16b, v21.16b, v18.16b, v17.16b+        bcax v17.16b, v17.16b, v19.16b, v18.16b+        bcax v18.16b, v18.16b, v20.16b, v19.16b+        bcax v19.16b, v19.16b, v21.16b, v20.16b+        bcax v20.16b, v0.16b, v22.16b, v1.16b+        bcax v21.16b, v1.16b, v23.16b, v22.16b+        bcax v22.16b, v22.16b, v24.16b, v23.16b+        bcax v23.16b, v23.16b, v0.16b, v24.16b+        bcax v24.16b, v24.16b, v1.16b, v0.16b+        bcax v0.16b, v30.16b, v2.16b, v27.16b+        bcax v1.16b, v27.16b, v3.16b, v2.16b+        bcax v2.16b, v2.16b, v4.16b, v3.16b+        bcax v3.16b, v3.16b, v30.16b, v4.16b+        bcax v4.16b, v4.16b, v27.16b, v30.16b+        eor v0.16b, v0.16b, v31.16b+        sub x2, x2, #0x1+        cbnz x2, Lmld_keccak_f1600_x1_v84a_loop+        stp d0, d1, [x0]+        stp d2, d3, [x0, #0x10]+        stp d4, d5, [x0, #0x20]+        stp d6, d7, [x0, #0x30]+        stp d8, d9, [x0, #0x40]+        stp d10, d11, [x0, #0x50]+        stp d12, d13, [x0, #0x60]+        stp d14, d15, [x0, #0x70]+        stp d16, d17, [x0, #0x80]+        stp d18, d19, [x0, #0x90]+        stp d20, d21, [x0, #0xa0]+        stp d22, d23, [x0, #0xb0]+        str d24, [x0, #0xc0]+        ldp d8, d9, [sp]+        .cfi_restore d8+        .cfi_restore d9+        ldp d10, d11, [sp, #0x10]+        .cfi_restore d10+        .cfi_restore d11+        ldp d12, d13, [sp, #0x20]+        .cfi_restore d12+        .cfi_restore d13+        ldp d14, d15, [sp, #0x30]+        .cfi_restore d14+        .cfi_restore d15+        add sp, sp, #0x40+        .cfi_adjust_cfa_offset -0x40+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(keccak_f1600_x1_v84a_aarch64_asm)++#endif /* __ARM_FEATURE_SHA3 */++#endif /* MLD_FIPS202_AARCH64_NEED_X1_V84A && !MLD_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x2_v84a_aarch64_asm.S view
@@ -0,0 +1,262 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * Copyright (c) 2021-2022 Arm Limited+ * Copyright (c) 2022 Matthias Kannwischer+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [HYBRID]+ *   Hybrid scalar/vector implementations of Keccak and SPHINCS+ on AArch64+ *   Becker, Kannwischer+ *   https://eprint.iacr.org/2022/1243+ */++/*yaml+  Name: keccak_f1600_x2_v84a_aarch64_asm+  Description: AArch64 ARMv8.4-A implementation of Keccak-f[1600] permutation for two sequential states+  Signature: void mld_keccak_f1600_x2_v84a_aarch64_asm(uint64_t state[50], const uint64_t rc[24])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON, SHA3]+    x0:+      type: buffer+      size_bytes: 400+      permissions: read/write+      c_parameter: uint64_t state[50]+      description: Two sequential Keccak states (state0[25], state1[25])+    x1:+      type: buffer+      size_bytes: 192+      permissions: read-only+      c_parameter: uint64_t const *rc+      description: Round constants (24 x uint64_t)+  Stack:+    bytes: 64+    description: register preservation+*/++//+// Author: Hanno Becker <hanno.becker@arm.com>+// Author: Matthias Kannwischer <matthias@kannwischer.eu>+//+// This implementation is essentially from the paper @[HYBRID].+// The only difference is interleaving/deinterleaving of Keccak state+// during load and store, so that the caller need not do this.+//++#include "../../../../common.h"+#if defined(MLD_FIPS202_AARCH64_NEED_X2_V84A) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++#if defined(__ARM_FEATURE_SHA3)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/fips202/aarch64/src/keccak_f1600_x2_v84a_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(keccak_f1600_x2_v84a_aarch64_asm)+MLD_ASM_FN_SYMBOL(keccak_f1600_x2_v84a_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0x40+        .cfi_adjust_cfa_offset 0x40+        stp d8, d9, [sp]+        .cfi_rel_offset d8, 0x0+        .cfi_rel_offset d9, 0x8+        stp d10, d11, [sp, #0x10]+        .cfi_rel_offset d10, 0x10+        .cfi_rel_offset d11, 0x18+        stp d12, d13, [sp, #0x20]+        .cfi_rel_offset d12, 0x20+        .cfi_rel_offset d13, 0x28+        stp d14, d15, [sp, #0x30]+        .cfi_rel_offset d14, 0x30+        .cfi_rel_offset d15, 0x38+        add x2, x0, #0xc8+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x2], #32+        trn1 v0.2d, v25.2d, v27.2d+        trn2 v1.2d, v25.2d, v27.2d+        trn1 v2.2d, v26.2d, v28.2d+        trn2 v3.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x2], #32+        trn1 v4.2d, v25.2d, v27.2d+        trn2 v5.2d, v25.2d, v27.2d+        trn1 v6.2d, v26.2d, v28.2d+        trn2 v7.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x2], #32+        trn1 v8.2d, v25.2d, v27.2d+        trn2 v9.2d, v25.2d, v27.2d+        trn1 v10.2d, v26.2d, v28.2d+        trn2 v11.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x2], #32+        trn1 v12.2d, v25.2d, v27.2d+        trn2 v13.2d, v25.2d, v27.2d+        trn1 v14.2d, v26.2d, v28.2d+        trn2 v15.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x2], #32+        trn1 v16.2d, v25.2d, v27.2d+        trn2 v17.2d, v25.2d, v27.2d+        trn1 v18.2d, v26.2d, v28.2d+        trn2 v19.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x2], #32+        trn1 v20.2d, v25.2d, v27.2d+        trn2 v21.2d, v25.2d, v27.2d+        trn1 v22.2d, v26.2d, v28.2d+        trn2 v23.2d, v26.2d, v28.2d+        ldr d25, [x0]+        ldr d27, [x2]+        trn1 v24.2d, v25.2d, v27.2d+        mov x2, #0x18               // =24++Lmld_keccak_f1600_x2_v84a_loop:+        eor3 v30.16b, v0.16b, v5.16b, v10.16b+        eor3 v29.16b, v1.16b, v6.16b, v11.16b+        eor3 v28.16b, v2.16b, v7.16b, v12.16b+        eor3 v27.16b, v3.16b, v8.16b, v13.16b+        eor3 v26.16b, v4.16b, v9.16b, v14.16b+        eor3 v30.16b, v30.16b, v15.16b, v20.16b+        eor3 v29.16b, v29.16b, v16.16b, v21.16b+        eor3 v28.16b, v28.16b, v17.16b, v22.16b+        eor3 v27.16b, v27.16b, v18.16b, v23.16b+        eor3 v26.16b, v26.16b, v19.16b, v24.16b+        rax1 v25.2d, v30.2d, v28.2d+        rax1 v28.2d, v28.2d, v26.2d+        rax1 v26.2d, v26.2d, v29.2d+        rax1 v29.2d, v29.2d, v27.2d+        rax1 v27.2d, v27.2d, v30.2d+        eor v30.16b, v0.16b, v26.16b+        xar v0.2d, v2.2d, v29.2d, #0x2+        xar v2.2d, v12.2d, v29.2d, #0x15+        xar v12.2d, v13.2d, v28.2d, #0x27+        xar v13.2d, v19.2d, v27.2d, #0x38+        xar v19.2d, v23.2d, v28.2d, #0x8+        xar v23.2d, v15.2d, v26.2d, #0x17+        xar v15.2d, v1.2d, v25.2d, #0x3f+        xar v1.2d, v8.2d, v28.2d, #0x9+        xar v8.2d, v16.2d, v25.2d, #0x13+        xar v16.2d, v7.2d, v29.2d, #0x3a+        xar v7.2d, v10.2d, v26.2d, #0x3d+        xar v10.2d, v3.2d, v28.2d, #0x24+        xar v3.2d, v18.2d, v28.2d, #0x2b+        xar v18.2d, v17.2d, v29.2d, #0x31+        xar v17.2d, v11.2d, v25.2d, #0x36+        xar v11.2d, v9.2d, v27.2d, #0x2c+        xar v9.2d, v22.2d, v29.2d, #0x3+        xar v22.2d, v14.2d, v27.2d, #0x19+        xar v14.2d, v20.2d, v26.2d, #0x2e+        xar v20.2d, v4.2d, v27.2d, #0x25+        xar v4.2d, v24.2d, v27.2d, #0x32+        xar v24.2d, v21.2d, v25.2d, #0x3e+        xar v21.2d, v5.2d, v26.2d, #0x1c+        xar v27.2d, v6.2d, v25.2d, #0x14+        ld1r { v31.2d }, [x1], #8+        bcax v5.16b, v10.16b, v7.16b, v11.16b+        bcax v6.16b, v11.16b, v8.16b, v7.16b+        bcax v7.16b, v7.16b, v9.16b, v8.16b+        bcax v8.16b, v8.16b, v10.16b, v9.16b+        bcax v9.16b, v9.16b, v11.16b, v10.16b+        bcax v10.16b, v15.16b, v12.16b, v16.16b+        bcax v11.16b, v16.16b, v13.16b, v12.16b+        bcax v12.16b, v12.16b, v14.16b, v13.16b+        bcax v13.16b, v13.16b, v15.16b, v14.16b+        bcax v14.16b, v14.16b, v16.16b, v15.16b+        bcax v15.16b, v20.16b, v17.16b, v21.16b+        bcax v16.16b, v21.16b, v18.16b, v17.16b+        bcax v17.16b, v17.16b, v19.16b, v18.16b+        bcax v18.16b, v18.16b, v20.16b, v19.16b+        bcax v19.16b, v19.16b, v21.16b, v20.16b+        bcax v20.16b, v0.16b, v22.16b, v1.16b+        bcax v21.16b, v1.16b, v23.16b, v22.16b+        bcax v22.16b, v22.16b, v24.16b, v23.16b+        bcax v23.16b, v23.16b, v0.16b, v24.16b+        bcax v24.16b, v24.16b, v1.16b, v0.16b+        bcax v0.16b, v30.16b, v2.16b, v27.16b+        bcax v1.16b, v27.16b, v3.16b, v2.16b+        bcax v2.16b, v2.16b, v4.16b, v3.16b+        bcax v3.16b, v3.16b, v30.16b, v4.16b+        bcax v4.16b, v4.16b, v27.16b, v30.16b+        eor v0.16b, v0.16b, v31.16b+        sub x2, x2, #0x1+        cbnz x2, Lmld_keccak_f1600_x2_v84a_loop+        sub x0, x0, #0xc0+        add x2, x0, #0xc8+        trn1 v25.2d, v0.2d, v1.2d+        trn1 v26.2d, v2.2d, v3.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v0.2d, v1.2d+        trn2 v28.2d, v2.2d, v3.2d+        st1 { v27.2d, v28.2d }, [x2], #32+        trn1 v25.2d, v4.2d, v5.2d+        trn1 v26.2d, v6.2d, v7.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v4.2d, v5.2d+        trn2 v28.2d, v6.2d, v7.2d+        st1 { v27.2d, v28.2d }, [x2], #32+        trn1 v25.2d, v8.2d, v9.2d+        trn1 v26.2d, v10.2d, v11.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v8.2d, v9.2d+        trn2 v28.2d, v10.2d, v11.2d+        st1 { v27.2d, v28.2d }, [x2], #32+        trn1 v25.2d, v12.2d, v13.2d+        trn1 v26.2d, v14.2d, v15.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v12.2d, v13.2d+        trn2 v28.2d, v14.2d, v15.2d+        st1 { v27.2d, v28.2d }, [x2], #32+        trn1 v25.2d, v16.2d, v17.2d+        trn1 v26.2d, v18.2d, v19.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v16.2d, v17.2d+        trn2 v28.2d, v18.2d, v19.2d+        st1 { v27.2d, v28.2d }, [x2], #32+        trn1 v25.2d, v20.2d, v21.2d+        trn1 v26.2d, v22.2d, v23.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v20.2d, v21.2d+        trn2 v28.2d, v22.2d, v23.2d+        st1 { v27.2d, v28.2d }, [x2], #32+        str d24, [x0]+        trn2 v25.2d, v24.2d, v24.2d+        str d25, [x2]+        ldp d8, d9, [sp]+        .cfi_restore d8+        .cfi_restore d9+        ldp d10, d11, [sp, #0x10]+        .cfi_restore d10+        .cfi_restore d11+        ldp d12, d13, [sp, #0x20]+        .cfi_restore d12+        .cfi_restore d13+        ldp d14, d15, [sp, #0x30]+        .cfi_restore d14+        .cfi_restore d15+        add sp, sp, #0x40+        .cfi_adjust_cfa_offset -0x40+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(keccak_f1600_x2_v84a_aarch64_asm)++#endif /* __ARM_FEATURE_SHA3 */++#endif /* MLD_FIPS202_AARCH64_NEED_X2_V84A && !MLD_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.S view
@@ -0,0 +1,1080 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * Copyright (c) 2021-2022 Arm Limited+ * Copyright (c) 2022 Matthias Kannwischer+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++// Author: Hanno Becker <hannobecker@posteo.de>+// Author: Matthias Kannwischer <matthias@kannwischer.eu>++/*yaml+  Name: keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm+  Description: AArch64 hybrid scalar/vector implementation of Keccak-f[1600] permutation for four sequential states+  Signature: void mld_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm(uint64_t state[100], const uint64_t rc[24])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 800+      permissions: read/write+      c_parameter: uint64_t state[100]+      description: Four sequential Keccak states (state0[25], state1[25], state2[25], state3[25])+    x1:+      type: buffer+      size_bytes: 192+      permissions: read-only+      c_parameter: uint64_t const *rc+      description: Round constants (24 x uint64_t)+  Stack:+    bytes: 224+    description: register preservation and temporary storage+*/++#include "../../../../common.h"+#if defined(MLD_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/fips202/aarch64/src/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm)+MLD_ASM_FN_SYMBOL(keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0xe0+        .cfi_adjust_cfa_offset 0xe0+        stp x19, x20, [sp, #0x30]+        .cfi_rel_offset x19, 0x30+        .cfi_rel_offset x20, 0x38+        stp x21, x22, [sp, #0x40]+        .cfi_rel_offset x21, 0x40+        .cfi_rel_offset x22, 0x48+        stp x23, x24, [sp, #0x50]+        .cfi_rel_offset x23, 0x50+        .cfi_rel_offset x24, 0x58+        stp x25, x26, [sp, #0x60]+        .cfi_rel_offset x25, 0x60+        .cfi_rel_offset x26, 0x68+        stp x27, x28, [sp, #0x70]+        .cfi_rel_offset x27, 0x70+        .cfi_rel_offset x28, 0x78+        stp x29, x30, [sp, #0x80]+        .cfi_rel_offset x29, 0x80+        .cfi_rel_offset x30, 0x88+        stp d8, d9, [sp, #0x90]+        .cfi_rel_offset d8, 0x90+        .cfi_rel_offset d9, 0x98+        stp d10, d11, [sp, #0xa0]+        .cfi_rel_offset d10, 0xa0+        .cfi_rel_offset d11, 0xa8+        stp d12, d13, [sp, #0xb0]+        .cfi_rel_offset d12, 0xb0+        .cfi_rel_offset d13, 0xb8+        stp d14, d15, [sp, #0xc0]+        .cfi_rel_offset d14, 0xc0+        .cfi_rel_offset d15, 0xc8+        mov x29, x1+        mov x30, #0x0               // =0+        str x30, [sp, #0x20]+        str x29, [sp, #0x8]+        str x29, [sp, #0x10]+        str x0, [sp]+        add x4, x0, #0xc8+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v0.2d, v25.2d, v27.2d+        trn2 v1.2d, v25.2d, v27.2d+        trn1 v2.2d, v26.2d, v28.2d+        trn2 v3.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v4.2d, v25.2d, v27.2d+        trn2 v5.2d, v25.2d, v27.2d+        trn1 v6.2d, v26.2d, v28.2d+        trn2 v7.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v8.2d, v25.2d, v27.2d+        trn2 v9.2d, v25.2d, v27.2d+        trn1 v10.2d, v26.2d, v28.2d+        trn2 v11.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v12.2d, v25.2d, v27.2d+        trn2 v13.2d, v25.2d, v27.2d+        trn1 v14.2d, v26.2d, v28.2d+        trn2 v15.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v16.2d, v25.2d, v27.2d+        trn2 v17.2d, v25.2d, v27.2d+        trn1 v18.2d, v26.2d, v28.2d+        trn2 v19.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v20.2d, v25.2d, v27.2d+        trn2 v21.2d, v25.2d, v27.2d+        trn1 v22.2d, v26.2d, v28.2d+        trn2 v23.2d, v26.2d, v28.2d+        ldr d25, [x0]+        ldr d27, [x4]+        trn1 v24.2d, v25.2d, v27.2d+        sub x0, x0, #0xc0+        add x0, x0, #0x190+        ldp x1, x6, [x0]+        ldp x11, x16, [x0, #0x10]+        ldp x21, x2, [x0, #0x20]+        ldp x7, x12, [x0, #0x30]+        ldp x17, x22, [x0, #0x40]+        ldp x3, x8, [x0, #0x50]+        ldp x13, x28, [x0, #0x60]+        ldp x23, x4, [x0, #0x70]+        ldp x9, x14, [x0, #0x80]+        ldp x19, x24, [x0, #0x90]+        ldp x5, x10, [x0, #0xa0]+        ldp x15, x20, [x0, #0xb0]+        ldr x25, [x0, #0xc0]+        sub x0, x0, #0x190++Lmld_keccak_f1600_x4_v8a_scalar_hybrid_initial:+        eor x30, x24, x25+        eor x27, x9, x10+        eor v30.16b, v0.16b, v5.16b+        eor v30.16b, v30.16b, v10.16b+        eor x0, x30, x21+        eor v30.16b, v30.16b, v15.16b+        eor x26, x27, x6+        eor x27, x26, x7+        eor v30.16b, v30.16b, v20.16b+        eor x29, x0, x22+        eor v29.16b, v1.16b, v6.16b+        eor x26, x29, x23+        eor v29.16b, v29.16b, v11.16b+        eor x29, x4, x5+        eor x30, x29, x1+        eor v29.16b, v29.16b, v16.16b+        eor x0, x27, x8+        eor v29.16b, v29.16b, v21.16b+        eor x29, x30, x2+        eor v28.16b, v2.16b, v7.16b+        eor x30, x19, x20+        eor x30, x30, x16+        eor v28.16b, v28.16b, v12.16b+        eor x27, x26, x0, ror #63+        eor v28.16b, v28.16b, v17.16b+        eor x4, x4, x27+        eor v28.16b, v28.16b, v22.16b+        eor x30, x30, x17+        eor x30, x30, x28+        eor v27.16b, v3.16b, v8.16b+        eor x29, x29, x3+        eor v27.16b, v27.16b, v13.16b+        eor x0, x0, x30, ror #63+        eor v27.16b, v27.16b, v18.16b+        eor x30, x30, x29, ror #63+        eor x22, x22, x30+        eor v27.16b, v27.16b, v23.16b+        eor x23, x23, x30+        eor v26.16b, v4.16b, v9.16b+        str x23, [sp, #0xd0]+        eor v26.16b, v26.16b, v14.16b+        eor x23, x14, x15+        eor x14, x14, x0+        eor v26.16b, v26.16b, v19.16b+        eor x23, x23, x11+        eor v26.16b, v26.16b, v24.16b+        eor x15, x15, x0+        eor x1, x1, x27+        add v31.2d, v28.2d, v28.2d+        eor x23, x23, x12+        sri v31.2d, v28.2d, #0x3f+        eor x23, x23, x13+        eor v25.16b, v31.16b, v30.16b+        eor x11, x11, x0+        eor x29, x29, x23, ror #63+        add v31.2d, v26.2d, v26.2d+        eor x23, x23, x26, ror #63+        sri v31.2d, v26.2d, #0x3f+        eor x26, x13, x0+        eor v28.16b, v31.16b, v28.16b+        eor x13, x28, x23+        eor x28, x24, x30+        add v31.2d, v29.2d, v29.2d+        eor x24, x16, x23+        sri v31.2d, v29.2d, #0x3f+        eor x16, x21, x30+        eor v26.16b, v31.16b, v26.16b+        eor x21, x25, x30+        eor x30, x19, x23+        add v31.2d, v27.2d, v27.2d+        eor x19, x20, x23+        sri v31.2d, v27.2d, #0x3f+        eor x20, x17, x23+        eor v29.16b, v31.16b, v29.16b+        eor x17, x12, x0+        eor x0, x2, x27+        add v31.2d, v30.2d, v30.2d+        eor x2, x6, x29+        sri v31.2d, v30.2d, #0x3f+        eor x6, x8, x29+        eor v27.16b, v31.16b, v27.16b+        bic x8, x28, x13, ror #47+        eor x12, x3, x27+        eor v30.16b, v0.16b, v26.16b+        bic x3, x13, x17, ror #19+        eor v31.16b, v2.16b, v29.16b+        eor x5, x5, x27+        ldr x27, [sp, #0xd0]+        shl v0.2d, v31.2d, #0x3e+        bic x25, x17, x2, ror #5+        sri v0.2d, v31.2d, #0x2+        eor x9, x9, x29+        eor v31.16b, v12.16b, v29.16b+        eor x23, x25, x5, ror #52+        eor x3, x3, x2, ror #24+        shl v2.2d, v31.2d, #0x2b+        eor x8, x8, x17, ror #2+        sri v2.2d, v31.2d, #0x15+        eor x17, x10, x29+        eor v31.16b, v13.16b, v28.16b+        bic x25, x12, x22, ror #47+        eor x29, x7, x29+        shl v12.2d, v31.2d, #0x19+        bic x10, x4, x27, ror #2+        sri v12.2d, v31.2d, #0x27+        bic x7, x5, x28, ror #10+        eor v31.16b, v19.16b, v27.16b+        eor x10, x10, x20, ror #50+        eor x13, x7, x13, ror #57+        shl v13.2d, v31.2d, #0x8+        bic x7, x2, x5, ror #47+        sri v13.2d, v31.2d, #0x38+        eor x2, x25, x24, ror #39+        eor v31.16b, v23.16b, v28.16b+        bic x25, x20, x11, ror #57+        bic x5, x17, x4, ror #25+        shl v19.2d, v31.2d, #0x38+        eor x25, x25, x17, ror #53+        sri v19.2d, v31.2d, #0x8+        bic x17, x11, x17, ror #60+        eor v31.16b, v15.16b, v26.16b+        eor x28, x7, x28, ror #57+        bic x7, x9, x12, ror #42+        shl v23.2d, v31.2d, #0x29+        eor x7, x7, x22, ror #25+        sri v23.2d, v31.2d, #0x17+        bic x22, x22, x24, ror #56+        bic x24, x24, x15, ror #31+        eor v31.16b, v1.16b, v25.16b+        eor x22, x22, x15, ror #23+        shl v15.2d, v31.2d, #0x1+        bic x20, x27, x20, ror #48+        sri v15.2d, v31.2d, #0x3f+        bic x15, x15, x9, ror #16+        eor x12, x15, x12, ror #58+        eor v31.16b, v8.16b, v28.16b+        eor x15, x5, x27, ror #27+        shl v1.2d, v31.2d, #0x37+        eor x5, x20, x11, ror #41+        sri v1.2d, v31.2d, #0x9+        ldr x11, [sp, #0x8]+        eor x20, x17, x4, ror #21+        eor v31.16b, v16.16b, v25.16b+        eor x17, x24, x9, ror #47+        shl v8.2d, v31.2d, #0x2d+        mov x24, #0x1               // =1+        sri v8.2d, v31.2d, #0x13+        bic x9, x0, x16, ror #9+        str x24, [sp, #0x18]+        eor v31.16b, v7.16b, v29.16b+        bic x24, x29, x1, ror #44+        shl v16.2d, v31.2d, #0x6+        bic x27, x1, x21, ror #50+        sri v16.2d, v31.2d, #0x3a+        bic x4, x26, x29, ror #63+        eor x1, x1, x4, ror #21+        eor v31.16b, v10.16b, v26.16b+        ldr x11, [x11]+        shl v7.2d, v31.2d, #0x3+        bic x4, x21, x30, ror #57+        sri v7.2d, v31.2d, #0x3d+        eor x21, x24, x21, ror #30+        eor x24, x9, x19, ror #44+        eor v31.16b, v3.16b, v28.16b+        bic x9, x14, x6, ror #5+        shl v10.2d, v31.2d, #0x1c+        eor x9, x9, x0, ror #43+        sri v10.2d, v31.2d, #0x24+        bic x0, x6, x0, ror #38+        eor x1, x1, x11+        eor v31.16b, v18.16b, v28.16b+        eor x11, x4, x26, ror #35+        shl v3.2d, v31.2d, #0x15+        eor x4, x0, x16, ror #47+        bic x0, x16, x19, ror #35+        sri v3.2d, v31.2d, #0x2b+        eor x16, x27, x30, ror #43+        eor v31.16b, v17.16b, v29.16b+        bic x27, x30, x26, ror #42+        shl v18.2d, v31.2d, #0xf+        bic x26, x19, x14, ror #41+        eor x19, x0, x14, ror #12+        sri v18.2d, v31.2d, #0x31+        eor x14, x26, x6, ror #46+        eor v31.16b, v11.16b, v25.16b+        eor x6, x27, x29, ror #41+        shl v17.2d, v31.2d, #0xa+        eor x0, x15, x11, ror #52+        eor x0, x0, x13, ror #48+        sri v17.2d, v31.2d, #0x36+        eor x26, x8, x9, ror #57+        eor v31.16b, v9.16b, v27.16b+        eor x27, x0, x14, ror #10+        shl v11.2d, v31.2d, #0x14+        eor x29, x16, x28, ror #63+        eor x26, x26, x6, ror #51+        sri v11.2d, v31.2d, #0x2c+        eor x30, x23, x22, ror #50+        eor v31.16b, v22.16b, v29.16b+        eor x0, x26, x10, ror #31+        shl v9.2d, v31.2d, #0x3d+        eor x29, x29, x19, ror #37+        eor x27, x27, x12, ror #5+        sri v9.2d, v31.2d, #0x3+        eor x30, x30, x24, ror #34+        eor v31.16b, v14.16b, v27.16b+        eor x0, x0, x7, ror #27+        shl v22.2d, v31.2d, #0x27+        eor x26, x30, x21, ror #26+        eor x26, x26, x25, ror #15+        sri v22.2d, v31.2d, #0x19+        ror x30, x27, #0x3e+        eor v31.16b, v20.16b, v26.16b+        eor x30, x30, x26, ror #57+        ror x26, x26, #0x3a+        shl v14.2d, v31.2d, #0x12+        eor x16, x30, x16+        sri v14.2d, v31.2d, #0x2e+        eor x28, x30, x28, ror #63+        eor v31.16b, v4.16b, v27.16b+        str x28, [sp, #0xd0]+        eor x29, x29, x17, ror #36+        shl v20.2d, v31.2d, #0x1b+        eor x28, x1, x2, ror #61+        sri v20.2d, v31.2d, #0x25+        eor x19, x30, x19, ror #37+        eor v31.16b, v24.16b, v27.16b+        eor x29, x29, x20, ror #2+        eor x28, x28, x4, ror #54+        shl v4.2d, v31.2d, #0xe+        eor x26, x26, x0, ror #55+        sri v4.2d, v31.2d, #0x32+        eor x28, x28, x3, ror #39+        eor v31.16b, v21.16b, v25.16b+        eor x28, x28, x5, ror #25+        ror x0, x0, #0x38+        shl v24.2d, v31.2d, #0x2+        eor x0, x0, x29, ror #63+        sri v24.2d, v31.2d, #0x3e+        eor x27, x28, x27, ror #61+        eor v31.16b, v5.16b, v26.16b+        eor x13, x0, x13, ror #46+        eor x28, x29, x28, ror #63+        shl v21.2d, v31.2d, #0x24+        eor x29, x30, x20, ror #2+        sri v21.2d, v31.2d, #0x1c+        eor x20, x26, x3, ror #39+        eor v31.16b, v6.16b, v25.16b+        eor x11, x0, x11, ror #50+        eor x25, x28, x25, ror #9+        shl v27.2d, v31.2d, #0x2c+        eor x3, x28, x21, ror #20+        sri v27.2d, v31.2d, #0x14+        eor x21, x26, x1+        eor x9, x27, x9, ror #49+        bic v31.16b, v7.16b, v11.16b+        eor x24, x28, x24, ror #28+        eor v5.16b, v31.16b, v10.16b+        eor x1, x30, x17, ror #36+        bic v31.16b, v8.16b, v7.16b+        eor x14, x0, x14, ror #8+        eor x22, x28, x22, ror #44+        eor v6.16b, v31.16b, v11.16b+        eor x8, x27, x8, ror #56+        bic v31.16b, v9.16b, v8.16b+        eor x17, x27, x7, ror #19+        eor v7.16b, v31.16b, v7.16b+        eor x15, x0, x15, ror #62+        bic x7, x20, x22, ror #47+        bic v31.16b, v10.16b, v9.16b+        eor x4, x26, x4, ror #54+        eor v8.16b, v31.16b, v8.16b+        eor x0, x0, x12, ror #3+        bic v31.16b, v11.16b, v10.16b+        eor x28, x28, x23, ror #58+        eor x23, x26, x2, ror #61+        eor v9.16b, v31.16b, v9.16b+        eor x26, x26, x5, ror #25+        bic v31.16b, v12.16b, v16.16b+        eor x2, x7, x16, ror #39+        eor v10.16b, v31.16b, v15.16b+        bic x7, x9, x20, ror #42+        bic x30, x15, x9, ror #16+        bic v31.16b, v13.16b, v12.16b+        eor x7, x7, x22, ror #25+        eor v11.16b, v31.16b, v16.16b+        eor x12, x30, x20, ror #58+        bic v31.16b, v14.16b, v13.16b+        bic x20, x22, x16, ror #56+        eor x30, x27, x6, ror #43+        eor v12.16b, v31.16b, v12.16b+        eor x22, x20, x15, ror #23+        bic v31.16b, v15.16b, v14.16b+        bic x6, x19, x13, ror #42+        eor v13.16b, v31.16b, v13.16b+        eor x6, x6, x17, ror #41+        bic x5, x13, x17, ror #63+        bic v31.16b, v16.16b, v15.16b+        eor x5, x21, x5, ror #21+        eor v14.16b, v31.16b, v14.16b+        bic x17, x17, x21, ror #44+        eor x27, x27, x10, ror #23+        bic v31.16b, v17.16b, v21.16b+        bic x21, x21, x25, ror #50+        eor v15.16b, v31.16b, v20.16b+        bic x20, x27, x4, ror #25+        bic v31.16b, v18.16b, v17.16b+        bic x10, x16, x15, ror #31+        eor x16, x21, x19, ror #43+        eor v16.16b, v31.16b, v21.16b+        eor x21, x17, x25, ror #30+        bic v31.16b, v19.16b, v18.16b+        bic x19, x25, x19, ror #57+        eor v17.16b, v31.16b, v17.16b+        ldr x25, [sp, #0x18]+        eor x17, x10, x9, ror #47+        bic v31.16b, v20.16b, v19.16b+        ldr x9, [sp, #0x8]+        eor v18.16b, v31.16b, v18.16b+        eor x15, x20, x28, ror #27+        bic v31.16b, v21.16b, v20.16b+        bic x20, x4, x28, ror #2+        eor x10, x20, x1, ror #50+        eor v19.16b, v31.16b, v19.16b+        bic x20, x11, x27, ror #60+        bic v31.16b, v22.16b, v1.16b+        eor x20, x20, x4, ror #21+        eor v20.16b, v31.16b, v0.16b+        bic x4, x28, x1, ror #48+        bic x1, x1, x11, ror #57+        bic v31.16b, v23.16b, v22.16b+        ldr x28, [x9, x25, lsl #3]+        eor v21.16b, v31.16b, v1.16b+        ldr x9, [sp, #0xd0]+        bic v31.16b, v24.16b, v23.16b+        add x25, x25, #0x1+        str x25, [sp, #0x18]+        eor v22.16b, v31.16b, v22.16b+        cmp x25, #0x17+        bic v31.16b, v0.16b, v24.16b+        eor x25, x1, x27, ror #53+        bic x27, x30, x26, ror #47+        eor v23.16b, v31.16b, v23.16b+        eor x1, x5, x28+        bic v31.16b, v1.16b, v0.16b+        eor x5, x4, x11, ror #41+        eor v24.16b, v31.16b, v24.16b+        eor x11, x19, x13, ror #35+        bic x13, x26, x24, ror #10+        bic v31.16b, v2.16b, v27.16b+        eor x28, x27, x24, ror #57+        eor v0.16b, v31.16b, v30.16b+        bic x27, x24, x9, ror #47+        bic v31.16b, v3.16b, v2.16b+        bic x19, x23, x3, ror #9+        bic x4, x29, x14, ror #41+        eor v1.16b, v31.16b, v27.16b+        eor x24, x19, x29, ror #44+        bic v31.16b, v4.16b, v3.16b+        bic x29, x3, x29, ror #35+        eor v2.16b, v31.16b, v2.16b+        eor x13, x13, x9, ror #57+        eor x19, x29, x14, ror #12+        bic v31.16b, v30.16b, v4.16b+        bic x29, x9, x0, ror #19+        eor v3.16b, v31.16b, v3.16b+        bic x14, x14, x8, ror #5+        bic v31.16b, v27.16b, v30.16b+        eor x9, x14, x23, ror #43+        eor x14, x4, x8, ror #46+        eor v4.16b, v31.16b, v4.16b+        bic x23, x8, x23, ror #38+        eor x8, x27, x0, ror #2+        eor x4, x23, x3, ror #47+        bic x3, x0, x30, ror #5+        eor x23, x3, x26, ror #52+        eor x3, x29, x30, ror #24+        ldr x30, [sp, #0x10]+        ld1r { v28.2d }, [x30], #8+        str x30, [sp, #0x10]+        eor v0.16b, v0.16b, v28.16b++Lmld_keccak_f1600_x4_v8a_scalar_hybrid_loop:+        eor x0, x15, x11, ror #52+        eor x0, x0, x13, ror #48+        eor v30.16b, v0.16b, v5.16b+        eor v30.16b, v30.16b, v10.16b+        eor x26, x8, x9, ror #57+        eor v30.16b, v30.16b, v15.16b+        eor x27, x0, x14, ror #10+        eor x29, x16, x28, ror #63+        eor v30.16b, v30.16b, v20.16b+        eor x26, x26, x6, ror #51+        eor v29.16b, v1.16b, v6.16b+        eor x30, x23, x22, ror #50+        eor v29.16b, v29.16b, v11.16b+        eor x0, x26, x10, ror #31+        eor x29, x29, x19, ror #37+        eor v29.16b, v29.16b, v16.16b+        eor x27, x27, x12, ror #5+        eor v29.16b, v29.16b, v21.16b+        eor x30, x30, x24, ror #34+        eor x0, x0, x7, ror #27+        eor v28.16b, v2.16b, v7.16b+        eor x26, x30, x21, ror #26+        eor v28.16b, v28.16b, v12.16b+        eor x26, x26, x25, ror #15+        eor v28.16b, v28.16b, v17.16b+        ror x30, x27, #0x3e+        eor x30, x30, x26, ror #57+        eor v28.16b, v28.16b, v22.16b+        ror x26, x26, #0x3a+        eor v27.16b, v3.16b, v8.16b+        eor x16, x30, x16+        eor v27.16b, v27.16b, v13.16b+        eor x28, x30, x28, ror #63+        str x28, [sp, #0xd0]+        eor v27.16b, v27.16b, v18.16b+        eor x29, x29, x17, ror #36+        eor v27.16b, v27.16b, v23.16b+        eor x28, x1, x2, ror #61+        eor x19, x30, x19, ror #37+        eor v26.16b, v4.16b, v9.16b+        eor x29, x29, x20, ror #2+        eor v26.16b, v26.16b, v14.16b+        eor x28, x28, x4, ror #54+        eor v26.16b, v26.16b, v19.16b+        eor x26, x26, x0, ror #55+        eor x28, x28, x3, ror #39+        eor v26.16b, v26.16b, v24.16b+        eor x28, x28, x5, ror #25+        add v31.2d, v28.2d, v28.2d+        ror x0, x0, #0x38+        eor x0, x0, x29, ror #63+        sri v31.2d, v28.2d, #0x3f+        eor x27, x28, x27, ror #61+        eor v25.16b, v31.16b, v30.16b+        eor x13, x0, x13, ror #46+        add v31.2d, v26.2d, v26.2d+        eor x28, x29, x28, ror #63+        eor x29, x30, x20, ror #2+        sri v31.2d, v26.2d, #0x3f+        eor x20, x26, x3, ror #39+        eor v28.16b, v31.16b, v28.16b+        eor x11, x0, x11, ror #50+        add v31.2d, v29.2d, v29.2d+        eor x25, x28, x25, ror #9+        eor x3, x28, x21, ror #20+        sri v31.2d, v29.2d, #0x3f+        eor x21, x26, x1+        eor v26.16b, v31.16b, v26.16b+        eor x9, x27, x9, ror #49+        eor x24, x28, x24, ror #28+        add v31.2d, v27.2d, v27.2d+        eor x1, x30, x17, ror #36+        sri v31.2d, v27.2d, #0x3f+        eor x14, x0, x14, ror #8+        eor v29.16b, v31.16b, v29.16b+        eor x22, x28, x22, ror #44+        eor x8, x27, x8, ror #56+        add v31.2d, v30.2d, v30.2d+        eor x17, x27, x7, ror #19+        sri v31.2d, v30.2d, #0x3f+        eor x15, x0, x15, ror #62+        bic x7, x20, x22, ror #47+        eor v27.16b, v31.16b, v27.16b+        eor x4, x26, x4, ror #54+        eor v30.16b, v0.16b, v26.16b+        eor x0, x0, x12, ror #3+        eor v31.16b, v2.16b, v29.16b+        eor x28, x28, x23, ror #58+        eor x23, x26, x2, ror #61+        shl v0.2d, v31.2d, #0x3e+        eor x26, x26, x5, ror #25+        sri v0.2d, v31.2d, #0x2+        eor x2, x7, x16, ror #39+        eor v31.16b, v12.16b, v29.16b+        bic x7, x9, x20, ror #42+        bic x30, x15, x9, ror #16+        shl v2.2d, v31.2d, #0x2b+        eor x7, x7, x22, ror #25+        sri v2.2d, v31.2d, #0x15+        eor x12, x30, x20, ror #58+        bic x20, x22, x16, ror #56+        eor v31.16b, v13.16b, v28.16b+        eor x30, x27, x6, ror #43+        shl v12.2d, v31.2d, #0x19+        eor x22, x20, x15, ror #23+        sri v12.2d, v31.2d, #0x27+        bic x6, x19, x13, ror #42+        eor x6, x6, x17, ror #41+        eor v31.16b, v19.16b, v27.16b+        bic x5, x13, x17, ror #63+        shl v13.2d, v31.2d, #0x8+        eor x5, x21, x5, ror #21+        sri v13.2d, v31.2d, #0x38+        bic x17, x17, x21, ror #44+        eor x27, x27, x10, ror #23+        eor v31.16b, v23.16b, v28.16b+        bic x21, x21, x25, ror #50+        shl v19.2d, v31.2d, #0x38+        bic x20, x27, x4, ror #25+        bic x10, x16, x15, ror #31+        sri v19.2d, v31.2d, #0x8+        eor x16, x21, x19, ror #43+        eor v31.16b, v15.16b, v26.16b+        eor x21, x17, x25, ror #30+        shl v23.2d, v31.2d, #0x29+        bic x19, x25, x19, ror #57+        ldr x25, [sp, #0x18]+        sri v23.2d, v31.2d, #0x17+        eor x17, x10, x9, ror #47+        eor v31.16b, v1.16b, v25.16b+        ldr x9, [sp, #0x8]+        eor x15, x20, x28, ror #27+        shl v15.2d, v31.2d, #0x1+        bic x20, x4, x28, ror #2+        sri v15.2d, v31.2d, #0x3f+        eor x10, x20, x1, ror #50+        eor v31.16b, v8.16b, v28.16b+        bic x20, x11, x27, ror #60+        eor x20, x20, x4, ror #21+        shl v1.2d, v31.2d, #0x37+        bic x4, x28, x1, ror #48+        sri v1.2d, v31.2d, #0x9+        bic x1, x1, x11, ror #57+        eor v31.16b, v16.16b, v25.16b+        ldr x28, [x9, x25, lsl #3]+        ldr x9, [sp, #0xd0]+        shl v8.2d, v31.2d, #0x2d+        add x25, x25, #0x1+        sri v8.2d, v31.2d, #0x13+        str x25, [sp, #0x18]+        cmp x25, #0x17+        eor v31.16b, v7.16b, v29.16b+        eor x25, x1, x27, ror #53+        shl v16.2d, v31.2d, #0x6+        bic x27, x30, x26, ror #47+        sri v16.2d, v31.2d, #0x3a+        eor x1, x5, x28+        eor x5, x4, x11, ror #41+        eor v31.16b, v10.16b, v26.16b+        eor x11, x19, x13, ror #35+        shl v7.2d, v31.2d, #0x3+        bic x13, x26, x24, ror #10+        eor x28, x27, x24, ror #57+        sri v7.2d, v31.2d, #0x3d+        bic x27, x24, x9, ror #47+        eor v31.16b, v3.16b, v28.16b+        bic x19, x23, x3, ror #9+        shl v10.2d, v31.2d, #0x1c+        bic x4, x29, x14, ror #41+        eor x24, x19, x29, ror #44+        sri v10.2d, v31.2d, #0x24+        bic x29, x3, x29, ror #35+        eor v31.16b, v18.16b, v28.16b+        eor x13, x13, x9, ror #57+        shl v3.2d, v31.2d, #0x15+        eor x19, x29, x14, ror #12+        bic x29, x9, x0, ror #19+        sri v3.2d, v31.2d, #0x2b+        bic x14, x14, x8, ror #5+        eor v31.16b, v17.16b, v29.16b+        eor x9, x14, x23, ror #43+        eor x14, x4, x8, ror #46+        shl v18.2d, v31.2d, #0xf+        bic x23, x8, x23, ror #38+        sri v18.2d, v31.2d, #0x31+        eor x8, x27, x0, ror #2+        eor v31.16b, v11.16b, v25.16b+        eor x4, x23, x3, ror #47+        bic x3, x0, x30, ror #5+        shl v17.2d, v31.2d, #0xa+        eor x23, x3, x26, ror #52+        sri v17.2d, v31.2d, #0x36+        eor x3, x29, x30, ror #24+        eor x0, x15, x11, ror #52+        eor v31.16b, v9.16b, v27.16b+        eor x0, x0, x13, ror #48+        shl v11.2d, v31.2d, #0x14+        eor x26, x8, x9, ror #57+        sri v11.2d, v31.2d, #0x2c+        eor x27, x0, x14, ror #10+        eor x29, x16, x28, ror #63+        eor v31.16b, v22.16b, v29.16b+        eor x26, x26, x6, ror #51+        shl v9.2d, v31.2d, #0x3d+        eor x30, x23, x22, ror #50+        sri v9.2d, v31.2d, #0x3+        eor x0, x26, x10, ror #31+        eor x29, x29, x19, ror #37+        eor v31.16b, v14.16b, v27.16b+        eor x27, x27, x12, ror #5+        shl v22.2d, v31.2d, #0x27+        eor x30, x30, x24, ror #34+        eor x0, x0, x7, ror #27+        sri v22.2d, v31.2d, #0x19+        eor x26, x30, x21, ror #26+        eor v31.16b, v20.16b, v26.16b+        eor x26, x26, x25, ror #15+        shl v14.2d, v31.2d, #0x12+        ror x30, x27, #0x3e+        eor x30, x30, x26, ror #57+        sri v14.2d, v31.2d, #0x2e+        ror x26, x26, #0x3a+        eor v31.16b, v4.16b, v27.16b+        eor x16, x30, x16+        shl v20.2d, v31.2d, #0x1b+        eor x28, x30, x28, ror #63+        str x28, [sp, #0xd0]+        sri v20.2d, v31.2d, #0x25+        eor x29, x29, x17, ror #36+        eor v31.16b, v24.16b, v27.16b+        eor x28, x1, x2, ror #61+        eor x19, x30, x19, ror #37+        shl v4.2d, v31.2d, #0xe+        eor x29, x29, x20, ror #2+        sri v4.2d, v31.2d, #0x32+        eor x28, x28, x4, ror #54+        eor v31.16b, v21.16b, v25.16b+        eor x26, x26, x0, ror #55+        eor x28, x28, x3, ror #39+        shl v24.2d, v31.2d, #0x2+        eor x28, x28, x5, ror #25+        sri v24.2d, v31.2d, #0x3e+        ror x0, x0, #0x38+        eor x0, x0, x29, ror #63+        eor v31.16b, v5.16b, v26.16b+        eor x27, x28, x27, ror #61+        shl v21.2d, v31.2d, #0x24+        eor x13, x0, x13, ror #46+        sri v21.2d, v31.2d, #0x1c+        eor x28, x29, x28, ror #63+        eor x29, x30, x20, ror #2+        eor v31.16b, v6.16b, v25.16b+        eor x20, x26, x3, ror #39+        shl v27.2d, v31.2d, #0x2c+        eor x11, x0, x11, ror #50+        sri v27.2d, v31.2d, #0x14+        eor x25, x28, x25, ror #9+        eor x3, x28, x21, ror #20+        bic v31.16b, v7.16b, v11.16b+        eor x21, x26, x1+        eor v5.16b, v31.16b, v10.16b+        eor x9, x27, x9, ror #49+        eor x24, x28, x24, ror #28+        bic v31.16b, v8.16b, v7.16b+        eor x1, x30, x17, ror #36+        eor v6.16b, v31.16b, v11.16b+        eor x14, x0, x14, ror #8+        bic v31.16b, v9.16b, v8.16b+        eor x22, x28, x22, ror #44+        eor x8, x27, x8, ror #56+        eor v7.16b, v31.16b, v7.16b+        eor x17, x27, x7, ror #19+        bic v31.16b, v10.16b, v9.16b+        eor x15, x0, x15, ror #62+        bic x7, x20, x22, ror #47+        eor v8.16b, v31.16b, v8.16b+        eor x4, x26, x4, ror #54+        bic v31.16b, v11.16b, v10.16b+        eor x0, x0, x12, ror #3+        eor v9.16b, v31.16b, v9.16b+        eor x28, x28, x23, ror #58+        eor x23, x26, x2, ror #61+        bic v31.16b, v12.16b, v16.16b+        eor x26, x26, x5, ror #25+        eor v10.16b, v31.16b, v15.16b+        eor x2, x7, x16, ror #39+        bic v31.16b, v13.16b, v12.16b+        bic x7, x9, x20, ror #42+        bic x30, x15, x9, ror #16+        eor v11.16b, v31.16b, v16.16b+        eor x7, x7, x22, ror #25+        bic v31.16b, v14.16b, v13.16b+        eor x12, x30, x20, ror #58+        bic x20, x22, x16, ror #56+        eor v12.16b, v31.16b, v12.16b+        eor x30, x27, x6, ror #43+        bic v31.16b, v15.16b, v14.16b+        eor x22, x20, x15, ror #23+        eor v13.16b, v31.16b, v13.16b+        bic x6, x19, x13, ror #42+        eor x6, x6, x17, ror #41+        bic v31.16b, v16.16b, v15.16b+        bic x5, x13, x17, ror #63+        eor v14.16b, v31.16b, v14.16b+        eor x5, x21, x5, ror #21+        bic v31.16b, v17.16b, v21.16b+        bic x17, x17, x21, ror #44+        eor x27, x27, x10, ror #23+        eor v15.16b, v31.16b, v20.16b+        bic x21, x21, x25, ror #50+        bic v31.16b, v18.16b, v17.16b+        bic x20, x27, x4, ror #25+        bic x10, x16, x15, ror #31+        eor v16.16b, v31.16b, v21.16b+        eor x16, x21, x19, ror #43+        bic v31.16b, v19.16b, v18.16b+        eor x21, x17, x25, ror #30+        eor v17.16b, v31.16b, v17.16b+        bic x19, x25, x19, ror #57+        ldr x25, [sp, #0x18]+        bic v31.16b, v20.16b, v19.16b+        eor x17, x10, x9, ror #47+        eor v18.16b, v31.16b, v18.16b+        ldr x9, [sp, #0x8]+        eor x15, x20, x28, ror #27+        bic v31.16b, v21.16b, v20.16b+        bic x20, x4, x28, ror #2+        eor v19.16b, v31.16b, v19.16b+        eor x10, x20, x1, ror #50+        bic v31.16b, v22.16b, v1.16b+        bic x20, x11, x27, ror #60+        eor x20, x20, x4, ror #21+        eor v20.16b, v31.16b, v0.16b+        bic x4, x28, x1, ror #48+        bic v31.16b, v23.16b, v22.16b+        bic x1, x1, x11, ror #57+        eor v21.16b, v31.16b, v1.16b+        ldr x28, [x9, x25, lsl #3]+        ldr x9, [sp, #0xd0]+        bic v31.16b, v24.16b, v23.16b+        add x25, x25, #0x1+        eor v22.16b, v31.16b, v22.16b+        str x25, [sp, #0x18]+        cmp x25, #0x17+        bic v31.16b, v0.16b, v24.16b+        eor x25, x1, x27, ror #53+        eor v23.16b, v31.16b, v23.16b+        bic x27, x30, x26, ror #47+        bic v31.16b, v1.16b, v0.16b+        eor x1, x5, x28+        eor x5, x4, x11, ror #41+        eor v24.16b, v31.16b, v24.16b+        eor x11, x19, x13, ror #35+        bic v31.16b, v2.16b, v27.16b+        bic x13, x26, x24, ror #10+        eor x28, x27, x24, ror #57+        eor v0.16b, v31.16b, v30.16b+        bic x27, x24, x9, ror #47+        bic v31.16b, v3.16b, v2.16b+        bic x19, x23, x3, ror #9+        eor v1.16b, v31.16b, v27.16b+        bic x4, x29, x14, ror #41+        eor x24, x19, x29, ror #44+        bic v31.16b, v4.16b, v3.16b+        bic x29, x3, x29, ror #35+        eor v2.16b, v31.16b, v2.16b+        eor x13, x13, x9, ror #57+        bic v31.16b, v30.16b, v4.16b+        eor x19, x29, x14, ror #12+        bic x29, x9, x0, ror #19+        eor v3.16b, v31.16b, v3.16b+        bic x14, x14, x8, ror #5+        bic v31.16b, v27.16b, v30.16b+        eor x9, x14, x23, ror #43+        eor x14, x4, x8, ror #46+        eor v4.16b, v31.16b, v4.16b+        bic x23, x8, x23, ror #38+        eor x8, x27, x0, ror #2+        eor x4, x23, x3, ror #47+        bic x3, x0, x30, ror #5+        eor x23, x3, x26, ror #52+        eor x3, x29, x30, ror #24+        ldr x30, [sp, #0x10]+        ld1r { v28.2d }, [x30], #8+        str x30, [sp, #0x10]+        eor v0.16b, v0.16b, v28.16b++Lmld_keccak_f1600_x4_v8a_scalar_hybrid_loop_end:+        b.le Lmld_keccak_f1600_x4_v8a_scalar_hybrid_loop+        ror x2, x2, #0x3d+        ror x3, x3, #0x27+        ror x4, x4, #0x36+        ror x5, x5, #0x19+        ror x6, x6, #0x2b+        ror x7, x7, #0x13+        ror x8, x8, #0x38+        ror x9, x9, #0x31+        ror x10, x10, #0x17+        ror x11, x11, #0x32+        ror x12, x12, #0x3+        ror x13, x13, #0x2e+        ror x14, x14, #0x8+        ror x15, x15, #0x3e+        ror x17, x17, #0x24+        ror x28, x28, #0x3f+        ror x19, x19, #0x25+        ror x20, x20, #0x2+        ror x21, x21, #0x14+        ror x22, x22, #0x2c+        ror x23, x23, #0x3a+        ror x24, x24, #0x1c+        ror x25, x25, #0x9+        ldr x30, [sp, #0x20]+        cmp x30, #0x1+        b.eq Lmld_keccak_f1600_x4_v8a_scalar_hybrid_done+        mov x30, #0x1               // =1+        str x30, [sp, #0x20]+        ldr x0, [sp]+        add x0, x0, #0x190+        stp x1, x6, [x0]+        stp x11, x16, [x0, #0x10]+        stp x21, x2, [x0, #0x20]+        stp x7, x12, [x0, #0x30]+        stp x17, x22, [x0, #0x40]+        stp x3, x8, [x0, #0x50]+        stp x13, x28, [x0, #0x60]+        stp x23, x4, [x0, #0x70]+        stp x9, x14, [x0, #0x80]+        stp x19, x24, [x0, #0x90]+        stp x5, x10, [x0, #0xa0]+        stp x15, x20, [x0, #0xb0]+        str x25, [x0, #0xc0]+        sub x0, x0, #0x190+        add x0, x0, #0x258+        ldp x1, x6, [x0]+        ldp x11, x16, [x0, #0x10]+        ldp x21, x2, [x0, #0x20]+        ldp x7, x12, [x0, #0x30]+        ldp x17, x22, [x0, #0x40]+        ldp x3, x8, [x0, #0x50]+        ldp x13, x28, [x0, #0x60]+        ldp x23, x4, [x0, #0x70]+        ldp x9, x14, [x0, #0x80]+        ldp x19, x24, [x0, #0x90]+        ldp x5, x10, [x0, #0xa0]+        ldp x15, x20, [x0, #0xb0]+        ldr x25, [x0, #0xc0]+        sub x0, x0, #0x258+        b Lmld_keccak_f1600_x4_v8a_scalar_hybrid_initial++Lmld_keccak_f1600_x4_v8a_scalar_hybrid_done:+        ldr x0, [sp]+        add x0, x0, #0x258+        stp x1, x6, [x0]+        stp x11, x16, [x0, #0x10]+        stp x21, x2, [x0, #0x20]+        stp x7, x12, [x0, #0x30]+        stp x17, x22, [x0, #0x40]+        stp x3, x8, [x0, #0x50]+        stp x13, x28, [x0, #0x60]+        stp x23, x4, [x0, #0x70]+        stp x9, x14, [x0, #0x80]+        stp x19, x24, [x0, #0x90]+        stp x5, x10, [x0, #0xa0]+        stp x15, x20, [x0, #0xb0]+        str x25, [x0, #0xc0]+        sub x0, x0, #0x258+        add x4, x0, #0xc8+        trn1 v25.2d, v0.2d, v1.2d+        trn1 v26.2d, v2.2d, v3.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v0.2d, v1.2d+        trn2 v28.2d, v2.2d, v3.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v4.2d, v5.2d+        trn1 v26.2d, v6.2d, v7.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v4.2d, v5.2d+        trn2 v28.2d, v6.2d, v7.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v8.2d, v9.2d+        trn1 v26.2d, v10.2d, v11.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v8.2d, v9.2d+        trn2 v28.2d, v10.2d, v11.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v12.2d, v13.2d+        trn1 v26.2d, v14.2d, v15.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v12.2d, v13.2d+        trn2 v28.2d, v14.2d, v15.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v16.2d, v17.2d+        trn1 v26.2d, v18.2d, v19.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v16.2d, v17.2d+        trn2 v28.2d, v18.2d, v19.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v20.2d, v21.2d+        trn1 v26.2d, v22.2d, v23.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v20.2d, v21.2d+        trn2 v28.2d, v22.2d, v23.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        str d24, [x0]+        trn2 v25.2d, v24.2d, v24.2d+        str d25, [x4]+        ldp d8, d9, [sp, #0x90]+        .cfi_restore d8+        .cfi_restore d9+        ldp d10, d11, [sp, #0xa0]+        .cfi_restore d10+        .cfi_restore d11+        ldp d12, d13, [sp, #0xb0]+        .cfi_restore d12+        .cfi_restore d13+        ldp d14, d15, [sp, #0xc0]+        .cfi_restore d14+        .cfi_restore d15+        ldp x19, x20, [sp, #0x30]+        .cfi_restore x19+        .cfi_restore x20+        ldp x21, x22, [sp, #0x40]+        .cfi_restore x21+        .cfi_restore x22+        ldp x23, x24, [sp, #0x50]+        .cfi_restore x23+        .cfi_restore x24+        ldp x25, x26, [sp, #0x60]+        .cfi_restore x25+        .cfi_restore x26+        ldp x27, x28, [sp, #0x70]+        .cfi_restore x27+        .cfi_restore x28+        ldp x29, x30, [sp, #0x80]+        .cfi_restore x29+        .cfi_restore x30+        add sp, sp, #0xe0+        .cfi_adjust_cfa_offset -0xe0+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm)++#endif /* MLD_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID && \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.S view
@@ -0,0 +1,990 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * Copyright (c) 2021-2022 Arm Limited+ * Copyright (c) 2022 Matthias Kannwischer+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++// Author: Hanno Becker <hannobecker@posteo.de>+// Author: Matthias Kannwischer <matthias@kannwischer.eu>++/*yaml+  Name: keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm+  Description: AArch64 hybrid scalar/vector implementation of Keccak-f[1600] permutation for four sequential states with ARMv8.4-A optimizations+  Signature: void mld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm(uint64_t state[100], const uint64_t rc[24])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON, SHA3]+    x0:+      type: buffer+      size_bytes: 800+      permissions: read/write+      c_parameter: uint64_t state[100]+      description: Four sequential Keccak states (state0[25], state1[25], state2[25], state3[25])+    x1:+      type: buffer+      size_bytes: 192+      permissions: read-only+      c_parameter: uint64_t const *rc+      description: Round constants (24 x uint64_t)+  Stack:+    bytes: 224+    description: register preservation and temporary storage+*/++#include "../../../../common.h"+#if defined(MLD_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++#if defined(__ARM_FEATURE_SHA3)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/fips202/aarch64/src/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm)+MLD_ASM_FN_SYMBOL(keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0xe0+        .cfi_adjust_cfa_offset 0xe0+        stp x19, x20, [sp, #0x30]+        .cfi_rel_offset x19, 0x30+        .cfi_rel_offset x20, 0x38+        stp x21, x22, [sp, #0x40]+        .cfi_rel_offset x21, 0x40+        .cfi_rel_offset x22, 0x48+        stp x23, x24, [sp, #0x50]+        .cfi_rel_offset x23, 0x50+        .cfi_rel_offset x24, 0x58+        stp x25, x26, [sp, #0x60]+        .cfi_rel_offset x25, 0x60+        .cfi_rel_offset x26, 0x68+        stp x27, x28, [sp, #0x70]+        .cfi_rel_offset x27, 0x70+        .cfi_rel_offset x28, 0x78+        stp x29, x30, [sp, #0x80]+        .cfi_rel_offset x29, 0x80+        .cfi_rel_offset x30, 0x88+        stp d8, d9, [sp, #0x90]+        .cfi_rel_offset d8, 0x90+        .cfi_rel_offset d9, 0x98+        stp d10, d11, [sp, #0xa0]+        .cfi_rel_offset d10, 0xa0+        .cfi_rel_offset d11, 0xa8+        stp d12, d13, [sp, #0xb0]+        .cfi_rel_offset d12, 0xb0+        .cfi_rel_offset d13, 0xb8+        stp d14, d15, [sp, #0xc0]+        .cfi_rel_offset d14, 0xc0+        .cfi_rel_offset d15, 0xc8+        mov x29, x1+        mov x30, #0x0               // =0+        str x30, [sp, #0x20]+        str x29, [sp, #0x8]+        str x29, [sp, #0x10]+        str x0, [sp]+        add x4, x0, #0xc8+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v0.2d, v25.2d, v27.2d+        trn2 v1.2d, v25.2d, v27.2d+        trn1 v2.2d, v26.2d, v28.2d+        trn2 v3.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v4.2d, v25.2d, v27.2d+        trn2 v5.2d, v25.2d, v27.2d+        trn1 v6.2d, v26.2d, v28.2d+        trn2 v7.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v8.2d, v25.2d, v27.2d+        trn2 v9.2d, v25.2d, v27.2d+        trn1 v10.2d, v26.2d, v28.2d+        trn2 v11.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v12.2d, v25.2d, v27.2d+        trn2 v13.2d, v25.2d, v27.2d+        trn1 v14.2d, v26.2d, v28.2d+        trn2 v15.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v16.2d, v25.2d, v27.2d+        trn2 v17.2d, v25.2d, v27.2d+        trn1 v18.2d, v26.2d, v28.2d+        trn2 v19.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v20.2d, v25.2d, v27.2d+        trn2 v21.2d, v25.2d, v27.2d+        trn1 v22.2d, v26.2d, v28.2d+        trn2 v23.2d, v26.2d, v28.2d+        ldr d25, [x0]+        ldr d27, [x4]+        trn1 v24.2d, v25.2d, v27.2d+        sub x0, x0, #0xc0+        add x0, x0, #0x190+        ldp x1, x6, [x0]+        ldp x11, x16, [x0, #0x10]+        ldp x21, x2, [x0, #0x20]+        ldp x7, x12, [x0, #0x30]+        ldp x17, x22, [x0, #0x40]+        ldp x3, x8, [x0, #0x50]+        ldp x13, x28, [x0, #0x60]+        ldp x23, x4, [x0, #0x70]+        ldp x9, x14, [x0, #0x80]+        ldp x19, x24, [x0, #0x90]+        ldp x5, x10, [x0, #0xa0]+        ldp x15, x20, [x0, #0xb0]+        ldr x25, [x0, #0xc0]+        sub x0, x0, #0x190++Lmld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_initial:+        eor x30, x24, x25+        eor x27, x9, x10+        eor3 v30.16b, v0.16b, v5.16b, v10.16b+        eor v30.16b, v30.16b, v15.16b+        eor x0, x30, x21+        eor x26, x27, x6+        eor v30.16b, v30.16b, v20.16b+        eor x27, x26, x7+        eor x29, x0, x22+        eor3 v29.16b, v1.16b, v6.16b, v11.16b+        eor x26, x29, x23+        eor x29, x4, x5+        eor v29.16b, v29.16b, v16.16b+        eor x30, x29, x1+        eor x0, x27, x8+        eor v29.16b, v29.16b, v21.16b+        eor x29, x30, x2+        eor x30, x19, x20+        eor3 v28.16b, v2.16b, v7.16b, v12.16b+        eor x30, x30, x16+        eor x27, x26, x0, ror #63+        eor v28.16b, v28.16b, v17.16b+        eor x4, x4, x27+        eor x30, x30, x17+        eor v28.16b, v28.16b, v22.16b+        eor x30, x30, x28+        eor x29, x29, x3+        eor3 v27.16b, v3.16b, v8.16b, v13.16b+        eor x0, x0, x30, ror #63+        eor x30, x30, x29, ror #63+        eor v27.16b, v27.16b, v18.16b+        eor x22, x22, x30+        eor v27.16b, v27.16b, v23.16b+        eor x23, x23, x30+        str x23, [sp, #0xd0]+        eor3 v26.16b, v4.16b, v9.16b, v14.16b+        eor x23, x14, x15+        eor x14, x14, x0+        eor v26.16b, v26.16b, v19.16b+        eor x23, x23, x11+        eor x15, x15, x0+        eor v26.16b, v26.16b, v24.16b+        eor x1, x1, x27+        eor x23, x23, x12+        rax1 v25.2d, v30.2d, v28.2d+        eor x23, x23, x13+        eor x11, x11, x0+        add v31.2d, v26.2d, v26.2d+        eor x29, x29, x23, ror #63+        eor x23, x23, x26, ror #63+        sri v31.2d, v26.2d, #0x3f+        eor x26, x13, x0+        eor x13, x28, x23+        eor v28.16b, v31.16b, v28.16b+        eor x28, x24, x30+        eor x24, x16, x23+        rax1 v26.2d, v26.2d, v29.2d+        eor x16, x21, x30+        eor x21, x25, x30+        add v31.2d, v27.2d, v27.2d+        eor x30, x19, x23+        sri v31.2d, v27.2d, #0x3f+        eor x19, x20, x23+        eor x20, x17, x23+        eor v29.16b, v31.16b, v29.16b+        eor x17, x12, x0+        eor x0, x2, x27+        rax1 v27.2d, v27.2d, v30.2d+        eor x2, x6, x29+        eor x6, x8, x29+        eor v30.16b, v0.16b, v26.16b+        bic x8, x28, x13, ror #47+        eor x12, x3, x27+        eor v31.16b, v2.16b, v29.16b+        bic x3, x13, x17, ror #19+        eor x5, x5, x27+        shl v0.2d, v31.2d, #0x3e+        ldr x27, [sp, #0xd0]+        bic x25, x17, x2, ror #5+        sri v0.2d, v31.2d, #0x2+        eor x9, x9, x29+        eor x23, x25, x5, ror #52+        xar v2.2d, v12.2d, v29.2d, #0x15+        eor x3, x3, x2, ror #24+        eor x8, x8, x17, ror #2+        eor v31.16b, v13.16b, v28.16b+        eor x17, x10, x29+        bic x25, x12, x22, ror #47+        shl v12.2d, v31.2d, #0x19+        eor x29, x7, x29+        bic x10, x4, x27, ror #2+        sri v12.2d, v31.2d, #0x27+        bic x7, x5, x28, ror #10+        xar v13.2d, v19.2d, v27.2d, #0x38+        eor x10, x10, x20, ror #50+        eor x13, x7, x13, ror #57+        eor v31.16b, v23.16b, v28.16b+        bic x7, x2, x5, ror #47+        eor x2, x25, x24, ror #39+        shl v19.2d, v31.2d, #0x38+        bic x25, x20, x11, ror #57+        bic x5, x17, x4, ror #25+        sri v19.2d, v31.2d, #0x8+        eor x25, x25, x17, ror #53+        bic x17, x11, x17, ror #60+        xar v23.2d, v15.2d, v26.2d, #0x17+        eor x28, x7, x28, ror #57+        bic x7, x9, x12, ror #42+        eor v31.16b, v1.16b, v25.16b+        eor x7, x7, x22, ror #25+        bic x22, x22, x24, ror #56+        shl v15.2d, v31.2d, #0x1+        bic x24, x24, x15, ror #31+        eor x22, x22, x15, ror #23+        sri v15.2d, v31.2d, #0x3f+        bic x20, x27, x20, ror #48+        bic x15, x15, x9, ror #16+        xar v1.2d, v8.2d, v28.2d, #0x9+        eor x12, x15, x12, ror #58+        eor x15, x5, x27, ror #27+        eor v31.16b, v16.16b, v25.16b+        eor x5, x20, x11, ror #41+        shl v8.2d, v31.2d, #0x2d+        ldr x11, [sp, #0x8]+        eor x20, x17, x4, ror #21+        sri v8.2d, v31.2d, #0x13+        eor x17, x24, x9, ror #47+        mov x24, #0x1               // =1+        xar v16.2d, v7.2d, v29.2d, #0x3a+        bic x9, x0, x16, ror #9+        str x24, [sp, #0x18]+        eor v31.16b, v10.16b, v26.16b+        bic x24, x29, x1, ror #44+        bic x27, x1, x21, ror #50+        shl v7.2d, v31.2d, #0x3+        bic x4, x26, x29, ror #63+        eor x1, x1, x4, ror #21+        sri v7.2d, v31.2d, #0x3d+        ldr x11, [x11]+        bic x4, x21, x30, ror #57+        xar v10.2d, v3.2d, v28.2d, #0x24+        eor x21, x24, x21, ror #30+        eor x24, x9, x19, ror #44+        eor v31.16b, v18.16b, v28.16b+        bic x9, x14, x6, ror #5+        eor x9, x9, x0, ror #43+        shl v3.2d, v31.2d, #0x15+        bic x0, x6, x0, ror #38+        eor x1, x1, x11+        sri v3.2d, v31.2d, #0x2b+        eor x11, x4, x26, ror #35+        eor x4, x0, x16, ror #47+        xar v18.2d, v17.2d, v29.2d, #0x31+        bic x0, x16, x19, ror #35+        eor v31.16b, v11.16b, v25.16b+        eor x16, x27, x30, ror #43+        bic x27, x30, x26, ror #42+        shl v17.2d, v31.2d, #0xa+        bic x26, x19, x14, ror #41+        eor x19, x0, x14, ror #12+        sri v17.2d, v31.2d, #0x36+        eor x14, x26, x6, ror #46+        eor x6, x27, x29, ror #41+        xar v11.2d, v9.2d, v27.2d, #0x2c+        eor x0, x15, x11, ror #52+        eor x0, x0, x13, ror #48+        eor v31.16b, v22.16b, v29.16b+        eor x26, x8, x9, ror #57+        eor x27, x0, x14, ror #10+        shl v9.2d, v31.2d, #0x3d+        eor x29, x16, x28, ror #63+        eor x26, x26, x6, ror #51+        sri v9.2d, v31.2d, #0x3+        eor x30, x23, x22, ror #50+        eor x0, x26, x10, ror #31+        xar v22.2d, v14.2d, v27.2d, #0x19+        eor x29, x29, x19, ror #37+        eor x27, x27, x12, ror #5+        eor v31.16b, v20.16b, v26.16b+        eor x30, x30, x24, ror #34+        eor x0, x0, x7, ror #27+        shl v14.2d, v31.2d, #0x12+        eor x26, x30, x21, ror #26+        sri v14.2d, v31.2d, #0x2e+        eor x26, x26, x25, ror #15+        ror x30, x27, #0x3e+        xar v20.2d, v4.2d, v27.2d, #0x25+        eor x30, x30, x26, ror #57+        ror x26, x26, #0x3a+        eor v31.16b, v24.16b, v27.16b+        eor x16, x30, x16+        eor x28, x30, x28, ror #63+        shl v4.2d, v31.2d, #0xe+        str x28, [sp, #0xd0]+        eor x29, x29, x17, ror #36+        sri v4.2d, v31.2d, #0x32+        eor x28, x1, x2, ror #61+        eor x19, x30, x19, ror #37+        xar v24.2d, v21.2d, v25.2d, #0x3e+        eor x29, x29, x20, ror #2+        eor x28, x28, x4, ror #54+        eor v31.16b, v5.16b, v26.16b+        eor x26, x26, x0, ror #55+        eor x28, x28, x3, ror #39+        shl v21.2d, v31.2d, #0x24+        eor x28, x28, x5, ror #25+        ror x0, x0, #0x38+        sri v21.2d, v31.2d, #0x1c+        eor x0, x0, x29, ror #63+        eor x27, x28, x27, ror #61+        xar v27.2d, v6.2d, v25.2d, #0x14+        eor x13, x0, x13, ror #46+        eor x28, x29, x28, ror #63+        bic v31.16b, v7.16b, v11.16b+        eor x29, x30, x20, ror #2+        eor v5.16b, v31.16b, v10.16b+        eor x20, x26, x3, ror #39+        eor x11, x0, x11, ror #50+        bcax v6.16b, v11.16b, v8.16b, v7.16b+        eor x25, x28, x25, ror #9+        eor x3, x28, x21, ror #20+        bic v31.16b, v9.16b, v8.16b+        eor x21, x26, x1+        eor x9, x27, x9, ror #49+        eor v7.16b, v31.16b, v7.16b+        eor x24, x28, x24, ror #28+        eor x1, x30, x17, ror #36+        bcax v8.16b, v8.16b, v10.16b, v9.16b+        eor x14, x0, x14, ror #8+        eor x22, x28, x22, ror #44+        bic v31.16b, v11.16b, v10.16b+        eor x8, x27, x8, ror #56+        eor x17, x27, x7, ror #19+        eor v9.16b, v31.16b, v9.16b+        eor x15, x0, x15, ror #62+        bic x7, x20, x22, ror #47+        bcax v10.16b, v15.16b, v12.16b, v16.16b+        eor x4, x26, x4, ror #54+        eor x0, x0, x12, ror #3+        bic v31.16b, v13.16b, v12.16b+        eor x28, x28, x23, ror #58+        eor x23, x26, x2, ror #61+        eor v11.16b, v31.16b, v16.16b+        eor x26, x26, x5, ror #25+        bcax v12.16b, v12.16b, v14.16b, v13.16b+        eor x2, x7, x16, ror #39+        bic x7, x9, x20, ror #42+        bic v31.16b, v15.16b, v14.16b+        bic x30, x15, x9, ror #16+        eor x7, x7, x22, ror #25+        eor v13.16b, v31.16b, v13.16b+        eor x12, x30, x20, ror #58+        bic x20, x22, x16, ror #56+        bic v31.16b, v16.16b, v15.16b+        eor x30, x27, x6, ror #43+        eor x22, x20, x15, ror #23+        eor v14.16b, v31.16b, v14.16b+        bic x6, x19, x13, ror #42+        eor x6, x6, x17, ror #41+        bcax v15.16b, v20.16b, v17.16b, v21.16b+        bic x5, x13, x17, ror #63+        eor x5, x21, x5, ror #21+        bic v31.16b, v18.16b, v17.16b+        bic x17, x17, x21, ror #44+        eor x27, x27, x10, ror #23+        eor v16.16b, v31.16b, v21.16b+        bic x21, x21, x25, ror #50+        bic x20, x27, x4, ror #25+        bcax v17.16b, v17.16b, v19.16b, v18.16b+        bic x10, x16, x15, ror #31+        eor x16, x21, x19, ror #43+        bic v31.16b, v20.16b, v19.16b+        eor x21, x17, x25, ror #30+        bic x19, x25, x19, ror #57+        eor v18.16b, v31.16b, v18.16b+        ldr x25, [sp, #0x18]+        bcax v19.16b, v19.16b, v21.16b, v20.16b+        eor x17, x10, x9, ror #47+        ldr x9, [sp, #0x8]+        bic v31.16b, v22.16b, v1.16b+        eor x15, x20, x28, ror #27+        bic x20, x4, x28, ror #2+        eor v20.16b, v31.16b, v0.16b+        eor x10, x20, x1, ror #50+        bic x20, x11, x27, ror #60+        bcax v21.16b, v1.16b, v23.16b, v22.16b+        eor x20, x20, x4, ror #21+        bic x4, x28, x1, ror #48+        bic v31.16b, v24.16b, v23.16b+        bic x1, x1, x11, ror #57+        ldr x28, [x9, x25, lsl #3]+        eor v22.16b, v31.16b, v22.16b+        ldr x9, [sp, #0xd0]+        add x25, x25, #0x1+        bcax v23.16b, v23.16b, v0.16b, v24.16b+        str x25, [sp, #0x18]+        cmp x25, #0x17+        bic v31.16b, v1.16b, v0.16b+        eor x25, x1, x27, ror #53+        bic x27, x30, x26, ror #47+        eor v24.16b, v31.16b, v24.16b+        eor x1, x5, x28+        eor x5, x4, x11, ror #41+        bcax v0.16b, v30.16b, v2.16b, v27.16b+        eor x11, x19, x13, ror #35+        bic v31.16b, v3.16b, v2.16b+        bic x13, x26, x24, ror #10+        eor x28, x27, x24, ror #57+        eor v1.16b, v31.16b, v27.16b+        bic x27, x24, x9, ror #47+        bic x19, x23, x3, ror #9+        bcax v2.16b, v2.16b, v4.16b, v3.16b+        bic x4, x29, x14, ror #41+        eor x24, x19, x29, ror #44+        bic v31.16b, v30.16b, v4.16b+        bic x29, x3, x29, ror #35+        eor x13, x13, x9, ror #57+        eor v3.16b, v31.16b, v3.16b+        eor x19, x29, x14, ror #12+        bic x29, x9, x0, ror #19+        bcax v4.16b, v4.16b, v27.16b, v30.16b+        bic x14, x14, x8, ror #5+        eor x9, x14, x23, ror #43+        eor x14, x4, x8, ror #46+        bic x23, x8, x23, ror #38+        eor x8, x27, x0, ror #2+        eor x4, x23, x3, ror #47+        bic x3, x0, x30, ror #5+        eor x23, x3, x26, ror #52+        eor x3, x29, x30, ror #24+        ldr x30, [sp, #0x10]+        ld1r { v28.2d }, [x30], #8+        str x30, [sp, #0x10]+        eor v0.16b, v0.16b, v28.16b++Lmld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_loop:+        eor x0, x15, x11, ror #52+        eor x0, x0, x13, ror #48+        eor3 v30.16b, v0.16b, v5.16b, v10.16b+        eor v30.16b, v30.16b, v15.16b+        eor x26, x8, x9, ror #57+        eor x27, x0, x14, ror #10+        eor v30.16b, v30.16b, v20.16b+        eor x29, x16, x28, ror #63+        eor x26, x26, x6, ror #51+        eor3 v29.16b, v1.16b, v6.16b, v11.16b+        eor x30, x23, x22, ror #50+        eor x0, x26, x10, ror #31+        eor v29.16b, v29.16b, v16.16b+        eor x29, x29, x19, ror #37+        eor x27, x27, x12, ror #5+        eor v29.16b, v29.16b, v21.16b+        eor x30, x30, x24, ror #34+        eor x0, x0, x7, ror #27+        eor3 v28.16b, v2.16b, v7.16b, v12.16b+        eor x26, x30, x21, ror #26+        eor x26, x26, x25, ror #15+        eor v28.16b, v28.16b, v17.16b+        ror x30, x27, #0x3e+        eor x30, x30, x26, ror #57+        eor v28.16b, v28.16b, v22.16b+        ror x26, x26, #0x3a+        eor x16, x30, x16+        eor3 v27.16b, v3.16b, v8.16b, v13.16b+        eor x28, x30, x28, ror #63+        str x28, [sp, #0xd0]+        eor v27.16b, v27.16b, v18.16b+        eor x29, x29, x17, ror #36+        eor x28, x1, x2, ror #61+        eor v27.16b, v27.16b, v23.16b+        eor x19, x30, x19, ror #37+        eor x29, x29, x20, ror #2+        eor3 v26.16b, v4.16b, v9.16b, v14.16b+        eor x28, x28, x4, ror #54+        eor x26, x26, x0, ror #55+        eor v26.16b, v26.16b, v19.16b+        eor x28, x28, x3, ror #39+        eor x28, x28, x5, ror #25+        eor v26.16b, v26.16b, v24.16b+        ror x0, x0, #0x38+        eor x0, x0, x29, ror #63+        rax1 v25.2d, v30.2d, v28.2d+        eor x27, x28, x27, ror #61+        eor x13, x0, x13, ror #46+        add v31.2d, v26.2d, v26.2d+        eor x28, x29, x28, ror #63+        eor x29, x30, x20, ror #2+        sri v31.2d, v26.2d, #0x3f+        eor x20, x26, x3, ror #39+        eor x11, x0, x11, ror #50+        eor v28.16b, v31.16b, v28.16b+        eor x25, x28, x25, ror #9+        eor x3, x28, x21, ror #20+        rax1 v26.2d, v26.2d, v29.2d+        eor x21, x26, x1+        add v31.2d, v27.2d, v27.2d+        eor x9, x27, x9, ror #49+        eor x24, x28, x24, ror #28+        sri v31.2d, v27.2d, #0x3f+        eor x1, x30, x17, ror #36+        eor x14, x0, x14, ror #8+        eor v29.16b, v31.16b, v29.16b+        eor x22, x28, x22, ror #44+        eor x8, x27, x8, ror #56+        rax1 v27.2d, v27.2d, v30.2d+        eor x17, x27, x7, ror #19+        eor x15, x0, x15, ror #62+        eor v30.16b, v0.16b, v26.16b+        bic x7, x20, x22, ror #47+        eor x4, x26, x4, ror #54+        eor v31.16b, v2.16b, v29.16b+        eor x0, x0, x12, ror #3+        eor x28, x28, x23, ror #58+        shl v0.2d, v31.2d, #0x3e+        eor x23, x26, x2, ror #61+        eor x26, x26, x5, ror #25+        sri v0.2d, v31.2d, #0x2+        eor x2, x7, x16, ror #39+        bic x7, x9, x20, ror #42+        xar v2.2d, v12.2d, v29.2d, #0x15+        bic x30, x15, x9, ror #16+        eor x7, x7, x22, ror #25+        eor v31.16b, v13.16b, v28.16b+        eor x12, x30, x20, ror #58+        bic x20, x22, x16, ror #56+        shl v12.2d, v31.2d, #0x19+        eor x30, x27, x6, ror #43+        eor x22, x20, x15, ror #23+        sri v12.2d, v31.2d, #0x27+        bic x6, x19, x13, ror #42+        eor x6, x6, x17, ror #41+        xar v13.2d, v19.2d, v27.2d, #0x38+        bic x5, x13, x17, ror #63+        eor x5, x21, x5, ror #21+        eor v31.16b, v23.16b, v28.16b+        bic x17, x17, x21, ror #44+        eor x27, x27, x10, ror #23+        shl v19.2d, v31.2d, #0x38+        bic x21, x21, x25, ror #50+        bic x20, x27, x4, ror #25+        sri v19.2d, v31.2d, #0x8+        bic x10, x16, x15, ror #31+        eor x16, x21, x19, ror #43+        xar v23.2d, v15.2d, v26.2d, #0x17+        eor x21, x17, x25, ror #30+        bic x19, x25, x19, ror #57+        eor v31.16b, v1.16b, v25.16b+        ldr x25, [sp, #0x18]+        eor x17, x10, x9, ror #47+        shl v15.2d, v31.2d, #0x1+        ldr x9, [sp, #0x8]+        sri v15.2d, v31.2d, #0x3f+        eor x15, x20, x28, ror #27+        bic x20, x4, x28, ror #2+        xar v1.2d, v8.2d, v28.2d, #0x9+        eor x10, x20, x1, ror #50+        bic x20, x11, x27, ror #60+        eor v31.16b, v16.16b, v25.16b+        eor x20, x20, x4, ror #21+        bic x4, x28, x1, ror #48+        shl v8.2d, v31.2d, #0x2d+        bic x1, x1, x11, ror #57+        ldr x28, [x9, x25, lsl #3]+        sri v8.2d, v31.2d, #0x13+        ldr x9, [sp, #0xd0]+        add x25, x25, #0x1+        xar v16.2d, v7.2d, v29.2d, #0x3a+        str x25, [sp, #0x18]+        cmp x25, #0x17+        eor v31.16b, v10.16b, v26.16b+        eor x25, x1, x27, ror #53+        bic x27, x30, x26, ror #47+        shl v7.2d, v31.2d, #0x3+        eor x1, x5, x28+        eor x5, x4, x11, ror #41+        sri v7.2d, v31.2d, #0x3d+        eor x11, x19, x13, ror #35+        bic x13, x26, x24, ror #10+        xar v10.2d, v3.2d, v28.2d, #0x24+        eor x28, x27, x24, ror #57+        bic x27, x24, x9, ror #47+        eor v31.16b, v18.16b, v28.16b+        bic x19, x23, x3, ror #9+        bic x4, x29, x14, ror #41+        shl v3.2d, v31.2d, #0x15+        eor x24, x19, x29, ror #44+        bic x29, x3, x29, ror #35+        sri v3.2d, v31.2d, #0x2b+        eor x13, x13, x9, ror #57+        eor x19, x29, x14, ror #12+        xar v18.2d, v17.2d, v29.2d, #0x31+        bic x29, x9, x0, ror #19+        bic x14, x14, x8, ror #5+        eor v31.16b, v11.16b, v25.16b+        eor x9, x14, x23, ror #43+        eor x14, x4, x8, ror #46+        shl v17.2d, v31.2d, #0xa+        bic x23, x8, x23, ror #38+        eor x8, x27, x0, ror #2+        sri v17.2d, v31.2d, #0x36+        eor x4, x23, x3, ror #47+        bic x3, x0, x30, ror #5+        xar v11.2d, v9.2d, v27.2d, #0x2c+        eor x23, x3, x26, ror #52+        eor x3, x29, x30, ror #24+        eor v31.16b, v22.16b, v29.16b+        eor x0, x15, x11, ror #52+        shl v9.2d, v31.2d, #0x3d+        eor x0, x0, x13, ror #48+        eor x26, x8, x9, ror #57+        sri v9.2d, v31.2d, #0x3+        eor x27, x0, x14, ror #10+        eor x29, x16, x28, ror #63+        xar v22.2d, v14.2d, v27.2d, #0x19+        eor x26, x26, x6, ror #51+        eor x30, x23, x22, ror #50+        eor v31.16b, v20.16b, v26.16b+        eor x0, x26, x10, ror #31+        eor x29, x29, x19, ror #37+        shl v14.2d, v31.2d, #0x12+        eor x27, x27, x12, ror #5+        eor x30, x30, x24, ror #34+        sri v14.2d, v31.2d, #0x2e+        eor x0, x0, x7, ror #27+        eor x26, x30, x21, ror #26+        xar v20.2d, v4.2d, v27.2d, #0x25+        eor x26, x26, x25, ror #15+        ror x30, x27, #0x3e+        eor v31.16b, v24.16b, v27.16b+        eor x30, x30, x26, ror #57+        ror x26, x26, #0x3a+        shl v4.2d, v31.2d, #0xe+        eor x16, x30, x16+        eor x28, x30, x28, ror #63+        sri v4.2d, v31.2d, #0x32+        str x28, [sp, #0xd0]+        eor x29, x29, x17, ror #36+        xar v24.2d, v21.2d, v25.2d, #0x3e+        eor x28, x1, x2, ror #61+        eor x19, x30, x19, ror #37+        eor v31.16b, v5.16b, v26.16b+        eor x29, x29, x20, ror #2+        eor x28, x28, x4, ror #54+        shl v21.2d, v31.2d, #0x24+        eor x26, x26, x0, ror #55+        eor x28, x28, x3, ror #39+        sri v21.2d, v31.2d, #0x1c+        eor x28, x28, x5, ror #25+        ror x0, x0, #0x38+        xar v27.2d, v6.2d, v25.2d, #0x14+        eor x0, x0, x29, ror #63+        eor x27, x28, x27, ror #61+        bic v31.16b, v7.16b, v11.16b+        eor x13, x0, x13, ror #46+        eor x28, x29, x28, ror #63+        eor v5.16b, v31.16b, v10.16b+        eor x29, x30, x20, ror #2+        eor x20, x26, x3, ror #39+        bcax v6.16b, v11.16b, v8.16b, v7.16b+        eor x11, x0, x11, ror #50+        eor x25, x28, x25, ror #9+        bic v31.16b, v9.16b, v8.16b+        eor x3, x28, x21, ror #20+        eor v7.16b, v31.16b, v7.16b+        eor x21, x26, x1+        eor x9, x27, x9, ror #49+        bcax v8.16b, v8.16b, v10.16b, v9.16b+        eor x24, x28, x24, ror #28+        eor x1, x30, x17, ror #36+        bic v31.16b, v11.16b, v10.16b+        eor x14, x0, x14, ror #8+        eor x22, x28, x22, ror #44+        eor v9.16b, v31.16b, v9.16b+        eor x8, x27, x8, ror #56+        eor x17, x27, x7, ror #19+        bcax v10.16b, v15.16b, v12.16b, v16.16b+        eor x15, x0, x15, ror #62+        bic x7, x20, x22, ror #47+        bic v31.16b, v13.16b, v12.16b+        eor x4, x26, x4, ror #54+        eor x0, x0, x12, ror #3+        eor v11.16b, v31.16b, v16.16b+        eor x28, x28, x23, ror #58+        eor x23, x26, x2, ror #61+        bcax v12.16b, v12.16b, v14.16b, v13.16b+        eor x26, x26, x5, ror #25+        eor x2, x7, x16, ror #39+        bic v31.16b, v15.16b, v14.16b+        bic x7, x9, x20, ror #42+        bic x30, x15, x9, ror #16+        eor v13.16b, v31.16b, v13.16b+        eor x7, x7, x22, ror #25+        eor x12, x30, x20, ror #58+        bic v31.16b, v16.16b, v15.16b+        bic x20, x22, x16, ror #56+        eor x30, x27, x6, ror #43+        eor v14.16b, v31.16b, v14.16b+        eor x22, x20, x15, ror #23+        bic x6, x19, x13, ror #42+        bcax v15.16b, v20.16b, v17.16b, v21.16b+        eor x6, x6, x17, ror #41+        bic x5, x13, x17, ror #63+        bic v31.16b, v18.16b, v17.16b+        eor x5, x21, x5, ror #21+        bic x17, x17, x21, ror #44+        eor v16.16b, v31.16b, v21.16b+        eor x27, x27, x10, ror #23+        bic x21, x21, x25, ror #50+        bcax v17.16b, v17.16b, v19.16b, v18.16b+        bic x20, x27, x4, ror #25+        bic x10, x16, x15, ror #31+        bic v31.16b, v20.16b, v19.16b+        eor x16, x21, x19, ror #43+        eor x21, x17, x25, ror #30+        eor v18.16b, v31.16b, v18.16b+        bic x19, x25, x19, ror #57+        ldr x25, [sp, #0x18]+        bcax v19.16b, v19.16b, v21.16b, v20.16b+        eor x17, x10, x9, ror #47+        bic v31.16b, v22.16b, v1.16b+        ldr x9, [sp, #0x8]+        eor x15, x20, x28, ror #27+        eor v20.16b, v31.16b, v0.16b+        bic x20, x4, x28, ror #2+        eor x10, x20, x1, ror #50+        bcax v21.16b, v1.16b, v23.16b, v22.16b+        bic x20, x11, x27, ror #60+        eor x20, x20, x4, ror #21+        bic v31.16b, v24.16b, v23.16b+        bic x4, x28, x1, ror #48+        bic x1, x1, x11, ror #57+        eor v22.16b, v31.16b, v22.16b+        ldr x28, [x9, x25, lsl #3]+        ldr x9, [sp, #0xd0]+        bcax v23.16b, v23.16b, v0.16b, v24.16b+        add x25, x25, #0x1+        str x25, [sp, #0x18]+        bic v31.16b, v1.16b, v0.16b+        cmp x25, #0x17+        eor x25, x1, x27, ror #53+        eor v24.16b, v31.16b, v24.16b+        bic x27, x30, x26, ror #47+        eor x1, x5, x28+        bcax v0.16b, v30.16b, v2.16b, v27.16b+        eor x5, x4, x11, ror #41+        eor x11, x19, x13, ror #35+        bic v31.16b, v3.16b, v2.16b+        bic x13, x26, x24, ror #10+        eor x28, x27, x24, ror #57+        eor v1.16b, v31.16b, v27.16b+        bic x27, x24, x9, ror #47+        bic x19, x23, x3, ror #9+        bcax v2.16b, v2.16b, v4.16b, v3.16b+        bic x4, x29, x14, ror #41+        eor x24, x19, x29, ror #44+        bic v31.16b, v30.16b, v4.16b+        bic x29, x3, x29, ror #35+        eor x13, x13, x9, ror #57+        eor v3.16b, v31.16b, v3.16b+        eor x19, x29, x14, ror #12+        bic x29, x9, x0, ror #19+        bcax v4.16b, v4.16b, v27.16b, v30.16b+        bic x14, x14, x8, ror #5+        eor x9, x14, x23, ror #43+        eor x14, x4, x8, ror #46+        bic x23, x8, x23, ror #38+        eor x8, x27, x0, ror #2+        eor x4, x23, x3, ror #47+        bic x3, x0, x30, ror #5+        eor x23, x3, x26, ror #52+        eor x3, x29, x30, ror #24+        ldr x30, [sp, #0x10]+        ld1r { v28.2d }, [x30], #8+        str x30, [sp, #0x10]+        eor v0.16b, v0.16b, v28.16b++Lmld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_loop_end:+        b.le Lmld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_loop+        ror x2, x2, #0x3d+        ror x3, x3, #0x27+        ror x4, x4, #0x36+        ror x5, x5, #0x19+        ror x6, x6, #0x2b+        ror x7, x7, #0x13+        ror x8, x8, #0x38+        ror x9, x9, #0x31+        ror x10, x10, #0x17+        ror x11, x11, #0x32+        ror x12, x12, #0x3+        ror x13, x13, #0x2e+        ror x14, x14, #0x8+        ror x15, x15, #0x3e+        ror x17, x17, #0x24+        ror x28, x28, #0x3f+        ror x19, x19, #0x25+        ror x20, x20, #0x2+        ror x21, x21, #0x14+        ror x22, x22, #0x2c+        ror x23, x23, #0x3a+        ror x24, x24, #0x1c+        ror x25, x25, #0x9+        ldr x30, [sp, #0x20]+        cmp x30, #0x1+        b.eq Lmld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_done+        mov x30, #0x1               // =1+        str x30, [sp, #0x20]+        ldr x0, [sp]+        add x0, x0, #0x190+        stp x1, x6, [x0]+        stp x11, x16, [x0, #0x10]+        stp x21, x2, [x0, #0x20]+        stp x7, x12, [x0, #0x30]+        stp x17, x22, [x0, #0x40]+        stp x3, x8, [x0, #0x50]+        stp x13, x28, [x0, #0x60]+        stp x23, x4, [x0, #0x70]+        stp x9, x14, [x0, #0x80]+        stp x19, x24, [x0, #0x90]+        stp x5, x10, [x0, #0xa0]+        stp x15, x20, [x0, #0xb0]+        str x25, [x0, #0xc0]+        sub x0, x0, #0x190+        add x0, x0, #0x258+        ldp x1, x6, [x0]+        ldp x11, x16, [x0, #0x10]+        ldp x21, x2, [x0, #0x20]+        ldp x7, x12, [x0, #0x30]+        ldp x17, x22, [x0, #0x40]+        ldp x3, x8, [x0, #0x50]+        ldp x13, x28, [x0, #0x60]+        ldp x23, x4, [x0, #0x70]+        ldp x9, x14, [x0, #0x80]+        ldp x19, x24, [x0, #0x90]+        ldp x5, x10, [x0, #0xa0]+        ldp x15, x20, [x0, #0xb0]+        ldr x25, [x0, #0xc0]+        sub x0, x0, #0x258+        b Lmld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_initial++Lmld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_done:+        ldr x0, [sp]+        add x0, x0, #0x258+        stp x1, x6, [x0]+        stp x11, x16, [x0, #0x10]+        stp x21, x2, [x0, #0x20]+        stp x7, x12, [x0, #0x30]+        stp x17, x22, [x0, #0x40]+        stp x3, x8, [x0, #0x50]+        stp x13, x28, [x0, #0x60]+        stp x23, x4, [x0, #0x70]+        stp x9, x14, [x0, #0x80]+        stp x19, x24, [x0, #0x90]+        stp x5, x10, [x0, #0xa0]+        stp x15, x20, [x0, #0xb0]+        str x25, [x0, #0xc0]+        sub x0, x0, #0x258+        add x4, x0, #0xc8+        trn1 v25.2d, v0.2d, v1.2d+        trn1 v26.2d, v2.2d, v3.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v0.2d, v1.2d+        trn2 v28.2d, v2.2d, v3.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v4.2d, v5.2d+        trn1 v26.2d, v6.2d, v7.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v4.2d, v5.2d+        trn2 v28.2d, v6.2d, v7.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v8.2d, v9.2d+        trn1 v26.2d, v10.2d, v11.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v8.2d, v9.2d+        trn2 v28.2d, v10.2d, v11.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v12.2d, v13.2d+        trn1 v26.2d, v14.2d, v15.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v12.2d, v13.2d+        trn2 v28.2d, v14.2d, v15.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v16.2d, v17.2d+        trn1 v26.2d, v18.2d, v19.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v16.2d, v17.2d+        trn2 v28.2d, v18.2d, v19.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v20.2d, v21.2d+        trn1 v26.2d, v22.2d, v23.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v20.2d, v21.2d+        trn2 v28.2d, v22.2d, v23.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        str d24, [x0]+        trn2 v25.2d, v24.2d, v24.2d+        str d25, [x4]+        ldp d8, d9, [sp, #0x90]+        .cfi_restore d8+        .cfi_restore d9+        ldp d10, d11, [sp, #0xa0]+        .cfi_restore d10+        .cfi_restore d11+        ldp d12, d13, [sp, #0xb0]+        .cfi_restore d12+        .cfi_restore d13+        ldp d14, d15, [sp, #0xc0]+        .cfi_restore d14+        .cfi_restore d15+        ldp x19, x20, [sp, #0x30]+        .cfi_restore x19+        .cfi_restore x20+        ldp x21, x22, [sp, #0x40]+        .cfi_restore x21+        .cfi_restore x22+        ldp x23, x24, [sp, #0x50]+        .cfi_restore x23+        .cfi_restore x24+        ldp x25, x26, [sp, #0x60]+        .cfi_restore x25+        .cfi_restore x26+        ldp x27, x28, [sp, #0x70]+        .cfi_restore x27+        .cfi_restore x28+        ldp x29, x30, [sp, #0x80]+        .cfi_restore x29+        .cfi_restore x30+        add sp, sp, #0xe0+        .cfi_adjust_cfa_offset -0xe0+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm)++#endif /* __ARM_FEATURE_SHA3 */++#endif /* MLD_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID && \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/fips202/native/aarch64/src/keccakf1600_round_constants.c view
@@ -0,0 +1,47 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mldsa-native repository.+ *          Do not modify it directly.+ */++#include "../../../../common.h"++#if (defined(MLD_FIPS202_AARCH64_NEED_X1_SCALAR) ||                  \+     defined(MLD_FIPS202_AARCH64_NEED_X1_V84A) ||                    \+     defined(MLD_FIPS202_AARCH64_NEED_X2_V84A) ||                    \+     defined(MLD_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID) ||       \+     defined(MLD_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID)) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++#include "fips202_native_aarch64.h"++MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint64_t+    mld_keccakf1600_round_constants[24] = {+        0x0000000000000001, 0x0000000000008082, 0x800000000000808a,+        0x8000000080008000, 0x000000000000808b, 0x0000000080000001,+        0x8000000080008081, 0x8000000000008009, 0x000000000000008a,+        0x0000000000000088, 0x0000000080008009, 0x000000008000000a,+        0x000000008000808b, 0x800000000000008b, 0x8000000000008089,+        0x8000000000008003, 0x8000000000008002, 0x8000000000000080,+        0x000000000000800a, 0x800000008000000a, 0x8000000080008081,+        0x8000000000008080, 0x0000000080000001, 0x8000000080008008,+};++#else /* (MLD_FIPS202_AARCH64_NEED_X1_SCALAR ||                               \+         MLD_FIPS202_AARCH64_NEED_X1_V84A || MLD_FIPS202_AARCH64_NEED_X2_V84A \+         || MLD_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID ||                  \+         MLD_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID) &&               \+         !MLD_CONFIG_MULTILEVEL_NO_SHARED */++MLD_EMPTY_CU(fips202_aarch64_round_constants)++#endif /* !((MLD_FIPS202_AARCH64_NEED_X1_SCALAR ||                             \+          MLD_FIPS202_AARCH64_NEED_X1_V84A || MLD_FIPS202_AARCH64_NEED_X2_V84A \+          || MLD_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID ||                  \+          MLD_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID) &&               \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED) */
+ cbits/mldsa/src/fips202/native/aarch64/x1_scalar.h view
@@ -0,0 +1,27 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLD_FIPS202_NATIVE_AARCH64_X1_SCALAR_H+#define MLD_FIPS202_NATIVE_AARCH64_X1_SCALAR_H++/* Part of backend API */+#define MLD_USE_NATIVE_FIPS202_X1+/* Guard for assembly file */+#define MLD_FIPS202_AARCH64_NEED_X1_SCALAR++#if !defined(__ASSEMBLER__)+#include "../api.h"+#include "src/fips202_native_aarch64.h"+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_keccak_f1600_x1_native(uint64_t *state)+{+  mld_keccak_f1600_x1_scalar_aarch64_asm(state,+                                         mld_keccakf1600_round_constants);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* !__ASSEMBLER__ */++#endif /* !MLD_FIPS202_NATIVE_AARCH64_X1_SCALAR_H */
+ cbits/mldsa/src/fips202/native/aarch64/x1_v84a.h view
@@ -0,0 +1,36 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLD_FIPS202_NATIVE_AARCH64_X1_V84A_H+#define MLD_FIPS202_NATIVE_AARCH64_X1_V84A_H++#if !defined(__ARM_FEATURE_SHA3)+#error This backend can only be used if SHA3 extensions are available.+#endif++/* Part of backend API */+#define MLD_USE_NATIVE_FIPS202_X1+/* Guard for assembly file */+#define MLD_FIPS202_AARCH64_NEED_X1_V84A++#if !defined(__ASSEMBLER__)+#include "../api.h"+#include "src/fips202_native_aarch64.h"+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_keccak_f1600_x1_native(uint64_t *state)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON) ||+      !mld_sys_check_capability(MLD_SYS_CAP_AARCH64_SHA3))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }++  mld_keccak_f1600_x1_v84a_aarch64_asm(state, mld_keccakf1600_round_constants);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* !__ASSEMBLER__ */++#endif /* !MLD_FIPS202_NATIVE_AARCH64_X1_V84A_H */
+ cbits/mldsa/src/fips202/native/aarch64/x2_v84a.h view
@@ -0,0 +1,40 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLD_FIPS202_NATIVE_AARCH64_X2_V84A_H+#define MLD_FIPS202_NATIVE_AARCH64_X2_V84A_H++#if !defined(__ARM_FEATURE_SHA3)+#error This backend can only be used if SHA3 extensions are available.+#endif++/* Part of backend API */+#define MLD_USE_NATIVE_FIPS202_X4+/* Guard for assembly file */+#define MLD_FIPS202_AARCH64_NEED_X2_V84A++#if !defined(__ASSEMBLER__)+#include "../api.h"+#include "src/fips202_native_aarch64.h"++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_keccak_f1600_x4_native(uint64_t *state)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON) ||+      !mld_sys_check_capability(MLD_SYS_CAP_AARCH64_SHA3))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }++  mld_keccak_f1600_x2_v84a_aarch64_asm(state + 0 * 25,+                                       mld_keccakf1600_round_constants);+  mld_keccak_f1600_x2_v84a_aarch64_asm(state + 2 * 25,+                                       mld_keccakf1600_round_constants);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* !__ASSEMBLER__ */++#endif /* !MLD_FIPS202_NATIVE_AARCH64_X2_V84A_H */
+ cbits/mldsa/src/fips202/native/aarch64/x4_v8a_scalar.h view
@@ -0,0 +1,32 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLD_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H+#define MLD_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H++/* Part of backend API */+#define MLD_USE_NATIVE_FIPS202_X4+/* Guard for assembly file */+#define MLD_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID++#if !defined(__ASSEMBLER__)+#include "../api.h"+#include "src/fips202_native_aarch64.h"+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_keccak_f1600_x4_native(uint64_t *state)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }++  mld_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm(+      state, mld_keccakf1600_round_constants);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* !__ASSEMBLER__ */++#endif /* !MLD_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H */
+ cbits/mldsa/src/fips202/native/aarch64/x4_v8a_v84a_scalar.h view
@@ -0,0 +1,37 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLD_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H+#define MLD_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H++#if !defined(__ARM_FEATURE_SHA3)+#error This backend can only be used if SHA3 extensions are available.+#endif++/* Part of backend API */+#define MLD_USE_NATIVE_FIPS202_X4+/* Guard for assembly file */+#define MLD_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID++#if !defined(__ASSEMBLER__)+#include "../api.h"+#include "src/fips202_native_aarch64.h"+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_keccak_f1600_x4_native(uint64_t *state)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON) ||+      !mld_sys_check_capability(MLD_SYS_CAP_AARCH64_SHA3))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }++  mld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm(+      state, mld_keccakf1600_round_constants);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* !__ASSEMBLER__ */++#endif /* !MLD_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H */
+ cbits/mldsa/src/fips202/native/api.h view
@@ -0,0 +1,129 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLD_FIPS202_NATIVE_API_H+#define MLD_FIPS202_NATIVE_API_H+/*+ * FIPS-202 native interface+ *+ * This header is primarily for documentation purposes.+ * It should not be included by backend implementations.+ */++#include "../../cbmc.h"+#include "../../common.h"++/* Backends must return MLD_NATIVE_FUNC_SUCCESS upon success. */+#define MLD_NATIVE_FUNC_SUCCESS (0)+/* Backends may return MLD_NATIVE_FUNC_FALLBACK to signal to the frontend that+ * the target/parameters are unsupported; typically, this would be because of+ * dependencies on CPU features not detected on the host CPU. In this case,+ * the frontend falls back to the default C implementation.+ *+ * IMPORTANT: Backend implementations must ensure that the decision of whether+ * to fallback (return MLD_NATIVE_FUNC_FALLBACK) or not must never depend on+ * the input data itself. Fallback decisions may only depend on system+ * capabilities (e.g., CPU features) and, where present, length information.+ * This requirement applies to all backend functions to maintain constant-time+ * properties.+ */+#define MLD_NATIVE_FUNC_FALLBACK (-1)++/*+ * This is the C<->native interface allowing for the drop-in+ * of custom Keccak-F1600 implementations.+ *+ * A _backend_ is a specific implementation of parts of this interface.+ *+ * You can replace 1-fold or 4-fold batched Keccak-F1600.+ * To enable, set MLD_USE_NATIVE_FIPS202_X1 or MLD_USE_NATIVE_FIPS202_X4+ * in your backend, and define the inline wrappers mld_keccak_f1600_x1_native()+ * and/or mld_keccak_f1600_x4_native(), respectively, to forward to your+ * implementation.+ */++#if defined(MLD_USE_NATIVE_FIPS202_X1)+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_keccak_f1600_x1_native(uint64_t *state)+__contract__(+    requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 1))+    assigns(memory_slice(state, sizeof(uint64_t) * 25 * 1))+    ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS)+    ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged_u64(state, 25 * 1))+);+#endif /* MLD_USE_NATIVE_FIPS202_X1 */+#if defined(MLD_USE_NATIVE_FIPS202_X4)+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_keccak_f1600_x4_native(uint64_t *state)+__contract__(+    requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4))+    assigns(memory_slice(state, sizeof(uint64_t) * 25 * 4))+    ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS)+    ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged_u64(state, 25 * 4))+);+#endif /* MLD_USE_NATIVE_FIPS202_X4 */++/*+ * Native x4 XOR bytes and extract bytes interface.+ *+ * These functions allow backends to provide optimized implementations for+ * XORing input data into the state and extracting output data from the state.+ * This is particularly useful for backends that use a different internal state+ * representation (e.g., bit-interleaved), as conversion can happen during+ * XOR/extract rather than before/after each permutation.+ *+ * NOTE: We assume that the custom representation of the zero state is the+ * all-zero state.+ *+ * MLD_USE_NATIVE_FIPS202_X4_XOR_BYTES: Backend provides native XOR bytes+ * MLD_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES: Backend provides native extract+ * bytes+ */++#if defined(MLD_USE_NATIVE_FIPS202_X4_XOR_BYTES)+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_keccakf1600_xor_bytes_x4_native(+    uint64_t *state, const unsigned char *data0, const unsigned char *data1,+    const unsigned char *data2, const unsigned char *data3, unsigned offset,+    unsigned length)+__contract__(+  requires(0 <= offset && offset <= 25 * sizeof(uint64_t) &&+           0 <= length && length <= 25 * sizeof(uint64_t) - offset)+  requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4))+  requires(memory_no_alias(data0, length))+  requires((data0 == data1 &&+            data0 == data2 &&+            data0 == data3) ||+           (memory_no_alias(data1, length) &&+            memory_no_alias(data2, length) &&+            memory_no_alias(data3, length)))+  assigns(memory_slice(state, sizeof(uint64_t) * 25 * 4))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged_u64(state, 25 * 4)));+#endif /* MLD_USE_NATIVE_FIPS202_X4_XOR_BYTES */++#if defined(MLD_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES)+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_keccakf1600_extract_bytes_x4_native(+    uint64_t *state, unsigned char *data0, unsigned char *data1,+    unsigned char *data2, unsigned char *data3, unsigned offset,+    unsigned length)+__contract__(+  requires(0 <= offset && offset <= 25 * sizeof(uint64_t) &&+           0 <= length && length <= 25 * sizeof(uint64_t) - offset)+  requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4))+  requires(memory_no_alias(data0, length))+  requires(memory_no_alias(data1, length))+  requires(memory_no_alias(data2, length))+  requires(memory_no_alias(data3, length))+  assigns(memory_slice(data0, length))+  assigns(memory_slice(data1, length))+  assigns(memory_slice(data2, length))+  assigns(memory_slice(data3, length))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS));+#endif /* MLD_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES */++#endif /* !MLD_FIPS202_NATIVE_API_H */
+ cbits/mldsa/src/fips202/native/auto.h view
@@ -0,0 +1,35 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLD_FIPS202_NATIVE_AUTO_H+#define MLD_FIPS202_NATIVE_AUTO_H++/*+ * Default FIPS202 backend+ */+#include "../../sys.h"++#if defined(MLD_SYS_AARCH64)+#include "aarch64/auto.h"+#endif++/* The x86_64 backend requires toolchain support for the SysV ABI */+#if defined(MLD_SYS_X86_64_AVX2) && defined(MLD_SYSV_ABI_SUPPORTED) &&       \+    (!defined(MLD_CONFIG_NO_KEYPAIR_API) ||                                  \+     !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_REDUCE_RAM)) && \+    !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY)+#include "x86_64/keccak_f1600_x4_avx2.h"+#endif /* MLD_SYS_X86_64_AVX2 && MLD_SYSV_ABI_SUPPORTED &&          \+          (!MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_SIGN_API || \+          !MLD_CONFIG_REDUCE_RAM) && !MLD_CONFIG_SERIAL_FIPS202_ONLY */++/* We do not yet include the FIPS202 backend for Armv8.1-M+MVE by default+ * as it is still experimental and undergoing review. */+/* #if defined(MLD_SYS_ARMV81M_MVE) */+/* #include "armv81m/mve.h" */+/* #endif */++#endif /* !MLD_FIPS202_NATIVE_AUTO_H */
+ cbits/mldsa/src/fips202/native/x86_64/keccak_f1600_x4_avx2.h view
@@ -0,0 +1,34 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLD_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H+#define MLD_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H++#include "../../../common.h"++#define MLD_FIPS202_X86_64_NEED_X4_AVX2++/* Part of backend API */+#define MLD_USE_NATIVE_FIPS202_X4++#if !defined(__ASSEMBLER__)+#include "../api.h"+#include "src/fips202_native_x86_64.h"+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_keccak_f1600_x4_native(uint64_t *state)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }++  mld_keccak_f1600_x4_avx2_asm(state, mld_keccakf1600_round_constants,+                               mld_keccak_rho8, mld_keccak_rho56);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* !__ASSEMBLER__ */++#endif /* !MLD_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H */
+ cbits/mldsa/src/fips202/native/x86_64/src/fips202_native_x86_64.h view
@@ -0,0 +1,45 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLD_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H+#define MLD_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H++#include "../../../../cbmc.h"+#include "../../../../common.h"++/* TODO: Reconsider whether this check is needed -- x86_64 is always+ * little-endian, so the backend selection already implies this. */+#ifndef MLD_SYS_LITTLE_ENDIAN+#error Expecting a little-endian platform+#endif++#define mld_keccakf1600_round_constants \+  MLD_NAMESPACE(keccakf1600_round_constants)+MLD_INTERNAL_DATA_DECLARATION const uint64_t+    mld_keccakf1600_round_constants[24];++#define mld_keccak_rho8 MLD_NAMESPACE(keccak_rho8)+MLD_INTERNAL_DATA_DECLARATION const uint64_t mld_keccak_rho8[4];++#define mld_keccak_rho56 MLD_NAMESPACE(keccak_rho56)+MLD_INTERNAL_DATA_DECLARATION const uint64_t mld_keccak_rho56[4];++#define mld_keccak_f1600_x4_avx2_asm MLD_NAMESPACE(keccak_f1600_x4_avx2_asm)+MLD_SYSV_ABI+void mld_keccak_f1600_x4_avx2_asm(uint64_t states[100], const uint64_t rc[24],+                                  const uint64_t rho8[4],+                                  const uint64_t rho56[4])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/keccak_f1600_x4_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(states, sizeof(uint64_t) * 25 * 4))+  requires(rc == mld_keccakf1600_round_constants)+  requires(rho8 == mld_keccak_rho8)+  requires(rho56 == mld_keccak_rho56)+  assigns(memory_slice(states, sizeof(uint64_t) * 25 * 4))+);++#endif /* !MLD_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H */
+ cbits/mldsa/src/fips202/native/x86_64/src/keccak_f1600_x4_avx2_asm.S view
@@ -0,0 +1,488 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+/*yaml+  Name: keccak_f1600_x4_avx2_asm+  Description: x86_64 AVX2 Keccak-f[1600] permutation for four sequential states+  Signature: void mld_keccak_f1600_x4_avx2_asm(uint64_t states[100], const uint64_t rc[24], const uint64_t rho8[4], const uint64_t rho56[4])+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 800+      permissions: read/write+      c_parameter: uint64_t states[100]+      description: Four sequential Keccak states (4 x 25 x uint64_t)+    rsi:+      type: buffer+      size_bytes: 192+      permissions: read-only+      c_parameter: const uint64_t rc[24]+      description: Round constants (24 x uint64_t)+    rdx:+      type: buffer+      size_bytes: 32+      permissions: read-only+      c_parameter: const uint64_t rho8[4]+      description: Rotation constant rho8 (4 x uint64_t)+    rcx:+      type: buffer+      size_bytes: 32+      permissions: read-only+      c_parameter: const uint64_t rho56[4]+      description: Rotation constant rho56 (4 x uint64_t)+*/++#include "../../../../common.h"++#if defined(MLD_FIPS202_X86_64_NEED_X4_AVX2) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/fips202/x86_64/src/keccak_f1600_x4_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(keccak_f1600_x4_avx2_asm)+MLD_ASM_FN_SYMBOL(keccak_f1600_x4_avx2_asm)++        .cfi_startproc+        movq %rsp, %r11+        .cfi_def_cfa_register %r11+        andq $-0x20, %rsp+        subq $0x300, %rsp            # imm = 0x300+        vmovdqu (%rdi), %ymm0+        vmovdqu 0xc8(%rdi), %ymm3+        vmovdqu 0x190(%rdi), %ymm1+        vmovdqu 0x258(%rdi), %ymm4+        vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2]+        vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3]+        vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2]+        vperm2i128 $0x20, %ymm3, %ymm2, %ymm7 # ymm7 = ymm2[0,1],ymm3[0,1]+        vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3]+        vperm2i128 $0x31, %ymm3, %ymm2, %ymm3 # ymm3 = ymm2[2,3],ymm3[2,3]+        vmovdqu 0x278(%rdi), %ymm4+        vmovdqu %ymm3, 0x40(%rsp)+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm3 # ymm3 = ymm0[2,3],ymm1[2,3]+        vmovdqu %ymm7, (%rsp)+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1]+        vmovdqu 0x20(%rdi), %ymm0+        vmovdqu 0x1b0(%rdi), %ymm1+        vmovdqu %ymm3, 0x60(%rsp)+        vmovdqu 0xe8(%rdi), %ymm3+        vmovdqu %ymm7, 0x20(%rsp)+        vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2]+        vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3]+        vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2]+        vperm2i128 $0x20, %ymm3, %ymm2, %ymm7 # ymm7 = ymm2[0,1],ymm3[0,1]+        vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3]+        vperm2i128 $0x31, %ymm3, %ymm2, %ymm3 # ymm3 = ymm2[2,3],ymm3[2,3]+        vmovdqu 0x298(%rdi), %ymm4+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm14 # ymm14 = ymm0[2,3],ymm1[2,3]+        vmovdqu %ymm7, 0x80(%rsp)+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1]+        vmovdqu 0x40(%rdi), %ymm0+        vmovdqu 0x1d0(%rdi), %ymm1+        vmovdqu %ymm3, 0xc0(%rsp)+        vmovdqu 0x108(%rdi), %ymm3+        vmovdqu %ymm14, %ymm10+        vmovdqu %ymm7, 0xa0(%rsp)+        vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2]+        vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3]+        vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3]+        vperm2i128 $0x20, %ymm3, %ymm2, %ymm11 # ymm11 = ymm2[0,1],ymm3[0,1]+        vperm2i128 $0x31, %ymm3, %ymm2, %ymm3 # ymm3 = ymm2[2,3],ymm3[2,3]+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1]+        vmovdqu %ymm3, 0x100(%rsp)+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm8 # ymm8 = ymm0[2,3],ymm1[2,3]+        vmovdqu 0x128(%rdi), %ymm3+        vmovdqu 0x60(%rdi), %ymm0+        vmovdqu 0x1f0(%rdi), %ymm1+        vmovdqu %ymm7, 0xe0(%rsp)+        vmovdqu %ymm11, %ymm14+        vmovdqu 0x2b8(%rdi), %ymm4+        vmovdqu 0x2f8(%rdi), %ymm5+        vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2]+        vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3]+        vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3]+        vmovdqu 0x2d8(%rdi), %ymm4+        vperm2i128 $0x20, %ymm3, %ymm2, %ymm15 # ymm15 = ymm2[0,1],ymm3[0,1]+        vperm2i128 $0x31, %ymm3, %ymm2, %ymm3 # ymm3 = ymm2[2,3],ymm3[2,3]+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1]+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm9 # ymm9 = ymm0[2,3],ymm1[2,3]+        vmovdqu %ymm3, 0x140(%rsp)+        vmovdqu 0x80(%rdi), %ymm0+        vmovdqu 0x148(%rdi), %ymm3+        vmovdqu 0x210(%rdi), %ymm1+        vmovdqu %ymm7, 0x120(%rsp)+        vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2]+        vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3]+        vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3]+        vperm2i128 $0x20, %ymm3, %ymm2, %ymm7 # ymm7 = ymm2[0,1],ymm3[0,1]+        vperm2i128 $0x31, %ymm3, %ymm2, %ymm13 # ymm13 = ymm2[2,3],ymm3[2,3]+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm3 # ymm3 = ymm0[2,3],ymm1[2,3]+        vmovdqu %ymm7, 0x160(%rsp)+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1]+        vmovdqu 0xa0(%rdi), %ymm0+        vmovdqu 0x230(%rdi), %ymm1+        vmovdqu %ymm3, 0x1a0(%rsp)+        vmovdqu 0x168(%rdi), %ymm3+        vpunpcklqdq %ymm5, %ymm1, %ymm4 # ymm4 = ymm1[0],ymm5[0],ymm1[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm5[1],ymm1[3],ymm5[3]+        vmovdqu %ymm7, 0x180(%rsp)+        vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2]+        vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3]+        vperm2i128 $0x20, %ymm4, %ymm2, %ymm12 # ymm12 = ymm2[0,1],ymm4[0,1]+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm3 # ymm3 = ymm0[0,1],ymm1[0,1]+        vperm2i128 $0x31, %ymm4, %ymm2, %ymm7 # ymm7 = ymm2[2,3],ymm4[2,3]+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm4 # ymm4 = ymm0[2,3],ymm1[2,3]+        vmovq 0x250(%rdi), %xmm0+        vmovq 0xc0(%rdi), %xmm1+        vmovdqu %ymm12, 0x1c0(%rsp)+        vmovdqu %ymm4, 0x1e0(%rsp)+        vpinsrq $0x1, 0x318(%rdi), %xmm0, %xmm0+        vpinsrq $0x1, 0x188(%rdi), %xmm1, %xmm1+        vinserti128 $0x1, %xmm0, %ymm1, %ymm2+        movq $0x0, %r10++LLmld_keccak_f1600_x4_avx2_asm:+        vmovdqu 0xa0(%rsp), %ymm4+        vpxor 0x1c0(%rsp), %ymm9, %ymm0+        vmovdqu %ymm9, 0x200(%rsp)+        vmovdqu %ymm10, %ymm9+        vmovdqu 0xc0(%rsp), %ymm11+        vmovdqu 0x160(%rsp), %ymm12+        vmovdqu %ymm3, 0x240(%rsp)+        vpxor 0x100(%rsp), %ymm4, %ymm1+        vmovdqu 0x40(%rsp), %ymm10+        vmovdqu %ymm4, 0x220(%rsp)+        vpxor %ymm3, %ymm12, %ymm12+        vmovdqu 0x20(%rsp), %ymm6+        vmovdqu 0x140(%rsp), %ymm4+        vmovdqu %ymm14, 0x2a0(%rsp)+        vpxor %ymm1, %ymm0, %ymm0+        vpxor %ymm8, %ymm11, %ymm1+        vpxor 0x180(%rsp), %ymm7, %ymm11+        vmovdqu %ymm10, 0x280(%rsp)+        vpxor %ymm1, %ymm12, %ymm12+        vpxor %ymm15, %ymm9, %ymm1+        vmovdqu 0xe0(%rsp), %ymm3+        vmovdqu %ymm8, 0x260(%rsp)+        vpxor %ymm1, %ymm11, %ymm11+        vpxor 0x120(%rsp), %ymm14, %ymm1+        vpxor %ymm6, %ymm12, %ymm12+        vmovdqu 0x60(%rsp), %ymm8+        vpxor %ymm10, %ymm11, %ymm11+        vpxor 0x1e0(%rsp), %ymm13, %ymm10+        vpxor %ymm4, %ymm3, %ymm3+        vmovdqu %ymm4, 0x2c0(%rsp)+        vpsrlq $0x3f, %ymm12, %ymm4+        vpsrlq $0x3f, %ymm11, %ymm5+        vpxor (%rsp), %ymm0, %ymm0+        vpxor %ymm1, %ymm10, %ymm10+        vmovdqu 0x80(%rsp), %ymm1+        vpxor %ymm8, %ymm10, %ymm10+        vmovdqu %ymm1, %ymm14+        vpxor 0x1a0(%rsp), %ymm2, %ymm1+        vmovdqu %ymm14, 0x2e0(%rsp)+        vpxor %ymm3, %ymm1, %ymm1+        vpsllq $0x1, %ymm12, %ymm3+        vpor %ymm4, %ymm3, %ymm3+        vpsllq $0x1, %ymm11, %ymm4+        vpxor %ymm14, %ymm1, %ymm1+        vpor %ymm5, %ymm4, %ymm4+        vpsrlq $0x3f, %ymm10, %ymm14+        vpxor %ymm1, %ymm3, %ymm3+        vpsllq $0x1, %ymm10, %ymm5+        vpxor %ymm0, %ymm4, %ymm4+        vpor %ymm14, %ymm5, %ymm5+        vpxor %ymm6, %ymm4, %ymm6+        vpxor %ymm12, %ymm5, %ymm5+        vpsrlq $0x3f, %ymm1, %ymm12+        vpsllq $0x1, %ymm1, %ymm1+        vpxor %ymm7, %ymm5, %ymm7+        vpxor %ymm9, %ymm5, %ymm9+        vpor %ymm12, %ymm1, %ymm1+        vpxor (%rsp), %ymm3, %ymm12+        vpxor %ymm11, %ymm1, %ymm1+        vpsrlq $0x3f, %ymm0, %ymm11+        vpsllq $0x1, %ymm0, %ymm0+        vpxor %ymm13, %ymm1, %ymm13+        vpxor %ymm8, %ymm1, %ymm8+        vpor %ymm11, %ymm0, %ymm0+        vpxor %ymm10, %ymm0, %ymm0+        vpxor 0xc0(%rsp), %ymm4, %ymm10+        vpxor %ymm2, %ymm0, %ymm2+        vpsrlq $0x14, %ymm10, %ymm11+        vpsllq $0x2c, %ymm10, %ymm10+        vpor %ymm11, %ymm10, %ymm10+        vpxor %ymm15, %ymm5, %ymm11+        vpbroadcastq (%rsi), %ymm15+        vpsrlq $0x15, %ymm11, %ymm14+        vpsllq $0x2b, %ymm11, %ymm11+        vpor %ymm14, %ymm11, %ymm11+        vpandn %ymm11, %ymm10, %ymm14+        vpxor %ymm15, %ymm14, %ymm14+        vpxor %ymm12, %ymm14, %ymm15+        vpsrlq $0x2b, %ymm13, %ymm14+        vpsllq $0x15, %ymm13, %ymm13+        vmovdqu %ymm15, (%rsp)+        vpor %ymm14, %ymm13, %ymm13+        vpandn %ymm13, %ymm11, %ymm14+        vpxor %ymm10, %ymm14, %ymm15+        vpsrlq $0x32, %ymm2, %ymm14+        vpsllq $0xe, %ymm2, %ymm2+        vmovdqu %ymm15, 0x20(%rsp)+        vpor %ymm14, %ymm2, %ymm2+        vpandn %ymm2, %ymm13, %ymm14+        vpxor %ymm11, %ymm14, %ymm11+        vmovdqu %ymm11, 0x40(%rsp)+        vpandn %ymm12, %ymm2, %ymm11+        vpandn %ymm10, %ymm12, %ymm12+        vpxor %ymm13, %ymm11, %ymm11+        vmovdqu %ymm11, 0x60(%rsp)+        vpxor %ymm2, %ymm12, %ymm11+        vpsrlq $0x24, %ymm8, %ymm2+        vpsllq $0x1c, %ymm8, %ymm8+        vmovdqu %ymm11, 0x80(%rsp)+        vpor %ymm2, %ymm8, %ymm8+        vpxor 0xe0(%rsp), %ymm0, %ymm2+        vpsrlq $0x2c, %ymm2, %ymm10+        vpsllq $0x14, %ymm2, %ymm2+        vpor %ymm10, %ymm2, %ymm2+        vpxor 0x100(%rsp), %ymm3, %ymm10+        vpsrlq $0x3d, %ymm10, %ymm11+        vpsllq $0x3, %ymm10, %ymm10+        vpor %ymm11, %ymm10, %ymm10+        vpandn %ymm10, %ymm2, %ymm11+        vpxor %ymm8, %ymm11, %ymm11+        vmovdqu %ymm11, 0xa0(%rsp)+        vpxor 0x160(%rsp), %ymm4, %ymm11+        vpsrlq $0x13, %ymm11, %ymm12+        vpsllq $0x2d, %ymm11, %ymm11+        vpor %ymm12, %ymm11, %ymm11+        vpandn %ymm11, %ymm10, %ymm12+        vpxor %ymm2, %ymm12, %ymm12+        vmovdqu %ymm12, 0xc0(%rsp)+        vpsrlq $0x3, %ymm7, %ymm12+        vpsllq $0x3d, %ymm7, %ymm7+        vpor %ymm12, %ymm7, %ymm7+        vpandn %ymm7, %ymm11, %ymm12+        vpxor %ymm10, %ymm12, %ymm10+        vpandn %ymm8, %ymm7, %ymm12+        vpandn %ymm2, %ymm8, %ymm8+        vpsrlq $0x3f, %ymm6, %ymm2+        vpsllq $0x1, %ymm6, %ymm6+        vpxor %ymm11, %ymm12, %ymm14+        vpor %ymm2, %ymm6, %ymm6+        vpsrlq $0x3a, %ymm9, %ymm2+        vpxor %ymm7, %ymm8, %ymm12+        vpsllq $0x6, %ymm9, %ymm9+        vmovdqu %ymm12, 0xe0(%rsp)+        vpxor 0x1a0(%rsp), %ymm0, %ymm7+        vpor %ymm2, %ymm9, %ymm9+        vpxor 0x120(%rsp), %ymm1, %ymm2+        vpshufb (%rdx), %ymm7, %ymm7+        vpsrlq $0x27, %ymm2, %ymm11+        vpsllq $0x19, %ymm2, %ymm2+        vpor %ymm2, %ymm11, %ymm11+        vpandn %ymm11, %ymm9, %ymm2+        vpandn %ymm7, %ymm11, %ymm8+        vpxor %ymm6, %ymm2, %ymm12+        vpxor 0x1c0(%rsp), %ymm3, %ymm2+        vpxor %ymm9, %ymm8, %ymm8+        vmovdqu %ymm12, 0x100(%rsp)+        vpsrlq $0x2e, %ymm2, %ymm12+        vpsllq $0x12, %ymm2, %ymm2+        vpor %ymm2, %ymm12, %ymm2+        vpandn %ymm2, %ymm7, %ymm12+        vpxor %ymm11, %ymm12, %ymm15+        vpandn %ymm6, %ymm2, %ymm11+        vpandn %ymm9, %ymm6, %ymm6+        vpxor %ymm7, %ymm11, %ymm12+        vmovdqu %ymm12, 0x120(%rsp)+        vpxor %ymm2, %ymm6, %ymm12+        vpxor 0x2e0(%rsp), %ymm0, %ymm6+        vpxor 0x2c0(%rsp), %ymm0, %ymm0+        vmovdqu %ymm12, 0x140(%rsp)+        vpsrlq $0x25, %ymm6, %ymm2+        vpsllq $0x1b, %ymm6, %ymm6+        vpor %ymm6, %ymm2, %ymm2+        vpxor 0x220(%rsp), %ymm3, %ymm6+        vpxor 0x200(%rsp), %ymm3, %ymm3+        vpsrlq $0x1c, %ymm6, %ymm7+        vpsllq $0x24, %ymm6, %ymm6+        vpor %ymm6, %ymm7, %ymm7+        vpxor 0x260(%rsp), %ymm4, %ymm6+        vpxor 0x240(%rsp), %ymm4, %ymm4+        vpsrlq $0x36, %ymm6, %ymm12+        vpsllq $0xa, %ymm6, %ymm6+        vpor %ymm6, %ymm12, %ymm12+        vpxor 0x180(%rsp), %ymm5, %ymm6+        vpxor 0x280(%rsp), %ymm5, %ymm5+        vpandn %ymm12, %ymm7, %ymm9+        vpsrlq $0x31, %ymm6, %ymm11+        vpsllq $0xf, %ymm6, %ymm6+        vpxor %ymm2, %ymm9, %ymm9+        vpor %ymm6, %ymm11, %ymm11+        vpandn %ymm11, %ymm12, %ymm6+        vpxor %ymm7, %ymm6, %ymm6+        vmovdqu %ymm6, 0x160(%rsp)+        vpxor 0x1e0(%rsp), %ymm1, %ymm6+        vpxor 0x2a0(%rsp), %ymm1, %ymm1+        vpshufb (%rcx), %ymm6, %ymm6+        vpandn %ymm6, %ymm11, %ymm13+        vpxor %ymm12, %ymm13, %ymm13+        vmovdqu %ymm13, 0x180(%rsp)+        vpandn %ymm2, %ymm6, %ymm13+        vpandn %ymm7, %ymm2, %ymm2+        vpxor %ymm6, %ymm2, %ymm2+        vpsrlq $0x3e, %ymm4, %ymm6+        vpxor %ymm11, %ymm13, %ymm13+        vmovdqu %ymm2, 0x1a0(%rsp)+        vpsrlq $0x2, %ymm5, %ymm2+        vpsllq $0x3e, %ymm5, %ymm5+        vpor %ymm5, %ymm2, %ymm2+        vpsrlq $0x9, %ymm1, %ymm5+        vpsllq $0x37, %ymm1, %ymm1+        vpsllq $0x2, %ymm4, %ymm4+        vpor %ymm1, %ymm5, %ymm1+        vpsrlq $0x19, %ymm0, %ymm5+        vpor %ymm4, %ymm6, %ymm4+        vpsllq $0x27, %ymm0, %ymm0+        vpor %ymm0, %ymm5, %ymm5+        vpandn %ymm5, %ymm1, %ymm0+        vpxor %ymm2, %ymm0, %ymm0+        vmovdqu %ymm0, 0x1c0(%rsp)+        vpsrlq $0x17, %ymm3, %ymm0+        vpsllq $0x29, %ymm3, %ymm3+        vpor %ymm3, %ymm0, %ymm0+        vpandn %ymm4, %ymm0, %ymm7+        vpandn %ymm0, %ymm5, %ymm3+        vpxor %ymm5, %ymm7, %ymm7+        vpandn %ymm2, %ymm4, %ymm5+        vpandn %ymm1, %ymm2, %ymm2+        vpxor %ymm0, %ymm5, %ymm5+        vpxor %ymm1, %ymm3, %ymm3+        vpxor %ymm4, %ymm2, %ymm2+        vmovdqu %ymm5, 0x1e0(%rsp)+        addq $0x8, %rsi+        addq $0x1, %r10+        cmpq $0x18, %r10+        jne LLmld_keccak_f1600_x4_avx2_asm+        vmovdqu (%rsp), %ymm4+        vmovdqu 0x40(%rsp), %ymm5+        vmovdqu 0x20(%rsp), %ymm0+        vmovdqu 0x60(%rsp), %ymm1+        vmovdqu 0x1c0(%rsp), %ymm12+        vmovdqu %ymm2, 0x1c0(%rsp)+        vpunpcklqdq %ymm0, %ymm4, %ymm2 # ymm2 = ymm4[0],ymm0[0],ymm4[2],ymm0[2]+        vpunpckhqdq %ymm0, %ymm4, %ymm0 # ymm0 = ymm4[1],ymm0[1],ymm4[3],ymm0[3]+        vpunpcklqdq %ymm1, %ymm5, %ymm4 # ymm4 = ymm5[0],ymm1[0],ymm5[2],ymm1[2]+        vpunpckhqdq %ymm1, %ymm5, %ymm1 # ymm1 = ymm5[1],ymm1[1],ymm5[3],ymm1[3]+        vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1]+        vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3]+        vmovdqu 0x80(%rsp), %ymm4+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm5 # ymm5 = ymm0[0,1],ymm1[0,1]+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm0 # ymm0 = ymm0[2,3],ymm1[2,3]+        vmovdqu %ymm6, (%rdi)+        vmovdqu %ymm5, 0xc8(%rdi)+        vmovdqu %ymm2, 0x190(%rdi)+        vmovdqu %ymm0, 0x258(%rdi)+        vmovdqu 0xa0(%rsp), %ymm0+        vpunpcklqdq %ymm0, %ymm4, %ymm2 # ymm2 = ymm4[0],ymm0[0],ymm4[2],ymm0[2]+        vpunpckhqdq %ymm0, %ymm4, %ymm1 # ymm1 = ymm4[1],ymm0[1],ymm4[3],ymm0[3]+        vmovdqu 0xc0(%rsp), %ymm0+        vpunpcklqdq %ymm10, %ymm0, %ymm4 # ymm4 = ymm0[0],ymm10[0],ymm0[2],ymm10[2]+        vpunpckhqdq %ymm10, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm10[1],ymm0[3],ymm10[3]+        vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1]+        vperm2i128 $0x20, %ymm0, %ymm1, %ymm5 # ymm5 = ymm1[0,1],ymm0[0,1]+        vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3]+        vmovdqu 0xe0(%rsp), %ymm4+        vperm2i128 $0x31, %ymm0, %ymm1, %ymm1 # ymm1 = ymm1[2,3],ymm0[2,3]+        vmovdqu 0x100(%rsp), %ymm0+        vmovdqu %ymm2, 0x1b0(%rdi)+        vmovdqu %ymm1, 0x278(%rdi)+        vpunpcklqdq %ymm4, %ymm14, %ymm2 # ymm2 = ymm14[0],ymm4[0],ymm14[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm14, %ymm1 # ymm1 = ymm14[1],ymm4[1],ymm14[3],ymm4[3]+        vpunpcklqdq %ymm8, %ymm0, %ymm4 # ymm4 = ymm0[0],ymm8[0],ymm0[2],ymm8[2]+        vpunpckhqdq %ymm8, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm8[1],ymm0[3],ymm8[3]+        vmovdqu %ymm6, 0x20(%rdi)+        vmovdqu %ymm5, 0xe8(%rdi)+        vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1]+        vperm2i128 $0x20, %ymm0, %ymm1, %ymm5 # ymm5 = ymm1[0,1],ymm0[0,1]+        vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3]+        vperm2i128 $0x31, %ymm0, %ymm1, %ymm1 # ymm1 = ymm1[2,3],ymm0[2,3]+        vmovdqu 0x120(%rsp), %ymm4+        vmovdqu 0x140(%rsp), %ymm0+        vmovdqu %ymm2, 0x1d0(%rdi)+        vmovdqu %ymm1, 0x298(%rdi)+        vpunpcklqdq %ymm4, %ymm15, %ymm2 # ymm2 = ymm15[0],ymm4[0],ymm15[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm15, %ymm1 # ymm1 = ymm15[1],ymm4[1],ymm15[3],ymm4[3]+        vpunpcklqdq %ymm9, %ymm0, %ymm4 # ymm4 = ymm0[0],ymm9[0],ymm0[2],ymm9[2]+        vmovdqu %ymm5, 0x108(%rdi)+        vpunpckhqdq %ymm9, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm9[1],ymm0[3],ymm9[3]+        vmovdqu %ymm6, 0x40(%rdi)+        vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1]+        vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3]+        vperm2i128 $0x20, %ymm0, %ymm1, %ymm5 # ymm5 = ymm1[0,1],ymm0[0,1]+        vmovdqu 0x160(%rsp), %ymm4+        vperm2i128 $0x31, %ymm0, %ymm1, %ymm1 # ymm1 = ymm1[2,3],ymm0[2,3]+        vmovdqu 0x180(%rsp), %ymm0+        vmovdqu %ymm5, 0x128(%rdi)+        vmovdqu 0x1a0(%rsp), %ymm5+        vmovdqu %ymm2, 0x1f0(%rdi)+        vpunpcklqdq %ymm0, %ymm4, %ymm2 # ymm2 = ymm4[0],ymm0[0],ymm4[2],ymm0[2]+        vpunpckhqdq %ymm0, %ymm4, %ymm0 # ymm0 = ymm4[1],ymm0[1],ymm4[3],ymm0[3]+        vpunpcklqdq %ymm5, %ymm13, %ymm4 # ymm4 = ymm13[0],ymm5[0],ymm13[2],ymm5[2]+        vmovdqu %ymm6, 0x60(%rdi)+        vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1]+        vmovdqu %ymm1, 0x2b8(%rdi)+        vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3]+        vpunpckhqdq %ymm5, %ymm13, %ymm1 # ymm1 = ymm13[1],ymm5[1],ymm13[3],ymm5[3]+        vmovdqu %ymm6, 0x80(%rdi)+        vmovdqu 0x1e0(%rsp), %ymm4+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm5 # ymm5 = ymm0[0,1],ymm1[0,1]+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm0 # ymm0 = ymm0[2,3],ymm1[2,3]+        vmovdqu %ymm2, 0x210(%rdi)+        vpunpcklqdq %ymm3, %ymm12, %ymm2 # ymm2 = ymm12[0],ymm3[0],ymm12[2],ymm3[2]+        vmovdqu %ymm0, 0x2d8(%rdi)+        vpunpckhqdq %ymm3, %ymm12, %ymm0 # ymm0 = ymm12[1],ymm3[1],ymm12[3],ymm3[3]+        vpunpcklqdq %ymm4, %ymm7, %ymm3 # ymm3 = ymm7[0],ymm4[0],ymm7[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm7, %ymm1 # ymm1 = ymm7[1],ymm4[1],ymm7[3],ymm4[3]+        vmovdqu %ymm5, 0x148(%rdi)+        vperm2i128 $0x20, %ymm3, %ymm2, %ymm5 # ymm5 = ymm2[0,1],ymm3[0,1]+        vperm2i128 $0x31, %ymm3, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm3[2,3]+        vmovdqu 0x1c0(%rsp), %ymm3+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm4 # ymm4 = ymm0[0,1],ymm1[0,1]+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm0 # ymm0 = ymm0[2,3],ymm1[2,3]+        vmovdqu %ymm5, 0xa0(%rdi)+        vextracti128 $0x1, %ymm3, %xmm15+        vmovdqu %ymm4, 0x168(%rdi)+        vmovdqu %ymm2, 0x230(%rdi)+        vmovdqu %ymm0, 0x2f8(%rdi)+        vmovq %xmm3, 0xc0(%rdi)+        vmovhpd %xmm3, 0x188(%rdi)+        vmovq %xmm15, 0x250(%rdi)+        vmovhpd %xmm15, 0x318(%rdi)+        movq %r11, %rsp+        .cfi_def_cfa_register %rsp+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(keccak_f1600_x4_avx2_asm)++#endif /* MLD_FIPS202_X86_64_NEED_X4_AVX2 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/fips202/native/x86_64/src/keccakf1600_constants.c view
@@ -0,0 +1,52 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mldsa-native repository.+ *          Do not modify it directly.+ */++#include "../../../../common.h"+#if defined(MLD_FIPS202_X86_64_NEED_X4_AVX2) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++#include <stdint.h>++#include "fips202_native_x86_64.h"++MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint64_t+    mld_keccakf1600_round_constants[24] = {+        0x0000000000000001, 0x0000000000008082, 0x800000000000808a,+        0x8000000080008000, 0x000000000000808b, 0x0000000080000001,+        0x8000000080008081, 0x8000000000008009, 0x000000000000008a,+        0x0000000000000088, 0x0000000080008009, 0x000000008000000a,+        0x000000008000808b, 0x800000000000008b, 0x8000000000008089,+        0x8000000000008003, 0x8000000000008002, 0x8000000000000080,+        0x000000000000800a, 0x800000008000000a, 0x8000000080008081,+        0x8000000000008080, 0x0000000080000001, 0x8000000080008008,+};++MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint64_t mld_keccak_rho8[4] = {+    0x0605040302010007,+    0x0e0d0c0b0a09080f,+    0x1615141312111017,+    0x1e1d1c1b1a19181f,+};++MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint64_t mld_keccak_rho56[4] = {+    0x0007060504030201,+    0x080f0e0d0c0b0a09,+    0x1017161514131211,+    0x181f1e1d1c1b1a19,+};++#else /* MLD_FIPS202_X86_64_NEED_X4_AVX2 && !MLD_CONFIG_MULTILEVEL_NO_SHARED \+       */++MLD_EMPTY_CU(fips202_x86_64_constants)++#endif /* !(MLD_FIPS202_X86_64_NEED_X4_AVX2 && \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED) */
+ cbits/mldsa/src/native/aarch64/meta.h view
@@ -0,0 +1,314 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLD_NATIVE_AARCH64_META_H+#define MLD_NATIVE_AARCH64_META_H++/* Set of primitives that this backend replaces */+#define MLD_USE_NATIVE_NTT+#define MLD_USE_NATIVE_INTT+#define MLD_USE_NATIVE_REJ_UNIFORM+#define MLD_USE_NATIVE_REJ_UNIFORM_ETA2+#define MLD_USE_NATIVE_REJ_UNIFORM_ETA4+#define MLD_USE_NATIVE_POLY_DECOMPOSE_32+#define MLD_USE_NATIVE_POLY_DECOMPOSE_88+#define MLD_USE_NATIVE_POLY_CADDQ+#define MLD_USE_NATIVE_POLY_USE_HINT_32+#define MLD_USE_NATIVE_POLY_USE_HINT_88+#define MLD_USE_NATIVE_POLY_CHKNORM+#define MLD_USE_NATIVE_POLYZ_UNPACK_17+#define MLD_USE_NATIVE_POLYZ_UNPACK_19+#define MLD_USE_NATIVE_POINTWISE_MONTGOMERY+#define MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4+#define MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5+#define MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7++/* Identifier for this backend so that source and assembly files+ * in the build can be appropriately guarded. */+#define MLD_ARITH_BACKEND_AARCH64+++#if !defined(__ASSEMBLER__)+#include "../api.h"+#include "src/arith_native_aarch64.h"++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_ntt_native(int32_t data[MLDSA_N])+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_ntt_aarch64_asm(data, mld_aarch64_ntt_zetas_layer123456,+                      mld_aarch64_ntt_zetas_layer78);+  return MLD_NATIVE_FUNC_SUCCESS;+}++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_intt_native(int32_t data[MLDSA_N])+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_intt_aarch64_asm(data, mld_aarch64_intt_zetas_layer78,+                       mld_aarch64_intt_zetas_layer123456);+  return MLD_NATIVE_FUNC_SUCCESS;+}++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_rej_uniform_native(int32_t *r, unsigned len,+                                             const uint8_t *buf,+                                             unsigned buflen)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON) || len != MLDSA_N ||+      buflen % 24 != 0)+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }++  /* Safety: outlen is at most MLDSA_N, hence, this cast is safe. */+  return (int)mld_rej_uniform_aarch64_asm(r, buf, buflen,+                                          mld_rej_uniform_table);+}++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_ETA == 2+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_rej_uniform_eta2_native(int32_t *r, unsigned len,+                                                  const uint8_t *buf,+                                                  unsigned buflen)+{+  uint64_t outlen;+  /* AArch64 implementation assumes specific buffer lengths */+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON) || len != MLDSA_N ||+      buflen != MLD_AARCH64_REJ_UNIFORM_ETA2_BUFLEN)+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  /* Constant time: Inputs and outputs to this function are secret.+   * It is safe to leak which coefficients are accepted/rejected.+   * The assembly implementation must not leak any other information about the+   * accepted coefficients. Constant-time testing cannot cover this, and we+   * hence have to manually verify the assembly.+   * We declassify prior the input data and mark the outputs as secret.+   */+  MLD_CT_TESTING_DECLASSIFY(buf, buflen);+  outlen = mld_rej_uniform_eta2_aarch64_asm(r, buf, buflen,+                                            mld_rej_uniform_eta_table);+  MLD_CT_TESTING_SECRET(r, sizeof(int32_t) * outlen);+  /* Safety: outlen is at most MLDSA_N and, hence, this cast is safe. */+  return (int)outlen;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_ETA == 2 */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_ETA == 4+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_rej_uniform_eta4_native(int32_t *r, unsigned len,+                                                  const uint8_t *buf,+                                                  unsigned buflen)+{+  uint64_t outlen;+  /* AArch64 implementation assumes specific buffer lengths */+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON) || len != MLDSA_N ||+      buflen != MLD_AARCH64_REJ_UNIFORM_ETA4_BUFLEN)+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  /* Constant time: Inputs and outputs to this function are secret.+   * It is safe to leak which coefficients are accepted/rejected.+   * The assembly implementation must not leak any other information about the+   * accepted coefficients. Constant-time testing cannot cover this, and we+   * hence have to manually verify the assembly.+   * We declassify prior the input data and mark the outputs as secret.+   */+  MLD_CT_TESTING_DECLASSIFY(buf, buflen);+  outlen = mld_rej_uniform_eta4_aarch64_asm(r, buf, buflen,+                                            mld_rej_uniform_eta_table);+  MLD_CT_TESTING_SECRET(r, sizeof(int32_t) * outlen);+  /* Safety: outlen is at most MLDSA_N and, hence, this cast is safe. */+  return (int)outlen;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_ETA == 4 */+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_decompose_32_native(int32_t *a1, int32_t *a0)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_poly_decompose_32_aarch64_asm(a1, a0);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87 */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_decompose_88_native(int32_t *a1, int32_t *a0)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_poly_decompose_88_aarch64_asm(a1, a0);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \+        */+#endif /* !MLD_CONFIG_NO_SIGN_API */++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_caddq_native(int32_t a[MLDSA_N])+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_poly_caddq_aarch64_asm(a);+  return MLD_NATIVE_FUNC_SUCCESS;+}++#if !defined(MLD_CONFIG_NO_VERIFY_API)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_use_hint_32_native(int32_t *a, const int32_t *h)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_poly_use_hint_32_aarch64_asm(a, h);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87 */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_use_hint_88_native(int32_t *a, const int32_t *h)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_poly_use_hint_88_aarch64_asm(a, h);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \+        */+#endif /* !MLD_CONFIG_NO_VERIFY_API */++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_chknorm_native(const int32_t *a, int32_t B)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  return mld_poly_chknorm_aarch64_asm(a, B);+}++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_polyz_unpack_17_native(int32_t *r, const uint8_t *buf)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_polyz_unpack_17_aarch64_asm(r, buf, mld_polyz_unpack_17_indices);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \+        */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_polyz_unpack_19_native(int32_t *r, const uint8_t *buf)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_polyz_unpack_19_aarch64_asm(r, buf, mld_polyz_unpack_19_indices);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87 */+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) || \+    defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_pointwise_montgomery_native(+    int32_t a[MLDSA_N], const int32_t b[MLDSA_N])+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_poly_pointwise_montgomery_aarch64_asm(a, b);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API || \+          MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 4+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l4_native(+    int32_t w[MLDSA_N], const int32_t u[4][MLDSA_N],+    const int32_t v[4][MLDSA_N])+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm(w, u, v);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 4 */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 5+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l5_native(+    int32_t w[MLDSA_N], const int32_t u[5][MLDSA_N],+    const int32_t v[5][MLDSA_N])+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm(w, u, v);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 5 */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 7+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l7_native(+    int32_t w[MLDSA_N], const int32_t u[7][MLDSA_N],+    const int32_t v[7][MLDSA_N])+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm(w, u, v);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 7 */++#endif /* !__ASSEMBLER__ */+#endif /* !MLD_NATIVE_AARCH64_META_H */
+ cbits/mldsa/src/native/aarch64/src/aarch64_zetas.c view
@@ -0,0 +1,248 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mldsa-native repository.+ *          Do not modify it directly.+ */++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_AARCH64) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++#include "arith_native_aarch64.h"++/*+ * Table of zeta values used in the AArch64 forward NTT+ * See autogen for details.+ */+MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const int32_t+    mld_aarch64_ntt_zetas_layer123456[144] = {+        -3572223, -915382907,  3765607,  964937599,   3761513,  963888510,+        -3201494, -820383522,  -2883726, -738955404,  -3145678, -806080660,+        -3201430, -820367122,  0,        0,           -601683,  -154181397,+        -3370349, -863652652,  -4063053, -1041158200, 3602218,  923069133,+        3182878,  815613168,   2740543,  702264730,   -3586446, -919027554,+        0,        0,           3542485,  907762539,   2663378,  682491182,+        -1674615, -429120452,  -3110818, -797147778,  2101410,  538486762,+        3704823,  949361686,   1159875,  297218217,   0,        0,+        2682288,  687336873,   -3524442, -903139016,  -434125,  -111244624,+        394148,   101000509,   928749,   237992130,   1095468,  280713909,+        -3506380, -898510625,  0,        0,           2129892,  545785280,+        676590,   173376332,   -1335936, -342333886,  2071829,  530906624,+        -4018989, -1029866791, 3241972,  830756018,   2156050,  552488273,+        0,        0,           3764867,  964747974,   -3227876, -827143915,+        1714295,  439288460,   3415069,  875112161,   1759347,  450833045,+        -817536,  -209493775,  -3574466, -915957677,  0,        0,+        -1005239, -257592709,  2453983,  628833668,   1460718,  374309300,+        3756790,  962678241,   -1935799, -496048908,  -1716988, -439978542,+        -3950053, -1012201926, 0,        0,           557458,   142848732,+        -642628,  -164673562,  -3585098, -918682129,  -2897314, -742437332,+        3192354,  818041395,   556856,   142694469,   3870317,  991769559,+        0,        0,           -1221177, -312926867,  2815639,  721508096,+        2283733,  585207070,   2917338,  747568486,   1853806,  475038184,+        3345963,  857403734,   1858416,  476219497,   0,        0,+};++MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const int32_t+    mld_aarch64_ntt_zetas_layer78[384] = {+        3073009,     1277625,     -2635473,   3852015,     787459213,+        327391679,   -675340520,  987079667,  1753,        -2659525,+        2660408,     -59148,      449207,     -681503850,  681730119,+        -15156688,   -1935420,    -1455890,   -1780227,    2772600,+        -495951789,  -373072124,  -456183549, 710479343,   4183372,+        -3222807,    -3121440,    -274060,    1071989969,  -825844983,+        -799869667,  -70227934,   1182243,    636927,      -3956745,+        -3284915,    302950022,   163212680,  -1013916752, -841760171,+        87208,       -3965306,    -2296397,   -3716946,    22347069,+        -1016110510, -588452222,  -952468207, 2508980,     2028118,+        1937570,     -3815725,    642926661,  519705671,   496502727,+        -977780347,  -27812,      1009365,    -1979497,    -3956944,+        -7126831,    258649997,   -507246529, -1013967746, 822541,+        -2454145,    1596822,     -3759465,   210776307,   -628875181,+        409185979,   -963363710,  2811291,    -2983781,    -1109516,+        4158088,     720393920,   -764594519, -284313712,  1065510939,+        -1685153,    2678278,     -3551006,   -250446,     -431820817,+        686309310,   -909946047,  -64176841,  -3410568,    -3768948,+        635956,      -2455377,    -873958779, -965793731,  162963861,+        -629190881,  1528066,     482649,     1148858,     -2962264,+        391567239,   123678909,   294395108,  -759080783,  -4146264,+        2192938,     2387513,     -268456,    -1062481036, 561940831,+        611800717,   -68791907,   -1772588,   -1727088,    -3611750,+        -3180456,    -454226054,  -442566669, -925511710,  -814992530,+        -565603,     169688,      2462444,    -3334383,    -144935890,+        43482586,    631001801,   -854436357, 3747250,     1239911,+        3195676,     1254190,     960233614,  317727459,   818892658,+        321386456,   2296099,     -3838479,   2642980,     -12417,+        588375860,   -983611064,  677264190,  -3181859,    -4166425,+        -3488383,    1987814,     -3197248,   -1067647297, -893898890,+        509377762,   -819295484,  2998219,    -89301,      -1354892,+        -1310261,    768294260,   -22883400,  -347191365,  -335754661,+        141835,      2513018,     613238,     -2218467,    36345249,+        643961400,   157142369,   -568482643, 1736313,     235407,+        -3250154,    3258457,     444930577,  60323094,    -832852657,+        834980303,   -458740,     4040196,    2039144,     -818761,+        -117552223,  1035301089,  522531086,  -209807681,  -1921994,+        -3472069,    -1879878,    -2178965,   -492511373,  -889718424,+        -481719139,  -558360247,  -2579253,   1787943,     -2391089,+        -2254727,    -660934133,  458160776,  -612717067,  -577774276,+        -1623354,    -2374402,    586241,     527981,      -415984810,+        -608441020,  150224382,   135295244,  2105286,     -2033807,+        -1179613,    -2743411,    539479988,  -521163479,  -302276083,+        -702999655,  3482206,     -4182915,   -1300016,    -2362063,+        892316032,   -1071872863, -333129378, -605279149,  -1476985,+        2491325,     507927,      -724804,    -378477722,  638402564,+        130156402,   -185731180,  1994046,    -1393159,    -1187885,+        -1834526,    510974714,   -356997292, -304395785,  -470097680,+        -1317678,    2461387,     3035980,    621164,      -337655269,+        630730945,   777970524,   159173408,  -3033742,    2647994,+        -2612853,    749577,      -777397036, 678549029,   -669544140,+        192079267,   -338420,     3009748,    4148469,     -4022750,+        -86720197,   771248568,   1063046068, -1030830548, 3901472,+        -1226661,    2925816,     3374250,    999753034,   -314332144,+        749740976,   864652284,   3980599,    -1615530,    1665318,+        1163598,     1020029345,  -413979908, 426738094,   298172236,+        2569011,     1723229,     2028038,    -3369273,    658309618,+        441577800,   519685171,   -863376927, 1356448,     -2775755,+        2683270,     -2778788,    347590090,  -711287812,  687588511,+        -712065019,  3994671,     -1370517,   3363542,     545376,+        1023635298,  -351195274,  861908357,  139752717,   -11879,+        3020393,     214880,      -770441,    -3043996,    773976352,+        55063046,    -197425671,  -3467665,   2312838,     -653275,+        -459163,     -888589898,  592665232,  -167401858,  -117660617,+        3105558,     508145,      860144,     140244,      795799901,+        130212265,   220412084,   35937555,   -1103344,    -553718,+        3430436,     -1514152,    -282732136, -141890356,  879049958,+        -388001774,  348812,      -327848,    1011223,     -2354215,+        89383150,    -84011120,   259126110,  -603268097,  -2185084,+        2358373,     -3014420,    2926054,    -559928242,  604333585,+        -772445769,  749801963,   3123762,    -2193087,    -1716814,+        -392707,     800464680,   -561979013, -439933955,  -100631253,+        -3818627,    -1922253,    -2236726,   1744507,     -978523985,+        -492577742,  -573161516,  447030292,  -303005,     -3974485,+        1900052,     1054478,     -77645096,  -1018462631, 486888731,+        270210213,   3531229,     -3773731,   -781875,     -731434,+        904878186,   -967019376,  -200355636, -187430119,+};++MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const int32_t+    mld_aarch64_intt_zetas_layer78[384] = {+        -1744507,    2236726,    1922253,     3818627,     -447030292,+        573161516,   492577742,  978523985,   731434,      781875,+        3773731,     -3531229,   187430119,   200355636,   967019376,+        -904878186,  -1054478,   -1900052,    3974485,     303005,+        -270210213,  -486888731, 1018462631,  77645096,    2354215,+        -1011223,    327848,     -348812,     603268097,   -259126110,+        84011120,    -89383150,  392707,      1716814,     2193087,+        -3123762,    100631253,  439933955,   561979013,   -800464680,+        -2926054,    3014420,    -2358373,    2185084,     -749801963,+        772445769,   -604333585, 559928242,   459163,      653275,+        -2312838,    3467665,    117660617,   167401858,   -592665232,+        888589898,   1514152,    -3430436,    553718,      1103344,+        388001774,   -879049958, 141890356,   282732136,   -140244,+        -860144,     -508145,    -3105558,    -35937555,   -220412084,+        -130212265,  -795799901, 2778788,     -2683270,    2775755,+        -1356448,    712065019,  -687588511,  711287812,   -347590090,+        770441,      -214880,    -3020393,    11879,       197425671,+        -55063046,   -773976352, 3043996,     -545376,     -3363542,+        1370517,     -3994671,   -139752717,  -861908357,  351195274,+        -1023635298, -3374250,   -2925816,    1226661,     -3901472,+        -864652284,  -749740976, 314332144,   -999753034,  3369273,+        -2028038,    -1723229,   -2569011,    863376927,   -519685171,+        -441577800,  -658309618, -1163598,    -1665318,    1615530,+        -3980599,    -298172236, -426738094,  413979908,   -1020029345,+        -621164,     -3035980,   -2461387,    1317678,     -159173408,+        -777970524,  -630730945, 337655269,   4022750,     -4148469,+        -3009748,    338420,     1030830548,  -1063046068, -771248568,+        86720197,    -749577,    2612853,     -2647994,    3033742,+        -192079267,  669544140,  -678549029,  777397036,   2362063,+        1300016,     4182915,    -3482206,    605279149,   333129378,+        1071872863,  -892316032, 1834526,     1187885,     1393159,+        -1994046,    470097680,  304395785,   356997292,   -510974714,+        724804,      -507927,    -2491325,    1476985,     185731180,+        -130156402,  -638402564, 378477722,   2254727,     2391089,+        -1787943,    2579253,    577774276,   612717067,   -458160776,+        660934133,   2743411,    1179613,     2033807,     -2105286,+        702999655,   302276083,  521163479,   -539479988,  -527981,+        -586241,     2374402,    1623354,     -135295244,  -150224382,+        608441020,   415984810,  -3258457,    3250154,     -235407,+        -1736313,    -834980303, 832852657,   -60323094,   -444930577,+        2178965,     1879878,    3472069,     1921994,     558360247,+        481719139,   889718424,  492511373,   818761,      -2039144,+        -4040196,    458740,     209807681,   -522531086,  -1035301089,+        117552223,   3197248,    -1987814,    3488383,     4166425,+        819295484,   -509377762, 893898890,   1067647297,  2218467,+        -613238,     -2513018,   -141835,     568482643,   -157142369,+        -643961400,  -36345249,  1310261,     1354892,     89301,+        -2998219,    335754661,  347191365,   22883400,    -768294260,+        3334383,     -2462444,   -169688,     565603,      854436357,+        -631001801,  -43482586,  144935890,   12417,       -2642980,+        3838479,     -2296099,   3181859,     -677264190,  983611064,+        -588375860,  -1254190,   -3195676,    -1239911,    -3747250,+        -321386456,  -818892658, -317727459,  -960233614,  2962264,+        -1148858,    -482649,    -1528066,    759080783,   -294395108,+        -123678909,  -391567239, 3180456,     3611750,     1727088,+        1772588,     814992530,  925511710,   442566669,   454226054,+        268456,      -2387513,   -2192938,    4146264,     68791907,+        -611800717,  -561940831, 1062481036,  -4158088,    1109516,+        2983781,     -2811291,   -1065510939, 284313712,   764594519,+        -720393920,  2455377,    -635956,     3768948,     3410568,+        629190881,   -162963861, 965793731,   873958779,   250446,+        3551006,     -2678278,   1685153,     64176841,    909946047,+        -686309310,  431820817,  3815725,     -1937570,    -2028118,+        -2508980,    977780347,  -496502727,  -519705671,  -642926661,+        3759465,     -1596822,   2454145,     -822541,     963363710,+        -409185979,  628875181,  -210776307,  3956944,     1979497,+        -1009365,    27812,      1013967746,  507246529,   -258649997,+        7126831,     274060,     3121440,     3222807,     -4183372,+        70227934,    799869667,  825844983,   -1071989969, 3716946,+        2296397,     3965306,    -87208,      952468207,   588452222,+        1016110510,  -22347069,  3284915,     3956745,     -636927,+        -1182243,    841760171,  1013916752,  -163212680,  -302950022,+        -3852015,    2635473,    -1277625,    -3073009,    -987079667,+        675340520,   -327391679, -787459213,  -2772600,    1780227,+        1455890,     1935420,    -710479343,  456183549,   373072124,+        495951789,   59148,      -2660408,    2659525,     -1753,+        15156688,    -681730119, 681503850,   -449207,+};++MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const int32_t+    mld_aarch64_intt_zetas_layer123456[160] = {+        -2283733, -585207070, 0,        0,          -1858416, -476219497,+        -3345963, -857403734, -2815639, -721508096, 0,        0,+        -1853806, -475038184, -2917338, -747568486, 3585098,  918682129,+        0,        0,          -3870317, -991769559, -556856,  -142694469,+        642628,   164673562,  0,        0,          -3192354, -818041395,+        2897314,  742437332,  -1460718, -374309300, 0,        0,+        3950053,  1012201926, 1716988,  439978542,  -2453983, -628833668,+        0,        0,          1935799,  496048908,  -3756790, -962678241,+        -1714295, -439288460, 0,        0,          3574466,  915957677,+        817536,   209493775,  3227876,  827143915,  0,        0,+        -1759347, -450833045, -3415069, -875112161, 1335936,  342333886,+        0,        0,          -2156050, -552488273, -3241972, -830756018,+        -676590,  -173376332, 0,        0,          4018989,  1029866791,+        -2071829, -530906624, 434125,   111244624,  0,        0,+        3506380,  898510625,  -1095468, -280713909, 3524442,  903139016,+        0,        0,          -928749,  -237992130, -394148,  -101000509,+        1674615,  429120452,  0,        0,          -1159875, -297218217,+        -3704823, -949361686, -2663378, -682491182, 0,        0,+        -2101410, -538486762, 3110818,  797147778,  4063053,  1041158200,+        0,        0,          3586446,  919027554,  -2740543, -702264730,+        3370349,  863652652,  0,        0,          -3182878, -815613168,+        -3602218, -923069133, -294725,  -75523344,  -3761513, -963888510,+        -3765607, -964937599, 3201430,  820367122,  3145678,  806080660,+        2883726,  738955404,  3201494,  820383522,  1221177,  312926867,+        -557458,  -142848732, 1005239,  257592709,  -3764867, -964747974,+        -2129892, -545785280, -2682288, -687336873, -3542485, -907762539,+        601683,   154181397,  0,        0,+};++#else /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */++MLD_EMPTY_CU(aarch64_zetas)++#endif /* !(MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED) */
+ cbits/mldsa/src/native/aarch64/src/arith_native_aarch64.h view
@@ -0,0 +1,367 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLD_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H+#define MLD_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H++#include "../../../cbmc.h"+#include "../../../common.h"++#define mld_aarch64_ntt_zetas_layer123456 \+  MLD_NAMESPACE(aarch64_ntt_zetas_layer123456)+#define mld_aarch64_ntt_zetas_layer78 MLD_NAMESPACE(aarch64_ntt_zetas_layer78)++#define mld_aarch64_intt_zetas_layer78 MLD_NAMESPACE(aarch64_intt_zetas_layer78)+#define mld_aarch64_intt_zetas_layer123456 \+  MLD_NAMESPACE(aarch64_intt_zetas_layer123456)++MLD_INTERNAL_DATA_DECLARATION const int32_t+    mld_aarch64_ntt_zetas_layer123456[144];+MLD_INTERNAL_DATA_DECLARATION const int32_t mld_aarch64_ntt_zetas_layer78[384];++MLD_INTERNAL_DATA_DECLARATION const int32_t mld_aarch64_intt_zetas_layer78[384];+MLD_INTERNAL_DATA_DECLARATION const int32_t+    mld_aarch64_intt_zetas_layer123456[160];++#define mld_rej_uniform_table MLD_NAMESPACE(rej_uniform_table)+MLD_INTERNAL_DATA_DECLARATION const uint8_t mld_rej_uniform_table[256];+#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+#define mld_rej_uniform_eta_table MLD_NAMESPACE(rej_uniform_eta_table)+MLD_INTERNAL_DATA_DECLARATION const uint8_t mld_rej_uniform_eta_table[4096];+#endif++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44+#define mld_polyz_unpack_17_indices MLD_NAMESPACE(polyz_unpack_17_indices)+MLD_INTERNAL_DATA_DECLARATION const uint8_t mld_polyz_unpack_17_indices[64];+#endif+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)+#define mld_polyz_unpack_19_indices MLD_NAMESPACE(polyz_unpack_19_indices)+MLD_INTERNAL_DATA_DECLARATION const uint8_t mld_polyz_unpack_19_indices[64];+#endif+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */+++/*+ * Sampling 256 coefficients mod 15 using rejection sampling from 4 bits.+ * Expected number of required bytes: (256 * (16/15))/2 = 136.5 bytes.+ * We sample 1 block (=136 bytes) of SHAKE256_RATE output initially.+ * Sampling 2 blocks initially results in slightly worse performance.+ */+#define MLD_AARCH64_REJ_UNIFORM_ETA2_BUFLEN (1 * 136)+/*+ * Sampling 256 coefficients mod 9 using rejection sampling from 4 bits.+ * Expected number of required bytes: (256 * (16/9))/2 = 227.5 bytes.+ * We sample 2 blocks (=272 bytes) of SHAKE256_RATE output initially.+ */+#define MLD_AARCH64_REJ_UNIFORM_ETA4_BUFLEN (2 * 136)++#define mld_ntt_aarch64_asm MLD_NAMESPACE(ntt_aarch64_asm)+void mld_ntt_aarch64_asm(int32_t r[MLDSA_N], const int32_t zetas_l123456[144],+                         const int32_t zetas_l78[384])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mldsa_ntt_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  /* check-magic: 8380417 == MLDSA_Q */+  requires(array_abs_bound(r, 0, MLDSA_N, 8380417))+  requires(zetas_l123456 == mld_aarch64_ntt_zetas_layer123456)+  requires(zetas_l78 == mld_aarch64_ntt_zetas_layer78)+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  /* check-magic: off */+  ensures(array_abs_bound(r, 0, MLDSA_N, 75423753))+  /* check-magic: on */+);++#define mld_intt_aarch64_asm MLD_NAMESPACE(intt_aarch64_asm)+void mld_intt_aarch64_asm(int32_t r[MLDSA_N], const int32_t zetas_l78[384],+                          const int32_t zetas_l123456[160])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mldsa_intt_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(array_abs_bound(r, 0, MLDSA_N, 8380417))+  requires(zetas_l78 == mld_aarch64_intt_zetas_layer78)+  requires(zetas_l123456 == mld_aarch64_intt_zetas_layer123456)+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  /* check-magic: off */+  ensures(array_abs_bound(r, 0, MLDSA_N, 8380417))+  /* check-magic: on */+);++#define mld_rej_uniform_aarch64_asm MLD_NAMESPACE(rej_uniform_aarch64_asm)+MLD_MUST_CHECK_RETURN_VALUE+uint64_t mld_rej_uniform_aarch64_asm(int32_t r[MLDSA_N], const uint8_t *buf,+                                     unsigned buflen, const uint8_t table[256])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mldsa_rej_uniform_aarch64_asm.ml. */+__contract__(+  requires(buflen % 24 == 0)+  requires(memory_no_alias(buf, buflen))+  requires(table == mld_rej_uniform_table)+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(return_value <= MLDSA_N)+  ensures(array_bound(r, 0, (unsigned) return_value, 0, MLDSA_Q))+);++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+#define mld_rej_uniform_eta2_aarch64_asm \+  MLD_NAMESPACE(rej_uniform_eta2_aarch64_asm)+MLD_MUST_CHECK_RETURN_VALUE+uint64_t mld_rej_uniform_eta2_aarch64_asm(int32_t r[MLDSA_N],+                                          const uint8_t *buf, unsigned buflen,+                                          const uint8_t table[4096])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mldsa_rej_uniform_eta2_aarch64_asm.ml */+__contract__(+  requires(buflen % 8 == 0)+  requires(buflen >= 8)+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(buf, buflen))+  requires(table == mld_rej_uniform_eta_table)+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(return_value <= MLDSA_N)+  /* check-magic: 3 == 2 + 1 (asm is eta=2-specific) */+  ensures(array_abs_bound(r, 0, return_value, 3))+);++#define mld_rej_uniform_eta4_aarch64_asm \+  MLD_NAMESPACE(rej_uniform_eta4_aarch64_asm)+MLD_MUST_CHECK_RETURN_VALUE+uint64_t mld_rej_uniform_eta4_aarch64_asm(int32_t r[MLDSA_N],+                                          const uint8_t *buf, unsigned buflen,+                                          const uint8_t table[4096])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mldsa_rej_uniform_eta4_aarch64_asm.ml */+__contract__(+  requires(buflen % 8 == 0)+  requires(buflen >= 8)+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(buf, buflen))+  requires(table == mld_rej_uniform_eta_table)+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(return_value <= MLDSA_N)+  /* check-magic: 5 == 4 + 1 (asm is eta=4-specific) */+  ensures(array_abs_bound(r, 0, return_value, 5))+);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+#define mld_poly_decompose_32_aarch64_asm \+  MLD_NAMESPACE(poly_decompose_32_aarch64_asm)+void mld_poly_decompose_32_aarch64_asm(int32_t a1[MLDSA_N], int32_t a0[MLDSA_N])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mldsa_poly_decompose_32_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(a1, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(a0, sizeof(int32_t) * MLDSA_N))+  requires(array_bound(a0, 0, MLDSA_N, 0, MLDSA_Q))+  assigns(memory_slice(a1, sizeof(int32_t) * MLDSA_N))+  assigns(memory_slice(a0, sizeof(int32_t) * MLDSA_N))+  /* check-magic: 16 == (MLDSA_Q - 1) / (2 * ((MLDSA_Q - 1) / 32)) */+  ensures(array_bound(a1, 0, MLDSA_N, 0, 16))+  /* check-magic: 261889 == (MLDSA_Q - 1) / 32 + 1 */+  ensures(array_abs_bound(a0, 0, MLDSA_N, 261889))+);++#define mld_poly_decompose_88_aarch64_asm \+  MLD_NAMESPACE(poly_decompose_88_aarch64_asm)+void mld_poly_decompose_88_aarch64_asm(int32_t a1[MLDSA_N], int32_t a0[MLDSA_N])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mldsa_poly_decompose_88_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(a1, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(a0, sizeof(int32_t) * MLDSA_N))+  requires(array_bound(a0, 0, MLDSA_N, 0, MLDSA_Q))+  assigns(memory_slice(a1, sizeof(int32_t) * MLDSA_N))+  assigns(memory_slice(a0, sizeof(int32_t) * MLDSA_N))+  /* check-magic: 44 == (MLDSA_Q - 1) / (2 * ((MLDSA_Q - 1) / 88)) */+  ensures(array_bound(a1, 0, MLDSA_N, 0, 44))+  /* check-magic: 95233 == (MLDSA_Q - 1) / 88 + 1 */+  ensures(array_abs_bound(a0, 0, MLDSA_N, 95233))+);+#endif /* !MLD_CONFIG_NO_SIGN_API */++#define mld_poly_caddq_aarch64_asm MLD_NAMESPACE(poly_caddq_aarch64_asm)+void mld_poly_caddq_aarch64_asm(int32_t a[MLDSA_N])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mldsa_poly_caddq_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N))+  requires(array_abs_bound(a, 0, MLDSA_N, MLDSA_Q))+  assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N))+  ensures(array_bound(a, 0, MLDSA_N, 0, MLDSA_Q))+);++#if !defined(MLD_CONFIG_NO_VERIFY_API)+#define mld_poly_use_hint_32_aarch64_asm \+  MLD_NAMESPACE(poly_use_hint_32_aarch64_asm)+void mld_poly_use_hint_32_aarch64_asm(int32_t a[MLDSA_N],+                                      const int32_t h[MLDSA_N])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mldsa_poly_use_hint_32_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(h, sizeof(int32_t) * MLDSA_N))+  requires(array_bound(a, 0, MLDSA_N, 0, MLDSA_Q))+  requires(array_bound(h, 0, MLDSA_N, 0, 2))+  assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N))+  ensures(array_bound(a, 0, MLDSA_N, 0, 16))+);++#define mld_poly_use_hint_88_aarch64_asm \+  MLD_NAMESPACE(poly_use_hint_88_aarch64_asm)+void mld_poly_use_hint_88_aarch64_asm(int32_t a[MLDSA_N],+                                      const int32_t h[MLDSA_N])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mldsa_poly_use_hint_88_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(h, sizeof(int32_t) * MLDSA_N))+  requires(array_bound(a, 0, MLDSA_N, 0, MLDSA_Q))+  requires(array_bound(h, 0, MLDSA_N, 0, 2))+  assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N))+  ensures(array_bound(a, 0, MLDSA_N, 0, 44))+);+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#define mld_poly_chknorm_aarch64_asm MLD_NAMESPACE(poly_chknorm_aarch64_asm)+MLD_MUST_CHECK_RETURN_VALUE+int mld_poly_chknorm_aarch64_asm(const int32_t a[MLDSA_N], int32_t B)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mldsa_poly_chknorm_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N))+  /* HOL Light precondition: abs(ival(x i)) < 2^31, i.e., a[i] != INT32_MIN */+  requires(forall(k0, 0, MLDSA_N, a[k0] > INT32_MIN))+  ensures(return_value == 0 || return_value == 1)+  ensures((return_value == 0) == array_abs_bound(a, 0, MLDSA_N, B))+);++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44+#define mld_polyz_unpack_17_aarch64_asm \+  MLD_NAMESPACE(polyz_unpack_17_aarch64_asm)+void mld_polyz_unpack_17_aarch64_asm(int32_t r[MLDSA_N], const uint8_t buf[576],+                                     const uint8_t indices[64])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mldsa_polyz_unpack_17_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(buf, 576))+  requires(indices == mld_polyz_unpack_17_indices)+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(array_bound(r, 0, MLDSA_N, -((1 << 17) - 1), (1 << 17) + 1))+);+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \+        */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)+#define mld_polyz_unpack_19_aarch64_asm \+  MLD_NAMESPACE(polyz_unpack_19_aarch64_asm)+void mld_polyz_unpack_19_aarch64_asm(int32_t r[MLDSA_N], const uint8_t buf[640],+                                     const uint8_t indices[64])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mldsa_polyz_unpack_19_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(buf, 640))+  requires(indices == mld_polyz_unpack_19_indices)+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(array_bound(r, 0, MLDSA_N, -((1 << 19) - 1), (1 << 19) + 1))+);+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87 */+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) || \+    defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+#define mld_poly_pointwise_montgomery_aarch64_asm \+  MLD_NAMESPACE(poly_pointwise_montgomery_aarch64_asm)+void mld_poly_pointwise_montgomery_aarch64_asm(int32_t a[MLDSA_N],+                                               const int32_t b[MLDSA_N])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mldsa_pointwise_montgomery_aarch64_asm.ml+ */+__contract__(+  requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(b, sizeof(int32_t) * MLDSA_N))+  /* Input bound MLD_NTT_BOUND = 9 * MLD_FQMUL_BOUND, the guaranteed bound of+   * any forward NTT implementation. Hardcoded here to keep this header free+   * of poly.h. */+  /* check-magic: 94279698 == 9 * ((5 * MLDSA_Q + 3) / 4) */+  requires(array_abs_bound(a, 0, MLDSA_N, 94279698))+  requires(array_abs_bound(b, 0, MLDSA_N, 94279698))+  assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N))+  ensures(array_abs_bound(a, 0, MLDSA_N, 8380417))+);+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API || \+          MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++#define mld_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm \+  MLD_NAMESPACE(polyvecl_pointwise_acc_montgomery_l4_aarch64_asm)+void mld_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm(+    int32_t r[MLDSA_N], const int32_t a[4][MLDSA_N],+    const int32_t b[4][MLDSA_N])+/* This must be kept in sync with the HOL-Light specification+ * in+ * proofs/hol_light/aarch64/proofs/mldsa_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm.ml+ */+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(a, sizeof(int32_t) * 4 * MLDSA_N))+  requires(memory_no_alias(b, sizeof(int32_t) * 4 * MLDSA_N))+  requires(forall(l0, 0, 4, array_abs_bound(a[l0], 0, MLDSA_N, 8380417)))+  /* check-magic: 94279698 == 9 * ((5 * MLDSA_Q + 3) / 4) */+  requires(forall(l1, 0, 4, array_abs_bound(b[l1], 0, MLDSA_N, 94279698)))+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(array_abs_bound(r, 0, MLDSA_N, 8380417))+);++#define mld_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm \+  MLD_NAMESPACE(polyvecl_pointwise_acc_montgomery_l5_aarch64_asm)+void mld_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm(+    int32_t r[MLDSA_N], const int32_t a[5][MLDSA_N],+    const int32_t b[5][MLDSA_N])+/* This must be kept in sync with the HOL-Light specification+ * in+ * proofs/hol_light/aarch64/proofs/mldsa_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm.ml+ */+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(a, sizeof(int32_t) * 5 * MLDSA_N))+  requires(memory_no_alias(b, sizeof(int32_t) * 5 * MLDSA_N))+  requires(forall(l0, 0, 5, array_abs_bound(a[l0], 0, MLDSA_N, 8380417)))+  /* check-magic: 94279698 == 9 * ((5 * MLDSA_Q + 3) / 4) */+  requires(forall(l1, 0, 5, array_abs_bound(b[l1], 0, MLDSA_N, 94279698)))+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(array_abs_bound(r, 0, MLDSA_N, 8380417))+);++#define mld_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm \+  MLD_NAMESPACE(polyvecl_pointwise_acc_montgomery_l7_aarch64_asm)+void mld_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm(+    int32_t r[MLDSA_N], const int32_t a[7][MLDSA_N],+    const int32_t b[7][MLDSA_N])+/* This must be kept in sync with the HOL-Light specification+ * in+ * proofs/hol_light/aarch64/proofs/mldsa_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm.ml+ */+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(a, sizeof(int32_t) * 7 * MLDSA_N))+  requires(memory_no_alias(b, sizeof(int32_t) * 7 * MLDSA_N))+  requires(forall(l0, 0, 7, array_abs_bound(a[l0], 0, MLDSA_N, 8380417)))+  /* check-magic: 94279698 == 9 * ((5 * MLDSA_Q + 3) / 4) */+  requires(forall(l1, 0, 7, array_abs_bound(b[l1], 0, MLDSA_N, 94279698)))+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(array_abs_bound(r, 0, MLDSA_N, 8380417))+);++#endif /* !MLD_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H */
+ cbits/mldsa/src/native/aarch64/src/mldsa_intt_aarch64_asm.S view
@@ -0,0 +1,786 @@+/* Copyright (c) 2022 Arm Limited+ * Copyright (c) 2022 Hanno Becker+ * Copyright (c) 2023 Amin Abdulrahman, Matthias Kannwischer+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [NeonNTT]+ *   Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1+ *   Becker, Hwang, Kannwischer, Yang, Yang+ *   https://eprint.iacr.org/2021/986+ *+ * - [NeonNTT_Autoformalised]+ *   Neon NTT - (Auto)formalised+ *   Hanno Becker+ *   https://eprint.iacr.org/2026/1223+ *+ * - [SLOTHY_Paper]+ *   Fast and Clean: Auditable high-performance assembly via constraint solving+ *   Abdulrahman, Becker, Kannwischer, Klein+ *   https://eprint.iacr.org/2022/1303+ */++/* AArch64 ML-DSA inverse NTT following @[NeonNTT], @[SLOTHY_Paper], and @[NeonNTT_Autoformalised] */++/*yaml+  Name: intt_aarch64_asm+  Description: AArch64 ML-DSA inverse NTT+  Signature: void mld_intt_aarch64_asm(int32_t r[256], const int32_t zetas_l78[384], const int32_t zetas_l123456[160])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 1024+      permissions: read/write+      c_parameter: int32_t r[256]+      description: Input/output polynomial (256 x int32_t)+    x1:+      type: buffer+      size_bytes: 1536+      permissions: read-only+      c_parameter: const int32_t zetas_l78[384]+      description: Twiddle factors for layers 7-8 (384 x int32_t)+    x2:+      type: buffer+      size_bytes: 640+      permissions: read-only+      c_parameter: const int32_t zetas_l123456[160]+      description: Twiddle factors for layers 1-6 (160 x int32_t)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_AARCH64) &&  !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/aarch64_opt/src/mldsa_intt_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(intt_aarch64_asm)+MLD_ASM_FN_SYMBOL(intt_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0x40+        .cfi_adjust_cfa_offset 0x40+        stp d8, d9, [sp]+        .cfi_rel_offset d8, 0x0+        .cfi_rel_offset d9, 0x8+        stp d10, d11, [sp, #0x10]+        .cfi_rel_offset d10, 0x10+        .cfi_rel_offset d11, 0x18+        stp d12, d13, [sp, #0x20]+        .cfi_rel_offset d12, 0x20+        .cfi_rel_offset d13, 0x28+        stp d14, d15, [sp, #0x30]+        .cfi_rel_offset d14, 0x30+        .cfi_rel_offset d15, 0x38+        mov w5, #0xe001             // =57345+        movk w5, #0x7f, lsl #16+        dup v31.4s, w5+        mov x3, x0+        mov x4, #0x10               // =16+        ldr q27, [x3, #0x10]+        ldr q18, [x3]+        ldr q3, [x3, #0x20]+        ldr q13, [x3, #0x30]+        ldr q2, [x1, #0x30]+        ldr q8, [x3, #0x70]+        ldr q21, [x3, #0x60]+        trn1 v10.4s, v18.4s, v27.4s+        trn2 v23.4s, v18.4s, v27.4s+        trn1 v6.4s, v3.4s, v13.4s+        trn2 v18.4s, v3.4s, v13.4s+        ldr q12, [x1, #0x50]+        ldr q17, [x1, #0x40]+        trn2 v29.2d, v10.2d, v6.2d+        trn2 v14.2d, v23.2d, v18.2d+        trn1 v10.2d, v10.2d, v6.2d+        trn1 v26.2d, v23.2d, v18.2d+        sub v3.4s, v29.4s, v14.4s+        trn1 v13.4s, v21.4s, v8.4s+        ldr q6, [x1, #0x10]+        add v24.4s, v10.4s, v26.4s+        sub v30.4s, v10.4s, v26.4s+        sqrdmulh v12.4s, v3.4s, v12.4s+        ldr q9, [x1, #0x20]+        sqrdmulh v5.4s, v30.4s, v2.4s+        mul v4.4s, v3.4s, v17.4s+        ldr q3, [x3, #0x50]+        add v10.4s, v29.4s, v14.4s+        ldr q26, [x3, #0x40]+        mul v15.4s, v30.4s, v9.4s+        trn2 v30.4s, v21.4s, v8.4s+        sub v21.4s, v24.4s, v10.4s+        add v29.4s, v24.4s, v10.4s+        mls v15.4s, v5.4s, v31.s[0]+        ldr q16, [x1], #0x60+        trn2 v10.4s, v26.4s, v3.4s+        mls v4.4s, v12.4s, v31.s[0]+        trn1 v3.4s, v26.4s, v3.4s+        ldr q0, [x1, #0x50]+        trn2 v25.2d, v10.2d, v30.2d+        sqrdmulh v12.4s, v21.4s, v6.4s+        trn2 v1.2d, v3.2d, v13.2d+        mul v21.4s, v21.4s, v16.4s+        sub v23.4s, v1.4s, v25.4s+        sub v2.4s, v15.4s, v4.4s+        add v20.4s, v15.4s, v4.4s+        sqrdmulh v4.4s, v23.4s, v0.4s+        trn1 v7.2d, v3.2d, v13.2d+        sqrdmulh v3.4s, v2.4s, v6.4s+        trn1 v15.2d, v10.2d, v30.2d+        mls v21.4s, v12.4s, v31.s[0]+        ldr q12, [x1, #0x30]+        sub v13.4s, v7.4s, v15.4s+        mul v10.4s, v2.4s, v16.4s+        ldr q16, [x1, #0x40]+        ldr q5, [x1, #0x20]+        mls v10.4s, v3.4s, v31.s[0]+        trn2 v11.4s, v29.4s, v20.4s+        sqrdmulh v2.4s, v13.4s, v12.4s+        ldr d9, [x2], #0x20+        mul v17.4s, v13.4s, v5.4s+        trn1 v24.4s, v29.4s, v20.4s+        trn1 v12.4s, v21.4s, v10.4s+        trn2 v10.4s, v21.4s, v10.4s+        mul v22.4s, v23.4s, v16.4s+        ldur q26, [x2, #-0x10]+        trn1 v13.2d, v24.2d, v12.2d+        trn1 v3.2d, v11.2d, v10.2d+        mls v17.4s, v2.4s, v31.s[0]+        mls v22.4s, v4.4s, v31.s[0]+        sub v23.4s, v13.4s, v3.4s+        trn2 v10.2d, v11.2d, v10.2d+        sqrdmulh v0.4s, v23.4s, v26.s[1]+        trn2 v21.2d, v24.2d, v12.2d+        mul v29.4s, v23.4s, v26.s[0]+        sub v23.4s, v21.4s, v10.4s+        sqrdmulh v8.4s, v23.4s, v26.s[3]+        add v6.4s, v1.4s, v25.4s+        add v11.4s, v21.4s, v10.4s+        mls v29.4s, v0.4s, v31.s[0]+        add v30.4s, v13.4s, v3.4s+        ldr q2, [x1, #0x10]+        mul v3.4s, v23.4s, v26.s[2]+        add v12.4s, v7.4s, v15.4s+        mls v3.4s, v8.4s, v31.s[0]+        sub v13.4s, v30.4s, v11.4s+        sub v21.4s, v12.4s, v6.4s+        add v26.4s, v17.4s, v22.4s+        sqrdmulh v28.4s, v13.4s, v9.s[1]+        mul v18.4s, v13.4s, v9.s[0]+        sub v8.4s, v29.4s, v3.4s+        sqrdmulh v24.4s, v21.4s, v2.4s+        add v15.4s, v30.4s, v11.4s+        add v10.4s, v29.4s, v3.4s+        add v14.4s, v12.4s, v6.4s+        sqrdmulh v16.4s, v8.4s, v9.s[1]+        sub v3.4s, v17.4s, v22.4s+        mul v8.4s, v8.4s, v9.s[0]+        ldr q17, [x1], #0x60+        sub x4, x4, #0x2++Lmld_intt_layer5678_start:+        ldr d4, [x2], #0x20+        mul v0.4s, v21.4s, v17.4s+        ldr q7, [x3, #0xa0]+        ldr q27, [x3, #0xb0]+        mls v8.4s, v16.4s, v31.s[0]+        ldr q29, [x3, #0x80]+        trn1 v9.4s, v14.4s, v26.4s+        ldr q25, [x3, #0x90]+        mls v0.4s, v24.4s, v31.s[0]+        str q15, [x3], #0x40+        trn1 v1.4s, v7.4s, v27.4s+        ldr q15, [x1, #0x50]+        trn2 v12.4s, v7.4s, v27.4s+        sqrdmulh v16.4s, v3.4s, v2.4s+        trn1 v21.4s, v29.4s, v25.4s+        stur q8, [x3, #-0x10]+        trn2 v19.4s, v29.4s, v25.4s+        mls v18.4s, v28.4s, v31.s[0]+        ldr q22, [x1, #0x30]+        trn2 v13.2d, v21.2d, v1.2d+        trn2 v5.2d, v19.2d, v12.2d+        mul v23.4s, v3.4s, v17.4s+        trn1 v17.2d, v21.2d, v1.2d+        ldr q29, [x1, #0x40]+        mls v23.4s, v16.4s, v31.s[0]+        sub v30.4s, v13.4s, v5.4s+        trn1 v24.2d, v19.2d, v12.2d+        stur q10, [x3, #-0x30]+        sqrdmulh v16.4s, v30.4s, v15.4s+        ldr q10, [x1, #0x20]+        trn2 v12.4s, v14.4s, v26.4s+        stur q18, [x3, #-0x20]+        mul v11.4s, v30.4s, v29.4s+        sub v8.4s, v17.4s, v24.4s+        trn1 v20.4s, v0.4s, v23.4s+        ldr q2, [x1, #0x10]+        trn2 v26.4s, v0.4s, v23.4s+        sqrdmulh v19.4s, v8.4s, v22.4s+        ldur q14, [x2, #-0x10]+        trn1 v21.2d, v9.2d, v20.2d+        mul v8.4s, v8.4s, v10.4s+        trn1 v6.2d, v12.2d, v26.2d+        trn2 v0.2d, v9.2d, v20.2d+        mls v11.4s, v16.4s, v31.s[0]+        sub v7.4s, v21.4s, v6.4s+        trn2 v28.2d, v12.2d, v26.2d+        add v10.4s, v21.4s, v6.4s+        mul v18.4s, v7.4s, v14.s[0]+        add v20.4s, v13.4s, v5.4s+        sqrdmulh v30.4s, v7.4s, v14.s[1]+        sub v21.4s, v0.4s, v28.4s+        add v13.4s, v0.4s, v28.4s+        add v9.4s, v17.4s, v24.4s+        sqrdmulh v1.4s, v21.4s, v14.s[3]+        ldr q17, [x1], #0x60+        mul v14.4s, v21.4s, v14.s[2]+        sub v21.4s, v9.4s, v20.4s+        mls v18.4s, v30.4s, v31.s[0]+        mls v14.4s, v1.4s, v31.s[0]+        mls v8.4s, v19.4s, v31.s[0]+        sub v3.4s, v10.4s, v13.4s+        add v15.4s, v10.4s, v13.4s+        sqrdmulh v28.4s, v3.4s, v4.s[1]+        add v10.4s, v18.4s, v14.4s+        sub v19.4s, v18.4s, v14.4s+        mul v18.4s, v3.4s, v4.s[0]+        sub v3.4s, v8.4s, v11.4s+        add v26.4s, v8.4s, v11.4s+        sqrdmulh v16.4s, v19.4s, v4.s[1]+        add v14.4s, v9.4s, v20.4s+        mul v8.4s, v19.4s, v4.s[0]+        sqrdmulh v24.4s, v21.4s, v2.4s+        subs x4, x4, #0x1+        cbnz x4, Lmld_intt_layer5678_start+        sqrdmulh v19.4s, v3.4s, v2.4s+        trn2 v0.4s, v14.4s, v26.4s+        str q15, [x3], #0x40+        trn1 v30.4s, v14.4s, v26.4s+        mul v26.4s, v21.4s, v17.4s+        stur q10, [x3, #-0x30]+        ldr d29, [x2], #0x20+        mls v26.4s, v24.4s, v31.s[0]+        mul v10.4s, v3.4s, v17.4s+        ldur q14, [x2, #-0x10]+        mls v10.4s, v19.4s, v31.s[0]+        mls v8.4s, v16.4s, v31.s[0]+        trn2 v22.4s, v26.4s, v10.4s+        trn1 v25.4s, v26.4s, v10.4s+        trn1 v1.2d, v30.2d, v25.2d+        trn1 v2.2d, v0.2d, v22.2d+        trn2 v13.2d, v30.2d, v25.2d+        trn2 v7.2d, v0.2d, v22.2d+        mls v18.4s, v28.4s, v31.s[0]+        sub v22.4s, v1.4s, v2.4s+        add v3.4s, v13.4s, v7.4s+        sqrdmulh v27.4s, v22.4s, v14.s[1]+        sub v4.4s, v13.4s, v7.4s+        stur q8, [x3, #-0x10]+        sqrdmulh v0.4s, v4.4s, v14.s[3]+        add v16.4s, v1.4s, v2.4s+        stur q18, [x3, #-0x20]+        mul v23.4s, v4.4s, v14.s[2]+        add v4.4s, v16.4s, v3.4s+        sub v17.4s, v16.4s, v3.4s+        mul v14.4s, v22.4s, v14.s[0]+        str q4, [x3], #0x40+        mls v23.4s, v0.4s, v31.s[0]+        mls v14.4s, v27.4s, v31.s[0]+        mul v11.4s, v17.4s, v29.s[0]+        sqrdmulh v10.4s, v17.4s, v29.s[1]+        sub v0.4s, v14.4s, v23.4s+        add v6.4s, v14.4s, v23.4s+        sqrdmulh v12.4s, v0.4s, v29.s[1]+        mul v0.4s, v0.4s, v29.s[0]+        stur q6, [x3, #-0x30]+        mls v11.4s, v10.4s, v31.s[0]+        mls v0.4s, v12.4s, v31.s[0]+        stur q11, [x3, #-0x20]+        stur q0, [x3, #-0x10]+        mov w5, #0x3ffe             // =16382+        dup v29.4s, w5+        mov w5, #0xe03              // =3587+        movk w5, #0x40, lsl #16+        dup v30.4s, w5+        mov x4, #0x4                // =4+        ldr q0, [x2], #0x80+        ldur q1, [x2, #-0x70]+        ldur q2, [x2, #-0x60]+        ldur q3, [x2, #-0x50]+        ldur q4, [x2, #-0x40]+        ldur q5, [x2, #-0x30]+        ldur q6, [x2, #-0x20]+        ldur q7, [x2, #-0x10]+        ldr q8, [x0, #0xc0]+        ldr q27, [x0, #0x80]+        ldr q20, [x0, #0x1c0]+        ldr q23, [x0, #0x180]+        ldr q24, [x0, #0x3c0]+        ldr q28, [x0, #0x40]+        ldr q25, [x0, #0x340]+        ldr q10, [x0, #0x380]+        sub v15.4s, v27.4s, v8.4s+        ldr q26, [x0]+        ldr q18, [x0, #0x300]+        add v9.4s, v23.4s, v20.4s+        mul v11.4s, v15.4s, v4.s[0]+        sub v19.4s, v23.4s, v20.4s+        sub v22.4s, v10.4s, v24.4s+        ldr q13, [x0, #0x240]+        sqrdmulh v12.4s, v15.4s, v4.s[1]+        sub v14.4s, v26.4s, v28.4s+        ldr q17, [x0, #0x200]+        add v23.4s, v18.4s, v25.4s+        sqrdmulh v15.4s, v14.4s, v3.s[3]+        sub v18.4s, v18.4s, v25.4s+        mul v25.4s, v14.4s, v3.s[2]+        sub v16.4s, v17.4s, v13.4s+        mls v11.4s, v12.4s, v31.s[0]+        add v21.4s, v10.4s, v24.4s+        mls v25.4s, v15.4s, v31.s[0]+        mul v12.4s, v16.4s, v5.s[2]+        sub v10.4s, v23.4s, v21.4s+        sqrdmulh v14.4s, v10.4s, v3.s[1]+        add v23.4s, v23.4s, v21.4s+        add v27.4s, v27.4s, v8.4s+        mul v15.4s, v10.4s, v3.s[0]+        add v24.4s, v25.4s, v11.4s+        sub v10.4s, v25.4s, v11.4s+        ldr q25, [x0, #0x100]+        sqrdmulh v8.4s, v18.4s, v6.s[3]+        add v21.4s, v26.4s, v28.4s+        ldr q26, [x0, #0x140]+        sqrdmulh v20.4s, v16.4s, v5.s[3]+        sqrdmulh v28.4s, v22.4s, v7.s[1]+        add v11.4s, v25.4s, v26.4s+        mls v15.4s, v14.4s, v31.s[0]+        add v17.4s, v17.4s, v13.4s+        mul v13.4s, v22.4s, v7.s[0]+        ldr q14, [x0, #0x280]+        ldr q22, [x0, #0x2c0]+        mls v13.4s, v28.4s, v31.s[0]+        sub v16.4s, v25.4s, v26.4s+        sub v28.4s, v21.4s, v27.4s+        mls v12.4s, v20.4s, v31.s[0]+        add v20.4s, v11.4s, v9.4s+        add v26.4s, v14.4s, v22.4s+        sub v14.4s, v14.4s, v22.4s+        sqrdmulh v22.4s, v28.4s, v1.s[3]+        sub v9.4s, v11.4s, v9.4s+        mul v25.4s, v14.4s, v6.s[0]+        sub v11.4s, v17.4s, v26.4s+        add v17.4s, v17.4s, v26.4s+        add v26.4s, v21.4s, v27.4s+        sqrdmulh v27.4s, v11.4s, v2.s[3]+        mul v21.4s, v11.4s, v2.s[2]+        mul v11.4s, v18.4s, v6.s[2]+        mls v11.4s, v8.4s, v31.s[0]+        sqrdmulh v8.4s, v14.4s, v6.s[1]+        add v18.4s, v26.4s, v20.4s+        mls v21.4s, v27.4s, v31.s[0]+        sub v27.4s, v17.4s, v23.4s+        mul v14.4s, v28.4s, v1.s[2]+        sub v28.4s, v26.4s, v20.4s+        add v23.4s, v17.4s, v23.4s+        mls v25.4s, v8.4s, v31.s[0]+        add v26.4s, v21.4s, v15.4s+        sub v15.4s, v21.4s, v15.4s+        mul v8.4s, v19.4s, v5.s[0]+        mls v14.4s, v22.4s, v31.s[0]+        sub v22.4s, v11.4s, v13.4s+        sub v20.4s, v12.4s, v25.4s+        add v21.4s, v12.4s, v25.4s+        sqrdmulh v12.4s, v22.4s, v3.s[1]+        mul v17.4s, v16.4s, v4.s[2]+        sqrdmulh v25.4s, v16.4s, v4.s[3]+        add v16.4s, v11.4s, v13.4s+        mul v11.4s, v22.4s, v3.s[0]+        sqrdmulh v19.4s, v19.4s, v5.s[1]+        sqrdmulh v22.4s, v15.4s, v1.s[1]+        sqrdmulh v13.4s, v20.4s, v2.s[3]+        mul v20.4s, v20.4s, v2.s[2]+        mls v11.4s, v12.4s, v31.s[0]+        mls v20.4s, v13.4s, v31.s[0]+        sub v13.4s, v18.4s, v23.4s+        add v23.4s, v18.4s, v23.4s+        mls v17.4s, v25.4s, v31.s[0]+        mls v8.4s, v19.4s, v31.s[0]+        sub v18.4s, v20.4s, v11.4s+        add v20.4s, v20.4s, v11.4s+        sqrdmulh v12.4s, v9.4s, v2.s[1]+        mul v15.4s, v15.4s, v1.s[0]+        sub v11.4s, v17.4s, v8.4s+        mls v15.4s, v22.4s, v31.s[0]+        add v19.4s, v17.4s, v8.4s+        sqrdmulh v25.4s, v11.4s, v2.s[1]+        add v17.4s, v24.4s, v19.4s+        sub v24.4s, v24.4s, v19.4s+        mul v19.4s, v11.4s, v2.s[0]+        sqrdmulh v8.4s, v24.4s, v0.s[3]+        mls v19.4s, v25.4s, v31.s[0]+        mul v25.4s, v9.4s, v2.s[0]+        sub x4, x4, #0x1++Lmld_intt_layer1234_start:+        sub v22.4s, v21.4s, v16.4s+        mls v25.4s, v12.4s, v31.s[0]+        add v12.4s, v21.4s, v16.4s+        mul v9.4s, v24.4s, v0.s[2]+        mls v9.4s, v8.4s, v31.s[0]+        sub v16.4s, v14.4s, v25.4s+        add v14.4s, v14.4s, v25.4s+        sqrdmulh v24.4s, v22.4s, v1.s[1]+        sub v11.4s, v14.4s, v26.4s+        sqrdmulh v21.4s, v16.4s, v0.s[3]+        add v25.4s, v14.4s, v26.4s+        mul v14.4s, v22.4s, v1.s[0]+        mls v14.4s, v24.4s, v31.s[0]+        mul v16.4s, v16.4s, v0.s[2]+        mls v16.4s, v21.4s, v31.s[0]+        add v24.4s, v9.4s, v14.4s+        sqrdmulh v21.4s, v25.4s, v30.4s+        sqrdmulh v26.4s, v11.4s, v0.s[1]+        mul v8.4s, v25.4s, v29.4s+        mls v8.4s, v21.4s, v31.s[0]+        mul v25.4s, v24.4s, v29.4s+        mul v22.4s, v11.4s, v0.s[0]+        sub v11.4s, v17.4s, v12.4s+        str q8, [x0, #0x80]+        sub v8.4s, v9.4s, v14.4s+        add v14.4s, v16.4s, v15.4s+        sqrdmulh v9.4s, v11.4s, v0.s[1]+        sub v21.4s, v16.4s, v15.4s+        mls v22.4s, v26.4s, v31.s[0]+        mul v15.4s, v11.4s, v0.s[0]+        mls v15.4s, v9.4s, v31.s[0]+        add v11.4s, v17.4s, v12.4s+        mul v16.4s, v10.4s, v1.s[2]+        str q22, [x0, #0x280]+        sqrdmulh v26.4s, v10.4s, v1.s[3]+        str q15, [x0, #0x240]+        sqrdmulh v9.4s, v8.4s, v0.s[1]+        mul v12.4s, v8.4s, v0.s[0]+        mls v16.4s, v26.4s, v31.s[0]+        sqrdmulh v10.4s, v27.4s, v1.s[1]+        sqrdmulh v15.4s, v14.4s, v30.4s+        add v22.4s, v16.4s, v19.4s+        sub v19.4s, v16.4s, v19.4s+        mls v12.4s, v9.4s, v31.s[0]+        add v8.4s, v22.4s, v20.4s+        sub v26.4s, v22.4s, v20.4s+        mul v27.4s, v27.4s, v1.s[0]+        sqrdmulh v17.4s, v11.4s, v30.4s+        mls v27.4s, v10.4s, v31.s[0]+        mul v20.4s, v11.4s, v29.4s+        sqrdmulh v22.4s, v24.4s, v30.4s+        mls v20.4s, v17.4s, v31.s[0]+        mul v14.4s, v14.4s, v29.4s+        sqrdmulh v16.4s, v23.4s, v30.4s+        str q20, [x0, #0x40]+        sqrdmulh v11.4s, v21.4s, v0.s[1]+        sqrdmulh v20.4s, v28.4s, v0.s[3]+        mul v17.4s, v23.4s, v29.4s+        mul v23.4s, v28.4s, v0.s[2]+        mls v17.4s, v16.4s, v31.s[0]+        mul v10.4s, v21.4s, v0.s[0]+        str q12, [x0, #0x340]+        mls v25.4s, v22.4s, v31.s[0]+        str q17, [x0], #0x10+        ldr q17, [x0, #0x2c0]+        mls v14.4s, v15.4s, v31.s[0]+        ldr q12, [x0, #0x200]+        sqrdmulh v16.4s, v13.4s, v0.s[1]+        str q25, [x0, #0x130]+        mul v28.4s, v13.4s, v0.s[0]+        ldr q13, [x0, #0x240]+        str q14, [x0, #0x170]+        mul v14.4s, v18.4s, v1.s[0]+        mls v28.4s, v16.4s, v31.s[0]+        ldr q21, [x0, #0xc0]+        mls v10.4s, v11.4s, v31.s[0]+        mls v23.4s, v20.4s, v31.s[0]+        str q28, [x0, #0x1f0]+        ldr q11, [x0, #0x300]+        sqrdmulh v28.4s, v18.4s, v1.s[1]+        str q10, [x0, #0x370]+        mul v16.4s, v19.4s, v0.s[2]+        sqrdmulh v19.4s, v19.4s, v0.s[3]+        sqrdmulh v10.4s, v26.4s, v0.s[1]+        add v18.4s, v12.4s, v13.4s+        sub v12.4s, v12.4s, v13.4s+        mls v16.4s, v19.4s, v31.s[0]+        ldr q9, [x0, #0x280]+        ldr q20, [x0, #0x380]+        add v24.4s, v23.4s, v27.4s+        sub v25.4s, v23.4s, v27.4s+        mul v19.4s, v26.4s, v0.s[0]+        mls v14.4s, v28.4s, v31.s[0]+        add v13.4s, v9.4s, v17.4s+        sub v28.4s, v18.4s, v13.4s+        mls v19.4s, v10.4s, v31.s[0]+        sqrdmulh v23.4s, v25.4s, v0.s[1]+        mul v15.4s, v25.4s, v0.s[0]+        str q19, [x0, #0x2b0]+        ldr q19, [x0, #0x80]+        sqrdmulh v22.4s, v8.4s, v30.4s+        mls v15.4s, v23.4s, v31.s[0]+        add v26.4s, v16.4s, v14.4s+        mul v10.4s, v8.4s, v29.4s+        add v8.4s, v18.4s, v13.4s+        sqrdmulh v13.4s, v26.4s, v30.4s+        str q15, [x0, #0x2f0]+        sub v18.4s, v9.4s, v17.4s+        mul v27.4s, v26.4s, v29.4s+        sqrdmulh v23.4s, v12.4s, v5.s[3]+        mls v27.4s, v13.4s, v31.s[0]+        mul v25.4s, v18.4s, v6.s[0]+        sqrdmulh v13.4s, v18.4s, v6.s[1]+        str q27, [x0, #0x1b0]+        sub v27.4s, v19.4s, v21.4s+        add v21.4s, v19.4s, v21.4s+        mul v18.4s, v12.4s, v5.s[2]+        sub v9.4s, v16.4s, v14.4s+        mul v26.4s, v24.4s, v29.4s+        ldr q12, [x0, #0x3c0]+        mls v25.4s, v13.4s, v31.s[0]+        sub v16.4s, v20.4s, v12.4s+        sqrdmulh v17.4s, v24.4s, v30.4s+        mul v13.4s, v16.4s, v7.s[0]+        mls v18.4s, v23.4s, v31.s[0]+        mls v26.4s, v17.4s, v31.s[0]+        sqrdmulh v17.4s, v9.4s, v0.s[1]+        sqrdmulh v24.4s, v16.4s, v7.s[1]+        ldr q16, [x0, #0x340]+        str q26, [x0, #0xf0]+        sub v15.4s, v18.4s, v25.4s+        mul v23.4s, v9.4s, v0.s[0]+        ldr q19, [x0, #0x180]+        ldr q14, [x0, #0x1c0]+        sqrdmulh v9.4s, v15.4s, v2.s[3]+        add v20.4s, v20.4s, v12.4s+        add v12.4s, v11.4s, v16.4s+        mul v26.4s, v15.4s, v2.s[2]+        sub v16.4s, v11.4s, v16.4s+        sub v11.4s, v12.4s, v20.4s+        mls v23.4s, v17.4s, v31.s[0]+        add v20.4s, v12.4s, v20.4s+        sub v12.4s, v19.4s, v14.4s+        mul v17.4s, v27.4s, v4.s[0]+        mls v10.4s, v22.4s, v31.s[0]+        add v22.4s, v19.4s, v14.4s+        ldr q14, [x0, #0x40]+        str q23, [x0, #0x3b0]+        sqrdmulh v23.4s, v27.4s, v4.s[1]+        ldr q19, [x0, #0x100]+        ldr q15, [x0, #0x140]+        mls v26.4s, v9.4s, v31.s[0]+        str q10, [x0, #0xb0]+        sqrdmulh v10.4s, v11.4s, v3.s[1]+        sub v27.4s, v8.4s, v20.4s+        mls v13.4s, v24.4s, v31.s[0]+        add v8.4s, v8.4s, v20.4s+        sub v20.4s, v19.4s, v15.4s+        sqrdmulh v9.4s, v16.4s, v6.s[3]+        add v19.4s, v19.4s, v15.4s+        ldr q15, [x0]+        mul v24.4s, v20.4s, v4.s[2]+        mls v17.4s, v23.4s, v31.s[0]+        add v23.4s, v15.4s, v14.4s+        sub v14.4s, v15.4s, v14.4s+        mul v15.4s, v11.4s, v3.s[0]+        sub v11.4s, v23.4s, v21.4s+        mul v16.4s, v16.4s, v6.s[2]+        add v23.4s, v23.4s, v21.4s+        add v21.4s, v18.4s, v25.4s+        sqrdmulh v25.4s, v20.4s, v4.s[3]+        sqrdmulh v18.4s, v12.4s, v5.s[1]+        sqrdmulh v20.4s, v14.4s, v3.s[3]+        mul v12.4s, v12.4s, v5.s[0]+        mls v16.4s, v9.4s, v31.s[0]+        mls v12.4s, v18.4s, v31.s[0]+        mls v24.4s, v25.4s, v31.s[0]+        sub v25.4s, v16.4s, v13.4s+        add v16.4s, v16.4s, v13.4s+        mul v13.4s, v28.4s, v2.s[2]+        sqrdmulh v9.4s, v25.4s, v3.s[1]+        mul v18.4s, v25.4s, v3.s[0]+        sqrdmulh v25.4s, v28.4s, v2.s[3]+        mls v18.4s, v9.4s, v31.s[0]+        mls v15.4s, v10.4s, v31.s[0]+        sub v9.4s, v24.4s, v12.4s+        mul v28.4s, v14.4s, v3.s[2]+        add v12.4s, v24.4s, v12.4s+        mls v28.4s, v20.4s, v31.s[0]+        add v20.4s, v26.4s, v18.4s+        add v24.4s, v19.4s, v22.4s+        mls v13.4s, v25.4s, v31.s[0]+        sub v18.4s, v26.4s, v18.4s+        sqrdmulh v26.4s, v11.4s, v1.s[3]+        add v25.4s, v23.4s, v24.4s+        mul v14.4s, v11.4s, v1.s[2]+        sub v10.4s, v28.4s, v17.4s+        add v17.4s, v28.4s, v17.4s+        sub v28.4s, v23.4s, v24.4s+        sqrdmulh v11.4s, v9.4s, v2.s[1]+        sub v22.4s, v19.4s, v22.4s+        sub v24.4s, v17.4s, v12.4s+        mls v14.4s, v26.4s, v31.s[0]+        add v17.4s, v17.4s, v12.4s+        sqrdmulh v12.4s, v22.4s, v2.s[1]+        sub v23.4s, v13.4s, v15.4s+        add v26.4s, v13.4s, v15.4s+        mul v19.4s, v9.4s, v2.s[0]+        sqrdmulh v9.4s, v23.4s, v1.s[1]+        mul v15.4s, v23.4s, v1.s[0]+        mls v19.4s, v11.4s, v31.s[0]+        sub v13.4s, v25.4s, v8.4s+        mls v15.4s, v9.4s, v31.s[0]+        add v23.4s, v25.4s, v8.4s+        sqrdmulh v8.4s, v24.4s, v0.s[3]+        mul v25.4s, v22.4s, v2.s[0]+        subs x4, x4, #0x1+        cbnz x4, Lmld_intt_layer1234_start+        mul v22.4s, v24.4s, v0.s[2]+        sqrdmulh v9.4s, v23.4s, v30.4s+        mls v25.4s, v12.4s, v31.s[0]+        mul v23.4s, v23.4s, v29.4s+        mls v23.4s, v9.4s, v31.s[0]+        mls v22.4s, v8.4s, v31.s[0]+        sqrdmulh v11.4s, v10.4s, v1.s[3]+        sub v9.4s, v14.4s, v25.4s+        str q23, [x0], #0x10+        mul v23.4s, v10.4s, v1.s[2]+        sqrdmulh v24.4s, v9.4s, v0.s[3]+        mls v23.4s, v11.4s, v31.s[0]+        mul v9.4s, v9.4s, v0.s[2]+        mls v9.4s, v24.4s, v31.s[0]+        sub v24.4s, v23.4s, v19.4s+        add v19.4s, v23.4s, v19.4s+        mul v8.4s, v28.4s, v0.s[2]+        add v23.4s, v14.4s, v25.4s+        mul v11.4s, v13.4s, v0.s[0]+        sub v25.4s, v21.4s, v16.4s+        sub v10.4s, v9.4s, v15.4s+        add v15.4s, v9.4s, v15.4s+        add v16.4s, v21.4s, v16.4s+        sqrdmulh v28.4s, v28.4s, v0.s[3]+        add v9.4s, v23.4s, v26.4s+        add v14.4s, v17.4s, v16.4s+        sqrdmulh v21.4s, v13.4s, v0.s[1]+        sub v13.4s, v17.4s, v16.4s+        mul v16.4s, v14.4s, v29.4s+        sub v12.4s, v23.4s, v26.4s+        mul v17.4s, v13.4s, v0.s[0]+        mls v8.4s, v28.4s, v31.s[0]+        sqrdmulh v28.4s, v14.4s, v30.4s+        sub v14.4s, v19.4s, v20.4s+        add v20.4s, v19.4s, v20.4s+        sqrdmulh v19.4s, v13.4s, v0.s[1]+        sqrdmulh v26.4s, v27.4s, v1.s[1]+        mls v16.4s, v28.4s, v31.s[0]+        sqrdmulh v28.4s, v12.4s, v0.s[1]+        mul v27.4s, v27.4s, v1.s[0]+        str q16, [x0, #0x30]+        mul v12.4s, v12.4s, v0.s[0]+        mls v12.4s, v28.4s, v31.s[0]+        mls v27.4s, v26.4s, v31.s[0]+        sqrdmulh v26.4s, v25.4s, v1.s[1]+        str q12, [x0, #0x270]+        sqrdmulh v12.4s, v14.4s, v0.s[1]+        sub v13.4s, v8.4s, v27.4s+        add v27.4s, v8.4s, v27.4s+        mul v16.4s, v14.4s, v0.s[0]+        mul v8.4s, v20.4s, v29.4s+        mls v16.4s, v12.4s, v31.s[0]+        mul v12.4s, v18.4s, v1.s[0]+        sqrdmulh v14.4s, v20.4s, v30.4s+        str q16, [x0, #0x2b0]+        sqrdmulh v20.4s, v18.4s, v1.s[1]+        mul v25.4s, v25.4s, v1.s[0]+        mls v25.4s, v26.4s, v31.s[0]+        mls v17.4s, v19.4s, v31.s[0]+        mul v18.4s, v9.4s, v29.4s+        add v28.4s, v22.4s, v25.4s+        sqrdmulh v9.4s, v9.4s, v30.4s+        sub v23.4s, v22.4s, v25.4s+        str q17, [x0, #0x230]+        mls v11.4s, v21.4s, v31.s[0]+        mls v8.4s, v14.4s, v31.s[0]+        sqrdmulh v21.4s, v28.4s, v30.4s+        str q11, [x0, #0x1f0]+        mul v17.4s, v28.4s, v29.4s+        str q8, [x0, #0xb0]+        mul v28.4s, v10.4s, v0.s[0]+        mls v17.4s, v21.4s, v31.s[0]+        sqrdmulh v21.4s, v10.4s, v0.s[1]+        mul v8.4s, v15.4s, v29.4s+        str q17, [x0, #0x130]+        sqrdmulh v15.4s, v15.4s, v30.4s+        mul v10.4s, v24.4s, v0.s[2]+        sqrdmulh v24.4s, v24.4s, v0.s[3]+        mls v12.4s, v20.4s, v31.s[0]+        mls v8.4s, v15.4s, v31.s[0]+        mls v10.4s, v24.4s, v31.s[0]+        sqrdmulh v24.4s, v27.4s, v30.4s+        str q8, [x0, #0x170]+        mul v14.4s, v13.4s, v0.s[0]+        sub v16.4s, v10.4s, v12.4s+        sqrdmulh v17.4s, v13.4s, v0.s[1]+        add v10.4s, v10.4s, v12.4s+        mul v13.4s, v27.4s, v29.4s+        sqrdmulh v20.4s, v10.4s, v30.4s+        mls v14.4s, v17.4s, v31.s[0]+        mls v13.4s, v24.4s, v31.s[0]+        sqrdmulh v24.4s, v23.4s, v0.s[1]+        str q14, [x0, #0x2f0]+        sqrdmulh v8.4s, v16.4s, v0.s[1]+        str q13, [x0, #0xf0]+        mls v18.4s, v9.4s, v31.s[0]+        mul v9.4s, v23.4s, v0.s[0]+        mul v14.4s, v16.4s, v0.s[0]+        str q18, [x0, #0x70]+        mul v27.4s, v10.4s, v29.4s+        mls v28.4s, v21.4s, v31.s[0]+        mls v14.4s, v8.4s, v31.s[0]+        mls v27.4s, v20.4s, v31.s[0]+        str q28, [x0, #0x370]+        mls v9.4s, v24.4s, v31.s[0]+        str q14, [x0, #0x3b0]+        str q27, [x0, #0x1b0]+        str q9, [x0, #0x330]+        ldp d8, d9, [sp]+        .cfi_restore d8+        .cfi_restore d9+        ldp d10, d11, [sp, #0x10]+        .cfi_restore d10+        .cfi_restore d11+        ldp d12, d13, [sp, #0x20]+        .cfi_restore d12+        .cfi_restore d13+        ldp d14, d15, [sp, #0x30]+        .cfi_restore d14+        .cfi_restore d15+        add sp, sp, #0x40+        .cfi_adjust_cfa_offset -0x40+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(intt_aarch64_asm)++#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/aarch64/src/mldsa_ntt_aarch64_asm.S view
@@ -0,0 +1,686 @@+/* Copyright (c) 2022 Arm Limited+ * Copyright (c) 2022 Hanno Becker+ * Copyright (c) 2023 Amin Abdulrahman, Matthias Kannwischer+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [NeonNTT]+ *   Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1+ *   Becker, Hwang, Kannwischer, Yang, Yang+ *   https://eprint.iacr.org/2021/986+ *+ * - [NeonNTT_Autoformalised]+ *   Neon NTT - (Auto)formalised+ *   Hanno Becker+ *   https://eprint.iacr.org/2026/1223+ *+ * - [SLOTHY_Paper]+ *   Fast and Clean: Auditable high-performance assembly via constraint solving+ *   Abdulrahman, Becker, Kannwischer, Klein+ *   https://eprint.iacr.org/2022/1303+ */++/* AArch64 ML-DSA forward NTT following @[NeonNTT], @[SLOTHY_Paper], and @[NeonNTT_Autoformalised] */++/*yaml+  Name: ntt_aarch64_asm+  Description: AArch64 ML-DSA forward NTT+  Signature: void mld_ntt_aarch64_asm(int32_t r[256], const int32_t zetas_l123456[144], const int32_t zetas_l78[384])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 1024+      permissions: read/write+      c_parameter: int32_t r[256]+      description: Input/output polynomial (256 x int32_t)+    x1:+      type: buffer+      size_bytes: 576+      permissions: read-only+      c_parameter: const int32_t zetas_l123456[144]+      description: Twiddle factors for layers 1-6 (144 x int32_t)+    x2:+      type: buffer+      size_bytes: 1536+      permissions: read-only+      c_parameter: const int32_t zetas_l78[384]+      description: Twiddle factors for layers 7-8 (384 x int32_t)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_AARCH64) &&  !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/aarch64_opt/src/mldsa_ntt_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(ntt_aarch64_asm)+MLD_ASM_FN_SYMBOL(ntt_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0x40+        .cfi_adjust_cfa_offset 0x40+        stp d8, d9, [sp]+        .cfi_rel_offset d8, 0x0+        .cfi_rel_offset d9, 0x8+        stp d10, d11, [sp, #0x10]+        .cfi_rel_offset d10, 0x10+        .cfi_rel_offset d11, 0x18+        stp d12, d13, [sp, #0x20]+        .cfi_rel_offset d12, 0x20+        .cfi_rel_offset d13, 0x28+        stp d14, d15, [sp, #0x30]+        .cfi_rel_offset d14, 0x30+        .cfi_rel_offset d15, 0x38+        mov w5, #0xe001             // =57345+        movk w5, #0x7f, lsl #16+        dup v7.4s, w5+        mov x3, x0+        mov x4, #0x8                // =8+        ldr q0, [x1], #0x40+        ldur q1, [x1, #-0x30]+        ldur q2, [x1, #-0x20]+        ldur q3, [x1, #-0x10]+        ldr q23, [x0, #0x390]+        ldr q13, [x0, #0x380]+        ldr q22, [x0, #0x80]+        ldr q26, [x0, #0x190]+        ldr q8, [x0, #0x280]+        ldr q6, [x0, #0x210]+        mul v10.4s, v13.4s, v0.s[0]+        sqrdmulh v13.4s, v13.4s, v0.s[1]+        mul v12.4s, v8.4s, v0.s[0]+        sqrdmulh v27.4s, v8.4s, v0.s[1]+        mul v4.4s, v6.4s, v0.s[0]+        mls v10.4s, v13.4s, v7.s[0]+        ldr q13, [x0, #0x180]+        sqrdmulh v14.4s, v23.4s, v0.s[1]+        mls v12.4s, v27.4s, v7.s[0]+        add v31.4s, v13.4s, v10.4s+        sub v13.4s, v13.4s, v10.4s+        mul v10.4s, v23.4s, v0.s[0]+        sqrdmulh v8.4s, v13.4s, v1.s[1]+        sub v18.4s, v22.4s, v12.4s+        mls v10.4s, v14.4s, v7.s[0]+        mul v13.4s, v13.4s, v1.s[0]+        mls v13.4s, v8.4s, v7.s[0]+        sub v29.4s, v26.4s, v10.4s+        add v25.4s, v26.4s, v10.4s+        mul v10.4s, v31.4s, v0.s[2]+        mul v14.4s, v25.4s, v0.s[2]+        add v17.4s, v18.4s, v13.4s+        sub v15.4s, v18.4s, v13.4s+        sqrdmulh v13.4s, v31.4s, v0.s[3]+        sqrdmulh v20.4s, v15.4s, v3.s[1]+        sqrdmulh v5.4s, v17.4s, v2.s[3]+        mls v10.4s, v13.4s, v7.s[0]+        ldr q13, [x0, #0x300]+        mul v18.4s, v17.4s, v2.s[2]+        add v31.4s, v22.4s, v12.4s+        mul v23.4s, v15.4s, v3.s[0]+        ldr q17, [x0, #0x90]+        add v19.4s, v31.4s, v10.4s+        sub v16.4s, v31.4s, v10.4s+        mul v10.4s, v13.4s, v0.s[0]+        sqrdmulh v13.4s, v13.4s, v0.s[1]+        sqrdmulh v27.4s, v16.4s, v2.s[1]+        mul v11.4s, v16.4s, v2.s[0]+        mls v10.4s, v13.4s, v7.s[0]+        ldr q13, [x0, #0x290]+        ldr q22, [x0, #0x100]+        mls v11.4s, v27.4s, v7.s[0]+        sqrdmulh v15.4s, v13.4s, v0.s[1]+        sub v12.4s, v22.4s, v10.4s+        add v30.4s, v22.4s, v10.4s+        mul v10.4s, v13.4s, v0.s[0]+        ldr q28, [x0]+        sqrdmulh v13.4s, v25.4s, v0.s[3]+        sqrdmulh v27.4s, v30.4s, v0.s[3]+        mls v10.4s, v15.4s, v7.s[0]+        mls v14.4s, v13.4s, v7.s[0]+        ldr q13, [x0, #0x200]+        sqrdmulh v25.4s, v12.4s, v1.s[1]+        add v24.4s, v17.4s, v10.4s+        sub v21.4s, v17.4s, v10.4s+        sqrdmulh v8.4s, v13.4s, v0.s[1]+        sub v9.4s, v24.4s, v14.4s+        mul v26.4s, v12.4s, v1.s[0]+        mul v13.4s, v13.4s, v0.s[0]+        mls v13.4s, v8.4s, v7.s[0]+        mul v8.4s, v30.4s, v0.s[2]+        mls v8.4s, v27.4s, v7.s[0]+        add v16.4s, v28.4s, v13.4s+        sub v10.4s, v28.4s, v13.4s+        mls v26.4s, v25.4s, v7.s[0]+        sqrdmulh v12.4s, v19.4s, v1.s[3]+        sub v25.4s, v16.4s, v8.4s+        mls v23.4s, v20.4s, v7.s[0]+        sub v22.4s, v25.4s, v11.4s+        sqrdmulh v20.4s, v9.4s, v2.s[1]+        sub v15.4s, v10.4s, v26.4s+        sub x4, x4, #0x2++Lmld_ntt_layer123_start:+        add v31.4s, v10.4s, v26.4s+        mul v17.4s, v19.4s, v1.s[2]+        add v26.4s, v15.4s, v23.4s+        ldr q30, [x0, #0x2a0]+        sub v13.4s, v15.4s, v23.4s+        mul v23.4s, v29.4s, v1.s[0]+        add v25.4s, v25.4s, v11.4s+        str q22, [x0, #0x180]+        mul v11.4s, v9.4s, v2.s[0]+        str q13, [x0, #0x380]+        ldr q28, [x0, #0x10]+        add v10.4s, v16.4s, v8.4s+        mls v17.4s, v12.4s, v7.s[0]+        ldr q13, [x0, #0x3a0]+        str q26, [x0, #0x300]+        sqrdmulh v27.4s, v30.4s, v0.s[1]+        mls v18.4s, v5.4s, v7.s[0]+        ldr q9, [x0, #0x1a0]+        sub v16.4s, v10.4s, v17.4s+        add v15.4s, v10.4s, v17.4s+        sqrdmulh v10.4s, v6.4s, v0.s[1]+        str q16, [x0, #0x80]+        str q15, [x0], #0x10+        sqrdmulh v19.4s, v13.4s, v0.s[1]+        sub v15.4s, v31.4s, v18.4s+        mul v8.4s, v13.4s, v0.s[0]+        add v26.4s, v31.4s, v18.4s+        str q15, [x0, #0x270]+        sqrdmulh v13.4s, v29.4s, v1.s[1]+        str q26, [x0, #0x1f0]+        mls v8.4s, v19.4s, v7.s[0]+        mls v11.4s, v20.4s, v7.s[0]+        mls v23.4s, v13.4s, v7.s[0]+        add v22.4s, v9.4s, v8.4s+        ldr q6, [x0, #0x210]+        sub v29.4s, v9.4s, v8.4s+        mul v17.4s, v30.4s, v0.s[0]+        ldr q9, [x0, #0x300]+        sqrdmulh v13.4s, v22.4s, v0.s[3]+        add v18.4s, v21.4s, v23.4s+        mls v4.4s, v10.4s, v7.s[0]+        sub v31.4s, v21.4s, v23.4s+        sqrdmulh v16.4s, v31.4s, v3.s[1]+        add v19.4s, v24.4s, v14.4s+        mul v14.4s, v22.4s, v0.s[2]+        sub v10.4s, v28.4s, v4.4s+        mls v14.4s, v13.4s, v7.s[0]+        ldr q13, [x0, #0x100]+        sqrdmulh v22.4s, v9.4s, v0.s[1]+        mul v8.4s, v9.4s, v0.s[0]+        mul v23.4s, v31.4s, v3.s[0]+        mls v8.4s, v22.4s, v7.s[0]+        mls v23.4s, v16.4s, v7.s[0]+        add v16.4s, v28.4s, v4.4s+        ldr q22, [x0, #0x90]+        mul v4.4s, v6.4s, v0.s[0]+        mls v17.4s, v27.4s, v7.s[0]+        add v21.4s, v13.4s, v8.4s+        sub v27.4s, v13.4s, v8.4s+        sqrdmulh v31.4s, v21.4s, v0.s[3]+        str q25, [x0, #0xf0]+        mul v8.4s, v21.4s, v0.s[2]+        add v24.4s, v22.4s, v17.4s+        sub v21.4s, v22.4s, v17.4s+        sqrdmulh v5.4s, v18.4s, v2.s[3]+        mls v8.4s, v31.4s, v7.s[0]+        sub v9.4s, v24.4s, v14.4s+        sqrdmulh v20.4s, v27.4s, v1.s[1]+        mul v26.4s, v27.4s, v1.s[0]+        sub v25.4s, v16.4s, v8.4s+        mul v18.4s, v18.4s, v2.s[2]+        sub v22.4s, v25.4s, v11.4s+        mls v26.4s, v20.4s, v7.s[0]+        sqrdmulh v20.4s, v9.4s, v2.s[1]+        sqrdmulh v12.4s, v19.4s, v1.s[3]+        sub v15.4s, v10.4s, v26.4s+        subs x4, x4, #0x1+        cbnz x4, Lmld_ntt_layer123_start+        add v13.4s, v10.4s, v26.4s+        mls v18.4s, v5.4s, v7.s[0]+        str q22, [x0, #0x180]+        add v27.4s, v16.4s, v8.4s+        mul v22.4s, v19.4s, v1.s[2]+        add v26.4s, v24.4s, v14.4s+        ldr q31, [x0, #0x110]+        sub v14.4s, v15.4s, v23.4s+        add v17.4s, v15.4s, v23.4s+        mls v22.4s, v12.4s, v7.s[0]+        add v28.4s, v13.4s, v18.4s+        str q14, [x0, #0x380]+        sqrdmulh v24.4s, v6.4s, v0.s[1]+        add v5.4s, v25.4s, v11.4s+        sub v19.4s, v13.4s, v18.4s+        str q17, [x0, #0x300]+        str q5, [x0, #0x100]+        mul v16.4s, v9.4s, v2.s[0]+        ldr q18, [x0, #0x310]+        str q19, [x0, #0x280]+        mls v16.4s, v20.4s, v7.s[0]+        str q28, [x0, #0x200]+        add v13.4s, v27.4s, v22.4s+        ldr q15, [x0, #0x10]+        sub v10.4s, v27.4s, v22.4s+        mls v4.4s, v24.4s, v7.s[0]+        str q13, [x0], #0x10+        str q10, [x0, #0x70]+        sqrdmulh v12.4s, v29.4s, v1.s[1]+        mul v23.4s, v29.4s, v1.s[0]+        mul v8.4s, v26.4s, v1.s[2]+        add v20.4s, v15.4s, v4.4s+        sub v6.4s, v15.4s, v4.4s+        mls v23.4s, v12.4s, v7.s[0]+        sqrdmulh v22.4s, v18.4s, v0.s[1]+        mul v5.4s, v18.4s, v0.s[0]+        sub v28.4s, v21.4s, v23.4s+        sqrdmulh v10.4s, v26.4s, v1.s[3]+        mls v5.4s, v22.4s, v7.s[0]+        sqrdmulh v30.4s, v28.4s, v3.s[1]+        add v4.4s, v21.4s, v23.4s+        mls v8.4s, v10.4s, v7.s[0]+        add v12.4s, v31.4s, v5.4s+        sub v9.4s, v31.4s, v5.4s+        sqrdmulh v25.4s, v4.4s, v2.s[3]+        sqrdmulh v15.4s, v9.4s, v1.s[1]+        sqrdmulh v31.4s, v12.4s, v0.s[3]+        mul v18.4s, v12.4s, v0.s[2]+        mul v11.4s, v9.4s, v1.s[0]+        mls v18.4s, v31.4s, v7.s[0]+        mul v29.4s, v4.4s, v2.s[2]+        mls v29.4s, v25.4s, v7.s[0]+        add v23.4s, v20.4s, v18.4s+        mls v11.4s, v15.4s, v7.s[0]+        sub v31.4s, v20.4s, v18.4s+        add v17.4s, v23.4s, v8.4s+        add v5.4s, v31.4s, v16.4s+        mul v24.4s, v28.4s, v3.s[0]+        str q17, [x0], #0x10+        sub v19.4s, v31.4s, v16.4s+        mls v24.4s, v30.4s, v7.s[0]+        str q5, [x0, #0xf0]+        add v31.4s, v6.4s, v11.4s+        sub v26.4s, v23.4s, v8.4s+        str q19, [x0, #0x170]+        add v4.4s, v31.4s, v29.4s+        sub v13.4s, v6.4s, v11.4s+        str q26, [x0, #0x70]+        sub v11.4s, v31.4s, v29.4s+        sub v22.4s, v13.4s, v24.4s+        add v23.4s, v13.4s, v24.4s+        str q4, [x0, #0x1f0]+        str q11, [x0, #0x270]+        str q23, [x0, #0x2f0]+        str q22, [x0, #0x370]+        mov x0, x3+        mov x4, #0x8                // =8+        ldr q9, [x0, #0x40]+        ldr q23, [x1], #0x40+        ldr q21, [x2, #0x60]+        ldr q1, [x0, #0x20]+        ldur q14, [x1, #-0x30]+        ldr q13, [x0]+        ldr q11, [x2, #0x50]+        sqrdmulh v16.4s, v9.4s, v23.s[1]+        ldr q17, [x0, #0x50]+        mul v15.4s, v9.4s, v23.s[0]+        ldr q30, [x0, #0x70]+        ldr q27, [x0, #0x60]+        ldr q8, [x2, #0x30]+        sqrdmulh v12.4s, v17.4s, v23.s[1]+        ldr q6, [x0, #0x30]+        mls v15.4s, v16.4s, v7.s[0]+        sqrdmulh v18.4s, v27.4s, v23.s[1]+        sqrdmulh v19.4s, v30.4s, v23.s[1]+        add v5.4s, v13.4s, v15.4s+        mul v25.4s, v27.4s, v23.s[0]+        sub v26.4s, v13.4s, v15.4s+        mls v25.4s, v18.4s, v7.s[0]+        mul v10.4s, v17.4s, v23.s[0]+        mls v10.4s, v12.4s, v7.s[0]+        mul v4.4s, v30.4s, v23.s[0]+        sub v22.4s, v1.4s, v25.4s+        mls v4.4s, v19.4s, v7.s[0]+        add v28.4s, v1.4s, v25.4s+        sqrdmulh v19.4s, v28.4s, v23.s[3]+        sqrdmulh v9.4s, v22.4s, v14.s[1]+        add v2.4s, v6.4s, v4.4s+        mul v0.4s, v28.4s, v23.s[2]+        sqrdmulh v27.4s, v2.4s, v23.s[3]+        sub v17.4s, v6.4s, v4.4s+        mul v3.4s, v2.4s, v23.s[2]+        sqrdmulh v20.4s, v17.4s, v14.s[1]+        ldr q1, [x0, #0x10]+        mls v3.4s, v27.4s, v7.s[0]+        mls v0.4s, v19.4s, v7.s[0]+        ldur q16, [x1, #-0x20]+        add v31.4s, v1.4s, v10.4s+        mul v30.4s, v17.4s, v14.s[0]+        mls v30.4s, v20.4s, v7.s[0]+        add v27.4s, v31.4s, v3.4s+        sub v23.4s, v1.4s, v10.4s+        sub v24.4s, v31.4s, v3.4s+        sqrdmulh v4.4s, v27.4s, v14.s[3]+        sqrdmulh v10.4s, v24.4s, v16.s[1]+        mul v18.4s, v24.4s, v16.s[0]+        add v15.4s, v23.4s, v30.4s+        sub v23.4s, v23.4s, v30.4s+        mul v29.4s, v27.4s, v14.s[2]+        sub v2.4s, v5.4s, v0.4s+        add v12.4s, v5.4s, v0.4s+        mls v18.4s, v10.4s, v7.s[0]+        ldur q3, [x1, #-0x10]+        mls v29.4s, v4.4s, v7.s[0]+        mul v4.4s, v22.4s, v14.s[0]+        add v1.4s, v2.4s, v18.4s+        sub v24.4s, v2.4s, v18.4s+        mls v4.4s, v9.4s, v7.s[0]+        ldr q20, [x2, #0x10]+        add v25.4s, v12.4s, v29.4s+        mul v9.4s, v23.4s, v3.s[0]+        sub v5.4s, v12.4s, v29.4s+        sqrdmulh v31.4s, v23.4s, v3.s[1]+        trn2 v6.4s, v1.4s, v24.4s+        trn2 v10.4s, v25.4s, v5.4s+        sqrdmulh v13.4s, v15.4s, v16.s[3]+        trn2 v30.2d, v10.2d, v6.2d+        ldr q3, [x2], #0xc0+        mul v12.4s, v15.4s, v16.s[2]+        trn1 v27.2d, v10.2d, v6.2d+        mls v9.4s, v31.4s, v7.s[0]+        trn1 v22.4s, v25.4s, v5.4s+        sub v6.4s, v26.4s, v4.4s+        mls v12.4s, v13.4s, v7.s[0]+        trn1 v1.4s, v1.4s, v24.4s+        add v13.4s, v26.4s, v4.4s+        trn2 v10.2d, v22.2d, v1.2d+        mul v28.4s, v30.4s, v3.4s+        sub v31.4s, v6.4s, v9.4s+        sub x4, x4, #0x1++Lmld_ntt_layer45678_start:+        add v2.4s, v13.4s, v12.4s+        sqrdmulh v5.4s, v30.4s, v20.4s+        sub v25.4s, v13.4s, v12.4s+        add v17.4s, v6.4s, v9.4s+        mul v19.4s, v10.4s, v3.4s+        trn2 v4.4s, v2.4s, v25.4s+        ldur q24, [x2, #-0x50]+        trn2 v29.4s, v17.4s, v31.4s+        sqrdmulh v15.4s, v10.4s, v20.4s+        mls v28.4s, v5.4s, v7.s[0]+        trn2 v3.2d, v4.2d, v29.2d+        sqrdmulh v12.4s, v3.4s, v24.4s+        mul v16.4s, v3.4s, v21.4s+        mls v19.4s, v15.4s, v7.s[0]+        ldur q10, [x2, #-0xa0]+        add v13.4s, v27.4s, v28.4s+        mls v16.4s, v12.4s, v7.s[0]+        sqrdmulh v9.4s, v13.4s, v8.4s+        sub v30.4s, v27.4s, v28.4s+        ldr q18, [x1], #0x40+        mul v8.4s, v13.4s, v10.4s+        ldr q10, [x0, #0xd0]+        sqrdmulh v14.4s, v30.4s, v11.4s+        ldr q23, [x0, #0xe0]+        sqrdmulh v13.4s, v10.4s, v18.s[1]+        sqrdmulh v12.4s, v23.4s, v18.s[1]+        ldur q6, [x2, #-0x80]+        mul v3.4s, v10.4s, v18.s[0]+        mls v3.4s, v13.4s, v7.s[0]+        ldr q13, [x0, #0xf0]+        trn1 v27.4s, v2.4s, v25.4s+        mul v2.4s, v30.4s, v6.4s+        trn1 v20.4s, v17.4s, v31.4s+        trn1 v25.2d, v4.2d, v29.2d+        sqrdmulh v10.4s, v13.4s, v18.s[1]+        trn2 v5.2d, v27.2d, v20.2d+        ldur q6, [x2, #-0x10]+        mls v8.4s, v9.4s, v7.s[0]+        sub v15.4s, v25.4s, v16.4s+        sqrdmulh v31.4s, v5.4s, v24.4s+        sqrdmulh v30.4s, v15.4s, v6.4s+        ldur q9, [x2, #-0x30]+        mul v4.4s, v5.4s, v21.4s+        ldur q21, [x2, #-0x40]+        ldur q6, [x2, #-0x20]+        add v5.4s, v25.4s, v16.4s+        mls v4.4s, v31.4s, v7.s[0]+        mul v0.4s, v5.4s, v21.4s+        mul v17.4s, v13.4s, v18.s[0]+        mls v17.4s, v10.4s, v7.s[0]+        ldr q28, [x0, #0xb0]+        sqrdmulh v26.4s, v5.4s, v9.4s+        mul v9.4s, v15.4s, v6.4s+        trn1 v6.2d, v22.2d, v1.2d+        mls v9.4s, v30.4s, v7.s[0]+        add v25.4s, v28.4s, v17.4s+        mls v2.4s, v14.4s, v7.s[0]+        trn1 v5.2d, v27.2d, v20.2d+        ldr q20, [x2, #0x10]+        mul v29.4s, v25.4s, v18.s[2]+        add v15.4s, v6.4s, v19.4s+        ldr q30, [x0, #0xc0]+        sub v19.4s, v6.4s, v19.4s+        add v31.4s, v15.4s, v8.4s+        mls v0.4s, v26.4s, v7.s[0]+        ldur q14, [x1, #-0x30]+        add v21.4s, v19.4s, v2.4s+        sub v24.4s, v19.4s, v2.4s+        sqrdmulh v27.4s, v25.4s, v18.s[3]+        sub v26.4s, v15.4s, v8.4s+        ldr q2, [x0, #0x90]+        mul v16.4s, v30.4s, v18.s[0]+        sub v25.4s, v28.4s, v17.4s+        trn1 v11.4s, v31.4s, v26.4s+        ldr q1, [x0, #0xa0]+        trn1 v6.4s, v21.4s, v24.4s+        sqrdmulh v13.4s, v25.4s, v14.s[1]+        add v8.4s, v2.4s, v3.4s+        trn2 v28.2d, v11.2d, v6.2d+        sqrdmulh v19.4s, v30.4s, v18.s[1]+        sub v10.4s, v5.4s, v4.4s+        ldur q22, [x1, #-0x20]+        str q28, [x0, #0x20]+        mls v29.4s, v27.4s, v7.s[0]+        add v15.4s, v10.4s, v9.4s+        mul v25.4s, v25.4s, v14.s[0]+        ldur q27, [x1, #-0x10]+        trn2 v17.4s, v31.4s, v26.4s+        trn2 v21.4s, v21.4s, v24.4s+        mls v16.4s, v19.4s, v7.s[0]+        sub v24.4s, v8.4s, v29.4s+        sub v10.4s, v10.4s, v9.4s+        mls v25.4s, v13.4s, v7.s[0]+        trn1 v13.2d, v11.2d, v6.2d+        ldr q28, [x0, #0x80]+        sqrdmulh v30.4s, v24.4s, v22.s[1]+        trn2 v19.2d, v17.2d, v21.2d+        trn1 v6.2d, v17.2d, v21.2d+        mul v31.4s, v23.4s, v18.s[0]+        str q13, [x0], #0x80+        stur q6, [x0, #-0x70]+        stur q19, [x0, #-0x50]+        ldr q11, [x2, #0x50]+        mls v31.4s, v12.4s, v7.s[0]+        ldr q21, [x2, #0x60]+        trn1 v9.4s, v15.4s, v10.4s+        trn2 v6.4s, v15.4s, v10.4s+        mul v24.4s, v24.4s, v22.s[0]+        sub v10.4s, v2.4s, v3.4s+        ldr q3, [x2], #0xc0+        mls v24.4s, v30.4s, v7.s[0]+        add v26.4s, v8.4s, v29.4s+        ldur q8, [x2, #-0x90]+        add v17.4s, v5.4s, v4.4s+        sqrdmulh v2.4s, v26.4s, v14.s[3]+        sub v13.4s, v1.4s, v31.4s+        add v30.4s, v1.4s, v31.4s+        add v15.4s, v10.4s, v25.4s+        sqrdmulh v19.4s, v13.4s, v14.s[1]+        sub v25.4s, v10.4s, v25.4s+        mul v29.4s, v13.4s, v14.s[0]+        sub v5.4s, v28.4s, v16.4s+        sqrdmulh v4.4s, v30.4s, v18.s[3]+        sub v23.4s, v17.4s, v0.4s+        add v31.4s, v17.4s, v0.4s+        mul v18.4s, v30.4s, v18.s[2]+        add v1.4s, v28.4s, v16.4s+        trn2 v12.4s, v31.4s, v23.4s+        mls v29.4s, v19.4s, v7.s[0]+        trn1 v13.4s, v31.4s, v23.4s+        trn2 v30.2d, v12.2d, v6.2d+        mls v18.4s, v4.4s, v7.s[0]+        trn2 v10.2d, v13.2d, v9.2d+        trn1 v31.2d, v13.2d, v9.2d+        mul v19.4s, v26.4s, v14.s[2]+        trn1 v12.2d, v12.2d, v6.2d+        sub v6.4s, v5.4s, v29.4s+        mls v19.4s, v2.4s, v7.s[0]+        add v13.4s, v5.4s, v29.4s+        stur q10, [x0, #-0x20]+        sub v10.4s, v1.4s, v18.4s+        add v28.4s, v1.4s, v18.4s+        sqrdmulh v5.4s, v25.4s, v27.s[1]+        stur q31, [x0, #-0x40]+        add v26.4s, v10.4s, v24.4s+        sub v31.4s, v10.4s, v24.4s+        mul v9.4s, v25.4s, v27.s[0]+        stur q12, [x0, #-0x30]+        sub v24.4s, v28.4s, v19.4s+        sqrdmulh v10.4s, v15.4s, v22.s[3]+        trn1 v1.4s, v26.4s, v31.4s+        stur q30, [x0, #-0x10]+        add v30.4s, v28.4s, v19.4s+        mls v9.4s, v5.4s, v7.s[0]+        trn2 v25.4s, v26.4s, v31.4s+        trn2 v14.4s, v30.4s, v24.4s+        mul v12.4s, v15.4s, v22.s[2]+        trn1 v22.4s, v30.4s, v24.4s+        trn1 v27.2d, v14.2d, v25.2d+        mls v12.4s, v10.4s, v7.s[0]+        trn2 v30.2d, v14.2d, v25.2d+        sub v31.4s, v6.4s, v9.4s+        trn2 v10.2d, v22.2d, v1.2d+        mul v28.4s, v30.4s, v3.4s+        subs x4, x4, #0x1+        cbnz x4, Lmld_ntt_layer45678_start+        add v9.4s, v6.4s, v9.4s+        sqrdmulh v6.4s, v30.4s, v20.4s+        ldur q24, [x2, #-0xa0]+        add v25.4s, v13.4s, v12.4s+        sub v15.4s, v13.4s, v12.4s+        mul v19.4s, v10.4s, v3.4s+        trn2 v5.4s, v9.4s, v31.4s+        sqrdmulh v3.4s, v10.4s, v20.4s+        trn2 v10.4s, v25.4s, v15.4s+        mls v28.4s, v6.4s, v7.s[0]+        trn2 v13.2d, v10.2d, v5.2d+        ldur q30, [x2, #-0x50]+        mul v12.4s, v13.4s, v21.4s+        mls v19.4s, v3.4s, v7.s[0]+        add v20.4s, v27.4s, v28.4s+        sqrdmulh v13.4s, v13.4s, v30.4s+        sub v3.4s, v27.4s, v28.4s+        mul v24.4s, v20.4s, v24.4s+        sqrdmulh v6.4s, v3.4s, v11.4s+        ldur q27, [x2, #-0x80]+        mls v12.4s, v13.4s, v7.s[0]+        trn1 v25.4s, v25.4s, v15.4s+        mul v27.4s, v3.4s, v27.4s+        trn1 v31.4s, v9.4s, v31.4s+        trn1 v3.2d, v10.2d, v5.2d+        ldur q13, [x2, #-0x30]+        ldur q15, [x2, #-0x40]+        sqrdmulh v9.4s, v20.4s, v8.4s+        trn2 v20.2d, v25.2d, v31.2d+        ldur q10, [x2, #-0x10]+        mls v27.4s, v6.4s, v7.s[0]+        add v5.4s, v3.4s, v12.4s+        sub v6.4s, v3.4s, v12.4s+        sqrdmulh v3.4s, v20.4s, v30.4s+        trn1 v12.2d, v22.2d, v1.2d+        sqrdmulh v10.4s, v6.4s, v10.4s+        mls v24.4s, v9.4s, v7.s[0]+        sub v9.4s, v12.4s, v19.4s+        trn1 v25.2d, v25.2d, v31.2d+        sqrdmulh v31.4s, v5.4s, v13.4s+        add v30.4s, v9.4s, v27.4s+        add v13.4s, v12.4s, v19.4s+        mul v1.4s, v20.4s, v21.4s+        ldur q12, [x2, #-0x20]+        add v21.4s, v13.4s, v24.4s+        sub v13.4s, v13.4s, v24.4s+        mls v1.4s, v3.4s, v7.s[0]+        sub v3.4s, v9.4s, v27.4s+        mul v9.4s, v6.4s, v12.4s+        trn2 v12.4s, v21.4s, v13.4s+        trn1 v6.4s, v30.4s, v3.4s+        trn2 v30.4s, v30.4s, v3.4s+        mls v9.4s, v10.4s, v7.s[0]+        trn1 v13.4s, v21.4s, v13.4s+        mul v15.4s, v5.4s, v15.4s+        sub v3.4s, v25.4s, v1.4s+        add v5.4s, v25.4s, v1.4s+        mls v15.4s, v31.4s, v7.s[0]+        trn1 v21.2d, v13.2d, v6.2d+        trn2 v6.2d, v13.2d, v6.2d+        add v10.4s, v3.4s, v9.4s+        sub v13.4s, v3.4s, v9.4s+        str q21, [x0], #0x80+        trn1 v3.2d, v12.2d, v30.2d+        trn2 v31.2d, v12.2d, v30.2d+        trn1 v21.4s, v10.4s, v13.4s+        sub v30.4s, v5.4s, v15.4s+        add v12.4s, v5.4s, v15.4s+        stur q3, [x0, #-0x70]+        trn2 v13.4s, v10.4s, v13.4s+        trn1 v19.4s, v12.4s, v30.4s+        trn2 v12.4s, v12.4s, v30.4s+        stur q6, [x0, #-0x60]+        stur q31, [x0, #-0x50]+        trn1 v10.2d, v19.2d, v21.2d+        trn2 v3.2d, v19.2d, v21.2d+        trn1 v21.2d, v12.2d, v13.2d+        trn2 v13.2d, v12.2d, v13.2d+        stur q10, [x0, #-0x40]+        stur q3, [x0, #-0x20]+        stur q13, [x0, #-0x10]+        stur q21, [x0, #-0x30]+        ldp d8, d9, [sp]+        .cfi_restore d8+        .cfi_restore d9+        ldp d10, d11, [sp, #0x10]+        .cfi_restore d10+        .cfi_restore d11+        ldp d12, d13, [sp, #0x20]+        .cfi_restore d12+        .cfi_restore d13+        ldp d14, d15, [sp, #0x30]+        .cfi_restore d14+        .cfi_restore d15+        add sp, sp, #0x40+        .cfi_adjust_cfa_offset -0x40+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(ntt_aarch64_asm)++#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/aarch64/src/mldsa_pointwise_montgomery_aarch64_asm.S view
@@ -0,0 +1,106 @@+/* Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*yaml+  Name: poly_pointwise_montgomery_aarch64_asm+  Description: AArch64 pointwise Montgomery multiplication of two polynomials+  Signature: void mld_poly_pointwise_montgomery_aarch64_asm(int32_t a[256], const int32_t b[256])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 1024+      permissions: read/write+      c_parameter: int32_t a[256]+      description: Input/output polynomial (256 x int32_t)+    x1:+      type: buffer+      size_bytes: 1024+      permissions: read-only+      c_parameter: const int32_t b[256]+      description: Input polynomial (256 x int32_t)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_AARCH64) &&                                      \+    (!defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) || \+     defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)) &&              \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/aarch64_opt/src/mldsa_pointwise_montgomery_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(poly_pointwise_montgomery_aarch64_asm)+MLD_ASM_FN_SYMBOL(poly_pointwise_montgomery_aarch64_asm)++        .cfi_startproc+        mov w3, #0xe001             // =57345+        movk w3, #0x7f, lsl #16+        dup v0.4s, w3+        mov w3, #0x2001             // =8193+        movk w3, #0x380, lsl #16+        dup v1.4s, w3+        mov x3, #0x40               // =64++Lmld_poly_pointwise_montgomery_loop_start:+        ldr q16, [x0]+        ldr q17, [x0, #0x10]+        ldr q18, [x0, #0x20]+        ldr q19, [x0, #0x30]+        ldr q21, [x1, #0x10]+        ldr q22, [x1, #0x20]+        ldr q23, [x1, #0x30]+        ldr q20, [x1], #0x40+        smull v24.2d, v16.2s, v20.2s+        smull2 v25.2d, v16.4s, v20.4s+        smull v26.2d, v17.2s, v21.2s+        smull2 v27.2d, v17.4s, v21.4s+        smull v28.2d, v18.2s, v22.2s+        smull2 v29.2d, v18.4s, v22.4s+        smull v30.2d, v19.2s, v23.2s+        smull2 v31.2d, v19.4s, v23.4s+        uzp1 v16.4s, v24.4s, v25.4s+        mul v16.4s, v16.4s, v1.4s+        smlsl v24.2d, v16.2s, v0.2s+        smlsl2 v25.2d, v16.4s, v0.4s+        uzp2 v16.4s, v24.4s, v25.4s+        uzp1 v17.4s, v26.4s, v27.4s+        mul v17.4s, v17.4s, v1.4s+        smlsl v26.2d, v17.2s, v0.2s+        smlsl2 v27.2d, v17.4s, v0.4s+        uzp2 v17.4s, v26.4s, v27.4s+        uzp1 v18.4s, v28.4s, v29.4s+        mul v18.4s, v18.4s, v1.4s+        smlsl v28.2d, v18.2s, v0.2s+        smlsl2 v29.2d, v18.4s, v0.4s+        uzp2 v18.4s, v28.4s, v29.4s+        uzp1 v19.4s, v30.4s, v31.4s+        mul v19.4s, v19.4s, v1.4s+        smlsl v30.2d, v19.2s, v0.2s+        smlsl2 v31.2d, v19.4s, v0.4s+        uzp2 v19.4s, v30.4s, v31.4s+        str q17, [x0, #0x10]+        str q18, [x0, #0x20]+        str q19, [x0, #0x30]+        str q16, [x0], #0x40+        subs x3, x3, #0x4+        cbnz x3, Lmld_poly_pointwise_montgomery_loop_start+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(poly_pointwise_montgomery_aarch64_asm)++#endif /* MLD_ARITH_BACKEND_AARCH64 && (!MLD_CONFIG_NO_SIGN_API || \+          !MLD_CONFIG_NO_VERIFY_API || MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) \+          && !MLD_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/aarch64/src/mldsa_poly_caddq_aarch64_asm.S view
@@ -0,0 +1,69 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+/*yaml+  Name: poly_caddq_aarch64_asm+  Description: AArch64 conditional addition of q to each coefficient+  Signature: void mld_poly_caddq_aarch64_asm(int32_t a[256])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 1024+      permissions: read/write+      c_parameter: int32_t a[256]+      description: Input/output polynomial (256 x int32_t)+*/++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/aarch64_opt/src/mldsa_poly_caddq_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(poly_caddq_aarch64_asm)+MLD_ASM_FN_SYMBOL(poly_caddq_aarch64_asm)++        .cfi_startproc+        mov w9, #0xe001             // =57345+        movk w9, #0x7f, lsl #16+        dup v4.4s, w9+        mov x1, #0x10               // =16++Lmld_poly_caddq_loop:+        ldr q0, [x0]+        ldr q1, [x0, #0x10]+        ldr q2, [x0, #0x20]+        ldr q3, [x0, #0x30]+        ushr v5.4s, v0.4s, #0x1f+        mla v0.4s, v5.4s, v4.4s+        ushr v5.4s, v1.4s, #0x1f+        mla v1.4s, v5.4s, v4.4s+        ushr v5.4s, v2.4s, #0x1f+        mla v2.4s, v5.4s, v4.4s+        ushr v5.4s, v3.4s, #0x1f+        mla v3.4s, v5.4s, v4.4s+        str q1, [x0, #0x10]+        str q2, [x0, #0x20]+        str q3, [x0, #0x30]+        str q0, [x0], #0x40+        subs x1, x1, #0x1+        b.ne Lmld_poly_caddq_loop+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(poly_caddq_aarch64_asm)++#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/aarch64/src/mldsa_poly_chknorm_aarch64_asm.S view
@@ -0,0 +1,76 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+/*yaml+  Name: poly_chknorm_aarch64_asm+  Description: AArch64 infinity-norm bound check on polynomial coefficients+  Signature: int mld_poly_chknorm_aarch64_asm(const int32_t a[256], int32_t B)+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 1024+      permissions: read-only+      c_parameter: const int32_t a[256]+      description: Input polynomial (256 x int32_t)+    x1:+      type: scalar+      c_parameter: int32_t B+      description: Norm bound+      test_with: 131072  # representative non-negative bound (1 << 17)+*/++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/aarch64_opt/src/mldsa_poly_chknorm_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(poly_chknorm_aarch64_asm)+MLD_ASM_FN_SYMBOL(poly_chknorm_aarch64_asm)++        .cfi_startproc+        dup v20.4s, w1+        eor v21.16b, v21.16b, v21.16b+        mov x2, #0x10               // =16++Lmld_poly_chknorm_loop:+        ldr q1, [x0, #0x10]+        ldr q2, [x0, #0x20]+        ldr q3, [x0, #0x30]+        ldr q0, [x0], #0x40+        abs v1.4s, v1.4s+        cmge v1.4s, v1.4s, v20.4s+        orr v21.16b, v21.16b, v1.16b+        abs v2.4s, v2.4s+        cmge v2.4s, v2.4s, v20.4s+        orr v21.16b, v21.16b, v2.16b+        abs v3.4s, v3.4s+        cmge v3.4s, v3.4s, v20.4s+        orr v21.16b, v21.16b, v3.16b+        abs v0.4s, v0.4s+        cmge v0.4s, v0.4s, v20.4s+        orr v21.16b, v21.16b, v0.16b+        subs x2, x2, #0x1+        b.ne Lmld_poly_chknorm_loop+        umaxv s21, v21.4s+        fmov w0, s21+        and w0, w0, #0x1+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(poly_chknorm_aarch64_asm)++#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/aarch64/src/mldsa_poly_decompose_32_aarch64_asm.S view
@@ -0,0 +1,108 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+/*yaml+  Name: poly_decompose_32_aarch64_asm+  Description: AArch64 coefficient decomposition (alpha = (Q-1)/32)+  Signature: void mld_poly_decompose_32_aarch64_asm(int32_t a1[256], int32_t a0[256])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t a1[256]+      description: Output high-part polynomial (256 x int32_t)+    x1:+      type: buffer+      size_bytes: 1024+      permissions: read/write+      c_parameter: int32_t a0[256]+      description: Input polynomial / output low-part (256 x int32_t)+*/++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_NO_SIGN_API) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \+           (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87))++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/aarch64_opt/src/mldsa_poly_decompose_32_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(poly_decompose_32_aarch64_asm)+MLD_ASM_FN_SYMBOL(poly_decompose_32_aarch64_asm)++        .cfi_startproc+        mov w4, #0xe001             // =57345+        movk w4, #0x7f, lsl #16+        dup v20.4s, w4+        mov w5, #0xe100             // =57600+        movk w5, #0x7b, lsl #16+        dup v21.4s, w5+        mov w7, #0xfe00             // =65024+        movk w7, #0x7, lsl #16+        dup v22.4s, w7+        mov w11, #0x401             // =1025+        movk w11, #0x4010, lsl #16+        dup v23.4s, w11+        mov x3, #0x10               // =16++Lmld_poly_decompose_32_loop:+        ldr q0, [x1]+        ldr q1, [x1, #0x10]+        ldr q2, [x1, #0x20]+        ldr q3, [x1, #0x30]+        sqdmulh v5.4s, v1.4s, v23.4s+        srshr v5.4s, v5.4s, #0x12+        cmgt v24.4s, v1.4s, v21.4s+        mls v1.4s, v5.4s, v22.4s+        bic v5.16b, v5.16b, v24.16b+        add v1.4s, v1.4s, v24.4s+        sqdmulh v6.4s, v2.4s, v23.4s+        srshr v6.4s, v6.4s, #0x12+        cmgt v24.4s, v2.4s, v21.4s+        mls v2.4s, v6.4s, v22.4s+        bic v6.16b, v6.16b, v24.16b+        add v2.4s, v2.4s, v24.4s+        sqdmulh v7.4s, v3.4s, v23.4s+        srshr v7.4s, v7.4s, #0x12+        cmgt v24.4s, v3.4s, v21.4s+        mls v3.4s, v7.4s, v22.4s+        bic v7.16b, v7.16b, v24.16b+        add v3.4s, v3.4s, v24.4s+        sqdmulh v4.4s, v0.4s, v23.4s+        srshr v4.4s, v4.4s, #0x12+        cmgt v24.4s, v0.4s, v21.4s+        mls v0.4s, v4.4s, v22.4s+        bic v4.16b, v4.16b, v24.16b+        add v0.4s, v0.4s, v24.4s+        str q5, [x0, #0x10]+        str q6, [x0, #0x20]+        str q7, [x0, #0x30]+        str q4, [x0], #0x40+        str q1, [x1, #0x10]+        str q2, [x1, #0x20]+        str q3, [x1, #0x30]+        str q0, [x1], #0x40+        subs x3, x3, #0x1+        b.ne Lmld_poly_decompose_32_loop+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(poly_decompose_32_aarch64_asm)++#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_NO_SIGN_API && \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/aarch64/src/mldsa_poly_decompose_88_aarch64_asm.S view
@@ -0,0 +1,108 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+/*yaml+  Name: poly_decompose_88_aarch64_asm+  Description: AArch64 coefficient decomposition (alpha = (Q-1)/88)+  Signature: void mld_poly_decompose_88_aarch64_asm(int32_t a1[256], int32_t a0[256])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t a1[256]+      description: Output high-part polynomial (256 x int32_t)+    x1:+      type: buffer+      size_bytes: 1024+      permissions: read/write+      c_parameter: int32_t a0[256]+      description: Input polynomial / output low-part (256 x int32_t)+*/++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_NO_SIGN_API) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \+           (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/aarch64_opt/src/mldsa_poly_decompose_88_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(poly_decompose_88_aarch64_asm)+MLD_ASM_FN_SYMBOL(poly_decompose_88_aarch64_asm)++        .cfi_startproc+        mov w4, #0xe001             // =57345+        movk w4, #0x7f, lsl #16+        dup v20.4s, w4+        mov w5, #0x6c00             // =27648+        movk w5, #0x7e, lsl #16+        dup v21.4s, w5+        mov w7, #0xe800             // =59392+        movk w7, #0x2, lsl #16+        dup v22.4s, w7+        mov w11, #0x581             // =1409+        movk w11, #0x5816, lsl #16+        dup v23.4s, w11+        mov x3, #0x10               // =16++Lmld_poly_decompose_88_loop:+        ldr q0, [x1]+        ldr q1, [x1, #0x10]+        ldr q2, [x1, #0x20]+        ldr q3, [x1, #0x30]+        sqdmulh v5.4s, v1.4s, v23.4s+        srshr v5.4s, v5.4s, #0x11+        cmgt v24.4s, v1.4s, v21.4s+        mls v1.4s, v5.4s, v22.4s+        bic v5.16b, v5.16b, v24.16b+        add v1.4s, v1.4s, v24.4s+        sqdmulh v6.4s, v2.4s, v23.4s+        srshr v6.4s, v6.4s, #0x11+        cmgt v24.4s, v2.4s, v21.4s+        mls v2.4s, v6.4s, v22.4s+        bic v6.16b, v6.16b, v24.16b+        add v2.4s, v2.4s, v24.4s+        sqdmulh v7.4s, v3.4s, v23.4s+        srshr v7.4s, v7.4s, #0x11+        cmgt v24.4s, v3.4s, v21.4s+        mls v3.4s, v7.4s, v22.4s+        bic v7.16b, v7.16b, v24.16b+        add v3.4s, v3.4s, v24.4s+        sqdmulh v4.4s, v0.4s, v23.4s+        srshr v4.4s, v4.4s, #0x11+        cmgt v24.4s, v0.4s, v21.4s+        mls v0.4s, v4.4s, v22.4s+        bic v4.16b, v4.16b, v24.16b+        add v0.4s, v0.4s, v24.4s+        str q5, [x0, #0x10]+        str q6, [x0, #0x20]+        str q7, [x0, #0x30]+        str q4, [x0], #0x40+        str q1, [x1, #0x10]+        str q2, [x1, #0x20]+        str q3, [x1, #0x30]+        str q0, [x1], #0x40+        subs x3, x3, #0x1+        b.ne Lmld_poly_decompose_88_loop+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(poly_decompose_88_aarch64_asm)++#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_NO_SIGN_API && \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44) \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/aarch64/src/mldsa_poly_use_hint_32_aarch64_asm.S view
@@ -0,0 +1,125 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+/*yaml+  Name: poly_use_hint_32_aarch64_asm+  Description: AArch64 hint application (alpha = (Q-1)/32)+  Signature: void mld_poly_use_hint_32_aarch64_asm(int32_t a[256], const int32_t h[256])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 1024+      permissions: read/write+      c_parameter: int32_t a[256]+      description: Input/output polynomial (256 x int32_t)+    x1:+      type: buffer+      size_bytes: 1024+      permissions: read-only+      c_parameter: const int32_t h[256]+      description: Hint polynomial (256 x int32_t)+*/++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_NO_VERIFY_API) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \+           (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87))++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/aarch64_opt/src/mldsa_poly_use_hint_32_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(poly_use_hint_32_aarch64_asm)+MLD_ASM_FN_SYMBOL(poly_use_hint_32_aarch64_asm)++        .cfi_startproc+        mov w4, #0xe001             // =57345+        movk w4, #0x7f, lsl #16+        dup v20.4s, w4+        mov w5, #0xe100             // =57600+        movk w5, #0x7b, lsl #16+        dup v21.4s, w5+        mov w7, #0xfe00             // =65024+        movk w7, #0x7, lsl #16+        dup v22.4s, w7+        mov w11, #0x401             // =1025+        movk w11, #0x4010, lsl #16+        dup v23.4s, w11+        movi v24.4s, #0xf+        mov x3, #0x10               // =16++Lmld_poly_use_hint_32_loop:+        ldr q1, [x0, #0x10]+        ldr q2, [x0, #0x20]+        ldr q3, [x0, #0x30]+        ldr q0, [x0]+        ldr q5, [x1, #0x10]+        ldr q6, [x1, #0x20]+        ldr q7, [x1, #0x30]+        ldr q4, [x1], #0x40+        sqdmulh v17.4s, v1.4s, v23.4s+        srshr v17.4s, v17.4s, #0x12+        cmgt v25.4s, v1.4s, v21.4s+        mls v1.4s, v17.4s, v22.4s+        bic v17.16b, v17.16b, v25.16b+        add v1.4s, v1.4s, v25.4s+        cmle v1.4s, v1.4s, #0+        orr v1.4s, #0x1+        mla v17.4s, v1.4s, v5.4s+        and v17.16b, v17.16b, v24.16b+        sqdmulh v18.4s, v2.4s, v23.4s+        srshr v18.4s, v18.4s, #0x12+        cmgt v25.4s, v2.4s, v21.4s+        mls v2.4s, v18.4s, v22.4s+        bic v18.16b, v18.16b, v25.16b+        add v2.4s, v2.4s, v25.4s+        cmle v2.4s, v2.4s, #0+        orr v2.4s, #0x1+        mla v18.4s, v2.4s, v6.4s+        and v18.16b, v18.16b, v24.16b+        sqdmulh v19.4s, v3.4s, v23.4s+        srshr v19.4s, v19.4s, #0x12+        cmgt v25.4s, v3.4s, v21.4s+        mls v3.4s, v19.4s, v22.4s+        bic v19.16b, v19.16b, v25.16b+        add v3.4s, v3.4s, v25.4s+        cmle v3.4s, v3.4s, #0+        orr v3.4s, #0x1+        mla v19.4s, v3.4s, v7.4s+        and v19.16b, v19.16b, v24.16b+        sqdmulh v16.4s, v0.4s, v23.4s+        srshr v16.4s, v16.4s, #0x12+        cmgt v25.4s, v0.4s, v21.4s+        mls v0.4s, v16.4s, v22.4s+        bic v16.16b, v16.16b, v25.16b+        add v0.4s, v0.4s, v25.4s+        cmle v0.4s, v0.4s, #0+        orr v0.4s, #0x1+        mla v16.4s, v0.4s, v4.4s+        and v16.16b, v16.16b, v24.16b+        str q17, [x0, #0x10]+        str q18, [x0, #0x20]+        str q19, [x0, #0x30]+        str q16, [x0], #0x40+        subs x3, x3, #0x1+        b.ne Lmld_poly_use_hint_32_loop+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(poly_use_hint_32_aarch64_asm)++#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_NO_VERIFY_API && \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/aarch64/src/mldsa_poly_use_hint_88_aarch64_asm.S view
@@ -0,0 +1,133 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+/*yaml+  Name: poly_use_hint_88_aarch64_asm+  Description: AArch64 hint application (alpha = (Q-1)/88)+  Signature: void mld_poly_use_hint_88_aarch64_asm(int32_t a[256], const int32_t h[256])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 1024+      permissions: read/write+      c_parameter: int32_t a[256]+      description: Input/output polynomial (256 x int32_t)+    x1:+      type: buffer+      size_bytes: 1024+      permissions: read-only+      c_parameter: const int32_t h[256]+      description: Hint polynomial (256 x int32_t)+*/++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_NO_VERIFY_API) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \+           (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/aarch64_opt/src/mldsa_poly_use_hint_88_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(poly_use_hint_88_aarch64_asm)+MLD_ASM_FN_SYMBOL(poly_use_hint_88_aarch64_asm)++        .cfi_startproc+        mov w4, #0xe001             // =57345+        movk w4, #0x7f, lsl #16+        dup v20.4s, w4+        mov w5, #0x6c00             // =27648+        movk w5, #0x7e, lsl #16+        dup v21.4s, w5+        mov w7, #0xe800             // =59392+        movk w7, #0x2, lsl #16+        dup v22.4s, w7+        mov w11, #0x581             // =1409+        movk w11, #0x5816, lsl #16+        dup v23.4s, w11+        movi v24.4s, #0x2b+        mov x3, #0x10               // =16++Lmld_poly_use_hint_88_loop:+        ldr q1, [x0, #0x10]+        ldr q2, [x0, #0x20]+        ldr q3, [x0, #0x30]+        ldr q0, [x0]+        ldr q5, [x1, #0x10]+        ldr q6, [x1, #0x20]+        ldr q7, [x1, #0x30]+        ldr q4, [x1], #0x40+        sqdmulh v17.4s, v1.4s, v23.4s+        srshr v17.4s, v17.4s, #0x11+        cmgt v25.4s, v1.4s, v21.4s+        mls v1.4s, v17.4s, v22.4s+        bic v17.16b, v17.16b, v25.16b+        add v1.4s, v1.4s, v25.4s+        cmle v1.4s, v1.4s, #0+        orr v1.4s, #0x1+        mla v17.4s, v1.4s, v5.4s+        cmgt v25.4s, v17.4s, v24.4s+        bic v17.16b, v17.16b, v25.16b+        umin v17.4s, v17.4s, v24.4s+        sqdmulh v18.4s, v2.4s, v23.4s+        srshr v18.4s, v18.4s, #0x11+        cmgt v25.4s, v2.4s, v21.4s+        mls v2.4s, v18.4s, v22.4s+        bic v18.16b, v18.16b, v25.16b+        add v2.4s, v2.4s, v25.4s+        cmle v2.4s, v2.4s, #0+        orr v2.4s, #0x1+        mla v18.4s, v2.4s, v6.4s+        cmgt v25.4s, v18.4s, v24.4s+        bic v18.16b, v18.16b, v25.16b+        umin v18.4s, v18.4s, v24.4s+        sqdmulh v19.4s, v3.4s, v23.4s+        srshr v19.4s, v19.4s, #0x11+        cmgt v25.4s, v3.4s, v21.4s+        mls v3.4s, v19.4s, v22.4s+        bic v19.16b, v19.16b, v25.16b+        add v3.4s, v3.4s, v25.4s+        cmle v3.4s, v3.4s, #0+        orr v3.4s, #0x1+        mla v19.4s, v3.4s, v7.4s+        cmgt v25.4s, v19.4s, v24.4s+        bic v19.16b, v19.16b, v25.16b+        umin v19.4s, v19.4s, v24.4s+        sqdmulh v16.4s, v0.4s, v23.4s+        srshr v16.4s, v16.4s, #0x11+        cmgt v25.4s, v0.4s, v21.4s+        mls v0.4s, v16.4s, v22.4s+        bic v16.16b, v16.16b, v25.16b+        add v0.4s, v0.4s, v25.4s+        cmle v0.4s, v0.4s, #0+        orr v0.4s, #0x1+        mla v16.4s, v0.4s, v4.4s+        cmgt v25.4s, v16.4s, v24.4s+        bic v16.16b, v16.16b, v25.16b+        umin v16.4s, v16.4s, v24.4s+        str q17, [x0, #0x10]+        str q18, [x0, #0x20]+        str q19, [x0, #0x30]+        str q16, [x0], #0x40+        subs x3, x3, #0x1+        b.ne Lmld_poly_use_hint_88_loop+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(poly_use_hint_88_aarch64_asm)++#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_NO_VERIFY_API && \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44) \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm.S view
@@ -0,0 +1,157 @@+/* Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*yaml+  Name: polyvecl_pointwise_acc_montgomery_l4_aarch64_asm+  Description: AArch64 pointwise multiply-accumulate of length-4 polynomial vectors+  Signature: void mld_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm(int32_t r[256], const int32_t a[4][256], const int32_t b[4][256])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t r[256]+      description: Output polynomial (256 x int32_t)+    x1:+      type: buffer+      size_bytes: 4096+      permissions: read-only+      c_parameter: const int32_t a[4][256]+      description: Input polynomial vector a (4 x 256 x int32_t)+    x2:+      type: buffer+      size_bytes: 4096+      permissions: read-only+      c_parameter: const int32_t b[4][256]+      description: Input polynomial vector b (4 x 256 x int32_t)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \+           (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 4)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/aarch64_opt/src/mldsa_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(polyvecl_pointwise_acc_montgomery_l4_aarch64_asm)+MLD_ASM_FN_SYMBOL(polyvecl_pointwise_acc_montgomery_l4_aarch64_asm)++        .cfi_startproc+        mov w3, #0xe001             // =57345+        movk w3, #0x7f, lsl #16+        dup v0.4s, w3+        mov w3, #0x2001             // =8193+        movk w3, #0x380, lsl #16+        dup v1.4s, w3+        mov x3, #0x40               // =64++Lmld_polyvecl_pointwise_acc_montgomery_l4_loop_start:+        ldr q17, [x1, #0x10]+        ldr q18, [x1, #0x20]+        ldr q19, [x1, #0x30]+        ldr q16, [x1], #0x40+        ldr q21, [x2, #0x10]+        ldr q22, [x2, #0x20]+        ldr q23, [x2, #0x30]+        ldr q20, [x2], #0x40+        smull v24.2d, v16.2s, v20.2s+        smull2 v25.2d, v16.4s, v20.4s+        smull v26.2d, v17.2s, v21.2s+        smull2 v27.2d, v17.4s, v21.4s+        smull v28.2d, v18.2s, v22.2s+        smull2 v29.2d, v18.4s, v22.4s+        smull v30.2d, v19.2s, v23.2s+        smull2 v31.2d, v19.4s, v23.4s+        ldr q16, [x1, #0x3c0]+        ldr q17, [x1, #0x3d0]+        ldr q18, [x1, #0x3e0]+        ldr q19, [x1, #0x3f0]+        ldr q20, [x2, #0x3c0]+        ldr q21, [x2, #0x3d0]+        ldr q22, [x2, #0x3e0]+        ldr q23, [x2, #0x3f0]+        smlal v24.2d, v16.2s, v20.2s+        smlal2 v25.2d, v16.4s, v20.4s+        smlal v26.2d, v17.2s, v21.2s+        smlal2 v27.2d, v17.4s, v21.4s+        smlal v28.2d, v18.2s, v22.2s+        smlal2 v29.2d, v18.4s, v22.4s+        smlal v30.2d, v19.2s, v23.2s+        smlal2 v31.2d, v19.4s, v23.4s+        ldr q16, [x1, #0x7c0]+        ldr q17, [x1, #0x7d0]+        ldr q18, [x1, #0x7e0]+        ldr q19, [x1, #0x7f0]+        ldr q20, [x2, #0x7c0]+        ldr q21, [x2, #0x7d0]+        ldr q22, [x2, #0x7e0]+        ldr q23, [x2, #0x7f0]+        smlal v24.2d, v16.2s, v20.2s+        smlal2 v25.2d, v16.4s, v20.4s+        smlal v26.2d, v17.2s, v21.2s+        smlal2 v27.2d, v17.4s, v21.4s+        smlal v28.2d, v18.2s, v22.2s+        smlal2 v29.2d, v18.4s, v22.4s+        smlal v30.2d, v19.2s, v23.2s+        smlal2 v31.2d, v19.4s, v23.4s+        ldr q16, [x1, #0xbc0]+        ldr q17, [x1, #0xbd0]+        ldr q18, [x1, #0xbe0]+        ldr q19, [x1, #0xbf0]+        ldr q20, [x2, #0xbc0]+        ldr q21, [x2, #0xbd0]+        ldr q22, [x2, #0xbe0]+        ldr q23, [x2, #0xbf0]+        smlal v24.2d, v16.2s, v20.2s+        smlal2 v25.2d, v16.4s, v20.4s+        smlal v26.2d, v17.2s, v21.2s+        smlal2 v27.2d, v17.4s, v21.4s+        smlal v28.2d, v18.2s, v22.2s+        smlal2 v29.2d, v18.4s, v22.4s+        smlal v30.2d, v19.2s, v23.2s+        smlal2 v31.2d, v19.4s, v23.4s+        uzp1 v16.4s, v24.4s, v25.4s+        mul v16.4s, v16.4s, v1.4s+        smlsl v24.2d, v16.2s, v0.2s+        smlsl2 v25.2d, v16.4s, v0.4s+        uzp2 v16.4s, v24.4s, v25.4s+        uzp1 v17.4s, v26.4s, v27.4s+        mul v17.4s, v17.4s, v1.4s+        smlsl v26.2d, v17.2s, v0.2s+        smlsl2 v27.2d, v17.4s, v0.4s+        uzp2 v17.4s, v26.4s, v27.4s+        uzp1 v18.4s, v28.4s, v29.4s+        mul v18.4s, v18.4s, v1.4s+        smlsl v28.2d, v18.2s, v0.2s+        smlsl2 v29.2d, v18.4s, v0.4s+        uzp2 v18.4s, v28.4s, v29.4s+        uzp1 v19.4s, v30.4s, v31.4s+        mul v19.4s, v19.4s, v1.4s+        smlsl v30.2d, v19.2s, v0.2s+        smlsl2 v31.2d, v19.4s, v0.4s+        uzp2 v19.4s, v30.4s, v31.4s+        str q17, [x0, #0x10]+        str q18, [x0, #0x20]+        str q19, [x0, #0x30]+        str q16, [x0], #0x40+        subs x3, x3, #0x4+        cbnz x3, Lmld_polyvecl_pointwise_acc_montgomery_l4_loop_start+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(polyvecl_pointwise_acc_montgomery_l4_aarch64_asm)++#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 4) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm.S view
@@ -0,0 +1,173 @@+/* Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*yaml+  Name: polyvecl_pointwise_acc_montgomery_l5_aarch64_asm+  Description: AArch64 pointwise multiply-accumulate of length-5 polynomial vectors+  Signature: void mld_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm(int32_t r[256], const int32_t a[5][256], const int32_t b[5][256])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t r[256]+      description: Output polynomial (256 x int32_t)+    x1:+      type: buffer+      size_bytes: 5120+      permissions: read-only+      c_parameter: const int32_t a[5][256]+      description: Input polynomial vector a (5 x 256 x int32_t)+    x2:+      type: buffer+      size_bytes: 5120+      permissions: read-only+      c_parameter: const int32_t b[5][256]+      description: Input polynomial vector b (5 x 256 x int32_t)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \+           (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 5)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/aarch64_opt/src/mldsa_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(polyvecl_pointwise_acc_montgomery_l5_aarch64_asm)+MLD_ASM_FN_SYMBOL(polyvecl_pointwise_acc_montgomery_l5_aarch64_asm)++        .cfi_startproc+        mov w3, #0xe001             // =57345+        movk w3, #0x7f, lsl #16+        dup v0.4s, w3+        mov w3, #0x2001             // =8193+        movk w3, #0x380, lsl #16+        dup v1.4s, w3+        mov x3, #0x40               // =64++Lmld_polyvecl_pointwise_acc_montgomery_l5_loop_start:+        ldr q17, [x1, #0x10]+        ldr q18, [x1, #0x20]+        ldr q19, [x1, #0x30]+        ldr q16, [x1], #0x40+        ldr q21, [x2, #0x10]+        ldr q22, [x2, #0x20]+        ldr q23, [x2, #0x30]+        ldr q20, [x2], #0x40+        smull v24.2d, v16.2s, v20.2s+        smull2 v25.2d, v16.4s, v20.4s+        smull v26.2d, v17.2s, v21.2s+        smull2 v27.2d, v17.4s, v21.4s+        smull v28.2d, v18.2s, v22.2s+        smull2 v29.2d, v18.4s, v22.4s+        smull v30.2d, v19.2s, v23.2s+        smull2 v31.2d, v19.4s, v23.4s+        ldr q16, [x1, #0x3c0]+        ldr q17, [x1, #0x3d0]+        ldr q18, [x1, #0x3e0]+        ldr q19, [x1, #0x3f0]+        ldr q20, [x2, #0x3c0]+        ldr q21, [x2, #0x3d0]+        ldr q22, [x2, #0x3e0]+        ldr q23, [x2, #0x3f0]+        smlal v24.2d, v16.2s, v20.2s+        smlal2 v25.2d, v16.4s, v20.4s+        smlal v26.2d, v17.2s, v21.2s+        smlal2 v27.2d, v17.4s, v21.4s+        smlal v28.2d, v18.2s, v22.2s+        smlal2 v29.2d, v18.4s, v22.4s+        smlal v30.2d, v19.2s, v23.2s+        smlal2 v31.2d, v19.4s, v23.4s+        ldr q16, [x1, #0x7c0]+        ldr q17, [x1, #0x7d0]+        ldr q18, [x1, #0x7e0]+        ldr q19, [x1, #0x7f0]+        ldr q20, [x2, #0x7c0]+        ldr q21, [x2, #0x7d0]+        ldr q22, [x2, #0x7e0]+        ldr q23, [x2, #0x7f0]+        smlal v24.2d, v16.2s, v20.2s+        smlal2 v25.2d, v16.4s, v20.4s+        smlal v26.2d, v17.2s, v21.2s+        smlal2 v27.2d, v17.4s, v21.4s+        smlal v28.2d, v18.2s, v22.2s+        smlal2 v29.2d, v18.4s, v22.4s+        smlal v30.2d, v19.2s, v23.2s+        smlal2 v31.2d, v19.4s, v23.4s+        ldr q16, [x1, #0xbc0]+        ldr q17, [x1, #0xbd0]+        ldr q18, [x1, #0xbe0]+        ldr q19, [x1, #0xbf0]+        ldr q20, [x2, #0xbc0]+        ldr q21, [x2, #0xbd0]+        ldr q22, [x2, #0xbe0]+        ldr q23, [x2, #0xbf0]+        smlal v24.2d, v16.2s, v20.2s+        smlal2 v25.2d, v16.4s, v20.4s+        smlal v26.2d, v17.2s, v21.2s+        smlal2 v27.2d, v17.4s, v21.4s+        smlal v28.2d, v18.2s, v22.2s+        smlal2 v29.2d, v18.4s, v22.4s+        smlal v30.2d, v19.2s, v23.2s+        smlal2 v31.2d, v19.4s, v23.4s+        ldr q16, [x1, #0xfc0]+        ldr q17, [x1, #0xfd0]+        ldr q18, [x1, #0xfe0]+        ldr q19, [x1, #0xff0]+        ldr q20, [x2, #0xfc0]+        ldr q21, [x2, #0xfd0]+        ldr q22, [x2, #0xfe0]+        ldr q23, [x2, #0xff0]+        smlal v24.2d, v16.2s, v20.2s+        smlal2 v25.2d, v16.4s, v20.4s+        smlal v26.2d, v17.2s, v21.2s+        smlal2 v27.2d, v17.4s, v21.4s+        smlal v28.2d, v18.2s, v22.2s+        smlal2 v29.2d, v18.4s, v22.4s+        smlal v30.2d, v19.2s, v23.2s+        smlal2 v31.2d, v19.4s, v23.4s+        uzp1 v16.4s, v24.4s, v25.4s+        mul v16.4s, v16.4s, v1.4s+        smlsl v24.2d, v16.2s, v0.2s+        smlsl2 v25.2d, v16.4s, v0.4s+        uzp2 v16.4s, v24.4s, v25.4s+        uzp1 v17.4s, v26.4s, v27.4s+        mul v17.4s, v17.4s, v1.4s+        smlsl v26.2d, v17.2s, v0.2s+        smlsl2 v27.2d, v17.4s, v0.4s+        uzp2 v17.4s, v26.4s, v27.4s+        uzp1 v18.4s, v28.4s, v29.4s+        mul v18.4s, v18.4s, v1.4s+        smlsl v28.2d, v18.2s, v0.2s+        smlsl2 v29.2d, v18.4s, v0.4s+        uzp2 v18.4s, v28.4s, v29.4s+        uzp1 v19.4s, v30.4s, v31.4s+        mul v19.4s, v19.4s, v1.4s+        smlsl v30.2d, v19.2s, v0.2s+        smlsl2 v31.2d, v19.4s, v0.4s+        uzp2 v19.4s, v30.4s, v31.4s+        str q17, [x0, #0x10]+        str q18, [x0, #0x20]+        str q19, [x0, #0x30]+        str q16, [x0], #0x40+        subs x3, x3, #0x4+        cbnz x3, Lmld_polyvecl_pointwise_acc_montgomery_l5_loop_start+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(polyvecl_pointwise_acc_montgomery_l5_aarch64_asm)++#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 5) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm.S view
@@ -0,0 +1,205 @@+/* Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*yaml+  Name: polyvecl_pointwise_acc_montgomery_l7_aarch64_asm+  Description: AArch64 pointwise multiply-accumulate of length-7 polynomial vectors+  Signature: void mld_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm(int32_t r[256], const int32_t a[7][256], const int32_t b[7][256])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t r[256]+      description: Output polynomial (256 x int32_t)+    x1:+      type: buffer+      size_bytes: 7168+      permissions: read-only+      c_parameter: const int32_t a[7][256]+      description: Input polynomial vector a (7 x 256 x int32_t)+    x2:+      type: buffer+      size_bytes: 7168+      permissions: read-only+      c_parameter: const int32_t b[7][256]+      description: Input polynomial vector b (7 x 256 x int32_t)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \+           (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 7)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/aarch64_opt/src/mldsa_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(polyvecl_pointwise_acc_montgomery_l7_aarch64_asm)+MLD_ASM_FN_SYMBOL(polyvecl_pointwise_acc_montgomery_l7_aarch64_asm)++        .cfi_startproc+        mov w3, #0xe001             // =57345+        movk w3, #0x7f, lsl #16+        dup v0.4s, w3+        mov w3, #0x2001             // =8193+        movk w3, #0x380, lsl #16+        dup v1.4s, w3+        mov x3, #0x40               // =64++Lmld_polyvecl_pointwise_acc_montgomery_l7_loop_start:+        ldr q17, [x1, #0x10]+        ldr q18, [x1, #0x20]+        ldr q19, [x1, #0x30]+        ldr q16, [x1], #0x40+        ldr q21, [x2, #0x10]+        ldr q22, [x2, #0x20]+        ldr q23, [x2, #0x30]+        ldr q20, [x2], #0x40+        smull v24.2d, v16.2s, v20.2s+        smull2 v25.2d, v16.4s, v20.4s+        smull v26.2d, v17.2s, v21.2s+        smull2 v27.2d, v17.4s, v21.4s+        smull v28.2d, v18.2s, v22.2s+        smull2 v29.2d, v18.4s, v22.4s+        smull v30.2d, v19.2s, v23.2s+        smull2 v31.2d, v19.4s, v23.4s+        ldr q16, [x1, #0x3c0]+        ldr q17, [x1, #0x3d0]+        ldr q18, [x1, #0x3e0]+        ldr q19, [x1, #0x3f0]+        ldr q20, [x2, #0x3c0]+        ldr q21, [x2, #0x3d0]+        ldr q22, [x2, #0x3e0]+        ldr q23, [x2, #0x3f0]+        smlal v24.2d, v16.2s, v20.2s+        smlal2 v25.2d, v16.4s, v20.4s+        smlal v26.2d, v17.2s, v21.2s+        smlal2 v27.2d, v17.4s, v21.4s+        smlal v28.2d, v18.2s, v22.2s+        smlal2 v29.2d, v18.4s, v22.4s+        smlal v30.2d, v19.2s, v23.2s+        smlal2 v31.2d, v19.4s, v23.4s+        ldr q16, [x1, #0x7c0]+        ldr q17, [x1, #0x7d0]+        ldr q18, [x1, #0x7e0]+        ldr q19, [x1, #0x7f0]+        ldr q20, [x2, #0x7c0]+        ldr q21, [x2, #0x7d0]+        ldr q22, [x2, #0x7e0]+        ldr q23, [x2, #0x7f0]+        smlal v24.2d, v16.2s, v20.2s+        smlal2 v25.2d, v16.4s, v20.4s+        smlal v26.2d, v17.2s, v21.2s+        smlal2 v27.2d, v17.4s, v21.4s+        smlal v28.2d, v18.2s, v22.2s+        smlal2 v29.2d, v18.4s, v22.4s+        smlal v30.2d, v19.2s, v23.2s+        smlal2 v31.2d, v19.4s, v23.4s+        ldr q16, [x1, #0xbc0]+        ldr q17, [x1, #0xbd0]+        ldr q18, [x1, #0xbe0]+        ldr q19, [x1, #0xbf0]+        ldr q20, [x2, #0xbc0]+        ldr q21, [x2, #0xbd0]+        ldr q22, [x2, #0xbe0]+        ldr q23, [x2, #0xbf0]+        smlal v24.2d, v16.2s, v20.2s+        smlal2 v25.2d, v16.4s, v20.4s+        smlal v26.2d, v17.2s, v21.2s+        smlal2 v27.2d, v17.4s, v21.4s+        smlal v28.2d, v18.2s, v22.2s+        smlal2 v29.2d, v18.4s, v22.4s+        smlal v30.2d, v19.2s, v23.2s+        smlal2 v31.2d, v19.4s, v23.4s+        ldr q16, [x1, #0xfc0]+        ldr q17, [x1, #0xfd0]+        ldr q18, [x1, #0xfe0]+        ldr q19, [x1, #0xff0]+        ldr q20, [x2, #0xfc0]+        ldr q21, [x2, #0xfd0]+        ldr q22, [x2, #0xfe0]+        ldr q23, [x2, #0xff0]+        smlal v24.2d, v16.2s, v20.2s+        smlal2 v25.2d, v16.4s, v20.4s+        smlal v26.2d, v17.2s, v21.2s+        smlal2 v27.2d, v17.4s, v21.4s+        smlal v28.2d, v18.2s, v22.2s+        smlal2 v29.2d, v18.4s, v22.4s+        smlal v30.2d, v19.2s, v23.2s+        smlal2 v31.2d, v19.4s, v23.4s+        ldr q16, [x1, #0x13c0]+        ldr q17, [x1, #0x13d0]+        ldr q18, [x1, #0x13e0]+        ldr q19, [x1, #0x13f0]+        ldr q20, [x2, #0x13c0]+        ldr q21, [x2, #0x13d0]+        ldr q22, [x2, #0x13e0]+        ldr q23, [x2, #0x13f0]+        smlal v24.2d, v16.2s, v20.2s+        smlal2 v25.2d, v16.4s, v20.4s+        smlal v26.2d, v17.2s, v21.2s+        smlal2 v27.2d, v17.4s, v21.4s+        smlal v28.2d, v18.2s, v22.2s+        smlal2 v29.2d, v18.4s, v22.4s+        smlal v30.2d, v19.2s, v23.2s+        smlal2 v31.2d, v19.4s, v23.4s+        ldr q16, [x1, #0x17c0]+        ldr q17, [x1, #0x17d0]+        ldr q18, [x1, #0x17e0]+        ldr q19, [x1, #0x17f0]+        ldr q20, [x2, #0x17c0]+        ldr q21, [x2, #0x17d0]+        ldr q22, [x2, #0x17e0]+        ldr q23, [x2, #0x17f0]+        smlal v24.2d, v16.2s, v20.2s+        smlal2 v25.2d, v16.4s, v20.4s+        smlal v26.2d, v17.2s, v21.2s+        smlal2 v27.2d, v17.4s, v21.4s+        smlal v28.2d, v18.2s, v22.2s+        smlal2 v29.2d, v18.4s, v22.4s+        smlal v30.2d, v19.2s, v23.2s+        smlal2 v31.2d, v19.4s, v23.4s+        uzp1 v16.4s, v24.4s, v25.4s+        mul v16.4s, v16.4s, v1.4s+        smlsl v24.2d, v16.2s, v0.2s+        smlsl2 v25.2d, v16.4s, v0.4s+        uzp2 v16.4s, v24.4s, v25.4s+        uzp1 v17.4s, v26.4s, v27.4s+        mul v17.4s, v17.4s, v1.4s+        smlsl v26.2d, v17.2s, v0.2s+        smlsl2 v27.2d, v17.4s, v0.4s+        uzp2 v17.4s, v26.4s, v27.4s+        uzp1 v18.4s, v28.4s, v29.4s+        mul v18.4s, v18.4s, v1.4s+        smlsl v28.2d, v18.2s, v0.2s+        smlsl2 v29.2d, v18.4s, v0.4s+        uzp2 v18.4s, v28.4s, v29.4s+        uzp1 v19.4s, v30.4s, v31.4s+        mul v19.4s, v19.4s, v1.4s+        smlsl v30.2d, v19.2s, v0.2s+        smlsl2 v31.2d, v19.4s, v0.4s+        uzp2 v19.4s, v30.4s, v31.4s+        str q17, [x0, #0x10]+        str q18, [x0, #0x20]+        str q19, [x0, #0x30]+        str q16, [x0], #0x40+        subs x3, x3, #0x4+        cbnz x3, Lmld_polyvecl_pointwise_acc_montgomery_l7_loop_start+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(polyvecl_pointwise_acc_montgomery_l7_aarch64_asm)++#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 7) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/aarch64/src/mldsa_polyz_unpack_17_aarch64_asm.S view
@@ -0,0 +1,103 @@+/*+ * Copyright (c) The mldsa-native project authors+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*yaml+  Name: polyz_unpack_17_aarch64_asm+  Description: AArch64 unpacking of 17-bit packed coefficients+  Signature: void mld_polyz_unpack_17_aarch64_asm(int32_t r[256], const uint8_t buf[576], const uint8_t indices[64])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t r[256]+      description: Output polynomial (256 x int32_t)+    x1:+      type: buffer+      size_bytes: 576+      permissions: read-only+      c_parameter: const uint8_t buf[576]+      description: Packed input bytes+    x2:+      type: buffer+      size_bytes: 64+      permissions: read-only+      c_parameter: const uint8_t indices[64]+      description: Permutation index table (64 x uint8_t)+*/++ #include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_AARCH64) && \+           (!defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)) && \+           !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \+           (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/aarch64_opt/src/mldsa_polyz_unpack_17_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(polyz_unpack_17_aarch64_asm)+MLD_ASM_FN_SYMBOL(polyz_unpack_17_aarch64_asm)++        .cfi_startproc+        ldr q24, [x2]+        ldr q25, [x2, #0x10]+        ldr q26, [x2, #0x20]+        ldr q27, [x2, #0x30]+        mov x3, #0xfe00000000       // =1090921693184+        mov v28.d[0], x3+        mov x3, #0xfc               // =252+        movk x3, #0xfa, lsl #32+        mov v28.d[1], x3+        movi v29.4s, #0x3, msl #16+        movi v30.4s, #0x2, lsl #16+        mov x9, #0x10               // =16++Lmld_polyz_unpack_17_loop:+        ld1 { v0.16b, v1.16b }, [x1]+        add x1, x1, #0x14+        ld1 { v2.16b }, [x1], #16+        tbl v4.16b, { v0.16b }, v24.16b+        tbl v5.16b, { v0.16b, v1.16b }, v25.16b+        tbl v6.16b, { v1.16b }, v26.16b+        tbl v7.16b, { v1.16b, v2.16b }, v27.16b+        ushl v4.4s, v4.4s, v28.4s+        and v4.16b, v4.16b, v29.16b+        sub v4.4s, v30.4s, v4.4s+        ushl v5.4s, v5.4s, v28.4s+        and v5.16b, v5.16b, v29.16b+        sub v5.4s, v30.4s, v5.4s+        ushl v6.4s, v6.4s, v28.4s+        and v6.16b, v6.16b, v29.16b+        sub v6.4s, v30.4s, v6.4s+        ushl v7.4s, v7.4s, v28.4s+        and v7.16b, v7.16b, v29.16b+        sub v7.4s, v30.4s, v7.4s+        str q5, [x0, #0x10]+        str q6, [x0, #0x20]+        str q7, [x0, #0x30]+        str q4, [x0], #0x40+        subs x9, x9, #0x1+        b.ne Lmld_polyz_unpack_17_loop+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(polyz_unpack_17_aarch64_asm)++#endif /* MLD_ARITH_BACKEND_AARCH64 && (!MLD_CONFIG_NO_SIGN_API || \+          !MLD_CONFIG_NO_VERIFY_API) && !MLD_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44) \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/aarch64/src/mldsa_polyz_unpack_19_aarch64_asm.S view
@@ -0,0 +1,100 @@+/*+ * Copyright (c) The mldsa-native project authors+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*yaml+  Name: polyz_unpack_19_aarch64_asm+  Description: AArch64 unpacking of 19-bit packed coefficients+  Signature: void mld_polyz_unpack_19_aarch64_asm(int32_t r[256], const uint8_t buf[640], const uint8_t indices[64])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t r[256]+      description: Output polynomial (256 x int32_t)+    x1:+      type: buffer+      size_bytes: 640+      permissions: read-only+      c_parameter: const uint8_t buf[640]+      description: Packed input bytes+    x2:+      type: buffer+      size_bytes: 64+      permissions: read-only+      c_parameter: const uint8_t indices[64]+      description: Permutation index table (64 x uint8_t)+*/++ #include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_AARCH64) && \+           (!defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)) && \+           !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \+           (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87))++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/aarch64_opt/src/mldsa_polyz_unpack_19_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(polyz_unpack_19_aarch64_asm)+MLD_ASM_FN_SYMBOL(polyz_unpack_19_aarch64_asm)++        .cfi_startproc+        ldr q24, [x2]+        ldr q25, [x2, #0x10]+        ldr q26, [x2, #0x20]+        ldr q27, [x2, #0x30]+        mov x3, #0xfc00000000       // =1082331758592+        dup v28.2d, x3+        movi v29.4s, #0xf, msl #16+        movi v30.4s, #0x8, lsl #16+        mov x9, #0x10               // =16++Lmld_polyz_unpack_19_loop:+        ld1 { v0.16b, v1.16b }, [x1]+        add x1, x1, #0x18+        ld1 { v2.16b }, [x1], #16+        tbl v4.16b, { v0.16b }, v24.16b+        tbl v5.16b, { v0.16b, v1.16b }, v25.16b+        tbl v6.16b, { v1.16b }, v26.16b+        tbl v7.16b, { v1.16b, v2.16b }, v27.16b+        ushl v4.4s, v4.4s, v28.4s+        and v4.16b, v4.16b, v29.16b+        sub v4.4s, v30.4s, v4.4s+        ushl v5.4s, v5.4s, v28.4s+        and v5.16b, v5.16b, v29.16b+        sub v5.4s, v30.4s, v5.4s+        ushl v6.4s, v6.4s, v28.4s+        and v6.16b, v6.16b, v29.16b+        sub v6.4s, v30.4s, v6.4s+        ushl v7.4s, v7.4s, v28.4s+        and v7.16b, v7.16b, v29.16b+        sub v7.4s, v30.4s, v7.4s+        str q5, [x0, #0x10]+        str q6, [x0, #0x20]+        str q7, [x0, #0x30]+        str q4, [x0], #0x40+        subs x9, x9, #0x1+        b.ne Lmld_polyz_unpack_19_loop+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(polyz_unpack_19_aarch64_asm)++#endif /* MLD_ARITH_BACKEND_AARCH64 && (!MLD_CONFIG_NO_SIGN_API || \+          !MLD_CONFIG_NO_VERIFY_API) && !MLD_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/aarch64/src/mldsa_rej_uniform_aarch64_asm.S view
@@ -0,0 +1,222 @@+/*+ * Copyright (c) The mldsa-native project authors+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*yaml+  Name: rej_uniform_aarch64_asm+  Description: AArch64 rejection sampling of uniform coefficients mod q+  Signature: uint64_t mld_rej_uniform_aarch64_asm(int32_t r[256], const uint8_t *buf, unsigned buflen, const uint8_t table[256])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t r[256]+      description: Output buffer (256 x int32_t)+    x1:+      type: buffer+      size_bytes: x2+      permissions: read-only+      c_parameter: const uint8_t *buf+      description: Input buffer+    x2:+      type: scalar+      c_parameter: unsigned buflen+      description: Length of input buffer (must be a multiple of 24)+      test_with: 840  # MLD_POLY_UNIFORM_NBLOCKS * SHAKE128_RATE = 5 * 168+    x3:+      type: buffer+      size_bytes: 256+      permissions: read-only+      c_parameter: const uint8_t table[256]+      description: Lookup table (256 x uint8_t)+*/++ #include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_AARCH64) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/aarch64_opt/src/mldsa_rej_uniform_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(rej_uniform_aarch64_asm)+MLD_ASM_FN_SYMBOL(rej_uniform_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0x440+        .cfi_adjust_cfa_offset 0x440+        mov x7, #0x1                // =1+        movk x7, #0x2, lsl #32+        mov v31.d[0], x7+        mov x7, #0x4                // =4+        movk x7, #0x8, lsl #32+        mov v31.d[1], x7+        mov w7, #0xe001             // =57345+        movk w7, #0x7f, lsl #16+        dup v30.4s, w7+        mov x8, sp+        mov x7, x8+        mov x11, #0x0               // =0+        eor v16.16b, v16.16b, v16.16b++Lmld_rej_uniform_initial_zero:+        str q16, [x7], #0x40+        stur q16, [x7, #-0x30]+        stur q16, [x7, #-0x20]+        stur q16, [x7, #-0x10]+        add x11, x11, #0x10+        cmp x11, #0x100+        b.lt Lmld_rej_uniform_initial_zero+        mov x7, x8+        mov x9, #0x0                // =0+        mov x4, #0x100              // =256+        cmp x2, #0x30+        b.lo Lmld_rej_uniform_loop48_end++Lmld_rej_uniform_loop48:+        cmp x9, x4+        b.hs Lmld_rej_uniform_memory_copy+        sub x2, x2, #0x30+        ld3 { v0.16b, v1.16b, v2.16b }, [x1], #48+        movi v4.16b, #0x80+        bic v2.16b, v2.16b, v4.16b+        zip1 v4.16b, v0.16b, v1.16b+        zip2 v5.16b, v0.16b, v1.16b+        ushll v6.8h, v2.8b, #0x0+        ushll2 v7.8h, v2.16b, #0x0+        zip1 v16.8h, v4.8h, v6.8h+        zip2 v17.8h, v4.8h, v6.8h+        zip1 v18.8h, v5.8h, v7.8h+        zip2 v19.8h, v5.8h, v7.8h+        cmhi v4.4s, v30.4s, v16.4s+        cmhi v5.4s, v30.4s, v17.4s+        cmhi v6.4s, v30.4s, v18.4s+        cmhi v7.4s, v30.4s, v19.4s+        and v4.16b, v4.16b, v31.16b+        and v5.16b, v5.16b, v31.16b+        and v6.16b, v6.16b, v31.16b+        and v7.16b, v7.16b, v31.16b+        uaddlv d20, v4.4s+        uaddlv d21, v5.4s+        uaddlv d22, v6.4s+        uaddlv d23, v7.4s+        fmov x12, d20+        fmov x13, d21+        fmov x14, d22+        fmov x15, d23+        ldr q24, [x3, x12, lsl #4]+        ldr q25, [x3, x13, lsl #4]+        ldr q26, [x3, x14, lsl #4]+        ldr q27, [x3, x15, lsl #4]+        cnt v4.16b, v4.16b+        cnt v5.16b, v5.16b+        cnt v6.16b, v6.16b+        cnt v7.16b, v7.16b+        uaddlv d20, v4.4s+        uaddlv d21, v5.4s+        uaddlv d22, v6.4s+        uaddlv d23, v7.4s+        fmov x12, d20+        fmov x13, d21+        fmov x14, d22+        fmov x15, d23+        tbl v16.16b, { v16.16b }, v24.16b+        tbl v17.16b, { v17.16b }, v25.16b+        tbl v18.16b, { v18.16b }, v26.16b+        tbl v19.16b, { v19.16b }, v27.16b+        st1 { v16.4s }, [x7]+        add x7, x7, x12, lsl #2+        st1 { v17.4s }, [x7]+        add x7, x7, x13, lsl #2+        st1 { v18.4s }, [x7]+        add x7, x7, x14, lsl #2+        st1 { v19.4s }, [x7]+        add x7, x7, x15, lsl #2+        add x12, x12, x13+        add x14, x14, x15+        add x9, x9, x12+        add x9, x9, x14+        cmp x2, #0x30+        b.hs Lmld_rej_uniform_loop48++Lmld_rej_uniform_loop48_end:+        cmp x9, x4+        b.hs Lmld_rej_uniform_memory_copy+        cmp x2, #0x18+        b.lo Lmld_rej_uniform_memory_copy+        sub x2, x2, #0x18+        ld3 { v0.8b, v1.8b, v2.8b }, [x1], #24+        movi v4.16b, #0x80+        bic v2.16b, v2.16b, v4.16b+        zip1 v4.16b, v0.16b, v1.16b+        ushll v6.8h, v2.8b, #0x0+        zip1 v16.8h, v4.8h, v6.8h+        zip2 v17.8h, v4.8h, v6.8h+        cmhi v4.4s, v30.4s, v16.4s+        cmhi v5.4s, v30.4s, v17.4s+        and v4.16b, v4.16b, v31.16b+        and v5.16b, v5.16b, v31.16b+        uaddlv d20, v4.4s+        uaddlv d21, v5.4s+        fmov x12, d20+        fmov x13, d21+        ldr q24, [x3, x12, lsl #4]+        ldr q25, [x3, x13, lsl #4]+        cnt v4.16b, v4.16b+        cnt v5.16b, v5.16b+        uaddlv d20, v4.4s+        uaddlv d21, v5.4s+        fmov x12, d20+        fmov x13, d21+        tbl v16.16b, { v16.16b }, v24.16b+        tbl v17.16b, { v17.16b }, v25.16b+        st1 { v16.4s }, [x7]+        add x7, x7, x12, lsl #2+        st1 { v17.4s }, [x7]+        add x7, x7, x13, lsl #2+        add x9, x9, x12+        add x9, x9, x13++Lmld_rej_uniform_memory_copy:+        cmp x9, x4+        csel x9, x9, x4, lo+        mov x11, #0x0               // =0+        mov x7, x8++Lmld_rej_uniform_final_copy:+        ldr q16, [x7], #0x40+        ldur q17, [x7, #-0x30]+        ldur q18, [x7, #-0x20]+        ldur q19, [x7, #-0x10]+        str q16, [x0], #0x40+        stur q17, [x0, #-0x30]+        stur q18, [x0, #-0x20]+        stur q19, [x0, #-0x10]+        add x11, x11, #0x10+        cmp x11, #0x100+        b.lt Lmld_rej_uniform_final_copy+        mov x0, x9+        b Lmld_rej_uniform_return++Lmld_rej_uniform_return:+        add sp, sp, #0x440+        .cfi_adjust_cfa_offset -0x440+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(rej_uniform_aarch64_asm)++#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/aarch64/src/mldsa_rej_uniform_eta2_aarch64_asm.S view
@@ -0,0 +1,170 @@+/*+ * Copyright (c) The mldsa-native project authors+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*yaml+  Name: rej_uniform_eta2_aarch64_asm+  Description: AArch64 rejection sampling of eta=2 secret coefficients+  Signature: uint64_t mld_rej_uniform_eta2_aarch64_asm(int32_t r[256], const uint8_t *buf, unsigned buflen, const uint8_t table[4096])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t r[256]+      description: Output buffer (256 x int32_t)+    x1:+      type: buffer+      size_bytes: x2+      permissions: read-only+      c_parameter: const uint8_t *buf+      description: Input buffer+    x2:+      type: scalar+      c_parameter: unsigned buflen+      description: Length of input buffer (must be a multiple of 8)+      test_with: 136  # MLD_AARCH64_REJ_UNIFORM_ETA2_BUFLEN+    x3:+      type: buffer+      size_bytes: 4096+      permissions: read-only+      c_parameter: const uint8_t table[4096]+      description: Lookup table (4096 x uint8_t)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_AARCH64) && \+    !defined(MLD_CONFIG_NO_KEYPAIR_API) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \+    (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_ETA == 2)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/aarch64_opt/src/mldsa_rej_uniform_eta2_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(rej_uniform_eta2_aarch64_asm)+MLD_ASM_FN_SYMBOL(rej_uniform_eta2_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0x240+        .cfi_adjust_cfa_offset 0x240+        mov x7, #0x1                // =1+        movk x7, #0x2, lsl #16+        movk x7, #0x4, lsl #32+        movk x7, #0x8, lsl #48+        mov v31.d[0], x7+        mov x7, #0x10               // =16+        movk x7, #0x20, lsl #16+        movk x7, #0x40, lsl #32+        movk x7, #0x80, lsl #48+        mov v31.d[1], x7+        movi v30.8h, #0xf+        mov x8, sp+        mov x7, x8+        mov x11, #0x0               // =0+        eor v16.16b, v16.16b, v16.16b++Lmld_rej_uniform_eta2_initial_zero:+        str q16, [x7], #0x40+        stur q16, [x7, #-0x30]+        stur q16, [x7, #-0x20]+        stur q16, [x7, #-0x10]+        add x11, x11, #0x20+        cmp x11, #0x100+        b.lt Lmld_rej_uniform_eta2_initial_zero+        mov x7, x8+        mov x9, #0x0                // =0+        mov x4, #0x100              // =256++Lmld_rej_uniform_eta2_loop8:+        cmp x9, x4+        b.hs Lmld_rej_uniform_eta2_memory_copy+        sub x2, x2, #0x8+        ld1 { v0.8b }, [x1], #8+        movi v26.8b, #0xf+        and v27.8b, v0.8b, v26.8b+        ushr v28.8b, v0.8b, #0x4+        zip1 v26.8b, v27.8b, v28.8b+        zip2 v29.8b, v27.8b, v28.8b+        ushll v16.8h, v26.8b, #0x0+        ushll v17.8h, v29.8b, #0x0+        cmhi v4.8h, v30.8h, v16.8h+        cmhi v5.8h, v30.8h, v17.8h+        and v4.16b, v4.16b, v31.16b+        and v5.16b, v5.16b, v31.16b+        uaddlv s20, v4.8h+        uaddlv s21, v5.8h+        fmov w12, s20+        fmov w13, s21+        ldr q24, [x3, x12, lsl #4]+        ldr q25, [x3, x13, lsl #4]+        cnt v4.16b, v4.16b+        cnt v5.16b, v5.16b+        uaddlv s20, v4.8h+        uaddlv s21, v5.8h+        fmov w12, s20+        fmov w13, s21+        tbl v16.16b, { v16.16b }, v24.16b+        tbl v17.16b, { v17.16b }, v25.16b+        st1 { v16.8h }, [x7]+        add x7, x7, x12, lsl #1+        st1 { v17.8h }, [x7]+        add x7, x7, x13, lsl #1+        add x12, x12, x13+        add x9, x9, x12+        cmp x2, #0x8+        b.hs Lmld_rej_uniform_eta2_loop8++Lmld_rej_uniform_eta2_memory_copy:+        cmp x9, x4+        csel x9, x9, x4, lo+        mov w7, #0x199a             // =6554+        dup v26.8h, w7+        movi v27.8h, #0x5+        movi v7.8h, #0x2+        mov x11, #0x0               // =0+        mov x7, x8++Lmld_rej_uniform_eta2_final_copy:+        ldr q16, [x7], #0x20+        ldur q18, [x7, #-0x10]+        sqdmulh v28.8h, v16.8h, v26.8h+        mls v16.8h, v28.8h, v27.8h+        sqdmulh v28.8h, v18.8h, v26.8h+        mls v18.8h, v28.8h, v27.8h+        sub v16.8h, v7.8h, v16.8h+        sub v18.8h, v7.8h, v18.8h+        sshll2 v17.4s, v16.8h, #0x0+        sshll v16.4s, v16.4h, #0x0+        sshll2 v19.4s, v18.8h, #0x0+        sshll v18.4s, v18.4h, #0x0+        str q16, [x0], #0x40+        stur q17, [x0, #-0x30]+        stur q18, [x0, #-0x20]+        stur q19, [x0, #-0x10]+        add x11, x11, #0x10+        cmp x11, #0x100+        b.lt Lmld_rej_uniform_eta2_final_copy+        mov x0, x9+        add sp, sp, #0x240+        .cfi_adjust_cfa_offset -0x240+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(rej_uniform_eta2_aarch64_asm)++#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_NO_KEYPAIR_API && \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_ETA == 2) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/aarch64/src/mldsa_rej_uniform_eta4_aarch64_asm.S view
@@ -0,0 +1,163 @@+/*+ * Copyright (c) The mldsa-native project authors+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*yaml+  Name: rej_uniform_eta4_aarch64_asm+  Description: AArch64 rejection sampling of eta=4 secret coefficients+  Signature: uint64_t mld_rej_uniform_eta4_aarch64_asm(int32_t r[256], const uint8_t *buf, unsigned buflen, const uint8_t table[4096])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t r[256]+      description: Output buffer (256 x int32_t)+    x1:+      type: buffer+      size_bytes: x2+      permissions: read-only+      c_parameter: const uint8_t *buf+      description: Input buffer+    x2:+      type: scalar+      c_parameter: unsigned buflen+      description: Length of input buffer (must be a multiple of 8)+      test_with: 272  # MLD_AARCH64_REJ_UNIFORM_ETA4_BUFLEN+    x3:+      type: buffer+      size_bytes: 4096+      permissions: read-only+      c_parameter: const uint8_t table[4096]+      description: Lookup table (4096 x uint8_t)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_AARCH64) && \+    !defined(MLD_CONFIG_NO_KEYPAIR_API) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \+    (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_ETA == 4)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/aarch64_opt/src/mldsa_rej_uniform_eta4_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(rej_uniform_eta4_aarch64_asm)+MLD_ASM_FN_SYMBOL(rej_uniform_eta4_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0x240+        .cfi_adjust_cfa_offset 0x240+        mov x7, #0x1                // =1+        movk x7, #0x2, lsl #16+        movk x7, #0x4, lsl #32+        movk x7, #0x8, lsl #48+        mov v31.d[0], x7+        mov x7, #0x10               // =16+        movk x7, #0x20, lsl #16+        movk x7, #0x40, lsl #32+        movk x7, #0x80, lsl #48+        mov v31.d[1], x7+        movi v30.8h, #0x9+        movi v7.8h, #0x4+        mov x8, sp+        mov x7, x8+        mov x11, #0x0               // =0+        eor v16.16b, v16.16b, v16.16b++Lmld_rej_uniform_eta4_initial_zero:+        str q16, [x7], #0x40+        stur q16, [x7, #-0x30]+        stur q16, [x7, #-0x20]+        stur q16, [x7, #-0x10]+        add x11, x11, #0x20+        cmp x11, #0x100+        b.lt Lmld_rej_uniform_eta4_initial_zero+        mov x7, x8+        mov x9, #0x0                // =0+        mov x4, #0x100              // =256++Lmld_rej_uniform_eta4_loop8:+        cmp x9, x4+        b.hs Lmld_rej_uniform_eta4_memory_copy+        sub x2, x2, #0x8+        ld1 { v0.8b }, [x1], #8+        movi v26.8b, #0xf+        and v27.8b, v0.8b, v26.8b+        ushr v28.8b, v0.8b, #0x4+        zip1 v26.8b, v27.8b, v28.8b+        zip2 v29.8b, v27.8b, v28.8b+        ushll v16.8h, v26.8b, #0x0+        ushll v17.8h, v29.8b, #0x0+        cmhi v4.8h, v30.8h, v16.8h+        cmhi v5.8h, v30.8h, v17.8h+        and v4.16b, v4.16b, v31.16b+        and v5.16b, v5.16b, v31.16b+        uaddlv s20, v4.8h+        uaddlv s21, v5.8h+        fmov w12, s20+        fmov w13, s21+        ldr q24, [x3, x12, lsl #4]+        ldr q25, [x3, x13, lsl #4]+        cnt v4.16b, v4.16b+        cnt v5.16b, v5.16b+        uaddlv s20, v4.8h+        uaddlv s21, v5.8h+        fmov w12, s20+        fmov w13, s21+        tbl v16.16b, { v16.16b }, v24.16b+        tbl v17.16b, { v17.16b }, v25.16b+        st1 { v16.8h }, [x7]+        add x7, x7, x12, lsl #1+        st1 { v17.8h }, [x7]+        add x7, x7, x13, lsl #1+        add x12, x12, x13+        add x9, x9, x12+        cmp x2, #0x8+        b.hs Lmld_rej_uniform_eta4_loop8++Lmld_rej_uniform_eta4_memory_copy:+        cmp x9, x4+        csel x9, x9, x4, lo+        mov x11, #0x0               // =0+        mov x7, x8++Lmld_rej_uniform_eta4_final_copy:+        ldr q16, [x7], #0x20+        ldur q18, [x7, #-0x10]+        sub v16.8h, v7.8h, v16.8h+        sub v18.8h, v7.8h, v18.8h+        sshll2 v17.4s, v16.8h, #0x0+        sshll v16.4s, v16.4h, #0x0+        sshll2 v19.4s, v18.8h, #0x0+        sshll v18.4s, v18.4h, #0x0+        str q16, [x0], #0x40+        stur q17, [x0, #-0x30]+        stur q18, [x0, #-0x20]+        stur q19, [x0, #-0x10]+        add x11, x11, #0x10+        cmp x11, #0x100+        b.lt Lmld_rej_uniform_eta4_final_copy+        mov x0, x9+        add sp, sp, #0x240+        .cfi_adjust_cfa_offset -0x240+        ret+        .cfi_endproc++MLD_ASM_FN_SIZE(rej_uniform_eta4_aarch64_asm)++#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_NO_KEYPAIR_API && \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_ETA == 4) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/aarch64/src/polyz_unpack_table.c view
@@ -0,0 +1,52 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mldsa-native repository.+ *          Do not modify it directly.+ */++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_AARCH64) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++#include "arith_native_aarch64.h"++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+/* Table of indices used for tbl instructions in polyz_unpack_{17,19}.+ * See autogen for details. */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44+MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint8_t+    mld_polyz_unpack_17_indices[64] = {+        0,  1,  2,  255, 2,  3,  4,  255, 4,  5,  6,  255, 6,  7,  8,  255,+        9,  10, 11, 255, 11, 12, 13, 255, 13, 14, 15, 255, 15, 16, 17, 255,+        2,  3,  4,  255, 4,  5,  6,  255, 6,  7,  8,  255, 8,  9,  10, 255,+        11, 12, 13, 255, 13, 14, 15, 255, 15, 28, 29, 255, 29, 30, 31, 255,+};+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \+        */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)+MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint8_t+    mld_polyz_unpack_19_indices[64] = {+        0,  1,  2,  255, 2,  3,  4,  255, 5,  6,  7,  255, 7,  8,  9,  255,+        10, 11, 12, 255, 12, 13, 14, 255, 15, 16, 17, 255, 17, 18, 19, 255,+        4,  5,  6,  255, 6,  7,  8,  255, 9,  10, 11, 255, 11, 12, 13, 255,+        14, 15, 24, 255, 24, 25, 26, 255, 27, 28, 29, 255, 29, 30, 31, 255,+};+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87 */++#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */++#else /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */++MLD_EMPTY_CU(aarch64_polyz_unpack_table)++#endif /* !(MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED) */
+ cbits/mldsa/src/native/aarch64/src/rej_uniform_eta_table.c view
@@ -0,0 +1,547 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mldsa-native repository.+ *          Do not modify it directly.+ */++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_AARCH64) &&  \+    !defined(MLD_CONFIG_NO_KEYPAIR_API) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++#include "arith_native_aarch64.h"++/*+ * Lookup table used by 16-bit rejection sampling (rej_eta).+ * Adapted from ML-KEM for ML-DSA eta rejection sampling.+ * See autogen for details.+ */+MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint8_t+    mld_rej_uniform_eta_table[4096] = {+        255, 255, 255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 0 */,+        0,   1,   255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 1 */,+        2,   3,   255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 2 */,+        0,   1,   2,   3,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 3 */,+        4,   5,   255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 4 */,+        0,   1,   4,   5,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 5 */,+        2,   3,   4,   5,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 6 */,+        0,   1,   2,   3,   4,   5,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 7 */,+        6,   7,   255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 8 */,+        0,   1,   6,   7,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 9 */,+        2,   3,   6,   7,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 10 */,+        0,   1,   2,   3,   6,   7,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 11 */,+        4,   5,   6,   7,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 12 */,+        0,   1,   4,   5,   6,   7,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 13 */,+        2,   3,   4,   5,   6,   7,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 14 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        255, 255, 255, 255, 255, 255, 255, 255 /* 15 */,+        8,   9,   255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 16 */,+        0,   1,   8,   9,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 17 */,+        2,   3,   8,   9,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 18 */,+        0,   1,   2,   3,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 19 */,+        4,   5,   8,   9,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 20 */,+        0,   1,   4,   5,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 21 */,+        2,   3,   4,   5,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 22 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        255, 255, 255, 255, 255, 255, 255, 255 /* 23 */,+        6,   7,   8,   9,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 24 */,+        0,   1,   6,   7,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 25 */,+        2,   3,   6,   7,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 26 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        255, 255, 255, 255, 255, 255, 255, 255 /* 27 */,+        4,   5,   6,   7,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 28 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        255, 255, 255, 255, 255, 255, 255, 255 /* 29 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        255, 255, 255, 255, 255, 255, 255, 255 /* 30 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   255, 255, 255, 255, 255, 255 /* 31 */,+        10,  11,  255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 32 */,+        0,   1,   10,  11,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 33 */,+        2,   3,   10,  11,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 34 */,+        0,   1,   2,   3,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 35 */,+        4,   5,   10,  11,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 36 */,+        0,   1,   4,   5,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 37 */,+        2,   3,   4,   5,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 38 */,+        0,   1,   2,   3,   4,   5,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 39 */,+        6,   7,   10,  11,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 40 */,+        0,   1,   6,   7,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 41 */,+        2,   3,   6,   7,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 42 */,+        0,   1,   2,   3,   6,   7,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 43 */,+        4,   5,   6,   7,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 44 */,+        0,   1,   4,   5,   6,   7,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 45 */,+        2,   3,   4,   5,   6,   7,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 46 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        10,  11,  255, 255, 255, 255, 255, 255 /* 47 */,+        8,   9,   10,  11,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 48 */,+        0,   1,   8,   9,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 49 */,+        2,   3,   8,   9,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 50 */,+        0,   1,   2,   3,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 51 */,+        4,   5,   8,   9,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 52 */,+        0,   1,   4,   5,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 53 */,+        2,   3,   4,   5,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 54 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        10,  11,  255, 255, 255, 255, 255, 255 /* 55 */,+        6,   7,   8,   9,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 56 */,+        0,   1,   6,   7,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 57 */,+        2,   3,   6,   7,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 58 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        10,  11,  255, 255, 255, 255, 255, 255 /* 59 */,+        4,   5,   6,   7,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 60 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        10,  11,  255, 255, 255, 255, 255, 255 /* 61 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        10,  11,  255, 255, 255, 255, 255, 255 /* 62 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   10,  11,  255, 255, 255, 255 /* 63 */,+        12,  13,  255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 64 */,+        0,   1,   12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 65 */,+        2,   3,   12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 66 */,+        0,   1,   2,   3,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 67 */,+        4,   5,   12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 68 */,+        0,   1,   4,   5,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 69 */,+        2,   3,   4,   5,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 70 */,+        0,   1,   2,   3,   4,   5,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 71 */,+        6,   7,   12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 72 */,+        0,   1,   6,   7,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 73 */,+        2,   3,   6,   7,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 74 */,+        0,   1,   2,   3,   6,   7,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 75 */,+        4,   5,   6,   7,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 76 */,+        0,   1,   4,   5,   6,   7,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 77 */,+        2,   3,   4,   5,   6,   7,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 78 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        12,  13,  255, 255, 255, 255, 255, 255 /* 79 */,+        8,   9,   12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 80 */,+        0,   1,   8,   9,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 81 */,+        2,   3,   8,   9,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 82 */,+        0,   1,   2,   3,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 83 */,+        4,   5,   8,   9,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 84 */,+        0,   1,   4,   5,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 85 */,+        2,   3,   4,   5,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 86 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        12,  13,  255, 255, 255, 255, 255, 255 /* 87 */,+        6,   7,   8,   9,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 88 */,+        0,   1,   6,   7,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 89 */,+        2,   3,   6,   7,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 90 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        12,  13,  255, 255, 255, 255, 255, 255 /* 91 */,+        4,   5,   6,   7,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 92 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        12,  13,  255, 255, 255, 255, 255, 255 /* 93 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        12,  13,  255, 255, 255, 255, 255, 255 /* 94 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   12,  13,  255, 255, 255, 255 /* 95 */,+        10,  11,  12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 96 */,+        0,   1,   10,  11,  12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 97 */,+        2,   3,   10,  11,  12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 98 */,+        0,   1,   2,   3,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 99 */,+        4,   5,   10,  11,  12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 100 */,+        0,   1,   4,   5,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 101 */,+        2,   3,   4,   5,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 102 */,+        0,   1,   2,   3,   4,   5,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 103 */,+        6,   7,   10,  11,  12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 104 */,+        0,   1,   6,   7,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 105 */,+        2,   3,   6,   7,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 106 */,+        0,   1,   2,   3,   6,   7,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 107 */,+        4,   5,   6,   7,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 108 */,+        0,   1,   4,   5,   6,   7,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 109 */,+        2,   3,   4,   5,   6,   7,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 110 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        10,  11,  12,  13,  255, 255, 255, 255 /* 111 */,+        8,   9,   10,  11,  12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 112 */,+        0,   1,   8,   9,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 113 */,+        2,   3,   8,   9,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 114 */,+        0,   1,   2,   3,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 115 */,+        4,   5,   8,   9,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 116 */,+        0,   1,   4,   5,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 117 */,+        2,   3,   4,   5,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 118 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        10,  11,  12,  13,  255, 255, 255, 255 /* 119 */,+        6,   7,   8,   9,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 120 */,+        0,   1,   6,   7,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 121 */,+        2,   3,   6,   7,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 122 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        10,  11,  12,  13,  255, 255, 255, 255 /* 123 */,+        4,   5,   6,   7,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 124 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        10,  11,  12,  13,  255, 255, 255, 255 /* 125 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        10,  11,  12,  13,  255, 255, 255, 255 /* 126 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   10,  11,  12,  13,  255, 255 /* 127 */,+        14,  15,  255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 128 */,+        0,   1,   14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 129 */,+        2,   3,   14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 130 */,+        0,   1,   2,   3,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 131 */,+        4,   5,   14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 132 */,+        0,   1,   4,   5,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 133 */,+        2,   3,   4,   5,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 134 */,+        0,   1,   2,   3,   4,   5,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 135 */,+        6,   7,   14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 136 */,+        0,   1,   6,   7,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 137 */,+        2,   3,   6,   7,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 138 */,+        0,   1,   2,   3,   6,   7,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 139 */,+        4,   5,   6,   7,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 140 */,+        0,   1,   4,   5,   6,   7,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 141 */,+        2,   3,   4,   5,   6,   7,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 142 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        14,  15,  255, 255, 255, 255, 255, 255 /* 143 */,+        8,   9,   14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 144 */,+        0,   1,   8,   9,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 145 */,+        2,   3,   8,   9,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 146 */,+        0,   1,   2,   3,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 147 */,+        4,   5,   8,   9,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 148 */,+        0,   1,   4,   5,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 149 */,+        2,   3,   4,   5,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 150 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        14,  15,  255, 255, 255, 255, 255, 255 /* 151 */,+        6,   7,   8,   9,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 152 */,+        0,   1,   6,   7,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 153 */,+        2,   3,   6,   7,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 154 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        14,  15,  255, 255, 255, 255, 255, 255 /* 155 */,+        4,   5,   6,   7,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 156 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        14,  15,  255, 255, 255, 255, 255, 255 /* 157 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        14,  15,  255, 255, 255, 255, 255, 255 /* 158 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   14,  15,  255, 255, 255, 255 /* 159 */,+        10,  11,  14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 160 */,+        0,   1,   10,  11,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 161 */,+        2,   3,   10,  11,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 162 */,+        0,   1,   2,   3,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 163 */,+        4,   5,   10,  11,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 164 */,+        0,   1,   4,   5,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 165 */,+        2,   3,   4,   5,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 166 */,+        0,   1,   2,   3,   4,   5,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 167 */,+        6,   7,   10,  11,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 168 */,+        0,   1,   6,   7,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 169 */,+        2,   3,   6,   7,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 170 */,+        0,   1,   2,   3,   6,   7,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 171 */,+        4,   5,   6,   7,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 172 */,+        0,   1,   4,   5,   6,   7,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 173 */,+        2,   3,   4,   5,   6,   7,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 174 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        10,  11,  14,  15,  255, 255, 255, 255 /* 175 */,+        8,   9,   10,  11,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 176 */,+        0,   1,   8,   9,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 177 */,+        2,   3,   8,   9,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 178 */,+        0,   1,   2,   3,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 179 */,+        4,   5,   8,   9,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 180 */,+        0,   1,   4,   5,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 181 */,+        2,   3,   4,   5,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 182 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        10,  11,  14,  15,  255, 255, 255, 255 /* 183 */,+        6,   7,   8,   9,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 184 */,+        0,   1,   6,   7,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 185 */,+        2,   3,   6,   7,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 186 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        10,  11,  14,  15,  255, 255, 255, 255 /* 187 */,+        4,   5,   6,   7,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 188 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        10,  11,  14,  15,  255, 255, 255, 255 /* 189 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        10,  11,  14,  15,  255, 255, 255, 255 /* 190 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   10,  11,  14,  15,  255, 255 /* 191 */,+        12,  13,  14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 192 */,+        0,   1,   12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 193 */,+        2,   3,   12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 194 */,+        0,   1,   2,   3,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 195 */,+        4,   5,   12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 196 */,+        0,   1,   4,   5,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 197 */,+        2,   3,   4,   5,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 198 */,+        0,   1,   2,   3,   4,   5,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 199 */,+        6,   7,   12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 200 */,+        0,   1,   6,   7,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 201 */,+        2,   3,   6,   7,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 202 */,+        0,   1,   2,   3,   6,   7,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 203 */,+        4,   5,   6,   7,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 204 */,+        0,   1,   4,   5,   6,   7,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 205 */,+        2,   3,   4,   5,   6,   7,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 206 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        12,  13,  14,  15,  255, 255, 255, 255 /* 207 */,+        8,   9,   12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 208 */,+        0,   1,   8,   9,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 209 */,+        2,   3,   8,   9,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 210 */,+        0,   1,   2,   3,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 211 */,+        4,   5,   8,   9,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 212 */,+        0,   1,   4,   5,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 213 */,+        2,   3,   4,   5,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 214 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        12,  13,  14,  15,  255, 255, 255, 255 /* 215 */,+        6,   7,   8,   9,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 216 */,+        0,   1,   6,   7,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 217 */,+        2,   3,   6,   7,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 218 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        12,  13,  14,  15,  255, 255, 255, 255 /* 219 */,+        4,   5,   6,   7,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 220 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        12,  13,  14,  15,  255, 255, 255, 255 /* 221 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        12,  13,  14,  15,  255, 255, 255, 255 /* 222 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   12,  13,  14,  15,  255, 255 /* 223 */,+        10,  11,  12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 224 */,+        0,   1,   10,  11,  12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 225 */,+        2,   3,   10,  11,  12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 226 */,+        0,   1,   2,   3,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 227 */,+        4,   5,   10,  11,  12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 228 */,+        0,   1,   4,   5,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 229 */,+        2,   3,   4,   5,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 230 */,+        0,   1,   2,   3,   4,   5,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 231 */,+        6,   7,   10,  11,  12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 232 */,+        0,   1,   6,   7,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 233 */,+        2,   3,   6,   7,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 234 */,+        0,   1,   2,   3,   6,   7,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 235 */,+        4,   5,   6,   7,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 236 */,+        0,   1,   4,   5,   6,   7,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 237 */,+        2,   3,   4,   5,   6,   7,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 238 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        10,  11,  12,  13,  14,  15,  255, 255 /* 239 */,+        8,   9,   10,  11,  12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 240 */,+        0,   1,   8,   9,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 241 */,+        2,   3,   8,   9,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 242 */,+        0,   1,   2,   3,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 243 */,+        4,   5,   8,   9,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 244 */,+        0,   1,   4,   5,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 245 */,+        2,   3,   4,   5,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 246 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        10,  11,  12,  13,  14,  15,  255, 255 /* 247 */,+        6,   7,   8,   9,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 248 */,+        0,   1,   6,   7,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 249 */,+        2,   3,   6,   7,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 250 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        10,  11,  12,  13,  14,  15,  255, 255 /* 251 */,+        4,   5,   6,   7,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 252 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        10,  11,  12,  13,  14,  15,  255, 255 /* 253 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        10,  11,  12,  13,  14,  15,  255, 255 /* 254 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   10,  11,  12,  13,  14,  15 /* 255 */,+};++#else /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_NO_KEYPAIR_API && \+         !MLD_CONFIG_MULTILEVEL_NO_SHARED */++MLD_EMPTY_CU(aarch64_rej_uniform_eta_table)++#endif /* !(MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_NO_KEYPAIR_API && \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED) */
+ cbits/mldsa/src/native/aarch64/src/rej_uniform_table.c view
@@ -0,0 +1,63 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mldsa-native repository.+ *          Do not modify it directly.+ */++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_AARCH64) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++#include "arith_native_aarch64.h"++/*+ * Lookup table used by rejection sampling of the public matrix.+ * See autogen for details.+ */+MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint8_t+    mld_rej_uniform_table[256] = {+        255, 255, 255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 0 */,+        0,   1,   2,   3,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 1 */,+        4,   5,   6,   7,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 2 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        255, 255, 255, 255, 255, 255, 255, 255 /* 3 */,+        8,   9,   10,  11,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 4 */,+        0,   1,   2,   3,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 5 */,+        4,   5,   6,   7,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 6 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   10,  11,  255, 255, 255, 255 /* 7 */,+        12,  13,  14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 8 */,+        0,   1,   2,   3,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 9 */,+        4,   5,   6,   7,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 10 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        12,  13,  14,  15,  255, 255, 255, 255 /* 11 */,+        8,   9,   10,  11,  12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 12 */,+        0,   1,   2,   3,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 13 */,+        4,   5,   6,   7,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 14 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   10,  11,  12,  13,  14,  15 /* 15 */,+};++#else /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */++MLD_EMPTY_CU(aarch64_rej_uniform_table)++#endif /* !(MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED) */
+ cbits/mldsa/src/native/api.h view
@@ -0,0 +1,617 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLD_NATIVE_API_H+#define MLD_NATIVE_API_H+/*+ * Native arithmetic interface+ *+ * This header is primarily for documentation purposes.+ * It should not be included by backend implementations.+ *+ * To ensure consistency with backends, the header will be+ * included automatically after inclusion of the active+ * backend, to ensure consistency of function signatures,+ * and run sanity checks.+ */++#include "../cbmc.h"+#include "../common.h"++/* Backends must return MLD_NATIVE_FUNC_SUCCESS upon success. */+#define MLD_NATIVE_FUNC_SUCCESS (0)+/* Backends may return MLD_NATIVE_FUNC_FALLBACK to signal to the frontend that+ * the target/parameters are unsupported; typically, this would be because of+ * dependencies on CPU features not detected on the host CPU. In this case,+ * the frontend falls back to the default C implementation.+ *+ * IMPORTANT: Backend implementations must ensure that the decision of whether+ * to fallback (return MLD_NATIVE_FUNC_FALLBACK) or not must never depend on+ * the input data itself. Fallback decisions may only depend on system+ * capabilities (e.g., CPU features) and, where present, length information.+ * This requirement applies to all backend functions to maintain constant-time+ * properties.+ */+#define MLD_NATIVE_FUNC_FALLBACK (-1)++/* Absolute exclusive upper bound for the output of fqmul.+ *+ * NOTE: This is the same bound as in poly.h and has to be kept+ * in sync. */+#define MLD_FQMUL_BOUND ((5 * MLDSA_Q + 3) / 4)++/* Bound on absolute value of coefficients after NTT.+ *+ * NOTE: This is the same bound as in poly.h and has to be kept+ * in sync. */+#define MLD_NTT_BOUND (9 * MLD_FQMUL_BOUND)++/* Absolute exclusive upper bound for the output of the inverse NTT+ *+ * NOTE: This is the same bound as in poly.h and has to be kept+ * in sync. */+#define MLD_INTT_BOUND MLDSA_Q++/* Absolute bound for range of mld_reduce32()+ *+ * NOTE: This is the same bound as in reduce.h and has to be kept+ * in sync. */+/* check-magic: 6283009 == (MLD_REDUCE32_DOMAIN_MAX - 255 * MLDSA_Q + 1) */+#define MLD_REDUCE32_RANGE_MAX 6283009+/*+ * This is the C<->native interface allowing for the drop-in of+ * native code for performance-critical arithmetic components of ML-DSA.+ *+ * A _backend_ is a specific implementation of (part of) this interface.+ *+ * To add a function to a backend, define MLD_USE_NATIVE_XXX and+ * implement `static inline xxx(...)` in the profile header.+ */++/*+ * Those functions are meant to be trivial wrappers around the chosen native+ * implementation. The are static inline to avoid unnecessary calls.+ * The macro before each declaration controls whether a native+ * implementation is present.+ */++#if defined(MLD_USE_NATIVE_NTT)+/**+ * Computes negacyclic number-theoretic transform (NTT) of a polynomial+ * in place.+ *+ * The input polynomial is assumed to be in normal order. The output+ * polynomial is in bitreversed order.+ *+ * @param[in,out] p Pointer to in/output polynomial.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_ntt_native(int32_t p[MLDSA_N])+__contract__(+    requires(memory_no_alias(p, sizeof(int32_t) * MLDSA_N))+    requires(array_abs_bound(p, 0, MLDSA_N, MLDSA_Q))+    assigns(memory_slice(p, sizeof(int32_t) * MLDSA_N))+    ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS)+    ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(p, 0, MLDSA_N, MLD_NTT_BOUND))+    ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_abs_bound(p, 0, MLDSA_N, MLDSA_Q))+    ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(p, MLDSA_N))+);+#endif /* MLD_USE_NATIVE_NTT */+++#if defined(MLD_USE_NATIVE_NTT_CUSTOM_ORDER)+/*+ * This must only be set if NTT and INTT have native implementations+ * that are adapted to the custom order.+ */+#if !defined(MLD_USE_NATIVE_NTT) || !defined(MLD_USE_NATIVE_INTT)+#error \+    "Invalid native profile: MLD_USE_NATIVE_NTT_CUSTOM_ORDER can only be \+set if there are native implementations for NTT and INTT."+#endif++/**+ * When MLD_USE_NATIVE_NTT_CUSTOM_ORDER is defined, convert a polynomial in+ * NTT domain from bitreversed order to the custom order output by the native+ * NTT.+ *+ * This must only be defined if there is native code for both the NTT and+ * INTT.+ *+ * @param[in,out] p Pointer to in/output polynomial.+ */+static MLD_INLINE void mld_poly_permute_bitrev_to_custom(int32_t p[MLDSA_N])+__contract__(+  /* We don't specify that this should be a permutation, but only+   * that it does not change the bound established at the end of+   * mld_polyvec_matrix_expand.+   */+  requires(memory_no_alias(p, sizeof(int32_t) * MLDSA_N))+  requires(array_bound(p, 0, MLDSA_N, 0, MLDSA_Q))+  assigns(memory_slice(p, sizeof(int32_t) * MLDSA_N))+  ensures(array_bound(p, 0, MLDSA_N, 0, MLDSA_Q)));+#endif /* MLD_USE_NATIVE_NTT_CUSTOM_ORDER */+++#if defined(MLD_USE_NATIVE_INTT)+/**+ * Computes inverse of negacyclic number-theoretic transform (NTT) of a+ * polynomial in place.+ *+ * The input polynomial is in bitreversed order. The output polynomial is+ * assumed to be in normal order.+ *+ * @param[in,out] p Pointer to in/output polynomial.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_intt_native(int32_t p[MLDSA_N])+__contract__(+  requires(memory_no_alias(p, sizeof(int32_t) * MLDSA_N))+  requires(array_abs_bound(p, 0, MLDSA_N, MLDSA_Q))+  assigns(memory_slice(p, sizeof(int32_t) * MLDSA_N))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(p, 0, MLDSA_N, MLD_INTT_BOUND))+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_abs_bound(p, 0, MLDSA_N, MLDSA_Q))+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(p, MLDSA_N))+);+#endif /* MLD_USE_NATIVE_INTT */++#if defined(MLD_USE_NATIVE_REJ_UNIFORM)+/**+ * Run rejection sampling on uniform random bytes to generate uniform random+ * integers in [0, MLDSA_Q-1].+ *+ * @param[out] r      Pointer to output buffer.+ * @param      len    Requested number of 32-bit integers (uniform mod+ *                    MLDSA_Q).+ * @param[in]  buf    Pointer to input buffer (assumed to be uniform random+ *                    bytes).+ * @param      buflen Length of input buffer in bytes.+ *+ * @return - MLD_NATIVE_FUNC_FALLBACK if the native implementation does not+ *           support the input lengths.+ *         - Otherwise, the non-negative number of sampled 32-bit integers+ *           (at most len).+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_rej_uniform_native(int32_t *r, unsigned len,+                                             const uint8_t *buf,+                                             unsigned buflen)+__contract__(+  requires(len <= MLDSA_N)+  requires(buflen <= ( 5 * 168) && buflen % 3 == 0)+  requires(memory_no_alias(r, sizeof(int32_t) * len))+  requires(memory_no_alias(buf, buflen))+  assigns(memory_slice(r, sizeof(int32_t) * len))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || (0 <= return_value && return_value <= len))+  ensures((return_value != MLD_NATIVE_FUNC_FALLBACK) ==> array_bound(r, 0, (unsigned) return_value, 0, MLDSA_Q))+);+#endif /* MLD_USE_NATIVE_REJ_UNIFORM */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+#if defined(MLD_USE_NATIVE_REJ_UNIFORM_ETA2)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_ETA == 2+/**+ * Run rejection sampling on uniform random bytes to generate uniform random+ * integers in [-2, +2].+ *+ * @param[out] r      Pointer to output buffer.+ * @param      len    Requested number of 32-bit integers (uniform in+ *                    [-2, +2]).+ * @param[in]  buf    Pointer to input buffer (assumed to be uniform random+ *                    bytes).+ * @param      buflen Length of input buffer in bytes.+ *+ * @return - MLD_NATIVE_FUNC_FALLBACK if the native implementation does not+ *           support the input lengths.+ *         - Otherwise, the non-negative number of sampled 32-bit integers+ *           (at most len).+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_rej_uniform_eta2_native(int32_t *r, unsigned len,+                                                  const uint8_t *buf,+                                                  unsigned buflen)+__contract__(+  requires(len <= MLDSA_N)+  requires(buflen <= (2 * 136))+  requires(memory_no_alias(r, sizeof(int32_t) * len))+  requires(memory_no_alias(buf, buflen))+  assigns(memory_slice(r, sizeof(int32_t) * len))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || (0 <= return_value && return_value <= len))+  /* check-magic: 3 == 2 + 1 (decl gated on MLDSA_ETA == 2) */+  ensures((return_value != MLD_NATIVE_FUNC_FALLBACK) ==> (array_abs_bound(r, 0, return_value, 3)))+);+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_ETA == 2 */+#endif /* MLD_USE_NATIVE_REJ_UNIFORM_ETA2 */++#if defined(MLD_USE_NATIVE_REJ_UNIFORM_ETA4)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_ETA == 4+/**+ * Run rejection sampling on uniform random bytes to generate uniform random+ * integers in [-4, +4].+ *+ * @param[out] r      Pointer to output buffer.+ * @param      len    Requested number of 32-bit integers (uniform in+ *                    [-4, +4]).+ * @param[in]  buf    Pointer to input buffer (assumed to be uniform random+ *                    bytes).+ * @param      buflen Length of input buffer in bytes.+ *+ * @return - MLD_NATIVE_FUNC_FALLBACK if the native implementation does not+ *           support the input lengths.+ *         - Otherwise, the non-negative number of sampled 32-bit integers+ *           (at most len).+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_rej_uniform_eta4_native(int32_t *r, unsigned len,+                                                  const uint8_t *buf,+                                                  unsigned buflen)+__contract__(+  requires(len <= MLDSA_N)+  requires(buflen <= (2 * 136))+  requires(memory_no_alias(r, sizeof(int32_t) * len))+  requires(memory_no_alias(buf, buflen))+  assigns(memory_slice(r, sizeof(int32_t) * len))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || (0 <= return_value && return_value <= len))+  /* check-magic: 5 == 4 + 1 (decl gated on MLDSA_ETA == 4) */+  ensures((return_value != MLD_NATIVE_FUNC_FALLBACK) ==> (array_abs_bound(r, 0, return_value, 5)))+);+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_ETA == 4 */+#endif /* MLD_USE_NATIVE_REJ_UNIFORM_ETA4 */+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+#if defined(MLD_USE_NATIVE_POLY_DECOMPOSE_32)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)+/**+ * Native implementation of poly_decompose for GAMMA2 = (MLDSA_Q-1)/32.+ *+ * For all coefficients c of the input polynomial, compute high and low bits+ * c0, c1 such c mod MLDSA_Q = c1*(2*GAMMA2) + c0 with+ * -(2*GAMMA2)/2 < c0 <= (2*GAMMA2)/2 except c1 = (MLDSA_Q-1)/(2*GAMMA2) where+ * we set c1 = 0 and -(2*GAMMA2)/2 <= c0 = c mod MLDSA_Q - MLDSA_Q < 0.+ * Assumes coefficients to be standard representatives.+ *+ * @param[out]    a1 Output polynomial with coefficients c1.+ * @param[in,out] a0 Input/output polynomial. Output has coefficients c0.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_decompose_32_native(int32_t *a1, int32_t *a0)+__contract__(+  requires(memory_no_alias(a1,  sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(a0, sizeof(int32_t) * MLDSA_N))+  requires(array_bound(a0, 0, MLDSA_N, 0, MLDSA_Q))+  assigns(memory_slice(a1, sizeof(int32_t) * MLDSA_N))+  assigns(memory_slice(a0, sizeof(int32_t) * MLDSA_N))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_bound(a1, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2)))+  ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(a0, 0, MLDSA_N, MLDSA_GAMMA2+1))+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_bound(a0, 0, MLDSA_N, 0, MLDSA_Q))+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(a0, MLDSA_N))+);+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87 */+#endif /* MLD_USE_NATIVE_POLY_DECOMPOSE_32 */++#if defined(MLD_USE_NATIVE_POLY_DECOMPOSE_88)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44+/**+ * Native implementation of poly_decompose for GAMMA2 = (MLDSA_Q-1)/88.+ *+ * For all coefficients c of the input polynomial, compute high and low bits+ * c0, c1 such c mod MLDSA_Q = c1*(2*GAMMA2) + c0 with+ * -(2*GAMMA2)/2 < c0 <= (2*GAMMA2)/2 except c1 = (MLDSA_Q-1)/(2*GAMMA2) where+ * we set c1 = 0 and -(2*GAMMA2)/2 <= c0 = c mod MLDSA_Q - MLDSA_Q < 0.+ * Assumes coefficients to be standard representatives.+ *+ * @param[out]    a1 Output polynomial with coefficients c1.+ * @param[in,out] a0 Input/output polynomial. Output has coefficients c0.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_decompose_88_native(int32_t *a1, int32_t *a0)+__contract__(+  requires(memory_no_alias(a1,  sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(a0, sizeof(int32_t) * MLDSA_N))+  requires(array_bound(a0, 0, MLDSA_N, 0, MLDSA_Q))+  assigns(memory_slice(a1, sizeof(int32_t) * MLDSA_N))+  assigns(memory_slice(a0, sizeof(int32_t) * MLDSA_N))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_bound(a1, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2)))+  ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(a0, 0, MLDSA_N, MLDSA_GAMMA2+1))+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_bound(a0, 0, MLDSA_N, 0, MLDSA_Q))+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(a0, MLDSA_N))+);+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \+        */+#endif /* MLD_USE_NATIVE_POLY_DECOMPOSE_88 */+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if defined(MLD_USE_NATIVE_POLY_CADDQ)+/**+ * For all coefficients of in/out polynomial add Q if coefficient is negative.+ *+ * @param[in,out] a Pointer to input/output polynomial.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_caddq_native(int32_t a[MLDSA_N])+__contract__(+  requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N))+  requires(array_abs_bound(a, 0, MLDSA_N, MLDSA_Q))+  assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_bound(a, 0, MLDSA_N, 0, MLDSA_Q))+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_abs_bound(a, 0, MLDSA_N, MLDSA_Q))+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(a, MLDSA_N))+);+#endif /* MLD_USE_NATIVE_POLY_CADDQ */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+#if defined(MLD_USE_NATIVE_POLY_USE_HINT_32)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)+/**+ * Native implementation of poly_use_hint for GAMMA2 = (MLDSA_Q-1)/32.+ *+ * Use hint h to correct the high bits of a in-place.+ *+ * @param[in,out] a Input/output polynomial.+ * @param[in]     h Hint polynomial.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_use_hint_32_native(int32_t *a, const int32_t *h)+__contract__(+  requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(h, sizeof(int32_t) * MLDSA_N))+  requires(array_bound(a, 0, MLDSA_N, 0, MLDSA_Q))+  requires(array_bound(h, 0, MLDSA_N, 0, 2))+  assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_bound(a, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2)))+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(a, MLDSA_N))+);+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87 */+#endif /* MLD_USE_NATIVE_POLY_USE_HINT_32 */++#if defined(MLD_USE_NATIVE_POLY_USE_HINT_88)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44+/**+ * Native implementation of poly_use_hint for GAMMA2 = (MLDSA_Q-1)/88.+ *+ * Use hint h to correct the high bits of a in-place.+ *+ * @param[in,out] a Input/output polynomial.+ * @param[in]     h Hint polynomial.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_use_hint_88_native(int32_t *a, const int32_t *h)+__contract__(+  requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(h, sizeof(int32_t) * MLDSA_N))+  requires(array_bound(a, 0, MLDSA_N, 0, MLDSA_Q))+  requires(array_bound(h, 0, MLDSA_N, 0, 2))+  assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_bound(a, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2)))+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(a, MLDSA_N))+);+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \+        */+#endif /* MLD_USE_NATIVE_POLY_USE_HINT_88 */+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#if defined(MLD_USE_NATIVE_POLY_CHKNORM)+/**+ * Check infinity norm of polynomial against given bound. Assumes input+ * coefficients were reduced by mld_reduce32().+ *+ * @param[in] a Pointer to polynomial.+ * @param     B Norm bound, which must be in the range+ *              0 .. MLDSA_Q - MLD_REDUCE32_RANGE_MAX inclusive.+ *+ * @return - MLD_NATIVE_FUNC_FALLBACK if the target CPU cannot support a+ *           native implementation of this function.+ *         - MLD_NATIVE_FUNC_SUCCESS if the infinity norm is strictly smaller+ *           than B.+ *         - 1 otherwise.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_chknorm_native(const int32_t *a, int32_t B)+__contract__(+  requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N))+  requires(0 <= B && B <= MLDSA_Q - MLD_REDUCE32_RANGE_MAX)+  requires(array_bound(a, 0, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == 0  ||+          return_value == 1)+  ensures((return_value != MLD_NATIVE_FUNC_FALLBACK) ==>+          ((return_value == 0) == array_abs_bound(a, 0, MLDSA_N, B)))+);+#endif /* MLD_USE_NATIVE_POLY_CHKNORM */++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+#if defined(MLD_USE_NATIVE_POLYZ_UNPACK_17)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44+/**+ * Native implementation of polyz_unpack for GAMMA1 = 2^17.+ *+ * Unpack polynomial z with coefficients in+ * [-(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1].+ *+ * @param[out] r Pointer to output polynomial.+ * @param[in]  a Byte array with bit-packed polynomial.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_polyz_unpack_17_native(int32_t *r, const uint8_t *a)+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(a, MLDSA_POLYZ_PACKEDBYTES))+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_bound(r, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(r, MLDSA_N))+);+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \+        */+#endif /* MLD_USE_NATIVE_POLYZ_UNPACK_17 */++#if defined(MLD_USE_NATIVE_POLYZ_UNPACK_19)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)+/**+ * Native implementation of polyz_unpack for GAMMA1 = 2^19.+ *+ * Unpack polynomial z with coefficients in+ * [-(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1].+ *+ * @param[out] r Pointer to output polynomial.+ * @param[in]  a Byte array with bit-packed polynomial.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_polyz_unpack_19_native(int32_t *r, const uint8_t *a)+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(a, MLDSA_POLYZ_PACKEDBYTES))+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_bound(r, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(r, MLDSA_N))+);+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87 */+#endif /* MLD_USE_NATIVE_POLYZ_UNPACK_19 */+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) || \+    defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+#if defined(MLD_USE_NATIVE_POINTWISE_MONTGOMERY)+/**+ * Pointwise multiplication of polynomials in NTT domain with Montgomery+ * reduction. Destructive in the first argument.+ *+ * Computes a[i] = a[i] * b[i] * R^(-1) mod MLDSA_Q for all i, where R = 2^32.+ *+ * @param[in,out] a First input/output polynomial.+ * @param[in]     b Second input polynomial.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_pointwise_montgomery_native(+    int32_t a[MLDSA_N], const int32_t b[MLDSA_N])+__contract__(+  requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(b, sizeof(int32_t) * MLDSA_N))+  requires(array_abs_bound(a, 0, MLDSA_N, MLD_NTT_BOUND))+  requires(array_abs_bound(b, 0, MLDSA_N, MLD_NTT_BOUND))+  assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(a, 0, MLDSA_N, MLDSA_Q))+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_abs_bound(a, 0, MLDSA_N, MLD_NTT_BOUND))+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_abs_bound(b, 0, MLDSA_N, MLD_NTT_BOUND))+);+#endif /* MLD_USE_NATIVE_POINTWISE_MONTGOMERY */+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API || \+          MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++#if defined(MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 4+/**+ * Native implementation of polyvecl_pointwise_acc_montgomery for MLDSA_L = 4.+ *+ * Pointwise multiply vectors of polynomials of length MLDSA_L, multiply+ * resulting vector by 2^{-32} and add (accumulate) polynomials in it.+ * Input/output vectors are in NTT domain representation.+ *+ * @param[out] w Output polynomial.+ * @param[in]  u First input vector.+ * @param[in]  v Second input vector.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l4_native(+    int32_t w[MLDSA_N], const int32_t u[4][MLDSA_N],+    const int32_t v[4][MLDSA_N])+__contract__(+  requires(memory_no_alias(w, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(u, sizeof(int32_t) * 4 * MLDSA_N))+  requires(memory_no_alias(v, sizeof(int32_t) * 4 * MLDSA_N))+  requires(forall(l0, 0, 4,+                  array_bound(u[l0], 0, MLDSA_N, 0, MLDSA_Q)))+  requires(forall(l1, 0, 4,+    array_abs_bound(v[l1], 0, MLDSA_N, MLD_NTT_BOUND)))+  assigns(memory_slice(w, sizeof(int32_t) * MLDSA_N))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(w, 0, MLDSA_N, MLDSA_Q))+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(w, MLDSA_N))+);+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 4 */+#endif /* MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4 */++#if defined(MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 5+/**+ * Native implementation of polyvecl_pointwise_acc_montgomery for MLDSA_L = 5.+ *+ * Pointwise multiply vectors of polynomials of length MLDSA_L, multiply+ * resulting vector by 2^{-32} and add (accumulate) polynomials in it.+ * Input/output vectors are in NTT domain representation.+ *+ * @param[out] w Output polynomial.+ * @param[in]  u First input vector.+ * @param[in]  v Second input vector.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l5_native(+    int32_t w[MLDSA_N], const int32_t u[5][MLDSA_N],+    const int32_t v[5][MLDSA_N])+__contract__(+  requires(memory_no_alias(w, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(u, sizeof(int32_t) * 5 * MLDSA_N))+  requires(memory_no_alias(v, sizeof(int32_t) * 5 * MLDSA_N))+  requires(forall(l0, 0, 5,+                  array_bound(u[l0], 0, MLDSA_N, 0, MLDSA_Q)))+  requires(forall(l1, 0, 5,+    array_abs_bound(v[l1], 0, MLDSA_N, MLD_NTT_BOUND)))+  assigns(memory_slice(w, sizeof(int32_t) * MLDSA_N))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(w, 0, MLDSA_N, MLDSA_Q))+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(w, MLDSA_N))+);+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 5 */+#endif /* MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5 */++#if defined(MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 7+/**+ * Native implementation of polyvecl_pointwise_acc_montgomery for MLDSA_L = 7.+ *+ * Pointwise multiply vectors of polynomials of length MLDSA_L, multiply+ * resulting vector by 2^{-32} and add (accumulate) polynomials in it.+ * Input/output vectors are in NTT domain representation.+ *+ * @param[out] w Output polynomial.+ * @param[in]  u First input vector.+ * @param[in]  v Second input vector.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l7_native(+    int32_t w[MLDSA_N], const int32_t u[7][MLDSA_N],+    const int32_t v[7][MLDSA_N])+__contract__(+  requires(memory_no_alias(w, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(u, sizeof(int32_t) * 7 * MLDSA_N))+  requires(memory_no_alias(v, sizeof(int32_t) * 7 * MLDSA_N))+  requires(forall(l0, 0, 7,+                  array_bound(u[l0], 0, MLDSA_N, 0, MLDSA_Q)))+  requires(forall(l1, 0, 7,+    array_abs_bound(v[l1], 0, MLDSA_N, MLD_NTT_BOUND)))+  assigns(memory_slice(w, sizeof(int32_t) * MLDSA_N))+  ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(w, 0, MLDSA_N, MLDSA_Q))+  ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(w, MLDSA_N))+);+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 7 */+#endif /* MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7 */++#endif /* !MLD_NATIVE_API_H */
+ cbits/mldsa/src/native/meta.h view
@@ -0,0 +1,24 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLD_NATIVE_META_H+#define MLD_NATIVE_META_H++/*+ * Default arithmetic backend+ */+#include "../sys.h"++#ifdef MLD_SYS_AARCH64_NEON+#include "aarch64/meta.h"+#endif++/* The x86_64 backend requires toolchain support for the SysV ABI */+#if defined(MLD_SYS_X86_64_AVX2) && defined(MLD_SYSV_ABI_SUPPORTED)+#include "x86_64/meta.h"+#endif++#endif /* !MLD_NATIVE_META_H */
+ cbits/mldsa/src/native/x86_64/meta.h view
@@ -0,0 +1,323 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLD_NATIVE_X86_64_META_H+#define MLD_NATIVE_X86_64_META_H++/* Identifier for this backend so that source and assembly files+ * in the build can be appropriately guarded. */+#define MLD_ARITH_BACKEND_X86_64_DEFAULT++#define MLD_USE_NATIVE_NTT_CUSTOM_ORDER+#define MLD_USE_NATIVE_NTT+#define MLD_USE_NATIVE_INTT+#define MLD_USE_NATIVE_REJ_UNIFORM+#define MLD_USE_NATIVE_REJ_UNIFORM_ETA2+#define MLD_USE_NATIVE_REJ_UNIFORM_ETA4+#define MLD_USE_NATIVE_POLY_DECOMPOSE_32+#define MLD_USE_NATIVE_POLY_DECOMPOSE_88+#define MLD_USE_NATIVE_POLY_CADDQ+#define MLD_USE_NATIVE_POLY_USE_HINT_32+#define MLD_USE_NATIVE_POLY_USE_HINT_88+#define MLD_USE_NATIVE_POLY_CHKNORM+#define MLD_USE_NATIVE_POLYZ_UNPACK_17+#define MLD_USE_NATIVE_POLYZ_UNPACK_19+#define MLD_USE_NATIVE_POINTWISE_MONTGOMERY+#define MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4+#define MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5+#define MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7++#if !defined(__ASSEMBLER__)+#include <string.h>+#include "../../common.h"+#include "../api.h"+#include "src/arith_native_x86_64.h"++static MLD_INLINE void mld_poly_permute_bitrev_to_custom(int32_t data[MLDSA_N])+{+  if (mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2))+  {+    mld_nttunpack_avx2_asm(data);+  }+}++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_ntt_native(int32_t data[MLDSA_N])+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }++  mld_ntt_avx2_asm(data, mld_qdata);+  return MLD_NATIVE_FUNC_SUCCESS;+}++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_intt_native(int32_t data[MLDSA_N])+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_invntt_avx2_asm(data, mld_qdata);+  return MLD_NATIVE_FUNC_SUCCESS;+}++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_rej_uniform_native(int32_t *r, unsigned len,+                                             const uint8_t *buf,+                                             unsigned buflen)+{+  /* AVX2 implementation assumes specific buffer lengths */+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2) || len != MLDSA_N ||+      buflen != MLD_AVX2_REJ_UNIFORM_BUFLEN)+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }++  /* Safety: outlen is at most MLDSA_N and, hence, this cast is safe. */+  return (int)mld_rej_uniform_avx2_asm(r, buf, mld_rej_uniform_table);+}++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_ETA == 2+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_rej_uniform_eta2_native(int32_t *r, unsigned len,+                                                  const uint8_t *buf,+                                                  unsigned buflen)+{+  unsigned int outlen;+  /* AVX2 implementation assumes specific buffer lengths */+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2) || len != MLDSA_N ||+      buflen != MLD_AVX2_REJ_UNIFORM_ETA2_BUFLEN)+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }++  /* Constant time: Inputs and outputs to this function are secret.+   * It is safe to leak which coefficients are accepted/rejected.+   * The assembly implementation must not leak any other information about the+   * accepted coefficients. Constant-time testing cannot cover this, and we+   * hence have to manually verify the assembly.+   * We declassify prior the input data and mark the outputs as secret.+   */+  MLD_CT_TESTING_DECLASSIFY(buf, buflen);+  outlen = mld_rej_uniform_eta2_avx2_asm(r, buf, mld_rej_uniform_table);+  MLD_CT_TESTING_SECRET(r, sizeof(int32_t) * outlen);+  /* Safety: outlen is at most MLDSA_N and, hence, this cast is safe. */+  return (int)outlen;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_ETA == 2 */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_ETA == 4+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_rej_uniform_eta4_native(int32_t *r, unsigned len,+                                                  const uint8_t *buf,+                                                  unsigned buflen)+{+  unsigned int outlen;+  /* AVX2 implementation assumes specific buffer lengths */+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2) || len != MLDSA_N ||+      buflen != MLD_AVX2_REJ_UNIFORM_ETA4_BUFLEN)+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }++  /* Constant time: Inputs and outputs to this function are secret.+   * It is safe to leak which coefficients are accepted/rejected.+   * The assembly implementation must not leak any other information about the+   * accepted coefficients. Constant-time testing cannot cover this, and we+   * hence have to manually verify the assembly.+   * We declassify prior the input data and mark the outputs as secret.+   */+  MLD_CT_TESTING_DECLASSIFY(buf, buflen);+  outlen = mld_rej_uniform_eta4_avx2_asm(r, buf, mld_rej_uniform_table);+  MLD_CT_TESTING_SECRET(r, sizeof(int32_t) * outlen);+  /* Safety: outlen is at most MLDSA_N and, hence, this cast is safe. */+  return (int)outlen;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_ETA == 4 */+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_decompose_32_native(int32_t *a1, int32_t *a0)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_poly_decompose_32_avx2_asm(a1, a0);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87 */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_decompose_88_native(int32_t *a1, int32_t *a0)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_poly_decompose_88_avx2_asm(a1, a0);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \+        */+#endif /* !MLD_CONFIG_NO_SIGN_API */++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_caddq_native(int32_t a[MLDSA_N])+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_poly_caddq_avx2_asm(a);+  return MLD_NATIVE_FUNC_SUCCESS;+}++#if !defined(MLD_CONFIG_NO_VERIFY_API)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_use_hint_32_native(int32_t *a, const int32_t *h)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_poly_use_hint_32_avx2_asm(a, h);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87 */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_use_hint_88_native(int32_t *a, const int32_t *h)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_poly_use_hint_88_avx2_asm(a, h);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \+        */+#endif /* !MLD_CONFIG_NO_VERIFY_API */++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_chknorm_native(const int32_t *a, int32_t B)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  return mld_poly_chknorm_avx2_asm(a, B);+}++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_polyz_unpack_17_native(int32_t *r, const uint8_t *a)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_polyz_unpack_17_avx2_asm(r, a);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \+        */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_polyz_unpack_19_native(int32_t *r, const uint8_t *a)+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_polyz_unpack_19_avx2_asm(r, a);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87 */+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) || \+    defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_poly_pointwise_montgomery_native(+    int32_t a[MLDSA_N], const int32_t b[MLDSA_N])+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_pointwise_avx2_asm(a, b, mld_qdata);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API || \+          MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 4+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l4_native(+    int32_t w[MLDSA_N], const int32_t u[4][MLDSA_N],+    const int32_t v[4][MLDSA_N])+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_pointwise_acc_l4_avx2_asm(w, u, v, mld_qdata);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 4 */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 5+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l5_native(+    int32_t w[MLDSA_N], const int32_t u[5][MLDSA_N],+    const int32_t v[5][MLDSA_N])+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_pointwise_acc_l5_avx2_asm(w, u, v, mld_qdata);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 5 */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 7+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l7_native(+    int32_t w[MLDSA_N], const int32_t u[7][MLDSA_N],+    const int32_t v[7][MLDSA_N])+{+  if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2))+  {+    return MLD_NATIVE_FUNC_FALLBACK;+  }+  mld_pointwise_acc_l7_avx2_asm(w, u, v, mld_qdata);+  return MLD_NATIVE_FUNC_SUCCESS;+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 7 */++#endif /* !__ASSEMBLER__ */++#endif /* !MLD_NATIVE_X86_64_META_H */
+ cbits/mldsa/src/native/x86_64/src/arith_native_x86_64.h view
@@ -0,0 +1,330 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLD_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H+#define MLD_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H+#include "../../../common.h"++#include "consts.h"++#define MLD_AVX2_REJ_UNIFORM_BUFLEN \+  (5 * 168) /* REJ_UNIFORM_NBLOCKS * SHAKE128_RATE */+++/*+ * Sampling 256 coefficients mod 15 using rejection sampling from 4 bits.+ * Expected number of required bytes: (256 * (16/15))/2 = 136.5 bytes.+ * We sample 1 block (=136 bytes) of SHAKE256_RATE output initially.+ * Sampling 2 blocks initially results in slightly worse performance.+ */+#define MLD_AVX2_REJ_UNIFORM_ETA2_BUFLEN (1 * 136)++/*+ * Sampling 256 coefficients mod 9 using rejection sampling from 4 bits.+ * Expected number of required bytes: (256 * (16/9))/2 = 227.5 bytes.+ * We sample 2 blocks (=272 bytes) of SHAKE256_RATE output initially.+ */+#define MLD_AVX2_REJ_UNIFORM_ETA4_BUFLEN (2 * 136)++#define mld_rej_uniform_table MLD_NAMESPACE(mld_rej_uniform_table)+MLD_INTERNAL_DATA_DECLARATION const uint8_t mld_rej_uniform_table[256][8];++#define mld_ntt_avx2_asm MLD_NAMESPACE(ntt_avx2_asm)+MLD_SYSV_ABI+void mld_ntt_avx2_asm(int32_t *r, const int32_t *qdata)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_ntt_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  /* check-magic: 8380417 == MLDSA_Q */+  requires(array_abs_bound(r, 0, MLDSA_N, 8380417))+  requires(qdata == mld_qdata)+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  /* check-magic: off */+  ensures(array_abs_bound(r, 0, MLDSA_N, 42035262))+  /* check-magic: on */+);++#define mld_invntt_avx2_asm MLD_NAMESPACE(invntt_avx2_asm)+MLD_SYSV_ABI+void mld_invntt_avx2_asm(int32_t *r, const int32_t *qdata)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_intt_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(array_abs_bound(r, 0, MLDSA_N, 8380417))+  requires(qdata == mld_qdata)+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  /* check-magic: off */+  ensures(array_abs_bound(r, 0, MLDSA_N, 6285313))+  /* check-magic: on */+);++#define mld_nttunpack_avx2_asm MLD_NAMESPACE(nttunpack_avx2_asm)+MLD_SYSV_ABI+void mld_nttunpack_avx2_asm(int32_t *r)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_nttunpack_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(array_abs_bound(r, 0, MLDSA_N, 8380417))+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  /* Output is a permutation of input: every output coefficient+   * is some input coefficient */+  ensures(forall(i, 0, MLDSA_N, exists(j, 0, MLDSA_N,+    r[i] == old(*(int32_t (*)[MLDSA_N])r)[j])))+);++#define mld_rej_uniform_avx2_asm MLD_NAMESPACE(rej_uniform_avx2_asm)+/* This contract must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_rej_uniform_avx2_asm.ml */+MLD_MUST_CHECK_RETURN_VALUE MLD_SYSV_ABI+unsigned mld_rej_uniform_avx2_asm(+    int32_t *r, const uint8_t buf[MLD_AVX2_REJ_UNIFORM_BUFLEN],+    const uint8_t table[256][8])+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(buf, 840))+  requires(table == mld_rej_uniform_table)+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(return_value <= MLDSA_N)+  ensures(array_bound(r, 0, return_value, 0, MLDSA_Q))+);++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+#define mld_rej_uniform_eta2_avx2_asm MLD_NAMESPACE(rej_uniform_eta2_avx2_asm)+/* This contract must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_rej_uniform_eta2_avx2_asm.ml */+MLD_MUST_CHECK_RETURN_VALUE MLD_SYSV_ABI+unsigned mld_rej_uniform_eta2_avx2_asm(+    int32_t *r, const uint8_t buf[MLD_AVX2_REJ_UNIFORM_ETA2_BUFLEN],+    const uint8_t table[256][8])+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(buf, MLD_AVX2_REJ_UNIFORM_ETA2_BUFLEN))+  requires(table == mld_rej_uniform_table)+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(return_value <= MLDSA_N)+  ensures(array_abs_bound(r, 0, return_value, 3))+);++#define mld_rej_uniform_eta4_avx2_asm MLD_NAMESPACE(rej_uniform_eta4_avx2_asm)+/* This contract must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_rej_uniform_eta4_avx2_asm.ml */+MLD_MUST_CHECK_RETURN_VALUE MLD_SYSV_ABI+unsigned mld_rej_uniform_eta4_avx2_asm(+    int32_t *r, const uint8_t buf[MLD_AVX2_REJ_UNIFORM_ETA4_BUFLEN],+    const uint8_t table[256][8])+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(buf, MLD_AVX2_REJ_UNIFORM_ETA4_BUFLEN))+  requires(table == mld_rej_uniform_table)+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(return_value <= MLDSA_N)+  ensures(array_abs_bound(r, 0, return_value, 5))+);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+#define mld_poly_decompose_32_avx2_asm MLD_NAMESPACE(poly_decompose_32_avx2_asm)+MLD_SYSV_ABI+void mld_poly_decompose_32_avx2_asm(int32_t *a1, int32_t *a0)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_poly_decompose_32_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(a1, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(a0, sizeof(int32_t) * MLDSA_N))+  requires(array_bound(a0, 0, MLDSA_N, 0, MLDSA_Q))+  assigns(memory_slice(a1, sizeof(int32_t) * MLDSA_N))+  assigns(memory_slice(a0, sizeof(int32_t) * MLDSA_N))+  /* check-magic: 16 == (MLDSA_Q - 1) / (2 * ((MLDSA_Q - 1) / 32)) */+  ensures(array_bound(a1, 0, MLDSA_N, 0, 16))+  /* check-magic: 261889 == (MLDSA_Q - 1) / 32 + 1 */+  ensures(array_abs_bound(a0, 0, MLDSA_N, 261889))+);++#define mld_poly_decompose_88_avx2_asm MLD_NAMESPACE(poly_decompose_88_avx2_asm)+MLD_SYSV_ABI+void mld_poly_decompose_88_avx2_asm(int32_t *a1, int32_t *a0)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_poly_decompose_88_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(a1, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(a0, sizeof(int32_t) * MLDSA_N))+  requires(array_bound(a0, 0, MLDSA_N, 0, MLDSA_Q))+  assigns(memory_slice(a1, sizeof(int32_t) * MLDSA_N))+  assigns(memory_slice(a0, sizeof(int32_t) * MLDSA_N))+  /* check-magic: 44 == (MLDSA_Q - 1) / (2 * ((MLDSA_Q - 1) / 88)) */+  ensures(array_bound(a1, 0, MLDSA_N, 0, 44))+  /* check-magic: 95233 == (MLDSA_Q - 1) / 88 + 1 */+  ensures(array_abs_bound(a0, 0, MLDSA_N, 95233))+);+#endif /* !MLD_CONFIG_NO_SIGN_API */++#define mld_poly_caddq_avx2_asm MLD_NAMESPACE(poly_caddq_avx2_asm)+MLD_SYSV_ABI+void mld_poly_caddq_avx2_asm(int32_t *r)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_poly_caddq_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(array_abs_bound(r, 0, MLDSA_N, MLDSA_Q))+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(array_bound(r, 0, MLDSA_N, 0, MLDSA_Q))+);++#if !defined(MLD_CONFIG_NO_VERIFY_API)+#define mld_poly_use_hint_32_avx2_asm MLD_NAMESPACE(poly_use_hint_32_avx2_asm)+MLD_SYSV_ABI+void mld_poly_use_hint_32_avx2_asm(int32_t *a, const int32_t *h)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_poly_use_hint_32_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(h, sizeof(int32_t) * MLDSA_N))+  requires(array_bound(a, 0, MLDSA_N, 0, MLDSA_Q))+  requires(array_bound(h, 0, MLDSA_N, 0, 2))+  assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N))+  /* check-magic: 16 == (MLDSA_Q - 1) / (2 * ((MLDSA_Q - 1) / 32)) */+  ensures(array_bound(a, 0, MLDSA_N, 0, 16))+);++#define mld_poly_use_hint_88_avx2_asm MLD_NAMESPACE(poly_use_hint_88_avx2_asm)+MLD_SYSV_ABI+void mld_poly_use_hint_88_avx2_asm(int32_t *a, const int32_t *h)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_poly_use_hint_88_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(h, sizeof(int32_t) * MLDSA_N))+  requires(array_bound(a, 0, MLDSA_N, 0, MLDSA_Q))+  requires(array_bound(h, 0, MLDSA_N, 0, 2))+  assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N))+  /* check-magic: 44 == (MLDSA_Q - 1) / (2 * ((MLDSA_Q - 1) / 88)) */+  ensures(array_bound(a, 0, MLDSA_N, 0, 44))+);+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#define mld_poly_chknorm_avx2_asm MLD_NAMESPACE(poly_chknorm_avx2_asm)+MLD_MUST_CHECK_RETURN_VALUE MLD_SYSV_ABI+int mld_poly_chknorm_avx2_asm(const int32_t *a, int32_t B)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_poly_chknorm_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N))+  /* HOL Light precondition: abs(ival(x i)) < 2^31, i.e., a[i] != INT32_MIN */+  requires(forall(k0, 0, MLDSA_N, a[k0] > INT32_MIN))+  /* HOL Light precondition: 0 <= ival bound (asm computes B-1 internally) */+  requires(B >= 0)+  ensures(return_value == 0 || return_value == 1)+  ensures((return_value == 0) == array_abs_bound(a, 0, MLDSA_N, B))+);++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+#define mld_polyz_unpack_17_avx2_asm MLD_NAMESPACE(polyz_unpack_17_avx2_asm)+MLD_SYSV_ABI+void mld_polyz_unpack_17_avx2_asm(int32_t *r, const uint8_t *a)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_polyz_unpack_17_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(a, 576))+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(array_bound(r, 0, MLDSA_N, -((1 << 17) - 1), (1 << 17) + 1))+);++#define mld_polyz_unpack_19_avx2_asm MLD_NAMESPACE(polyz_unpack_19_avx2_asm)+MLD_SYSV_ABI+void mld_polyz_unpack_19_avx2_asm(int32_t *r, const uint8_t *a)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_polyz_unpack_19_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(a, 640))+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(array_bound(r, 0, MLDSA_N, -((1 << 19) - 1), (1 << 19) + 1))+);+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */++#define mld_pointwise_avx2_asm MLD_NAMESPACE(pointwise_avx2_asm)+MLD_SYSV_ABI+void mld_pointwise_avx2_asm(int32_t *a, const int32_t *b, const int32_t *qdata)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_pointwise_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(b, sizeof(int32_t) * MLDSA_N))+  /* Input bound MLD_NTT_BOUND = 9 * MLD_FQMUL_BOUND, the guaranteed bound of+   * any forward NTT implementation. Hardcoded here to keep this header free+   * of poly.h. */+  /* check-magic: 94279698 == 9 * ((5 * MLDSA_Q + 3) / 4) */+  requires(array_abs_bound(a, 0, MLDSA_N, 94279698))+  requires(array_abs_bound(b, 0, MLDSA_N, 94279698))+  requires(qdata == mld_qdata)+  assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N))+  ensures(array_abs_bound(a, 0, MLDSA_N, 8380417))+);++#define mld_pointwise_acc_l4_avx2_asm MLD_NAMESPACE(pointwise_acc_l4_avx2_asm)+MLD_SYSV_ABI+void mld_pointwise_acc_l4_avx2_asm(int32_t c[MLDSA_N],+                                   const int32_t a[4][MLDSA_N],+                                   const int32_t b[4][MLDSA_N],+                                   const int32_t *qdata)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_pointwise_acc_l4_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(c, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(a, sizeof(int32_t) * 4 * MLDSA_N))+  requires(memory_no_alias(b, sizeof(int32_t) * 4 * MLDSA_N))+  requires(forall(l0, 0, 4, array_abs_bound(a[l0], 0, MLDSA_N, 8380417)))+  /* check-magic: 94279698 == 9 * ((5 * MLDSA_Q + 3) / 4) */+  requires(forall(l1, 0, 4, array_abs_bound(b[l1], 0, MLDSA_N, 94279698)))+  requires(qdata == mld_qdata)+  assigns(memory_slice(c, sizeof(int32_t) * MLDSA_N))+  ensures(array_abs_bound(c, 0, MLDSA_N, 8380417))+);++#define mld_pointwise_acc_l5_avx2_asm MLD_NAMESPACE(pointwise_acc_l5_avx2_asm)+MLD_SYSV_ABI+void mld_pointwise_acc_l5_avx2_asm(int32_t c[MLDSA_N],+                                   const int32_t a[5][MLDSA_N],+                                   const int32_t b[5][MLDSA_N],+                                   const int32_t *qdata)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_pointwise_acc_l5_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(c, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(a, sizeof(int32_t) * 5 * MLDSA_N))+  requires(memory_no_alias(b, sizeof(int32_t) * 5 * MLDSA_N))+  requires(forall(l0, 0, 5, array_abs_bound(a[l0], 0, MLDSA_N, 8380417)))+  /* check-magic: 94279698 == 9 * ((5 * MLDSA_Q + 3) / 4) */+  requires(forall(l1, 0, 5, array_abs_bound(b[l1], 0, MLDSA_N, 94279698)))+  requires(qdata == mld_qdata)+  assigns(memory_slice(c, sizeof(int32_t) * MLDSA_N))+  ensures(array_abs_bound(c, 0, MLDSA_N, 8380417))+);++#define mld_pointwise_acc_l7_avx2_asm MLD_NAMESPACE(pointwise_acc_l7_avx2_asm)+MLD_SYSV_ABI+void mld_pointwise_acc_l7_avx2_asm(int32_t c[MLDSA_N],+                                   const int32_t a[7][MLDSA_N],+                                   const int32_t b[7][MLDSA_N],+                                   const int32_t *qdata)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mldsa_pointwise_acc_l7_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(c, sizeof(int32_t) * MLDSA_N))+  requires(memory_no_alias(a, sizeof(int32_t) * 7 * MLDSA_N))+  requires(memory_no_alias(b, sizeof(int32_t) * 7 * MLDSA_N))+  requires(forall(l0, 0, 7, array_abs_bound(a[l0], 0, MLDSA_N, 8380417)))+  /* check-magic: 94279698 == 9 * ((5 * MLDSA_Q + 3) / 4) */+  requires(forall(l1, 0, 7, array_abs_bound(b[l1], 0, MLDSA_N, 94279698)))+  requires(qdata == mld_qdata)+  assigns(memory_slice(c, sizeof(int32_t) * MLDSA_N))+  ensures(array_abs_bound(c, 0, MLDSA_N, 8380417))+);++#endif /* !MLD_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H */
+ cbits/mldsa/src/native/x86_64/src/consts.c view
@@ -0,0 +1,157 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mldsa-native repository.+ *          Do not modify it directly.+ */++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++#include "consts.h"++/*+ * Table of zeta values used in the AVX2 forward and inverse NTT+ * See autogen for details.+ */+MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const int32_t mld_qdata[624] = {+    8380417,     8380417,     8380417,     8380417,     8380417,+    8380417,     8380417,     8380417,     58728449,    58728449,+    58728449,    58728449,    58728449,    58728449,    58728449,+    58728449,    -8395782,    -8395782,    -8395782,    -8395782,+    -8395782,    -8395782,    -8395782,    -8395782,    41978,+    41978,       41978,       41978,       41978,       41978,+    41978,       41978,       -151046689,  1830765815,  -1929875198,+    -1927777021, 1640767044,  1477910808,  1612161320,  1640734244,+    308362795,   308362795,   308362795,   308362795,   -1815525077,+    -1815525077, -1815525077, -1815525077, -1374673747, -1374673747,+    -1374673747, -1374673747, -1091570561, -1091570561, -1091570561,+    -1091570561, -1929495947, -1929495947, -1929495947, -1929495947,+    515185417,   515185417,   515185417,   515185417,   -285697463,+    -285697463,  -285697463,  -285697463,  625853735,   625853735,+    625853735,   625853735,   1727305304,  1727305304,  2082316400,+    2082316400,  -1364982364, -1364982364, 858240904,   858240904,+    1806278032,  1806278032,  222489248,   222489248,   -346752664,+    -346752664,  684667771,   684667771,   1654287830,  1654287830,+    -878576921,  -878576921,  -1257667337, -1257667337, -748618600,+    -748618600,  329347125,   329347125,   1837364258,  1837364258,+    -1443016191, -1443016191, -1170414139, -1170414139, -1846138265,+    -1631226336, -1404529459, 1838055109,  1594295555,  -1076973524,+    -1898723372, -594436433,  -202001019,  -475984260,  -561427818,+    1797021249,  -1061813248, 2059733581,  -1661512036, -1104976547,+    -1750224323, -901666090,  418987550,   1831915353,  -1925356481,+    992097815,   879957084,   2024403852,  1484874664,  -1636082790,+    -285388938,  -1983539117, -1495136972, -950076368,  -1714807468,+    -952438995,  -1574918427, 1350681039,  -2143979939, 1599739335,+    -1285853323, -993005454,  -1440787840, 568627424,   -783134478,+    -588790216,  289871779,   -1262003603, 2135294594,  -1018755525,+    -889861155,  1665705315,  1321868265,  1225434135,  -1784632064,+    666258756,   675310538,   -1555941048, -1999506068, -1499481951,+    -695180180,  -1375177022, 1777179795,  334803717,   -178766299,+    -518252220,  1957047970,  1146323031,  -654783359,  -1974159335,+    1651689966,  140455867,   -1039411342, 1955560694,  1529189038,+    -2131021878, -247357819,  1518161567,  -86965173,   1708872713,+    1787797779,  1638590967,  -120646188,  -1669960606, -916321552,+    1155548552,  2143745726,  1210558298,  -1261461890, -318346816,+    628664287,   -1729304568, 1422575624,  1424130038,  -1185330464,+    235321234,   168022240,   1206536194,  985155484,   -894060583,+    -898413,     -1363460238, -605900043,  2027833504,  14253662,+    1014493059,  863641633,   1819892093,  2124962073,  -1223601433,+    -1920467227, -1637785316, -1536588520, 694382729,   235104446,+    -1045062172, 831969619,   -300448763,  756955444,   -260312805,+    1554794072,  1339088280,  -2040058690, -853476187,  -2047270596,+    -1723816713, -1591599803, -440824168,  1119856484,  1544891539,+    155290192,   -973777462,  991903578,   912367099,   -44694137,+    1176904444,  -421552614,  -818371958,  1747917558,  -325927722,+    908452108,   1851023419,  -1176751719, -1354528380, -72690498,+    -314284737,  985022747,   963438279,   -1078959975, 604552167,+    -1021949428, 608791570,   173440395,   -2126092136, -1316619236,+    -1039370342, 6087993,     -110126092,  565464272,   -1758099917,+    -1600929361, 879867909,   -1809756372, 400711272,   1363007700,+    30313375,    -326425360,  1683520342,  -517299994,  2027935492,+    -1372618620, 128353682,   -1123881663, 137583815,   -635454918,+    -642772911,  45766801,    671509323,   -2070602178, 419615363,+    1216882040,  -270590488,  -1276805128, 371462360,   -1357098057,+    -384158533,  827959816,   -596344473,  702390549,   -279505433,+    -260424530,  -71875110,   -1208667171, -1499603926, 2036925262,+    -540420426,  746144248,   -1420958686, 2032221021,  1904936414,+    1257750362,  1926727420,  1931587462,  1258381762,  885133339,+    1629985060,  1967222129,  6363718,     -1287922800, 1136965286,+    1779436847,  1116720494,  1042326957,  1405999311,  713994583,+    940195359,   -1542497137, 2061661095,  -883155599,  1726753853,+    -1547952704, 394851342,   283780712,   776003547,   1123958025,+    201262505,   1934038751,  374860238,   -3975713,    25847,+    -2608894,    -518909,     237124,      -777960,     -876248,+    466468,      1826347,     1826347,     1826347,     1826347,+    2353451,     2353451,     2353451,     2353451,     -359251,+    -359251,     -359251,     -359251,     -2091905,    -2091905,+    -2091905,    -2091905,    3119733,     3119733,     3119733,+    3119733,     -2884855,    -2884855,    -2884855,    -2884855,+    3111497,     3111497,     3111497,     3111497,     2680103,+    2680103,     2680103,     2680103,     2725464,     2725464,+    1024112,     1024112,     -1079900,    -1079900,    3585928,+    3585928,     -549488,     -549488,     -1119584,    -1119584,+    2619752,     2619752,     -2108549,    -2108549,    -2118186,+    -2118186,    -3859737,    -3859737,    -1399561,    -1399561,+    -3277672,    -3277672,    1757237,     1757237,     -19422,+    -19422,      4010497,     4010497,     280005,      280005,+    2706023,     95776,       3077325,     3530437,     -1661693,+    -3592148,    -2537516,    3915439,     -3861115,    -3043716,+    3574422,     -2867647,    3539968,     -300467,     2348700,+    -539299,     -1699267,    -1643818,    3505694,     -3821735,+    3507263,     -2140649,    -1600420,    3699596,     811944,+    531354,      954230,      3881043,     3900724,     -2556880,+    2071892,     -2797779,    -3930395,    -3677745,    -1452451,+    2176455,     -1257611,    -4083598,    -3190144,    -3632928,+    3412210,     2147896,     -2967645,    -411027,     -671102,+    -22981,      -381987,     1852771,     -3343383,    508951,+    44288,       904516,      -3724342,    1653064,     2389356,+    759969,      189548,      3159746,     -2409325,    1315589,+    1285669,     -812732,     -3019102,    -3628969,    -1528703,+    -3041255,    3475950,     -1585221,    1939314,     -1000202,+    -3157330,    126922,      -983419,     2715295,     -3693493,+    -2477047,    -1228525,    -1308169,    1349076,     -1430430,+    264944,      3097992,     -1100098,    3958618,     -8578,+    -3249728,    -210977,     -1316856,    -3553272,    -1851402,+    -177440,     1341330,     -1584928,    -1439742,    -3881060,+    3839961,     2091667,     -3342478,    266997,      -3520352,+    900702,      495491,      -655327,     -3556995,    342297,+    3437287,     2842341,     4055324,     -3767016,    -2994039,+    -1333058,    -451100,     -1279661,    1500165,     -542412,+    -2584293,    -2013608,    1957272,     -3183426,    810149,+    -3038916,    2213111,     -426683,     -1667432,    -2939036,+    183443,      -554416,     3937738,     3407706,     2244091,+    2434439,     -3759364,    1859098,     -1613174,    -3122442,+    -525098,     286988,      -3342277,    2691481,     1247620,+    1250494,     1869119,     1237275,     1312455,     1917081,+    777191,      -2831860,    -3724270,    2432395,     3369112,+    162844,      1652634,     3523897,     -975884,     1723600,+    -1104333,    -2235985,    -976891,     3919660,     1400424,+    2316500,     -2446433,    -1235728,    -1197226,    909542,+    -43260,      2031748,     -768622,     -2437823,    1735879,+    -2590150,    2486353,     2635921,     1903435,     -3318210,+    3306115,     -2546312,    2235880,     -1671176,    594136,+    2454455,     185531,      1616392,     -3694233,    3866901,+    1717735,     -1803090,    -260646,     -420899,     1612842,+    -48306,      -846154,     3817976,     -3562462,    3513181,+    -3193378,    819034,      -522500,     3207046,     -3595838,+    4108315,     203044,      1265009,     1595974,     -3548272,+    -1050970,    -1430225,    -1962642,    -1374803,    3406031,+    -1846953,    -3776993,    -164721,     -1207385,    3014001,+    -1799107,    269760,      472078,      1910376,     -3833893,+    -2286327,    -3545687,    -1362209,    1976782,+};++#else /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \+       */++MLD_EMPTY_CU(avx2_consts)++#endif /* !(MLD_ARITH_BACKEND_X86_64_DEFAULT && \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED) */
+ cbits/mldsa/src/native/x86_64/src/consts.h view
@@ -0,0 +1,27 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mldsa-native repository.+ *          Do not modify it directly.+ */++#ifndef MLD_NATIVE_X86_64_SRC_CONSTS_H+#define MLD_NATIVE_X86_64_SRC_CONSTS_H+#include "../../../common.h"+#define MLD_AVX2_BACKEND_DATA_OFFSET_8XQ 0+#define MLD_AVX2_BACKEND_DATA_OFFSET_8XQINV 8+#define MLD_AVX2_BACKEND_DATA_OFFSET_8XDIV_QINV 16+#define MLD_AVX2_BACKEND_DATA_OFFSET_8XDIV 24+#define MLD_AVX2_BACKEND_DATA_OFFSET_ZETAS_QINV 32+#define MLD_AVX2_BACKEND_DATA_OFFSET_ZETAS 328++#ifndef __ASSEMBLER__+#define mld_qdata MLD_NAMESPACE(qdata)+MLD_INTERNAL_DATA_DECLARATION const int32_t mld_qdata[624];+#endif++#endif /* !MLD_NATIVE_X86_64_SRC_CONSTS_H */
+ cbits/mldsa/src/native/x86_64/src/mldsa_intt_avx2_asm.S view
@@ -0,0 +1,2333 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++ /*+  * This file is derived from the public domain+  * AVX2 Dilithium implementation @[REF_AVX2].+  */++/*yaml+  Name: invntt_avx2_asm+  Description: x86_64 AVX2 inverse NTT+  Signature: void mld_invntt_avx2_asm(int32_t *r, const int32_t *qdata)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: read/write+      c_parameter: int32_t *r+      description: Input/output polynomial (256 x int32_t)+    rsi:+      type: buffer+      size_bytes: 2496+      permissions: read-only+      c_parameter: const int32_t *qdata+      description: Precomputed constants (624 x int32_t)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_intt_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(invntt_avx2_asm)+MLD_ASM_FN_SYMBOL(invntt_avx2_asm)++        .cfi_startproc+        vmovdqa (%rsi), %ymm0+        vmovdqa (%rdi), %ymm4+        vmovdqa 0x20(%rdi), %ymm5+        vmovdqa 0x40(%rdi), %ymm6+        vmovdqa 0x60(%rdi), %ymm7+        vmovdqa 0x80(%rdi), %ymm8+        vmovdqa 0xa0(%rdi), %ymm9+        vmovdqa 0xc0(%rdi), %ymm10+        vmovdqa 0xe0(%rdi), %ymm11+        vpermq $0x1b, 0x500(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x9a0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm4, %ymm5, %ymm12+        vpaddd %ymm5, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm5   # ymm5 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm5, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm5, %ymm5+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm5, %ymm5+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7]+        vpermq $0x1b, 0x480(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x920(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm6, %ymm7, %ymm12+        vpaddd %ymm7, %ymm6, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpermq $0x1b, 0x400(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x8a0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm8, %ymm9, %ymm12+        vpaddd %ymm9, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm9   # ymm9 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm9, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm9, %ymm9+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7]+        vpermq $0x1b, 0x380(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x820(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm10, %ymm11, %ymm12+        vpaddd %ymm11, %ymm10, %ymm10+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vpermq $0x1b, 0x300(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x7a0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm4, %ymm6, %ymm12+        vpaddd %ymm6, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm6   # ymm6 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm6, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm6, %ymm6+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm6, %ymm6+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm6, %ymm12, %ymm6 # ymm6 = ymm12[0],ymm6[1],ymm12[2],ymm6[3],ymm12[4],ymm6[5],ymm12[6],ymm6[7]+        vpsubd %ymm5, %ymm7, %ymm12+        vpaddd %ymm7, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpermq $0x1b, 0x280(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x720(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm8, %ymm10, %ymm12+        vpaddd %ymm10, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm10  # ymm10 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm10, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm10, %ymm10+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm10, %ymm10+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7]+        vpsubd %ymm9, %ymm11, %ymm12+        vpaddd %ymm11, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vpermq $0x1b, 0x200(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x6a0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm4, %ymm8, %ymm12+        vpaddd %ymm4, %ymm8, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm8   # ymm8 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm8, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm8, %ymm8+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm8, %ymm8+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7]+        vpsubd %ymm5, %ymm9, %ymm12+        vpaddd %ymm5, %ymm9, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm9   # ymm9 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm9, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm9, %ymm9+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7]+        vpsubd %ymm6, %ymm10, %ymm12+        vpaddd %ymm6, %ymm10, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm10  # ymm10 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm10, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm10, %ymm10+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm10, %ymm10+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7]+        vpsubd %ymm7, %ymm11, %ymm12+        vpaddd %ymm7, %ymm11, %ymm7+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vmovsldup %ymm5, %ymm3    # ymm3 = ymm5[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7]+        vpsrlq $0x20, %ymm4, %ymm4+        vpblendd $0xaa, %ymm5, %ymm4, %ymm5 # ymm5 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7]+        vmovsldup %ymm7, %ymm4    # ymm4 = ymm7[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7]+        vpsrlq $0x20, %ymm6, %ymm6+        vpblendd $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7]+        vmovsldup %ymm9, %ymm6    # ymm6 = ymm9[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7]+        vpsrlq $0x20, %ymm8, %ymm8+        vpblendd $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7]+        vmovsldup %ymm11, %ymm8   # ymm8 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm8, %ymm10, %ymm8 # ymm8 = ymm10[0],ymm8[1],ymm10[2],ymm8[3],ymm10[4],ymm8[5],ymm10[6],ymm8[7]+        vpsrlq $0x20, %ymm10, %ymm10+        vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7]+        vpermq $0x1b, 0x180(%rsi), %ymm1 # ymm1 = mem[3,2,1,0]+        vpermq $0x1b, 0x620(%rsi), %ymm2 # ymm2 = mem[3,2,1,0]+        vpsubd %ymm3, %ymm5, %ymm12+        vpaddd %ymm5, %ymm3, %ymm3+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm5   # ymm5 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm5, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm5, %ymm5+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7]+        vpsubd %ymm4, %ymm7, %ymm12+        vpaddd %ymm7, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpsubd %ymm6, %ymm9, %ymm12+        vpaddd %ymm6, %ymm9, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm9   # ymm9 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm9, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm9, %ymm9+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7]+        vpsubd %ymm8, %ymm11, %ymm12+        vpaddd %ymm11, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vpunpcklqdq %ymm4, %ymm3, %ymm10 # ymm10 = ymm3[0],ymm4[0],ymm3[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[1],ymm4[1],ymm3[3],ymm4[3]+        vpunpcklqdq %ymm8, %ymm6, %ymm3 # ymm3 = ymm6[0],ymm8[0],ymm6[2],ymm8[2]+        vpunpckhqdq %ymm8, %ymm6, %ymm8 # ymm8 = ymm6[1],ymm8[1],ymm6[3],ymm8[3]+        vpunpcklqdq %ymm7, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm7[0],ymm5[2],ymm7[2]+        vpunpckhqdq %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[1],ymm7[1],ymm5[3],ymm7[3]+        vpunpcklqdq %ymm11, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm11[0],ymm9[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3]+        vpermq $0x1b, 0x100(%rsi), %ymm1 # ymm1 = mem[3,2,1,0]+        vpermq $0x1b, 0x5a0(%rsi), %ymm2 # ymm2 = mem[3,2,1,0]+        vpsubd %ymm10, %ymm4, %ymm12+        vpaddd %ymm4, %ymm10, %ymm10+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm4   # ymm4 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm4, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm4, %ymm4+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm4, %ymm12, %ymm4 # ymm4 = ymm12[0],ymm4[1],ymm12[2],ymm4[3],ymm12[4],ymm4[5],ymm12[6],ymm4[7]+        vpsubd %ymm3, %ymm8, %ymm12+        vpaddd %ymm3, %ymm8, %ymm3+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm8   # ymm8 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm8, %ymm8+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7]+        vpsubd %ymm6, %ymm7, %ymm12+        vpaddd %ymm7, %ymm6, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpsubd %ymm5, %ymm11, %ymm12+        vpaddd %ymm5, %ymm11, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vperm2i128 $0x20, %ymm3, %ymm10, %ymm9 # ymm9 = ymm10[0,1],ymm3[0,1]+        vperm2i128 $0x31, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[2,3],ymm3[2,3]+        vperm2i128 $0x20, %ymm5, %ymm6, %ymm10 # ymm10 = ymm6[0,1],ymm5[0,1]+        vperm2i128 $0x31, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[2,3],ymm5[2,3]+        vperm2i128 $0x20, %ymm8, %ymm4, %ymm6 # ymm6 = ymm4[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm11, %ymm7, %ymm4 # ymm4 = ymm7[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3]+        vpbroadcastd 0x9c(%rsi), %ymm1+        vpbroadcastd 0x53c(%rsi), %ymm2+        vpsubd %ymm9, %ymm3, %ymm12+        vpaddd %ymm3, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm3   # ymm3 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm3, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm3, %ymm3+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm3, %ymm3+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm3, %ymm12, %ymm3 # ymm3 = ymm12[0],ymm3[1],ymm12[2],ymm3[3],ymm12[4],ymm3[5],ymm12[6],ymm3[7]+        vpsubd %ymm10, %ymm5, %ymm12+        vpaddd %ymm5, %ymm10, %ymm10+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm5   # ymm5 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm5, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm5, %ymm5+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7]+        vpsubd %ymm6, %ymm8, %ymm12+        vpaddd %ymm6, %ymm8, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm8   # ymm8 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm8, %ymm8+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7]+        vpsubd %ymm4, %ymm11, %ymm12+        vpaddd %ymm4, %ymm11, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vmovdqa %ymm9, (%rdi)+        vmovdqa %ymm10, 0x20(%rdi)+        vmovdqa %ymm6, 0x40(%rdi)+        vmovdqa %ymm4, 0x60(%rdi)+        vmovdqa %ymm3, 0x80(%rdi)+        vmovdqa %ymm5, 0xa0(%rdi)+        vmovdqa %ymm8, 0xc0(%rdi)+        vmovdqa %ymm11, 0xe0(%rdi)+        vmovdqa 0x100(%rdi), %ymm4+        vmovdqa 0x120(%rdi), %ymm5+        vmovdqa 0x140(%rdi), %ymm6+        vmovdqa 0x160(%rdi), %ymm7+        vmovdqa 0x180(%rdi), %ymm8+        vmovdqa 0x1a0(%rdi), %ymm9+        vmovdqa 0x1c0(%rdi), %ymm10+        vmovdqa 0x1e0(%rdi), %ymm11+        vpermq $0x1b, 0x4e0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x980(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm4, %ymm5, %ymm12+        vpaddd %ymm5, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm5   # ymm5 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm5, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm5, %ymm5+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm5, %ymm5+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7]+        vpermq $0x1b, 0x460(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x900(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm6, %ymm7, %ymm12+        vpaddd %ymm7, %ymm6, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpermq $0x1b, 0x3e0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x880(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm8, %ymm9, %ymm12+        vpaddd %ymm9, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm9   # ymm9 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm9, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm9, %ymm9+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7]+        vpermq $0x1b, 0x360(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x800(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm10, %ymm11, %ymm12+        vpaddd %ymm11, %ymm10, %ymm10+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vpermq $0x1b, 0x2e0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x780(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm4, %ymm6, %ymm12+        vpaddd %ymm6, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm6   # ymm6 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm6, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm6, %ymm6+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm6, %ymm6+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm6, %ymm12, %ymm6 # ymm6 = ymm12[0],ymm6[1],ymm12[2],ymm6[3],ymm12[4],ymm6[5],ymm12[6],ymm6[7]+        vpsubd %ymm5, %ymm7, %ymm12+        vpaddd %ymm7, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpermq $0x1b, 0x260(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x700(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm8, %ymm10, %ymm12+        vpaddd %ymm10, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm10  # ymm10 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm10, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm10, %ymm10+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm10, %ymm10+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7]+        vpsubd %ymm9, %ymm11, %ymm12+        vpaddd %ymm11, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vpermq $0x1b, 0x1e0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x680(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm4, %ymm8, %ymm12+        vpaddd %ymm4, %ymm8, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm8   # ymm8 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm8, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm8, %ymm8+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm8, %ymm8+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7]+        vpsubd %ymm5, %ymm9, %ymm12+        vpaddd %ymm5, %ymm9, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm9   # ymm9 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm9, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm9, %ymm9+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7]+        vpsubd %ymm6, %ymm10, %ymm12+        vpaddd %ymm6, %ymm10, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm10  # ymm10 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm10, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm10, %ymm10+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm10, %ymm10+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7]+        vpsubd %ymm7, %ymm11, %ymm12+        vpaddd %ymm7, %ymm11, %ymm7+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vmovsldup %ymm5, %ymm3    # ymm3 = ymm5[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7]+        vpsrlq $0x20, %ymm4, %ymm4+        vpblendd $0xaa, %ymm5, %ymm4, %ymm5 # ymm5 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7]+        vmovsldup %ymm7, %ymm4    # ymm4 = ymm7[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7]+        vpsrlq $0x20, %ymm6, %ymm6+        vpblendd $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7]+        vmovsldup %ymm9, %ymm6    # ymm6 = ymm9[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7]+        vpsrlq $0x20, %ymm8, %ymm8+        vpblendd $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7]+        vmovsldup %ymm11, %ymm8   # ymm8 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm8, %ymm10, %ymm8 # ymm8 = ymm10[0],ymm8[1],ymm10[2],ymm8[3],ymm10[4],ymm8[5],ymm10[6],ymm8[7]+        vpsrlq $0x20, %ymm10, %ymm10+        vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7]+        vpermq $0x1b, 0x160(%rsi), %ymm1 # ymm1 = mem[3,2,1,0]+        vpermq $0x1b, 0x600(%rsi), %ymm2 # ymm2 = mem[3,2,1,0]+        vpsubd %ymm3, %ymm5, %ymm12+        vpaddd %ymm5, %ymm3, %ymm3+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm5   # ymm5 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm5, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm5, %ymm5+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7]+        vpsubd %ymm4, %ymm7, %ymm12+        vpaddd %ymm7, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpsubd %ymm6, %ymm9, %ymm12+        vpaddd %ymm6, %ymm9, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm9   # ymm9 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm9, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm9, %ymm9+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7]+        vpsubd %ymm8, %ymm11, %ymm12+        vpaddd %ymm11, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vpunpcklqdq %ymm4, %ymm3, %ymm10 # ymm10 = ymm3[0],ymm4[0],ymm3[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[1],ymm4[1],ymm3[3],ymm4[3]+        vpunpcklqdq %ymm8, %ymm6, %ymm3 # ymm3 = ymm6[0],ymm8[0],ymm6[2],ymm8[2]+        vpunpckhqdq %ymm8, %ymm6, %ymm8 # ymm8 = ymm6[1],ymm8[1],ymm6[3],ymm8[3]+        vpunpcklqdq %ymm7, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm7[0],ymm5[2],ymm7[2]+        vpunpckhqdq %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[1],ymm7[1],ymm5[3],ymm7[3]+        vpunpcklqdq %ymm11, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm11[0],ymm9[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3]+        vpermq $0x1b, 0xe0(%rsi), %ymm1 # ymm1 = mem[3,2,1,0]+        vpermq $0x1b, 0x580(%rsi), %ymm2 # ymm2 = mem[3,2,1,0]+        vpsubd %ymm10, %ymm4, %ymm12+        vpaddd %ymm4, %ymm10, %ymm10+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm4   # ymm4 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm4, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm4, %ymm4+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm4, %ymm12, %ymm4 # ymm4 = ymm12[0],ymm4[1],ymm12[2],ymm4[3],ymm12[4],ymm4[5],ymm12[6],ymm4[7]+        vpsubd %ymm3, %ymm8, %ymm12+        vpaddd %ymm3, %ymm8, %ymm3+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm8   # ymm8 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm8, %ymm8+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7]+        vpsubd %ymm6, %ymm7, %ymm12+        vpaddd %ymm7, %ymm6, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpsubd %ymm5, %ymm11, %ymm12+        vpaddd %ymm5, %ymm11, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vperm2i128 $0x20, %ymm3, %ymm10, %ymm9 # ymm9 = ymm10[0,1],ymm3[0,1]+        vperm2i128 $0x31, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[2,3],ymm3[2,3]+        vperm2i128 $0x20, %ymm5, %ymm6, %ymm10 # ymm10 = ymm6[0,1],ymm5[0,1]+        vperm2i128 $0x31, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[2,3],ymm5[2,3]+        vperm2i128 $0x20, %ymm8, %ymm4, %ymm6 # ymm6 = ymm4[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm11, %ymm7, %ymm4 # ymm4 = ymm7[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3]+        vpbroadcastd 0x98(%rsi), %ymm1+        vpbroadcastd 0x538(%rsi), %ymm2+        vpsubd %ymm9, %ymm3, %ymm12+        vpaddd %ymm3, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm3   # ymm3 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm3, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm3, %ymm3+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm3, %ymm3+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm3, %ymm12, %ymm3 # ymm3 = ymm12[0],ymm3[1],ymm12[2],ymm3[3],ymm12[4],ymm3[5],ymm12[6],ymm3[7]+        vpsubd %ymm10, %ymm5, %ymm12+        vpaddd %ymm5, %ymm10, %ymm10+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm5   # ymm5 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm5, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm5, %ymm5+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7]+        vpsubd %ymm6, %ymm8, %ymm12+        vpaddd %ymm6, %ymm8, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm8   # ymm8 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm8, %ymm8+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7]+        vpsubd %ymm4, %ymm11, %ymm12+        vpaddd %ymm4, %ymm11, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vmovdqa %ymm9, 0x100(%rdi)+        vmovdqa %ymm10, 0x120(%rdi)+        vmovdqa %ymm6, 0x140(%rdi)+        vmovdqa %ymm4, 0x160(%rdi)+        vmovdqa %ymm3, 0x180(%rdi)+        vmovdqa %ymm5, 0x1a0(%rdi)+        vmovdqa %ymm8, 0x1c0(%rdi)+        vmovdqa %ymm11, 0x1e0(%rdi)+        vmovdqa 0x200(%rdi), %ymm4+        vmovdqa 0x220(%rdi), %ymm5+        vmovdqa 0x240(%rdi), %ymm6+        vmovdqa 0x260(%rdi), %ymm7+        vmovdqa 0x280(%rdi), %ymm8+        vmovdqa 0x2a0(%rdi), %ymm9+        vmovdqa 0x2c0(%rdi), %ymm10+        vmovdqa 0x2e0(%rdi), %ymm11+        vpermq $0x1b, 0x4c0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x960(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm4, %ymm5, %ymm12+        vpaddd %ymm5, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm5   # ymm5 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm5, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm5, %ymm5+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm5, %ymm5+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7]+        vpermq $0x1b, 0x440(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x8e0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm6, %ymm7, %ymm12+        vpaddd %ymm7, %ymm6, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpermq $0x1b, 0x3c0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x860(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm8, %ymm9, %ymm12+        vpaddd %ymm9, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm9   # ymm9 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm9, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm9, %ymm9+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7]+        vpermq $0x1b, 0x340(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x7e0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm10, %ymm11, %ymm12+        vpaddd %ymm11, %ymm10, %ymm10+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vpermq $0x1b, 0x2c0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x760(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm4, %ymm6, %ymm12+        vpaddd %ymm6, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm6   # ymm6 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm6, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm6, %ymm6+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm6, %ymm6+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm6, %ymm12, %ymm6 # ymm6 = ymm12[0],ymm6[1],ymm12[2],ymm6[3],ymm12[4],ymm6[5],ymm12[6],ymm6[7]+        vpsubd %ymm5, %ymm7, %ymm12+        vpaddd %ymm7, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpermq $0x1b, 0x240(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x6e0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm8, %ymm10, %ymm12+        vpaddd %ymm10, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm10  # ymm10 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm10, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm10, %ymm10+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm10, %ymm10+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7]+        vpsubd %ymm9, %ymm11, %ymm12+        vpaddd %ymm11, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vpermq $0x1b, 0x1c0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x660(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm4, %ymm8, %ymm12+        vpaddd %ymm4, %ymm8, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm8   # ymm8 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm8, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm8, %ymm8+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm8, %ymm8+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7]+        vpsubd %ymm5, %ymm9, %ymm12+        vpaddd %ymm5, %ymm9, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm9   # ymm9 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm9, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm9, %ymm9+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7]+        vpsubd %ymm6, %ymm10, %ymm12+        vpaddd %ymm6, %ymm10, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm10  # ymm10 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm10, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm10, %ymm10+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm10, %ymm10+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7]+        vpsubd %ymm7, %ymm11, %ymm12+        vpaddd %ymm7, %ymm11, %ymm7+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vmovsldup %ymm5, %ymm3    # ymm3 = ymm5[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7]+        vpsrlq $0x20, %ymm4, %ymm4+        vpblendd $0xaa, %ymm5, %ymm4, %ymm5 # ymm5 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7]+        vmovsldup %ymm7, %ymm4    # ymm4 = ymm7[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7]+        vpsrlq $0x20, %ymm6, %ymm6+        vpblendd $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7]+        vmovsldup %ymm9, %ymm6    # ymm6 = ymm9[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7]+        vpsrlq $0x20, %ymm8, %ymm8+        vpblendd $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7]+        vmovsldup %ymm11, %ymm8   # ymm8 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm8, %ymm10, %ymm8 # ymm8 = ymm10[0],ymm8[1],ymm10[2],ymm8[3],ymm10[4],ymm8[5],ymm10[6],ymm8[7]+        vpsrlq $0x20, %ymm10, %ymm10+        vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7]+        vpermq $0x1b, 0x140(%rsi), %ymm1 # ymm1 = mem[3,2,1,0]+        vpermq $0x1b, 0x5e0(%rsi), %ymm2 # ymm2 = mem[3,2,1,0]+        vpsubd %ymm3, %ymm5, %ymm12+        vpaddd %ymm5, %ymm3, %ymm3+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm5   # ymm5 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm5, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm5, %ymm5+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7]+        vpsubd %ymm4, %ymm7, %ymm12+        vpaddd %ymm7, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpsubd %ymm6, %ymm9, %ymm12+        vpaddd %ymm6, %ymm9, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm9   # ymm9 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm9, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm9, %ymm9+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7]+        vpsubd %ymm8, %ymm11, %ymm12+        vpaddd %ymm11, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vpunpcklqdq %ymm4, %ymm3, %ymm10 # ymm10 = ymm3[0],ymm4[0],ymm3[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[1],ymm4[1],ymm3[3],ymm4[3]+        vpunpcklqdq %ymm8, %ymm6, %ymm3 # ymm3 = ymm6[0],ymm8[0],ymm6[2],ymm8[2]+        vpunpckhqdq %ymm8, %ymm6, %ymm8 # ymm8 = ymm6[1],ymm8[1],ymm6[3],ymm8[3]+        vpunpcklqdq %ymm7, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm7[0],ymm5[2],ymm7[2]+        vpunpckhqdq %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[1],ymm7[1],ymm5[3],ymm7[3]+        vpunpcklqdq %ymm11, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm11[0],ymm9[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3]+        vpermq $0x1b, 0xc0(%rsi), %ymm1 # ymm1 = mem[3,2,1,0]+        vpermq $0x1b, 0x560(%rsi), %ymm2 # ymm2 = mem[3,2,1,0]+        vpsubd %ymm10, %ymm4, %ymm12+        vpaddd %ymm4, %ymm10, %ymm10+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm4   # ymm4 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm4, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm4, %ymm4+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm4, %ymm12, %ymm4 # ymm4 = ymm12[0],ymm4[1],ymm12[2],ymm4[3],ymm12[4],ymm4[5],ymm12[6],ymm4[7]+        vpsubd %ymm3, %ymm8, %ymm12+        vpaddd %ymm3, %ymm8, %ymm3+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm8   # ymm8 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm8, %ymm8+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7]+        vpsubd %ymm6, %ymm7, %ymm12+        vpaddd %ymm7, %ymm6, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpsubd %ymm5, %ymm11, %ymm12+        vpaddd %ymm5, %ymm11, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vperm2i128 $0x20, %ymm3, %ymm10, %ymm9 # ymm9 = ymm10[0,1],ymm3[0,1]+        vperm2i128 $0x31, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[2,3],ymm3[2,3]+        vperm2i128 $0x20, %ymm5, %ymm6, %ymm10 # ymm10 = ymm6[0,1],ymm5[0,1]+        vperm2i128 $0x31, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[2,3],ymm5[2,3]+        vperm2i128 $0x20, %ymm8, %ymm4, %ymm6 # ymm6 = ymm4[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm11, %ymm7, %ymm4 # ymm4 = ymm7[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3]+        vpbroadcastd 0x94(%rsi), %ymm1+        vpbroadcastd 0x534(%rsi), %ymm2+        vpsubd %ymm9, %ymm3, %ymm12+        vpaddd %ymm3, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm3   # ymm3 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm3, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm3, %ymm3+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm3, %ymm3+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm3, %ymm12, %ymm3 # ymm3 = ymm12[0],ymm3[1],ymm12[2],ymm3[3],ymm12[4],ymm3[5],ymm12[6],ymm3[7]+        vpsubd %ymm10, %ymm5, %ymm12+        vpaddd %ymm5, %ymm10, %ymm10+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm5   # ymm5 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm5, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm5, %ymm5+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7]+        vpsubd %ymm6, %ymm8, %ymm12+        vpaddd %ymm6, %ymm8, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm8   # ymm8 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm8, %ymm8+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7]+        vpsubd %ymm4, %ymm11, %ymm12+        vpaddd %ymm4, %ymm11, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vmovdqa %ymm9, 0x200(%rdi)+        vmovdqa %ymm10, 0x220(%rdi)+        vmovdqa %ymm6, 0x240(%rdi)+        vmovdqa %ymm4, 0x260(%rdi)+        vmovdqa %ymm3, 0x280(%rdi)+        vmovdqa %ymm5, 0x2a0(%rdi)+        vmovdqa %ymm8, 0x2c0(%rdi)+        vmovdqa %ymm11, 0x2e0(%rdi)+        vmovdqa 0x300(%rdi), %ymm4+        vmovdqa 0x320(%rdi), %ymm5+        vmovdqa 0x340(%rdi), %ymm6+        vmovdqa 0x360(%rdi), %ymm7+        vmovdqa 0x380(%rdi), %ymm8+        vmovdqa 0x3a0(%rdi), %ymm9+        vmovdqa 0x3c0(%rdi), %ymm10+        vmovdqa 0x3e0(%rdi), %ymm11+        vpermq $0x1b, 0x4a0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x940(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm4, %ymm5, %ymm12+        vpaddd %ymm5, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm5   # ymm5 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm5, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm5, %ymm5+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm5, %ymm5+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7]+        vpermq $0x1b, 0x420(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x8c0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm6, %ymm7, %ymm12+        vpaddd %ymm7, %ymm6, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpermq $0x1b, 0x3a0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x840(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm8, %ymm9, %ymm12+        vpaddd %ymm9, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm9   # ymm9 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm9, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm9, %ymm9+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7]+        vpermq $0x1b, 0x320(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x7c0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm10, %ymm11, %ymm12+        vpaddd %ymm11, %ymm10, %ymm10+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vpermq $0x1b, 0x2a0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x740(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm4, %ymm6, %ymm12+        vpaddd %ymm6, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm6   # ymm6 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm6, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm6, %ymm6+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm6, %ymm6+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm6, %ymm12, %ymm6 # ymm6 = ymm12[0],ymm6[1],ymm12[2],ymm6[3],ymm12[4],ymm6[5],ymm12[6],ymm6[7]+        vpsubd %ymm5, %ymm7, %ymm12+        vpaddd %ymm7, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpermq $0x1b, 0x220(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x6c0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm8, %ymm10, %ymm12+        vpaddd %ymm10, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm10  # ymm10 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm10, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm10, %ymm10+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm10, %ymm10+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7]+        vpsubd %ymm9, %ymm11, %ymm12+        vpaddd %ymm11, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vpermq $0x1b, 0x1a0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0]+        vpermq $0x1b, 0x640(%rsi), %ymm15 # ymm15 = mem[3,2,1,0]+        vmovshdup %ymm3, %ymm1    # ymm1 = ymm3[1,1,3,3,5,5,7,7]+        vmovshdup %ymm15, %ymm2   # ymm2 = ymm15[1,1,3,3,5,5,7,7]+        vpsubd %ymm4, %ymm8, %ymm12+        vpaddd %ymm4, %ymm8, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm8   # ymm8 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm8, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm8, %ymm8+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm8, %ymm8+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7]+        vpsubd %ymm5, %ymm9, %ymm12+        vpaddd %ymm5, %ymm9, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm9   # ymm9 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm9, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm9, %ymm9+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7]+        vpsubd %ymm6, %ymm10, %ymm12+        vpaddd %ymm6, %ymm10, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm10  # ymm10 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm10, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm10, %ymm10+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm10, %ymm10+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7]+        vpsubd %ymm7, %ymm11, %ymm12+        vpaddd %ymm7, %ymm11, %ymm7+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm3, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm15, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vmovsldup %ymm5, %ymm3    # ymm3 = ymm5[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7]+        vpsrlq $0x20, %ymm4, %ymm4+        vpblendd $0xaa, %ymm5, %ymm4, %ymm5 # ymm5 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7]+        vmovsldup %ymm7, %ymm4    # ymm4 = ymm7[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7]+        vpsrlq $0x20, %ymm6, %ymm6+        vpblendd $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7]+        vmovsldup %ymm9, %ymm6    # ymm6 = ymm9[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7]+        vpsrlq $0x20, %ymm8, %ymm8+        vpblendd $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7]+        vmovsldup %ymm11, %ymm8   # ymm8 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm8, %ymm10, %ymm8 # ymm8 = ymm10[0],ymm8[1],ymm10[2],ymm8[3],ymm10[4],ymm8[5],ymm10[6],ymm8[7]+        vpsrlq $0x20, %ymm10, %ymm10+        vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7]+        vpermq $0x1b, 0x120(%rsi), %ymm1 # ymm1 = mem[3,2,1,0]+        vpermq $0x1b, 0x5c0(%rsi), %ymm2 # ymm2 = mem[3,2,1,0]+        vpsubd %ymm3, %ymm5, %ymm12+        vpaddd %ymm5, %ymm3, %ymm3+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm5   # ymm5 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm5, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm5, %ymm5+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7]+        vpsubd %ymm4, %ymm7, %ymm12+        vpaddd %ymm7, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpsubd %ymm6, %ymm9, %ymm12+        vpaddd %ymm6, %ymm9, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm9   # ymm9 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm9, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm9, %ymm9+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7]+        vpsubd %ymm8, %ymm11, %ymm12+        vpaddd %ymm11, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vpunpcklqdq %ymm4, %ymm3, %ymm10 # ymm10 = ymm3[0],ymm4[0],ymm3[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[1],ymm4[1],ymm3[3],ymm4[3]+        vpunpcklqdq %ymm8, %ymm6, %ymm3 # ymm3 = ymm6[0],ymm8[0],ymm6[2],ymm8[2]+        vpunpckhqdq %ymm8, %ymm6, %ymm8 # ymm8 = ymm6[1],ymm8[1],ymm6[3],ymm8[3]+        vpunpcklqdq %ymm7, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm7[0],ymm5[2],ymm7[2]+        vpunpckhqdq %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[1],ymm7[1],ymm5[3],ymm7[3]+        vpunpcklqdq %ymm11, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm11[0],ymm9[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3]+        vpermq $0x1b, 0xa0(%rsi), %ymm1 # ymm1 = mem[3,2,1,0]+        vpermq $0x1b, 0x540(%rsi), %ymm2 # ymm2 = mem[3,2,1,0]+        vpsubd %ymm10, %ymm4, %ymm12+        vpaddd %ymm4, %ymm10, %ymm10+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm4   # ymm4 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm4, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm4, %ymm4+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm4, %ymm12, %ymm4 # ymm4 = ymm12[0],ymm4[1],ymm12[2],ymm4[3],ymm12[4],ymm4[5],ymm12[6],ymm4[7]+        vpsubd %ymm3, %ymm8, %ymm12+        vpaddd %ymm3, %ymm8, %ymm3+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm8   # ymm8 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm8, %ymm8+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7]+        vpsubd %ymm6, %ymm7, %ymm12+        vpaddd %ymm7, %ymm6, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpsubd %ymm5, %ymm11, %ymm12+        vpaddd %ymm5, %ymm11, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vperm2i128 $0x20, %ymm3, %ymm10, %ymm9 # ymm9 = ymm10[0,1],ymm3[0,1]+        vperm2i128 $0x31, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[2,3],ymm3[2,3]+        vperm2i128 $0x20, %ymm5, %ymm6, %ymm10 # ymm10 = ymm6[0,1],ymm5[0,1]+        vperm2i128 $0x31, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[2,3],ymm5[2,3]+        vperm2i128 $0x20, %ymm8, %ymm4, %ymm6 # ymm6 = ymm4[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm11, %ymm7, %ymm4 # ymm4 = ymm7[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3]+        vpbroadcastd 0x90(%rsi), %ymm1+        vpbroadcastd 0x530(%rsi), %ymm2+        vpsubd %ymm9, %ymm3, %ymm12+        vpaddd %ymm3, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm3   # ymm3 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm3, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm3, %ymm3+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm3, %ymm3+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm3, %ymm12, %ymm3 # ymm3 = ymm12[0],ymm3[1],ymm12[2],ymm3[3],ymm12[4],ymm3[5],ymm12[6],ymm3[7]+        vpsubd %ymm10, %ymm5, %ymm12+        vpaddd %ymm5, %ymm10, %ymm10+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm5   # ymm5 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm5, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm5, %ymm5+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7]+        vpsubd %ymm6, %ymm8, %ymm12+        vpaddd %ymm6, %ymm8, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm8   # ymm8 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm8, %ymm8+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7]+        vpsubd %ymm4, %ymm11, %ymm12+        vpaddd %ymm4, %ymm11, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vmovdqa %ymm9, 0x300(%rdi)+        vmovdqa %ymm10, 0x320(%rdi)+        vmovdqa %ymm6, 0x340(%rdi)+        vmovdqa %ymm4, 0x360(%rdi)+        vmovdqa %ymm3, 0x380(%rdi)+        vmovdqa %ymm5, 0x3a0(%rdi)+        vmovdqa %ymm8, 0x3c0(%rdi)+        vmovdqa %ymm11, 0x3e0(%rdi)+        vmovdqa (%rdi), %ymm4+        vmovdqa 0x80(%rdi), %ymm5+        vmovdqa 0x100(%rdi), %ymm6+        vmovdqa 0x180(%rdi), %ymm7+        vmovdqa 0x200(%rdi), %ymm8+        vmovdqa 0x280(%rdi), %ymm9+        vmovdqa 0x300(%rdi), %ymm10+        vmovdqa 0x380(%rdi), %ymm11+        vpbroadcastd 0x8c(%rsi), %ymm1+        vpbroadcastd 0x52c(%rsi), %ymm2+        vpsubd %ymm4, %ymm6, %ymm12+        vpaddd %ymm6, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm6   # ymm6 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm6, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm6, %ymm6+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm6, %ymm12, %ymm6 # ymm6 = ymm12[0],ymm6[1],ymm12[2],ymm6[3],ymm12[4],ymm6[5],ymm12[6],ymm6[7]+        vpsubd %ymm5, %ymm7, %ymm12+        vpaddd %ymm7, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpbroadcastd 0x88(%rsi), %ymm1+        vpbroadcastd 0x528(%rsi), %ymm2+        vpsubd %ymm8, %ymm10, %ymm12+        vpaddd %ymm10, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm10  # ymm10 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm10, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm10, %ymm10+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7]+        vpsubd %ymm9, %ymm11, %ymm12+        vpaddd %ymm11, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vpbroadcastd 0x80(%rsi), %ymm1+        vpbroadcastd 0x520(%rsi), %ymm2+        vpsubd %ymm4, %ymm8, %ymm12+        vpaddd %ymm4, %ymm8, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm8   # ymm8 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm8, %ymm8+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7]+        vpsubd %ymm5, %ymm9, %ymm12+        vpaddd %ymm5, %ymm9, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm9   # ymm9 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm9, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm9, %ymm9+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7]+        vpsubd %ymm6, %ymm10, %ymm12+        vpaddd %ymm6, %ymm10, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm10  # ymm10 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm10, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm10, %ymm10+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7]+        vpsubd %ymm7, %ymm11, %ymm12+        vpaddd %ymm7, %ymm11, %ymm7+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vmovdqa %ymm8, 0x200(%rdi)+        vmovdqa %ymm9, 0x280(%rdi)+        vmovdqa %ymm10, 0x300(%rdi)+        vmovdqa %ymm11, 0x380(%rdi)+        vmovdqa 0x40(%rsi), %ymm1+        vmovdqa 0x60(%rsi), %ymm2+        vpmuldq %ymm1, %ymm4, %ymm12+        vpmuldq %ymm1, %ymm5, %ymm13+        vmovshdup %ymm4, %ymm8    # ymm8 = ymm4[1,1,3,3,5,5,7,7]+        vmovshdup %ymm5, %ymm9    # ymm9 = ymm5[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm1, %ymm9, %ymm15+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpmuldq %ymm0, %ymm15, %ymm15+        vpsubd %ymm12, %ymm4, %ymm4+        vpsubd %ymm13, %ymm5, %ymm5+        vpsubd %ymm14, %ymm8, %ymm8+        vpsubd %ymm15, %ymm9, %ymm9+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vmovshdup %ymm5, %ymm5    # ymm5 = ymm5[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm8[1],ymm4[2],ymm8[3],ymm4[4],ymm8[5],ymm4[6],ymm8[7]+        vpblendd $0xaa, %ymm9, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm9[1],ymm5[2],ymm9[3],ymm5[4],ymm9[5],ymm5[6],ymm9[7]+        vpmuldq %ymm1, %ymm6, %ymm12+        vpmuldq %ymm1, %ymm7, %ymm13+        vmovshdup %ymm6, %ymm8    # ymm8 = ymm6[1,1,3,3,5,5,7,7]+        vmovshdup %ymm7, %ymm9    # ymm9 = ymm7[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm1, %ymm9, %ymm15+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpmuldq %ymm0, %ymm15, %ymm15+        vpsubd %ymm12, %ymm6, %ymm6+        vpsubd %ymm13, %ymm7, %ymm7+        vpsubd %ymm14, %ymm8, %ymm8+        vpsubd %ymm15, %ymm9, %ymm9+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vmovshdup %ymm7, %ymm7    # ymm7 = ymm7[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm8[1],ymm6[2],ymm8[3],ymm6[4],ymm8[5],ymm6[6],ymm8[7]+        vpblendd $0xaa, %ymm9, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7]+        vmovdqa %ymm4, (%rdi)+        vmovdqa %ymm5, 0x80(%rdi)+        vmovdqa %ymm6, 0x100(%rdi)+        vmovdqa %ymm7, 0x180(%rdi)+        vmovdqa 0x20(%rdi), %ymm4+        vmovdqa 0xa0(%rdi), %ymm5+        vmovdqa 0x120(%rdi), %ymm6+        vmovdqa 0x1a0(%rdi), %ymm7+        vmovdqa 0x220(%rdi), %ymm8+        vmovdqa 0x2a0(%rdi), %ymm9+        vmovdqa 0x320(%rdi), %ymm10+        vmovdqa 0x3a0(%rdi), %ymm11+        vpbroadcastd 0x8c(%rsi), %ymm1+        vpbroadcastd 0x52c(%rsi), %ymm2+        vpsubd %ymm4, %ymm6, %ymm12+        vpaddd %ymm6, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm6   # ymm6 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm6, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm6, %ymm6+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm6, %ymm12, %ymm6 # ymm6 = ymm12[0],ymm6[1],ymm12[2],ymm6[3],ymm12[4],ymm6[5],ymm12[6],ymm6[7]+        vpsubd %ymm5, %ymm7, %ymm12+        vpaddd %ymm7, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpbroadcastd 0x88(%rsi), %ymm1+        vpbroadcastd 0x528(%rsi), %ymm2+        vpsubd %ymm8, %ymm10, %ymm12+        vpaddd %ymm10, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm10  # ymm10 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm10, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm10, %ymm10+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7]+        vpsubd %ymm9, %ymm11, %ymm12+        vpaddd %ymm11, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vpbroadcastd 0x80(%rsi), %ymm1+        vpbroadcastd 0x520(%rsi), %ymm2+        vpsubd %ymm4, %ymm8, %ymm12+        vpaddd %ymm4, %ymm8, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm8   # ymm8 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm8, %ymm8+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7]+        vpsubd %ymm5, %ymm9, %ymm12+        vpaddd %ymm5, %ymm9, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm9   # ymm9 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm9, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm9, %ymm9+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7]+        vpsubd %ymm6, %ymm10, %ymm12+        vpaddd %ymm6, %ymm10, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm10  # ymm10 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm10, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm10, %ymm10+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7]+        vpsubd %ymm7, %ymm11, %ymm12+        vpaddd %ymm7, %ymm11, %ymm7+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vmovdqa %ymm8, 0x220(%rdi)+        vmovdqa %ymm9, 0x2a0(%rdi)+        vmovdqa %ymm10, 0x320(%rdi)+        vmovdqa %ymm11, 0x3a0(%rdi)+        vmovdqa 0x40(%rsi), %ymm1+        vmovdqa 0x60(%rsi), %ymm2+        vpmuldq %ymm1, %ymm4, %ymm12+        vpmuldq %ymm1, %ymm5, %ymm13+        vmovshdup %ymm4, %ymm8    # ymm8 = ymm4[1,1,3,3,5,5,7,7]+        vmovshdup %ymm5, %ymm9    # ymm9 = ymm5[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm1, %ymm9, %ymm15+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpmuldq %ymm0, %ymm15, %ymm15+        vpsubd %ymm12, %ymm4, %ymm4+        vpsubd %ymm13, %ymm5, %ymm5+        vpsubd %ymm14, %ymm8, %ymm8+        vpsubd %ymm15, %ymm9, %ymm9+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vmovshdup %ymm5, %ymm5    # ymm5 = ymm5[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm8[1],ymm4[2],ymm8[3],ymm4[4],ymm8[5],ymm4[6],ymm8[7]+        vpblendd $0xaa, %ymm9, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm9[1],ymm5[2],ymm9[3],ymm5[4],ymm9[5],ymm5[6],ymm9[7]+        vpmuldq %ymm1, %ymm6, %ymm12+        vpmuldq %ymm1, %ymm7, %ymm13+        vmovshdup %ymm6, %ymm8    # ymm8 = ymm6[1,1,3,3,5,5,7,7]+        vmovshdup %ymm7, %ymm9    # ymm9 = ymm7[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm1, %ymm9, %ymm15+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpmuldq %ymm0, %ymm15, %ymm15+        vpsubd %ymm12, %ymm6, %ymm6+        vpsubd %ymm13, %ymm7, %ymm7+        vpsubd %ymm14, %ymm8, %ymm8+        vpsubd %ymm15, %ymm9, %ymm9+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vmovshdup %ymm7, %ymm7    # ymm7 = ymm7[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm8[1],ymm6[2],ymm8[3],ymm6[4],ymm8[5],ymm6[6],ymm8[7]+        vpblendd $0xaa, %ymm9, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7]+        vmovdqa %ymm4, 0x20(%rdi)+        vmovdqa %ymm5, 0xa0(%rdi)+        vmovdqa %ymm6, 0x120(%rdi)+        vmovdqa %ymm7, 0x1a0(%rdi)+        vmovdqa 0x40(%rdi), %ymm4+        vmovdqa 0xc0(%rdi), %ymm5+        vmovdqa 0x140(%rdi), %ymm6+        vmovdqa 0x1c0(%rdi), %ymm7+        vmovdqa 0x240(%rdi), %ymm8+        vmovdqa 0x2c0(%rdi), %ymm9+        vmovdqa 0x340(%rdi), %ymm10+        vmovdqa 0x3c0(%rdi), %ymm11+        vpbroadcastd 0x8c(%rsi), %ymm1+        vpbroadcastd 0x52c(%rsi), %ymm2+        vpsubd %ymm4, %ymm6, %ymm12+        vpaddd %ymm6, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm6   # ymm6 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm6, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm6, %ymm6+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm6, %ymm12, %ymm6 # ymm6 = ymm12[0],ymm6[1],ymm12[2],ymm6[3],ymm12[4],ymm6[5],ymm12[6],ymm6[7]+        vpsubd %ymm5, %ymm7, %ymm12+        vpaddd %ymm7, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpbroadcastd 0x88(%rsi), %ymm1+        vpbroadcastd 0x528(%rsi), %ymm2+        vpsubd %ymm8, %ymm10, %ymm12+        vpaddd %ymm10, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm10  # ymm10 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm10, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm10, %ymm10+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7]+        vpsubd %ymm9, %ymm11, %ymm12+        vpaddd %ymm11, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vpbroadcastd 0x80(%rsi), %ymm1+        vpbroadcastd 0x520(%rsi), %ymm2+        vpsubd %ymm4, %ymm8, %ymm12+        vpaddd %ymm4, %ymm8, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm8   # ymm8 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm8, %ymm8+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7]+        vpsubd %ymm5, %ymm9, %ymm12+        vpaddd %ymm5, %ymm9, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm9   # ymm9 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm9, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm9, %ymm9+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7]+        vpsubd %ymm6, %ymm10, %ymm12+        vpaddd %ymm6, %ymm10, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm10  # ymm10 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm10, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm10, %ymm10+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7]+        vpsubd %ymm7, %ymm11, %ymm12+        vpaddd %ymm7, %ymm11, %ymm7+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vmovdqa %ymm8, 0x240(%rdi)+        vmovdqa %ymm9, 0x2c0(%rdi)+        vmovdqa %ymm10, 0x340(%rdi)+        vmovdqa %ymm11, 0x3c0(%rdi)+        vmovdqa 0x40(%rsi), %ymm1+        vmovdqa 0x60(%rsi), %ymm2+        vpmuldq %ymm1, %ymm4, %ymm12+        vpmuldq %ymm1, %ymm5, %ymm13+        vmovshdup %ymm4, %ymm8    # ymm8 = ymm4[1,1,3,3,5,5,7,7]+        vmovshdup %ymm5, %ymm9    # ymm9 = ymm5[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm1, %ymm9, %ymm15+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpmuldq %ymm0, %ymm15, %ymm15+        vpsubd %ymm12, %ymm4, %ymm4+        vpsubd %ymm13, %ymm5, %ymm5+        vpsubd %ymm14, %ymm8, %ymm8+        vpsubd %ymm15, %ymm9, %ymm9+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vmovshdup %ymm5, %ymm5    # ymm5 = ymm5[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm8[1],ymm4[2],ymm8[3],ymm4[4],ymm8[5],ymm4[6],ymm8[7]+        vpblendd $0xaa, %ymm9, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm9[1],ymm5[2],ymm9[3],ymm5[4],ymm9[5],ymm5[6],ymm9[7]+        vpmuldq %ymm1, %ymm6, %ymm12+        vpmuldq %ymm1, %ymm7, %ymm13+        vmovshdup %ymm6, %ymm8    # ymm8 = ymm6[1,1,3,3,5,5,7,7]+        vmovshdup %ymm7, %ymm9    # ymm9 = ymm7[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm1, %ymm9, %ymm15+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpmuldq %ymm0, %ymm15, %ymm15+        vpsubd %ymm12, %ymm6, %ymm6+        vpsubd %ymm13, %ymm7, %ymm7+        vpsubd %ymm14, %ymm8, %ymm8+        vpsubd %ymm15, %ymm9, %ymm9+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vmovshdup %ymm7, %ymm7    # ymm7 = ymm7[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm8[1],ymm6[2],ymm8[3],ymm6[4],ymm8[5],ymm6[6],ymm8[7]+        vpblendd $0xaa, %ymm9, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7]+        vmovdqa %ymm4, 0x40(%rdi)+        vmovdqa %ymm5, 0xc0(%rdi)+        vmovdqa %ymm6, 0x140(%rdi)+        vmovdqa %ymm7, 0x1c0(%rdi)+        vmovdqa 0x60(%rdi), %ymm4+        vmovdqa 0xe0(%rdi), %ymm5+        vmovdqa 0x160(%rdi), %ymm6+        vmovdqa 0x1e0(%rdi), %ymm7+        vmovdqa 0x260(%rdi), %ymm8+        vmovdqa 0x2e0(%rdi), %ymm9+        vmovdqa 0x360(%rdi), %ymm10+        vmovdqa 0x3e0(%rdi), %ymm11+        vpbroadcastd 0x8c(%rsi), %ymm1+        vpbroadcastd 0x52c(%rsi), %ymm2+        vpsubd %ymm4, %ymm6, %ymm12+        vpaddd %ymm6, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm6   # ymm6 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm6, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm6, %ymm6+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm6, %ymm12, %ymm6 # ymm6 = ymm12[0],ymm6[1],ymm12[2],ymm6[3],ymm12[4],ymm6[5],ymm12[6],ymm6[7]+        vpsubd %ymm5, %ymm7, %ymm12+        vpaddd %ymm7, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm7   # ymm7 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm7, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm7, %ymm7+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7]+        vpbroadcastd 0x88(%rsi), %ymm1+        vpbroadcastd 0x528(%rsi), %ymm2+        vpsubd %ymm8, %ymm10, %ymm12+        vpaddd %ymm10, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm10  # ymm10 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm10, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm10, %ymm10+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7]+        vpsubd %ymm9, %ymm11, %ymm12+        vpaddd %ymm11, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vpbroadcastd 0x80(%rsi), %ymm1+        vpbroadcastd 0x520(%rsi), %ymm2+        vpsubd %ymm4, %ymm8, %ymm12+        vpaddd %ymm4, %ymm8, %ymm4+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm8   # ymm8 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm8, %ymm8+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7]+        vpsubd %ymm5, %ymm9, %ymm12+        vpaddd %ymm5, %ymm9, %ymm5+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm9   # ymm9 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm9, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm9, %ymm9+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7]+        vpsubd %ymm6, %ymm10, %ymm12+        vpaddd %ymm6, %ymm10, %ymm6+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm10  # ymm10 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm10, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm10, %ymm10+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7]+        vpsubd %ymm7, %ymm11, %ymm12+        vpaddd %ymm7, %ymm11, %ymm7+        vpmuldq %ymm1, %ymm12, %ymm13+        vmovshdup %ymm12, %ymm11  # ymm11 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm14+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpsubd %ymm13, %ymm12, %ymm12+        vpsubd %ymm14, %ymm11, %ymm11+        vmovshdup %ymm12, %ymm12  # ymm12 = ymm12[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7]+        vmovdqa %ymm8, 0x260(%rdi)+        vmovdqa %ymm9, 0x2e0(%rdi)+        vmovdqa %ymm10, 0x360(%rdi)+        vmovdqa %ymm11, 0x3e0(%rdi)+        vmovdqa 0x40(%rsi), %ymm1+        vmovdqa 0x60(%rsi), %ymm2+        vpmuldq %ymm1, %ymm4, %ymm12+        vpmuldq %ymm1, %ymm5, %ymm13+        vmovshdup %ymm4, %ymm8    # ymm8 = ymm4[1,1,3,3,5,5,7,7]+        vmovshdup %ymm5, %ymm9    # ymm9 = ymm5[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm1, %ymm9, %ymm15+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpmuldq %ymm0, %ymm15, %ymm15+        vpsubd %ymm12, %ymm4, %ymm4+        vpsubd %ymm13, %ymm5, %ymm5+        vpsubd %ymm14, %ymm8, %ymm8+        vpsubd %ymm15, %ymm9, %ymm9+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vmovshdup %ymm5, %ymm5    # ymm5 = ymm5[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm8[1],ymm4[2],ymm8[3],ymm4[4],ymm8[5],ymm4[6],ymm8[7]+        vpblendd $0xaa, %ymm9, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm9[1],ymm5[2],ymm9[3],ymm5[4],ymm9[5],ymm5[6],ymm9[7]+        vpmuldq %ymm1, %ymm6, %ymm12+        vpmuldq %ymm1, %ymm7, %ymm13+        vmovshdup %ymm6, %ymm8    # ymm8 = ymm6[1,1,3,3,5,5,7,7]+        vmovshdup %ymm7, %ymm9    # ymm9 = ymm7[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm14+        vpmuldq %ymm1, %ymm9, %ymm15+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm0, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vpmuldq %ymm0, %ymm15, %ymm15+        vpsubd %ymm12, %ymm6, %ymm6+        vpsubd %ymm13, %ymm7, %ymm7+        vpsubd %ymm14, %ymm8, %ymm8+        vpsubd %ymm15, %ymm9, %ymm9+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vmovshdup %ymm7, %ymm7    # ymm7 = ymm7[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm8, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm8[1],ymm6[2],ymm8[3],ymm6[4],ymm8[5],ymm6[6],ymm8[7]+        vpblendd $0xaa, %ymm9, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7]+        vmovdqa %ymm4, 0x60(%rdi)+        vmovdqa %ymm5, 0xe0(%rdi)+        vmovdqa %ymm6, 0x160(%rdi)+        vmovdqa %ymm7, 0x1e0(%rdi)+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(invntt_avx2_asm)++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/mldsa_ntt_avx2_asm.S view
@@ -0,0 +1,2405 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++ /*+  * This file is derived from the public domain+  * AVX2 Dilithium implementation @[REF_AVX2].+  */++/*yaml+  Name: ntt_avx2_asm+  Description: x86_64 AVX2 forward NTT+  Signature: void mld_ntt_avx2_asm(int32_t *r, const int32_t *qdata)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: read/write+      c_parameter: int32_t *r+      description: Input/output polynomial (256 x int32_t)+    rsi:+      type: buffer+      size_bytes: 2496+      permissions: read-only+      c_parameter: const int32_t *qdata+      description: Precomputed constants (624 x int32_t)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_ntt_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(ntt_avx2_asm)+MLD_ASM_FN_SYMBOL(ntt_avx2_asm)++        .cfi_startproc+        vmovdqa (%rsi), %ymm0+        vpbroadcastd 0x84(%rsi), %ymm1+        vpbroadcastd 0x524(%rsi), %ymm2+        vmovdqa (%rdi), %ymm4+        vmovdqa 0x80(%rdi), %ymm5+        vmovdqa 0x100(%rdi), %ymm6+        vmovdqa 0x180(%rdi), %ymm7+        vmovdqa 0x200(%rdi), %ymm8+        vmovdqa 0x280(%rdi), %ymm9+        vmovdqa 0x300(%rdi), %ymm10+        vmovdqa 0x380(%rdi), %ymm11+        vpmuldq %ymm1, %ymm8, %ymm13+        vmovshdup %ymm8, %ymm12   # ymm12 = ymm8[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm8, %ymm8    # ymm8 = ymm8[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7]+        vpsubd %ymm8, %ymm4, %ymm12+        vpaddd %ymm4, %ymm8, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm8+        vpsubd %ymm13, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm9, %ymm13+        vmovshdup %ymm9, %ymm12   # ymm12 = ymm9[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm9, %ymm9    # ymm9 = ymm9[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm9, %ymm9 # ymm9 = ymm9[0],ymm12[1],ymm9[2],ymm12[3],ymm9[4],ymm12[5],ymm9[6],ymm12[7]+        vpsubd %ymm9, %ymm5, %ymm12+        vpaddd %ymm5, %ymm9, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm9+        vpsubd %ymm13, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm10, %ymm13+        vmovshdup %ymm10, %ymm12  # ymm12 = ymm10[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm10, %ymm10  # ymm10 = ymm10[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7]+        vpsubd %ymm10, %ymm6, %ymm12+        vpaddd %ymm6, %ymm10, %ymm6+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm10+        vpsubd %ymm13, %ymm6, %ymm6+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm7, %ymm12+        vpaddd %ymm7, %ymm11, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm7, %ymm7+        vpbroadcastd 0x88(%rsi), %ymm1+        vpbroadcastd 0x528(%rsi), %ymm2+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm4, %ymm12+        vpaddd %ymm6, %ymm4, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm7, %ymm13+        vmovshdup %ymm7, %ymm12   # ymm12 = ymm7[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm7, %ymm7    # ymm7 = ymm7[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm12[1],ymm7[2],ymm12[3],ymm7[4],ymm12[5],ymm7[6],ymm12[7]+        vpsubd %ymm7, %ymm5, %ymm12+        vpaddd %ymm7, %ymm5, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm7+        vpsubd %ymm13, %ymm5, %ymm5+        vpbroadcastd 0x8c(%rsi), %ymm1+        vpbroadcastd 0x52c(%rsi), %ymm2+        vpmuldq %ymm1, %ymm10, %ymm13+        vmovshdup %ymm10, %ymm12  # ymm12 = ymm10[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm10, %ymm10  # ymm10 = ymm10[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7]+        vpsubd %ymm10, %ymm8, %ymm12+        vpaddd %ymm10, %ymm8, %ymm8+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm10+        vpsubd %ymm13, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm9, %ymm12+        vpaddd %ymm11, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm9, %ymm9+        vmovdqa %ymm4, (%rdi)+        vmovdqa %ymm5, 0x80(%rdi)+        vmovdqa %ymm6, 0x100(%rdi)+        vmovdqa %ymm7, 0x180(%rdi)+        vmovdqa %ymm8, 0x200(%rdi)+        vmovdqa %ymm9, 0x280(%rdi)+        vmovdqa %ymm10, 0x300(%rdi)+        vmovdqa %ymm11, 0x380(%rdi)+        vpbroadcastd 0x84(%rsi), %ymm1+        vpbroadcastd 0x524(%rsi), %ymm2+        vmovdqa 0x20(%rdi), %ymm4+        vmovdqa 0xa0(%rdi), %ymm5+        vmovdqa 0x120(%rdi), %ymm6+        vmovdqa 0x1a0(%rdi), %ymm7+        vmovdqa 0x220(%rdi), %ymm8+        vmovdqa 0x2a0(%rdi), %ymm9+        vmovdqa 0x320(%rdi), %ymm10+        vmovdqa 0x3a0(%rdi), %ymm11+        vpmuldq %ymm1, %ymm8, %ymm13+        vmovshdup %ymm8, %ymm12   # ymm12 = ymm8[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm8, %ymm8    # ymm8 = ymm8[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7]+        vpsubd %ymm8, %ymm4, %ymm12+        vpaddd %ymm4, %ymm8, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm8+        vpsubd %ymm13, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm9, %ymm13+        vmovshdup %ymm9, %ymm12   # ymm12 = ymm9[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm9, %ymm9    # ymm9 = ymm9[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm9, %ymm9 # ymm9 = ymm9[0],ymm12[1],ymm9[2],ymm12[3],ymm9[4],ymm12[5],ymm9[6],ymm12[7]+        vpsubd %ymm9, %ymm5, %ymm12+        vpaddd %ymm5, %ymm9, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm9+        vpsubd %ymm13, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm10, %ymm13+        vmovshdup %ymm10, %ymm12  # ymm12 = ymm10[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm10, %ymm10  # ymm10 = ymm10[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7]+        vpsubd %ymm10, %ymm6, %ymm12+        vpaddd %ymm6, %ymm10, %ymm6+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm10+        vpsubd %ymm13, %ymm6, %ymm6+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm7, %ymm12+        vpaddd %ymm7, %ymm11, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm7, %ymm7+        vpbroadcastd 0x88(%rsi), %ymm1+        vpbroadcastd 0x528(%rsi), %ymm2+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm4, %ymm12+        vpaddd %ymm6, %ymm4, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm7, %ymm13+        vmovshdup %ymm7, %ymm12   # ymm12 = ymm7[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm7, %ymm7    # ymm7 = ymm7[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm12[1],ymm7[2],ymm12[3],ymm7[4],ymm12[5],ymm7[6],ymm12[7]+        vpsubd %ymm7, %ymm5, %ymm12+        vpaddd %ymm7, %ymm5, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm7+        vpsubd %ymm13, %ymm5, %ymm5+        vpbroadcastd 0x8c(%rsi), %ymm1+        vpbroadcastd 0x52c(%rsi), %ymm2+        vpmuldq %ymm1, %ymm10, %ymm13+        vmovshdup %ymm10, %ymm12  # ymm12 = ymm10[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm10, %ymm10  # ymm10 = ymm10[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7]+        vpsubd %ymm10, %ymm8, %ymm12+        vpaddd %ymm10, %ymm8, %ymm8+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm10+        vpsubd %ymm13, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm9, %ymm12+        vpaddd %ymm11, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm9, %ymm9+        vmovdqa %ymm4, 0x20(%rdi)+        vmovdqa %ymm5, 0xa0(%rdi)+        vmovdqa %ymm6, 0x120(%rdi)+        vmovdqa %ymm7, 0x1a0(%rdi)+        vmovdqa %ymm8, 0x220(%rdi)+        vmovdqa %ymm9, 0x2a0(%rdi)+        vmovdqa %ymm10, 0x320(%rdi)+        vmovdqa %ymm11, 0x3a0(%rdi)+        vpbroadcastd 0x84(%rsi), %ymm1+        vpbroadcastd 0x524(%rsi), %ymm2+        vmovdqa 0x40(%rdi), %ymm4+        vmovdqa 0xc0(%rdi), %ymm5+        vmovdqa 0x140(%rdi), %ymm6+        vmovdqa 0x1c0(%rdi), %ymm7+        vmovdqa 0x240(%rdi), %ymm8+        vmovdqa 0x2c0(%rdi), %ymm9+        vmovdqa 0x340(%rdi), %ymm10+        vmovdqa 0x3c0(%rdi), %ymm11+        vpmuldq %ymm1, %ymm8, %ymm13+        vmovshdup %ymm8, %ymm12   # ymm12 = ymm8[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm8, %ymm8    # ymm8 = ymm8[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7]+        vpsubd %ymm8, %ymm4, %ymm12+        vpaddd %ymm4, %ymm8, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm8+        vpsubd %ymm13, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm9, %ymm13+        vmovshdup %ymm9, %ymm12   # ymm12 = ymm9[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm9, %ymm9    # ymm9 = ymm9[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm9, %ymm9 # ymm9 = ymm9[0],ymm12[1],ymm9[2],ymm12[3],ymm9[4],ymm12[5],ymm9[6],ymm12[7]+        vpsubd %ymm9, %ymm5, %ymm12+        vpaddd %ymm5, %ymm9, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm9+        vpsubd %ymm13, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm10, %ymm13+        vmovshdup %ymm10, %ymm12  # ymm12 = ymm10[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm10, %ymm10  # ymm10 = ymm10[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7]+        vpsubd %ymm10, %ymm6, %ymm12+        vpaddd %ymm6, %ymm10, %ymm6+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm10+        vpsubd %ymm13, %ymm6, %ymm6+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm7, %ymm12+        vpaddd %ymm7, %ymm11, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm7, %ymm7+        vpbroadcastd 0x88(%rsi), %ymm1+        vpbroadcastd 0x528(%rsi), %ymm2+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm4, %ymm12+        vpaddd %ymm6, %ymm4, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm7, %ymm13+        vmovshdup %ymm7, %ymm12   # ymm12 = ymm7[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm7, %ymm7    # ymm7 = ymm7[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm12[1],ymm7[2],ymm12[3],ymm7[4],ymm12[5],ymm7[6],ymm12[7]+        vpsubd %ymm7, %ymm5, %ymm12+        vpaddd %ymm7, %ymm5, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm7+        vpsubd %ymm13, %ymm5, %ymm5+        vpbroadcastd 0x8c(%rsi), %ymm1+        vpbroadcastd 0x52c(%rsi), %ymm2+        vpmuldq %ymm1, %ymm10, %ymm13+        vmovshdup %ymm10, %ymm12  # ymm12 = ymm10[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm10, %ymm10  # ymm10 = ymm10[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7]+        vpsubd %ymm10, %ymm8, %ymm12+        vpaddd %ymm10, %ymm8, %ymm8+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm10+        vpsubd %ymm13, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm9, %ymm12+        vpaddd %ymm11, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm9, %ymm9+        vmovdqa %ymm4, 0x40(%rdi)+        vmovdqa %ymm5, 0xc0(%rdi)+        vmovdqa %ymm6, 0x140(%rdi)+        vmovdqa %ymm7, 0x1c0(%rdi)+        vmovdqa %ymm8, 0x240(%rdi)+        vmovdqa %ymm9, 0x2c0(%rdi)+        vmovdqa %ymm10, 0x340(%rdi)+        vmovdqa %ymm11, 0x3c0(%rdi)+        vpbroadcastd 0x84(%rsi), %ymm1+        vpbroadcastd 0x524(%rsi), %ymm2+        vmovdqa 0x60(%rdi), %ymm4+        vmovdqa 0xe0(%rdi), %ymm5+        vmovdqa 0x160(%rdi), %ymm6+        vmovdqa 0x1e0(%rdi), %ymm7+        vmovdqa 0x260(%rdi), %ymm8+        vmovdqa 0x2e0(%rdi), %ymm9+        vmovdqa 0x360(%rdi), %ymm10+        vmovdqa 0x3e0(%rdi), %ymm11+        vpmuldq %ymm1, %ymm8, %ymm13+        vmovshdup %ymm8, %ymm12   # ymm12 = ymm8[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm8, %ymm8    # ymm8 = ymm8[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7]+        vpsubd %ymm8, %ymm4, %ymm12+        vpaddd %ymm4, %ymm8, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm8+        vpsubd %ymm13, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm9, %ymm13+        vmovshdup %ymm9, %ymm12   # ymm12 = ymm9[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm9, %ymm9    # ymm9 = ymm9[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm9, %ymm9 # ymm9 = ymm9[0],ymm12[1],ymm9[2],ymm12[3],ymm9[4],ymm12[5],ymm9[6],ymm12[7]+        vpsubd %ymm9, %ymm5, %ymm12+        vpaddd %ymm5, %ymm9, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm9+        vpsubd %ymm13, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm10, %ymm13+        vmovshdup %ymm10, %ymm12  # ymm12 = ymm10[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm10, %ymm10  # ymm10 = ymm10[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7]+        vpsubd %ymm10, %ymm6, %ymm12+        vpaddd %ymm6, %ymm10, %ymm6+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm10+        vpsubd %ymm13, %ymm6, %ymm6+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm7, %ymm12+        vpaddd %ymm7, %ymm11, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm7, %ymm7+        vpbroadcastd 0x88(%rsi), %ymm1+        vpbroadcastd 0x528(%rsi), %ymm2+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm4, %ymm12+        vpaddd %ymm6, %ymm4, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm7, %ymm13+        vmovshdup %ymm7, %ymm12   # ymm12 = ymm7[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm7, %ymm7    # ymm7 = ymm7[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm12[1],ymm7[2],ymm12[3],ymm7[4],ymm12[5],ymm7[6],ymm12[7]+        vpsubd %ymm7, %ymm5, %ymm12+        vpaddd %ymm7, %ymm5, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm7+        vpsubd %ymm13, %ymm5, %ymm5+        vpbroadcastd 0x8c(%rsi), %ymm1+        vpbroadcastd 0x52c(%rsi), %ymm2+        vpmuldq %ymm1, %ymm10, %ymm13+        vmovshdup %ymm10, %ymm12  # ymm12 = ymm10[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm10, %ymm10  # ymm10 = ymm10[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7]+        vpsubd %ymm10, %ymm8, %ymm12+        vpaddd %ymm10, %ymm8, %ymm8+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm10+        vpsubd %ymm13, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm9, %ymm12+        vpaddd %ymm11, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm9, %ymm9+        vmovdqa %ymm4, 0x60(%rdi)+        vmovdqa %ymm5, 0xe0(%rdi)+        vmovdqa %ymm6, 0x160(%rdi)+        vmovdqa %ymm7, 0x1e0(%rdi)+        vmovdqa %ymm8, 0x260(%rdi)+        vmovdqa %ymm9, 0x2e0(%rdi)+        vmovdqa %ymm10, 0x360(%rdi)+        vmovdqa %ymm11, 0x3e0(%rdi)+        vmovdqa (%rdi), %ymm4+        vmovdqa 0x20(%rdi), %ymm5+        vmovdqa 0x40(%rdi), %ymm6+        vmovdqa 0x60(%rdi), %ymm7+        vmovdqa 0x80(%rdi), %ymm8+        vmovdqa 0xa0(%rdi), %ymm9+        vmovdqa 0xc0(%rdi), %ymm10+        vmovdqa 0xe0(%rdi), %ymm11+        vpbroadcastd 0x90(%rsi), %ymm1+        vpbroadcastd 0x530(%rsi), %ymm2+        vpmuldq %ymm1, %ymm8, %ymm13+        vmovshdup %ymm8, %ymm12   # ymm12 = ymm8[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm8, %ymm8    # ymm8 = ymm8[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7]+        vpsubd %ymm8, %ymm4, %ymm12+        vpaddd %ymm4, %ymm8, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm8+        vpsubd %ymm13, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm9, %ymm13+        vmovshdup %ymm9, %ymm12   # ymm12 = ymm9[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm9, %ymm9    # ymm9 = ymm9[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm9, %ymm9 # ymm9 = ymm9[0],ymm12[1],ymm9[2],ymm12[3],ymm9[4],ymm12[5],ymm9[6],ymm12[7]+        vpsubd %ymm9, %ymm5, %ymm12+        vpaddd %ymm5, %ymm9, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm9+        vpsubd %ymm13, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm10, %ymm13+        vmovshdup %ymm10, %ymm12  # ymm12 = ymm10[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm10, %ymm10  # ymm10 = ymm10[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7]+        vpsubd %ymm10, %ymm6, %ymm12+        vpaddd %ymm6, %ymm10, %ymm6+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm10+        vpsubd %ymm13, %ymm6, %ymm6+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm7, %ymm12+        vpaddd %ymm7, %ymm11, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm7, %ymm7+        vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1]+        vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3]+        vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1]+        vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3]+        vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3]+        vmovdqa 0xa0(%rsi), %ymm1+        vmovdqa 0x540(%rsi), %ymm2+        vpmuldq %ymm1, %ymm5, %ymm13+        vmovshdup %ymm5, %ymm12   # ymm12 = ymm5[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm5, %ymm5    # ymm5 = ymm5[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm12[1],ymm5[2],ymm12[3],ymm5[4],ymm12[5],ymm5[6],ymm12[7]+        vpsubd %ymm5, %ymm3, %ymm12+        vpaddd %ymm5, %ymm3, %ymm3+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm5+        vpsubd %ymm13, %ymm3, %ymm3+        vpmuldq %ymm1, %ymm10, %ymm13+        vmovshdup %ymm10, %ymm12  # ymm12 = ymm10[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm10, %ymm10  # ymm10 = ymm10[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7]+        vpsubd %ymm10, %ymm8, %ymm12+        vpaddd %ymm10, %ymm8, %ymm8+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm10+        vpsubd %ymm13, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm4, %ymm12+        vpaddd %ymm6, %ymm4, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm9, %ymm12+        vpaddd %ymm11, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm9, %ymm9+        vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3]+        vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2]+        vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3]+        vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2]+        vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3]+        vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3]+        vmovdqa 0x120(%rsi), %ymm1+        vmovdqa 0x5c0(%rsi), %ymm2+        vpmuldq %ymm1, %ymm8, %ymm13+        vmovshdup %ymm8, %ymm12   # ymm12 = ymm8[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm8, %ymm8    # ymm8 = ymm8[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7]+        vpsubd %ymm8, %ymm7, %ymm12+        vpaddd %ymm7, %ymm8, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm8+        vpsubd %ymm13, %ymm7, %ymm7+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm5, %ymm12+        vpaddd %ymm6, %ymm5, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm4, %ymm13+        vmovshdup %ymm4, %ymm12   # ymm12 = ymm4[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7]+        vpsubd %ymm4, %ymm3, %ymm12+        vpaddd %ymm4, %ymm3, %ymm3+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm4+        vpsubd %ymm13, %ymm3, %ymm3+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm10, %ymm12+        vpaddd %ymm11, %ymm10, %ymm10+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm10, %ymm10+        vmovsldup %ymm8, %ymm9    # ymm9 = ymm8[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7]+        vpsrlq $0x20, %ymm7, %ymm7+        vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7]+        vmovsldup %ymm6, %ymm7    # ymm7 = ymm6[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7]+        vpsrlq $0x20, %ymm5, %ymm5+        vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7]+        vmovsldup %ymm4, %ymm5    # ymm5 = ymm4[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7]+        vmovsldup %ymm11, %ymm3   # ymm3 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7]+        vpsrlq $0x20, %ymm10, %ymm10+        vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7]+        vmovdqa 0x1a0(%rsi), %ymm1+        vmovdqa 0x640(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm5, %ymm13+        vmovshdup %ymm5, %ymm12   # ymm12 = ymm5[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm5, %ymm5    # ymm5 = ymm5[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm12[1],ymm5[2],ymm12[3],ymm5[4],ymm12[5],ymm5[6],ymm12[7]+        vpsubd %ymm5, %ymm9, %ymm12+        vpaddd %ymm5, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm5+        vpsubd %ymm13, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm4, %ymm13+        vmovshdup %ymm4, %ymm12   # ymm12 = ymm4[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7]+        vpsubd %ymm4, %ymm8, %ymm12+        vpaddd %ymm4, %ymm8, %ymm8+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm4+        vpsubd %ymm13, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm3, %ymm13+        vmovshdup %ymm3, %ymm12   # ymm12 = ymm3[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm3, %ymm3+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm3, %ymm3    # ymm3 = ymm3[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm3, %ymm3 # ymm3 = ymm3[0],ymm12[1],ymm3[2],ymm12[3],ymm3[4],ymm12[5],ymm3[6],ymm12[7]+        vpsubd %ymm3, %ymm7, %ymm12+        vpaddd %ymm3, %ymm7, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm3+        vpsubd %ymm13, %ymm7, %ymm7+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm6, %ymm12+        vpaddd %ymm6, %ymm11, %ymm6+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm6, %ymm6+        vmovdqa 0x220(%rsi), %ymm1+        vmovdqa 0x6c0(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm7, %ymm13+        vmovshdup %ymm7, %ymm12   # ymm12 = ymm7[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm7, %ymm7    # ymm7 = ymm7[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm12[1],ymm7[2],ymm12[3],ymm7[4],ymm12[5],ymm7[6],ymm12[7]+        vpsubd %ymm7, %ymm9, %ymm12+        vpaddd %ymm7, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm7+        vpsubd %ymm13, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm8, %ymm12+        vpaddd %ymm6, %ymm8, %ymm8+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm8, %ymm8+        vmovdqa 0x2a0(%rsi), %ymm1+        vmovdqa 0x740(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm3, %ymm13+        vmovshdup %ymm3, %ymm12   # ymm12 = ymm3[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm3, %ymm3+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm3, %ymm3    # ymm3 = ymm3[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm3, %ymm3 # ymm3 = ymm3[0],ymm12[1],ymm3[2],ymm12[3],ymm3[4],ymm12[5],ymm3[6],ymm12[7]+        vpsubd %ymm3, %ymm5, %ymm12+        vpaddd %ymm3, %ymm5, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm3+        vpsubd %ymm13, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm4, %ymm12+        vpaddd %ymm4, %ymm11, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm4, %ymm4+        vmovdqa 0x320(%rsi), %ymm1+        vmovdqa 0x7c0(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm13+        vmovshdup %ymm8, %ymm12   # ymm12 = ymm8[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm8, %ymm8    # ymm8 = ymm8[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7]+        vpsubd %ymm8, %ymm9, %ymm12+        vpaddd %ymm8, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm8+        vpsubd %ymm13, %ymm9, %ymm9+        vmovdqa 0x3a0(%rsi), %ymm1+        vmovdqa 0x840(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm7, %ymm12+        vpaddd %ymm6, %ymm7, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm7, %ymm7+        vmovdqa 0x420(%rsi), %ymm1+        vmovdqa 0x8c0(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm4, %ymm13+        vmovshdup %ymm4, %ymm12   # ymm12 = ymm4[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7]+        vpsubd %ymm4, %ymm5, %ymm12+        vpaddd %ymm4, %ymm5, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm4+        vpsubd %ymm13, %ymm5, %ymm5+        vmovdqa 0x4a0(%rsi), %ymm1+        vmovdqa 0x940(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm3, %ymm12+        vpaddd %ymm3, %ymm11, %ymm3+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm3, %ymm3+        vmovdqa %ymm9, (%rdi)+        vmovdqa %ymm8, 0x20(%rdi)+        vmovdqa %ymm7, 0x40(%rdi)+        vmovdqa %ymm6, 0x60(%rdi)+        vmovdqa %ymm5, 0x80(%rdi)+        vmovdqa %ymm4, 0xa0(%rdi)+        vmovdqa %ymm3, 0xc0(%rdi)+        vmovdqa %ymm11, 0xe0(%rdi)+        vmovdqa 0x100(%rdi), %ymm4+        vmovdqa 0x120(%rdi), %ymm5+        vmovdqa 0x140(%rdi), %ymm6+        vmovdqa 0x160(%rdi), %ymm7+        vmovdqa 0x180(%rdi), %ymm8+        vmovdqa 0x1a0(%rdi), %ymm9+        vmovdqa 0x1c0(%rdi), %ymm10+        vmovdqa 0x1e0(%rdi), %ymm11+        vpbroadcastd 0x94(%rsi), %ymm1+        vpbroadcastd 0x534(%rsi), %ymm2+        vpmuldq %ymm1, %ymm8, %ymm13+        vmovshdup %ymm8, %ymm12   # ymm12 = ymm8[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm8, %ymm8    # ymm8 = ymm8[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7]+        vpsubd %ymm8, %ymm4, %ymm12+        vpaddd %ymm4, %ymm8, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm8+        vpsubd %ymm13, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm9, %ymm13+        vmovshdup %ymm9, %ymm12   # ymm12 = ymm9[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm9, %ymm9    # ymm9 = ymm9[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm9, %ymm9 # ymm9 = ymm9[0],ymm12[1],ymm9[2],ymm12[3],ymm9[4],ymm12[5],ymm9[6],ymm12[7]+        vpsubd %ymm9, %ymm5, %ymm12+        vpaddd %ymm5, %ymm9, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm9+        vpsubd %ymm13, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm10, %ymm13+        vmovshdup %ymm10, %ymm12  # ymm12 = ymm10[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm10, %ymm10  # ymm10 = ymm10[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7]+        vpsubd %ymm10, %ymm6, %ymm12+        vpaddd %ymm6, %ymm10, %ymm6+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm10+        vpsubd %ymm13, %ymm6, %ymm6+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm7, %ymm12+        vpaddd %ymm7, %ymm11, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm7, %ymm7+        vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1]+        vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3]+        vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1]+        vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3]+        vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3]+        vmovdqa 0xc0(%rsi), %ymm1+        vmovdqa 0x560(%rsi), %ymm2+        vpmuldq %ymm1, %ymm5, %ymm13+        vmovshdup %ymm5, %ymm12   # ymm12 = ymm5[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm5, %ymm5    # ymm5 = ymm5[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm12[1],ymm5[2],ymm12[3],ymm5[4],ymm12[5],ymm5[6],ymm12[7]+        vpsubd %ymm5, %ymm3, %ymm12+        vpaddd %ymm5, %ymm3, %ymm3+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm5+        vpsubd %ymm13, %ymm3, %ymm3+        vpmuldq %ymm1, %ymm10, %ymm13+        vmovshdup %ymm10, %ymm12  # ymm12 = ymm10[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm10, %ymm10  # ymm10 = ymm10[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7]+        vpsubd %ymm10, %ymm8, %ymm12+        vpaddd %ymm10, %ymm8, %ymm8+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm10+        vpsubd %ymm13, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm4, %ymm12+        vpaddd %ymm6, %ymm4, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm9, %ymm12+        vpaddd %ymm11, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm9, %ymm9+        vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3]+        vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2]+        vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3]+        vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2]+        vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3]+        vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3]+        vmovdqa 0x140(%rsi), %ymm1+        vmovdqa 0x5e0(%rsi), %ymm2+        vpmuldq %ymm1, %ymm8, %ymm13+        vmovshdup %ymm8, %ymm12   # ymm12 = ymm8[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm8, %ymm8    # ymm8 = ymm8[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7]+        vpsubd %ymm8, %ymm7, %ymm12+        vpaddd %ymm7, %ymm8, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm8+        vpsubd %ymm13, %ymm7, %ymm7+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm5, %ymm12+        vpaddd %ymm6, %ymm5, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm4, %ymm13+        vmovshdup %ymm4, %ymm12   # ymm12 = ymm4[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7]+        vpsubd %ymm4, %ymm3, %ymm12+        vpaddd %ymm4, %ymm3, %ymm3+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm4+        vpsubd %ymm13, %ymm3, %ymm3+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm10, %ymm12+        vpaddd %ymm11, %ymm10, %ymm10+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm10, %ymm10+        vmovsldup %ymm8, %ymm9    # ymm9 = ymm8[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7]+        vpsrlq $0x20, %ymm7, %ymm7+        vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7]+        vmovsldup %ymm6, %ymm7    # ymm7 = ymm6[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7]+        vpsrlq $0x20, %ymm5, %ymm5+        vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7]+        vmovsldup %ymm4, %ymm5    # ymm5 = ymm4[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7]+        vmovsldup %ymm11, %ymm3   # ymm3 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7]+        vpsrlq $0x20, %ymm10, %ymm10+        vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7]+        vmovdqa 0x1c0(%rsi), %ymm1+        vmovdqa 0x660(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm5, %ymm13+        vmovshdup %ymm5, %ymm12   # ymm12 = ymm5[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm5, %ymm5    # ymm5 = ymm5[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm12[1],ymm5[2],ymm12[3],ymm5[4],ymm12[5],ymm5[6],ymm12[7]+        vpsubd %ymm5, %ymm9, %ymm12+        vpaddd %ymm5, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm5+        vpsubd %ymm13, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm4, %ymm13+        vmovshdup %ymm4, %ymm12   # ymm12 = ymm4[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7]+        vpsubd %ymm4, %ymm8, %ymm12+        vpaddd %ymm4, %ymm8, %ymm8+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm4+        vpsubd %ymm13, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm3, %ymm13+        vmovshdup %ymm3, %ymm12   # ymm12 = ymm3[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm3, %ymm3+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm3, %ymm3    # ymm3 = ymm3[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm3, %ymm3 # ymm3 = ymm3[0],ymm12[1],ymm3[2],ymm12[3],ymm3[4],ymm12[5],ymm3[6],ymm12[7]+        vpsubd %ymm3, %ymm7, %ymm12+        vpaddd %ymm3, %ymm7, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm3+        vpsubd %ymm13, %ymm7, %ymm7+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm6, %ymm12+        vpaddd %ymm6, %ymm11, %ymm6+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm6, %ymm6+        vmovdqa 0x240(%rsi), %ymm1+        vmovdqa 0x6e0(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm7, %ymm13+        vmovshdup %ymm7, %ymm12   # ymm12 = ymm7[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm7, %ymm7    # ymm7 = ymm7[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm12[1],ymm7[2],ymm12[3],ymm7[4],ymm12[5],ymm7[6],ymm12[7]+        vpsubd %ymm7, %ymm9, %ymm12+        vpaddd %ymm7, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm7+        vpsubd %ymm13, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm8, %ymm12+        vpaddd %ymm6, %ymm8, %ymm8+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm8, %ymm8+        vmovdqa 0x2c0(%rsi), %ymm1+        vmovdqa 0x760(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm3, %ymm13+        vmovshdup %ymm3, %ymm12   # ymm12 = ymm3[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm3, %ymm3+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm3, %ymm3    # ymm3 = ymm3[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm3, %ymm3 # ymm3 = ymm3[0],ymm12[1],ymm3[2],ymm12[3],ymm3[4],ymm12[5],ymm3[6],ymm12[7]+        vpsubd %ymm3, %ymm5, %ymm12+        vpaddd %ymm3, %ymm5, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm3+        vpsubd %ymm13, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm4, %ymm12+        vpaddd %ymm4, %ymm11, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm4, %ymm4+        vmovdqa 0x340(%rsi), %ymm1+        vmovdqa 0x7e0(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm13+        vmovshdup %ymm8, %ymm12   # ymm12 = ymm8[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm8, %ymm8    # ymm8 = ymm8[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7]+        vpsubd %ymm8, %ymm9, %ymm12+        vpaddd %ymm8, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm8+        vpsubd %ymm13, %ymm9, %ymm9+        vmovdqa 0x3c0(%rsi), %ymm1+        vmovdqa 0x860(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm7, %ymm12+        vpaddd %ymm6, %ymm7, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm7, %ymm7+        vmovdqa 0x440(%rsi), %ymm1+        vmovdqa 0x8e0(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm4, %ymm13+        vmovshdup %ymm4, %ymm12   # ymm12 = ymm4[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7]+        vpsubd %ymm4, %ymm5, %ymm12+        vpaddd %ymm4, %ymm5, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm4+        vpsubd %ymm13, %ymm5, %ymm5+        vmovdqa 0x4c0(%rsi), %ymm1+        vmovdqa 0x960(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm3, %ymm12+        vpaddd %ymm3, %ymm11, %ymm3+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm3, %ymm3+        vmovdqa %ymm9, 0x100(%rdi)+        vmovdqa %ymm8, 0x120(%rdi)+        vmovdqa %ymm7, 0x140(%rdi)+        vmovdqa %ymm6, 0x160(%rdi)+        vmovdqa %ymm5, 0x180(%rdi)+        vmovdqa %ymm4, 0x1a0(%rdi)+        vmovdqa %ymm3, 0x1c0(%rdi)+        vmovdqa %ymm11, 0x1e0(%rdi)+        vmovdqa 0x200(%rdi), %ymm4+        vmovdqa 0x220(%rdi), %ymm5+        vmovdqa 0x240(%rdi), %ymm6+        vmovdqa 0x260(%rdi), %ymm7+        vmovdqa 0x280(%rdi), %ymm8+        vmovdqa 0x2a0(%rdi), %ymm9+        vmovdqa 0x2c0(%rdi), %ymm10+        vmovdqa 0x2e0(%rdi), %ymm11+        vpbroadcastd 0x98(%rsi), %ymm1+        vpbroadcastd 0x538(%rsi), %ymm2+        vpmuldq %ymm1, %ymm8, %ymm13+        vmovshdup %ymm8, %ymm12   # ymm12 = ymm8[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm8, %ymm8    # ymm8 = ymm8[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7]+        vpsubd %ymm8, %ymm4, %ymm12+        vpaddd %ymm4, %ymm8, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm8+        vpsubd %ymm13, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm9, %ymm13+        vmovshdup %ymm9, %ymm12   # ymm12 = ymm9[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm9, %ymm9    # ymm9 = ymm9[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm9, %ymm9 # ymm9 = ymm9[0],ymm12[1],ymm9[2],ymm12[3],ymm9[4],ymm12[5],ymm9[6],ymm12[7]+        vpsubd %ymm9, %ymm5, %ymm12+        vpaddd %ymm5, %ymm9, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm9+        vpsubd %ymm13, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm10, %ymm13+        vmovshdup %ymm10, %ymm12  # ymm12 = ymm10[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm10, %ymm10  # ymm10 = ymm10[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7]+        vpsubd %ymm10, %ymm6, %ymm12+        vpaddd %ymm6, %ymm10, %ymm6+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm10+        vpsubd %ymm13, %ymm6, %ymm6+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm7, %ymm12+        vpaddd %ymm7, %ymm11, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm7, %ymm7+        vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1]+        vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3]+        vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1]+        vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3]+        vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3]+        vmovdqa 0xe0(%rsi), %ymm1+        vmovdqa 0x580(%rsi), %ymm2+        vpmuldq %ymm1, %ymm5, %ymm13+        vmovshdup %ymm5, %ymm12   # ymm12 = ymm5[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm5, %ymm5    # ymm5 = ymm5[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm12[1],ymm5[2],ymm12[3],ymm5[4],ymm12[5],ymm5[6],ymm12[7]+        vpsubd %ymm5, %ymm3, %ymm12+        vpaddd %ymm5, %ymm3, %ymm3+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm5+        vpsubd %ymm13, %ymm3, %ymm3+        vpmuldq %ymm1, %ymm10, %ymm13+        vmovshdup %ymm10, %ymm12  # ymm12 = ymm10[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm10, %ymm10  # ymm10 = ymm10[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7]+        vpsubd %ymm10, %ymm8, %ymm12+        vpaddd %ymm10, %ymm8, %ymm8+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm10+        vpsubd %ymm13, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm4, %ymm12+        vpaddd %ymm6, %ymm4, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm9, %ymm12+        vpaddd %ymm11, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm9, %ymm9+        vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3]+        vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2]+        vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3]+        vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2]+        vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3]+        vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3]+        vmovdqa 0x160(%rsi), %ymm1+        vmovdqa 0x600(%rsi), %ymm2+        vpmuldq %ymm1, %ymm8, %ymm13+        vmovshdup %ymm8, %ymm12   # ymm12 = ymm8[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm8, %ymm8    # ymm8 = ymm8[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7]+        vpsubd %ymm8, %ymm7, %ymm12+        vpaddd %ymm7, %ymm8, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm8+        vpsubd %ymm13, %ymm7, %ymm7+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm5, %ymm12+        vpaddd %ymm6, %ymm5, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm4, %ymm13+        vmovshdup %ymm4, %ymm12   # ymm12 = ymm4[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7]+        vpsubd %ymm4, %ymm3, %ymm12+        vpaddd %ymm4, %ymm3, %ymm3+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm4+        vpsubd %ymm13, %ymm3, %ymm3+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm10, %ymm12+        vpaddd %ymm11, %ymm10, %ymm10+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm10, %ymm10+        vmovsldup %ymm8, %ymm9    # ymm9 = ymm8[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7]+        vpsrlq $0x20, %ymm7, %ymm7+        vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7]+        vmovsldup %ymm6, %ymm7    # ymm7 = ymm6[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7]+        vpsrlq $0x20, %ymm5, %ymm5+        vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7]+        vmovsldup %ymm4, %ymm5    # ymm5 = ymm4[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7]+        vmovsldup %ymm11, %ymm3   # ymm3 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7]+        vpsrlq $0x20, %ymm10, %ymm10+        vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7]+        vmovdqa 0x1e0(%rsi), %ymm1+        vmovdqa 0x680(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm5, %ymm13+        vmovshdup %ymm5, %ymm12   # ymm12 = ymm5[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm5, %ymm5    # ymm5 = ymm5[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm12[1],ymm5[2],ymm12[3],ymm5[4],ymm12[5],ymm5[6],ymm12[7]+        vpsubd %ymm5, %ymm9, %ymm12+        vpaddd %ymm5, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm5+        vpsubd %ymm13, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm4, %ymm13+        vmovshdup %ymm4, %ymm12   # ymm12 = ymm4[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7]+        vpsubd %ymm4, %ymm8, %ymm12+        vpaddd %ymm4, %ymm8, %ymm8+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm4+        vpsubd %ymm13, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm3, %ymm13+        vmovshdup %ymm3, %ymm12   # ymm12 = ymm3[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm3, %ymm3+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm3, %ymm3    # ymm3 = ymm3[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm3, %ymm3 # ymm3 = ymm3[0],ymm12[1],ymm3[2],ymm12[3],ymm3[4],ymm12[5],ymm3[6],ymm12[7]+        vpsubd %ymm3, %ymm7, %ymm12+        vpaddd %ymm3, %ymm7, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm3+        vpsubd %ymm13, %ymm7, %ymm7+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm6, %ymm12+        vpaddd %ymm6, %ymm11, %ymm6+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm6, %ymm6+        vmovdqa 0x260(%rsi), %ymm1+        vmovdqa 0x700(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm7, %ymm13+        vmovshdup %ymm7, %ymm12   # ymm12 = ymm7[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm7, %ymm7    # ymm7 = ymm7[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm12[1],ymm7[2],ymm12[3],ymm7[4],ymm12[5],ymm7[6],ymm12[7]+        vpsubd %ymm7, %ymm9, %ymm12+        vpaddd %ymm7, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm7+        vpsubd %ymm13, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm8, %ymm12+        vpaddd %ymm6, %ymm8, %ymm8+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm8, %ymm8+        vmovdqa 0x2e0(%rsi), %ymm1+        vmovdqa 0x780(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm3, %ymm13+        vmovshdup %ymm3, %ymm12   # ymm12 = ymm3[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm3, %ymm3+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm3, %ymm3    # ymm3 = ymm3[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm3, %ymm3 # ymm3 = ymm3[0],ymm12[1],ymm3[2],ymm12[3],ymm3[4],ymm12[5],ymm3[6],ymm12[7]+        vpsubd %ymm3, %ymm5, %ymm12+        vpaddd %ymm3, %ymm5, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm3+        vpsubd %ymm13, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm4, %ymm12+        vpaddd %ymm4, %ymm11, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm4, %ymm4+        vmovdqa 0x360(%rsi), %ymm1+        vmovdqa 0x800(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm13+        vmovshdup %ymm8, %ymm12   # ymm12 = ymm8[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm8, %ymm8    # ymm8 = ymm8[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7]+        vpsubd %ymm8, %ymm9, %ymm12+        vpaddd %ymm8, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm8+        vpsubd %ymm13, %ymm9, %ymm9+        vmovdqa 0x3e0(%rsi), %ymm1+        vmovdqa 0x880(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm7, %ymm12+        vpaddd %ymm6, %ymm7, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm7, %ymm7+        vmovdqa 0x460(%rsi), %ymm1+        vmovdqa 0x900(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm4, %ymm13+        vmovshdup %ymm4, %ymm12   # ymm12 = ymm4[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7]+        vpsubd %ymm4, %ymm5, %ymm12+        vpaddd %ymm4, %ymm5, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm4+        vpsubd %ymm13, %ymm5, %ymm5+        vmovdqa 0x4e0(%rsi), %ymm1+        vmovdqa 0x980(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm3, %ymm12+        vpaddd %ymm3, %ymm11, %ymm3+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm3, %ymm3+        vmovdqa %ymm9, 0x200(%rdi)+        vmovdqa %ymm8, 0x220(%rdi)+        vmovdqa %ymm7, 0x240(%rdi)+        vmovdqa %ymm6, 0x260(%rdi)+        vmovdqa %ymm5, 0x280(%rdi)+        vmovdqa %ymm4, 0x2a0(%rdi)+        vmovdqa %ymm3, 0x2c0(%rdi)+        vmovdqa %ymm11, 0x2e0(%rdi)+        vmovdqa 0x300(%rdi), %ymm4+        vmovdqa 0x320(%rdi), %ymm5+        vmovdqa 0x340(%rdi), %ymm6+        vmovdqa 0x360(%rdi), %ymm7+        vmovdqa 0x380(%rdi), %ymm8+        vmovdqa 0x3a0(%rdi), %ymm9+        vmovdqa 0x3c0(%rdi), %ymm10+        vmovdqa 0x3e0(%rdi), %ymm11+        vpbroadcastd 0x9c(%rsi), %ymm1+        vpbroadcastd 0x53c(%rsi), %ymm2+        vpmuldq %ymm1, %ymm8, %ymm13+        vmovshdup %ymm8, %ymm12   # ymm12 = ymm8[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm8, %ymm8    # ymm8 = ymm8[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7]+        vpsubd %ymm8, %ymm4, %ymm12+        vpaddd %ymm4, %ymm8, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm8+        vpsubd %ymm13, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm9, %ymm13+        vmovshdup %ymm9, %ymm12   # ymm12 = ymm9[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm9, %ymm9+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm9, %ymm9    # ymm9 = ymm9[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm9, %ymm9 # ymm9 = ymm9[0],ymm12[1],ymm9[2],ymm12[3],ymm9[4],ymm12[5],ymm9[6],ymm12[7]+        vpsubd %ymm9, %ymm5, %ymm12+        vpaddd %ymm5, %ymm9, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm9+        vpsubd %ymm13, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm10, %ymm13+        vmovshdup %ymm10, %ymm12  # ymm12 = ymm10[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm10, %ymm10  # ymm10 = ymm10[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7]+        vpsubd %ymm10, %ymm6, %ymm12+        vpaddd %ymm6, %ymm10, %ymm6+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm10+        vpsubd %ymm13, %ymm6, %ymm6+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm7, %ymm12+        vpaddd %ymm7, %ymm11, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm7, %ymm7+        vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1]+        vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3]+        vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1]+        vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3]+        vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3]+        vmovdqa 0x100(%rsi), %ymm1+        vmovdqa 0x5a0(%rsi), %ymm2+        vpmuldq %ymm1, %ymm5, %ymm13+        vmovshdup %ymm5, %ymm12   # ymm12 = ymm5[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm5, %ymm5    # ymm5 = ymm5[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm12[1],ymm5[2],ymm12[3],ymm5[4],ymm12[5],ymm5[6],ymm12[7]+        vpsubd %ymm5, %ymm3, %ymm12+        vpaddd %ymm5, %ymm3, %ymm3+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm5+        vpsubd %ymm13, %ymm3, %ymm3+        vpmuldq %ymm1, %ymm10, %ymm13+        vmovshdup %ymm10, %ymm12  # ymm12 = ymm10[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm10, %ymm10+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm10, %ymm10  # ymm10 = ymm10[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7]+        vpsubd %ymm10, %ymm8, %ymm12+        vpaddd %ymm10, %ymm8, %ymm8+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm10+        vpsubd %ymm13, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm4, %ymm12+        vpaddd %ymm6, %ymm4, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm4, %ymm4+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm9, %ymm12+        vpaddd %ymm11, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm9, %ymm9+        vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3]+        vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2]+        vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3]+        vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2]+        vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3]+        vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3]+        vmovdqa 0x180(%rsi), %ymm1+        vmovdqa 0x620(%rsi), %ymm2+        vpmuldq %ymm1, %ymm8, %ymm13+        vmovshdup %ymm8, %ymm12   # ymm12 = ymm8[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm8, %ymm8    # ymm8 = ymm8[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7]+        vpsubd %ymm8, %ymm7, %ymm12+        vpaddd %ymm7, %ymm8, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm8+        vpsubd %ymm13, %ymm7, %ymm7+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm5, %ymm12+        vpaddd %ymm6, %ymm5, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm4, %ymm13+        vmovshdup %ymm4, %ymm12   # ymm12 = ymm4[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7]+        vpsubd %ymm4, %ymm3, %ymm12+        vpaddd %ymm4, %ymm3, %ymm3+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm4+        vpsubd %ymm13, %ymm3, %ymm3+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm2, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm10, %ymm12+        vpaddd %ymm11, %ymm10, %ymm10+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm10, %ymm10+        vmovsldup %ymm8, %ymm9    # ymm9 = ymm8[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7]+        vpsrlq $0x20, %ymm7, %ymm7+        vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7]+        vmovsldup %ymm6, %ymm7    # ymm7 = ymm6[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7]+        vpsrlq $0x20, %ymm5, %ymm5+        vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7]+        vmovsldup %ymm4, %ymm5    # ymm5 = ymm4[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7]+        vmovsldup %ymm11, %ymm3   # ymm3 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7]+        vpsrlq $0x20, %ymm10, %ymm10+        vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7]+        vmovdqa 0x200(%rsi), %ymm1+        vmovdqa 0x6a0(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm5, %ymm13+        vmovshdup %ymm5, %ymm12   # ymm12 = ymm5[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm5, %ymm5+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm5, %ymm5    # ymm5 = ymm5[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm12[1],ymm5[2],ymm12[3],ymm5[4],ymm12[5],ymm5[6],ymm12[7]+        vpsubd %ymm5, %ymm9, %ymm12+        vpaddd %ymm5, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm5+        vpsubd %ymm13, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm4, %ymm13+        vmovshdup %ymm4, %ymm12   # ymm12 = ymm4[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7]+        vpsubd %ymm4, %ymm8, %ymm12+        vpaddd %ymm4, %ymm8, %ymm8+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm4+        vpsubd %ymm13, %ymm8, %ymm8+        vpmuldq %ymm1, %ymm3, %ymm13+        vmovshdup %ymm3, %ymm12   # ymm12 = ymm3[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm3, %ymm3+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm3, %ymm3    # ymm3 = ymm3[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm3, %ymm3 # ymm3 = ymm3[0],ymm12[1],ymm3[2],ymm12[3],ymm3[4],ymm12[5],ymm3[6],ymm12[7]+        vpsubd %ymm3, %ymm7, %ymm12+        vpaddd %ymm3, %ymm7, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm3+        vpsubd %ymm13, %ymm7, %ymm7+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm6, %ymm12+        vpaddd %ymm6, %ymm11, %ymm6+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm6, %ymm6+        vmovdqa 0x280(%rsi), %ymm1+        vmovdqa 0x720(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm7, %ymm13+        vmovshdup %ymm7, %ymm12   # ymm12 = ymm7[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm7, %ymm7+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm7, %ymm7    # ymm7 = ymm7[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm12[1],ymm7[2],ymm12[3],ymm7[4],ymm12[5],ymm7[6],ymm12[7]+        vpsubd %ymm7, %ymm9, %ymm12+        vpaddd %ymm7, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm7+        vpsubd %ymm13, %ymm9, %ymm9+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm8, %ymm12+        vpaddd %ymm6, %ymm8, %ymm8+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm8, %ymm8+        vmovdqa 0x300(%rsi), %ymm1+        vmovdqa 0x7a0(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm3, %ymm13+        vmovshdup %ymm3, %ymm12   # ymm12 = ymm3[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm3, %ymm3+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm3, %ymm3    # ymm3 = ymm3[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm3, %ymm3 # ymm3 = ymm3[0],ymm12[1],ymm3[2],ymm12[3],ymm3[4],ymm12[5],ymm3[6],ymm12[7]+        vpsubd %ymm3, %ymm5, %ymm12+        vpaddd %ymm3, %ymm5, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm3+        vpsubd %ymm13, %ymm5, %ymm5+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm4, %ymm12+        vpaddd %ymm4, %ymm11, %ymm4+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm4, %ymm4+        vmovdqa 0x380(%rsi), %ymm1+        vmovdqa 0x820(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm8, %ymm13+        vmovshdup %ymm8, %ymm12   # ymm12 = ymm8[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm8, %ymm8+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm8, %ymm8    # ymm8 = ymm8[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7]+        vpsubd %ymm8, %ymm9, %ymm12+        vpaddd %ymm8, %ymm9, %ymm9+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm8+        vpsubd %ymm13, %ymm9, %ymm9+        vmovdqa 0x400(%rsi), %ymm1+        vmovdqa 0x8a0(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm6, %ymm13+        vmovshdup %ymm6, %ymm12   # ymm12 = ymm6[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm6, %ymm6+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7]+        vpsubd %ymm6, %ymm7, %ymm12+        vpaddd %ymm6, %ymm7, %ymm7+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm6+        vpsubd %ymm13, %ymm7, %ymm7+        vmovdqa 0x480(%rsi), %ymm1+        vmovdqa 0x920(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm4, %ymm13+        vmovshdup %ymm4, %ymm12   # ymm12 = ymm4[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm4, %ymm4+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7]+        vpsubd %ymm4, %ymm5, %ymm12+        vpaddd %ymm4, %ymm5, %ymm5+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm4+        vpsubd %ymm13, %ymm5, %ymm5+        vmovdqa 0x500(%rsi), %ymm1+        vmovdqa 0x9a0(%rsi), %ymm2+        vpsrlq $0x20, %ymm1, %ymm10+        vmovshdup %ymm2, %ymm15   # ymm15 = ymm2[1,1,3,3,5,5,7,7]+        vpmuldq %ymm1, %ymm11, %ymm13+        vmovshdup %ymm11, %ymm12  # ymm12 = ymm11[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm12, %ymm14+        vpmuldq %ymm2, %ymm11, %ymm11+        vpmuldq %ymm15, %ymm12, %ymm12+        vpmuldq %ymm0, %ymm13, %ymm13+        vpmuldq %ymm0, %ymm14, %ymm14+        vmovshdup %ymm11, %ymm11  # ymm11 = ymm11[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7]+        vpsubd %ymm11, %ymm3, %ymm12+        vpaddd %ymm3, %ymm11, %ymm3+        vmovshdup %ymm13, %ymm13  # ymm13 = ymm13[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7]+        vpaddd %ymm13, %ymm12, %ymm11+        vpsubd %ymm13, %ymm3, %ymm3+        vmovdqa %ymm9, 0x300(%rdi)+        vmovdqa %ymm8, 0x320(%rdi)+        vmovdqa %ymm7, 0x340(%rdi)+        vmovdqa %ymm6, 0x360(%rdi)+        vmovdqa %ymm5, 0x380(%rdi)+        vmovdqa %ymm4, 0x3a0(%rdi)+        vmovdqa %ymm3, 0x3c0(%rdi)+        vmovdqa %ymm11, 0x3e0(%rdi)+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(ntt_avx2_asm)++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/mldsa_nttunpack_avx2_asm.S view
@@ -0,0 +1,254 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Dilithium implementation @[REF_AVX2].+ */++/*yaml+  Name: nttunpack_avx2_asm+  Description: x86_64 AVX2 NTT coefficient unpacking/permutation+  Signature: void mld_nttunpack_avx2_asm(int32_t *r)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: read/write+      c_parameter: int32_t *r+      description: Input/output polynomial (256 x int32_t)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_nttunpack_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(nttunpack_avx2_asm)+MLD_ASM_FN_SYMBOL(nttunpack_avx2_asm)++        .cfi_startproc+        vmovdqa (%rdi), %ymm4+        vmovdqa 0x20(%rdi), %ymm5+        vmovdqa 0x40(%rdi), %ymm6+        vmovdqa 0x60(%rdi), %ymm7+        vmovdqa 0x80(%rdi), %ymm8+        vmovdqa 0xa0(%rdi), %ymm9+        vmovdqa 0xc0(%rdi), %ymm10+        vmovdqa 0xe0(%rdi), %ymm11+        vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1]+        vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3]+        vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1]+        vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3]+        vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3]+        vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3]+        vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2]+        vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3]+        vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2]+        vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3]+        vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3]+        vmovsldup %ymm8, %ymm9    # ymm9 = ymm8[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7]+        vpsrlq $0x20, %ymm7, %ymm7+        vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7]+        vmovsldup %ymm6, %ymm7    # ymm7 = ymm6[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7]+        vpsrlq $0x20, %ymm5, %ymm5+        vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7]+        vmovsldup %ymm4, %ymm5    # ymm5 = ymm4[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7]+        vmovsldup %ymm11, %ymm3   # ymm3 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7]+        vpsrlq $0x20, %ymm10, %ymm10+        vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7]+        vmovdqa %ymm9, (%rdi)+        vmovdqa %ymm8, 0x20(%rdi)+        vmovdqa %ymm7, 0x40(%rdi)+        vmovdqa %ymm6, 0x60(%rdi)+        vmovdqa %ymm5, 0x80(%rdi)+        vmovdqa %ymm4, 0xa0(%rdi)+        vmovdqa %ymm3, 0xc0(%rdi)+        vmovdqa %ymm11, 0xe0(%rdi)+        vmovdqa 0x100(%rdi), %ymm4+        vmovdqa 0x120(%rdi), %ymm5+        vmovdqa 0x140(%rdi), %ymm6+        vmovdqa 0x160(%rdi), %ymm7+        vmovdqa 0x180(%rdi), %ymm8+        vmovdqa 0x1a0(%rdi), %ymm9+        vmovdqa 0x1c0(%rdi), %ymm10+        vmovdqa 0x1e0(%rdi), %ymm11+        vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1]+        vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3]+        vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1]+        vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3]+        vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3]+        vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3]+        vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2]+        vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3]+        vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2]+        vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3]+        vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3]+        vmovsldup %ymm8, %ymm9    # ymm9 = ymm8[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7]+        vpsrlq $0x20, %ymm7, %ymm7+        vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7]+        vmovsldup %ymm6, %ymm7    # ymm7 = ymm6[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7]+        vpsrlq $0x20, %ymm5, %ymm5+        vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7]+        vmovsldup %ymm4, %ymm5    # ymm5 = ymm4[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7]+        vmovsldup %ymm11, %ymm3   # ymm3 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7]+        vpsrlq $0x20, %ymm10, %ymm10+        vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7]+        vmovdqa %ymm9, 0x100(%rdi)+        vmovdqa %ymm8, 0x120(%rdi)+        vmovdqa %ymm7, 0x140(%rdi)+        vmovdqa %ymm6, 0x160(%rdi)+        vmovdqa %ymm5, 0x180(%rdi)+        vmovdqa %ymm4, 0x1a0(%rdi)+        vmovdqa %ymm3, 0x1c0(%rdi)+        vmovdqa %ymm11, 0x1e0(%rdi)+        vmovdqa 0x200(%rdi), %ymm4+        vmovdqa 0x220(%rdi), %ymm5+        vmovdqa 0x240(%rdi), %ymm6+        vmovdqa 0x260(%rdi), %ymm7+        vmovdqa 0x280(%rdi), %ymm8+        vmovdqa 0x2a0(%rdi), %ymm9+        vmovdqa 0x2c0(%rdi), %ymm10+        vmovdqa 0x2e0(%rdi), %ymm11+        vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1]+        vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3]+        vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1]+        vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3]+        vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3]+        vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3]+        vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2]+        vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3]+        vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2]+        vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3]+        vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3]+        vmovsldup %ymm8, %ymm9    # ymm9 = ymm8[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7]+        vpsrlq $0x20, %ymm7, %ymm7+        vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7]+        vmovsldup %ymm6, %ymm7    # ymm7 = ymm6[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7]+        vpsrlq $0x20, %ymm5, %ymm5+        vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7]+        vmovsldup %ymm4, %ymm5    # ymm5 = ymm4[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7]+        vmovsldup %ymm11, %ymm3   # ymm3 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7]+        vpsrlq $0x20, %ymm10, %ymm10+        vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7]+        vmovdqa %ymm9, 0x200(%rdi)+        vmovdqa %ymm8, 0x220(%rdi)+        vmovdqa %ymm7, 0x240(%rdi)+        vmovdqa %ymm6, 0x260(%rdi)+        vmovdqa %ymm5, 0x280(%rdi)+        vmovdqa %ymm4, 0x2a0(%rdi)+        vmovdqa %ymm3, 0x2c0(%rdi)+        vmovdqa %ymm11, 0x2e0(%rdi)+        vmovdqa 0x300(%rdi), %ymm4+        vmovdqa 0x320(%rdi), %ymm5+        vmovdqa 0x340(%rdi), %ymm6+        vmovdqa 0x360(%rdi), %ymm7+        vmovdqa 0x380(%rdi), %ymm8+        vmovdqa 0x3a0(%rdi), %ymm9+        vmovdqa 0x3c0(%rdi), %ymm10+        vmovdqa 0x3e0(%rdi), %ymm11+        vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1]+        vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3]+        vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1]+        vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3]+        vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3]+        vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3]+        vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2]+        vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3]+        vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2]+        vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3]+        vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3]+        vmovsldup %ymm8, %ymm9    # ymm9 = ymm8[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7]+        vpsrlq $0x20, %ymm7, %ymm7+        vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7]+        vmovsldup %ymm6, %ymm7    # ymm7 = ymm6[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7]+        vpsrlq $0x20, %ymm5, %ymm5+        vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7]+        vmovsldup %ymm4, %ymm5    # ymm5 = ymm4[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7]+        vmovsldup %ymm11, %ymm3   # ymm3 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7]+        vpsrlq $0x20, %ymm10, %ymm10+        vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7]+        vmovdqa %ymm9, 0x300(%rdi)+        vmovdqa %ymm8, 0x320(%rdi)+        vmovdqa %ymm7, 0x340(%rdi)+        vmovdqa %ymm6, 0x360(%rdi)+        vmovdqa %ymm5, 0x380(%rdi)+        vmovdqa %ymm4, 0x3a0(%rdi)+        vmovdqa %ymm3, 0x3c0(%rdi)+        vmovdqa %ymm11, 0x3e0(%rdi)+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(nttunpack_avx2_asm)++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/mldsa_pointwise_acc_l4_avx2_asm.S view
@@ -0,0 +1,173 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Dilithium implementation @[REF_AVX2].+ */++/*yaml+  Name: pointwise_acc_l4_avx2_asm+  Description: x86_64 AVX2 pointwise multiply-accumulate of length-4 polynomial vectors+  Signature: void mld_pointwise_acc_l4_avx2_asm(int32_t *c, const int32_t a[4][256], const int32_t b[4][256], const int32_t *qdata)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t *c+      description: Output polynomial (256 x int32_t)+    rsi:+      type: buffer+      size_bytes: 4096+      permissions: read-only+      c_parameter: const int32_t a[4][256]+      description: Input polynomial vector a (4 x 256 x int32_t)+    rdx:+      type: buffer+      size_bytes: 4096+      permissions: read-only+      c_parameter: const int32_t b[4][256]+      description: Input polynomial vector b (4 x 256 x int32_t)+    rcx:+      type: buffer+      size_bytes: 2496+      permissions: read-only+      c_parameter: const int32_t *qdata+      description: Precomputed constants (624 x int32_t)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \+    (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 4)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_pointwise_acc_l4_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(pointwise_acc_l4_avx2_asm)+MLD_ASM_FN_SYMBOL(pointwise_acc_l4_avx2_asm)++        .cfi_startproc+        vmovdqa 0x20(%rcx), %ymm0+        vmovdqa (%rcx), %ymm1+        xorl %eax, %eax++Lmld_pointwise_acc_l4_avx2_looptop2:+        vmovdqa (%rsi), %ymm6+        vmovdqa 0x20(%rsi), %ymm8+        vmovdqa (%rdx), %ymm10+        vmovdqa 0x20(%rdx), %ymm12+        vpsrlq $0x20, %ymm6, %ymm7+        vpsrlq $0x20, %ymm8, %ymm9+        vmovshdup %ymm10, %ymm11  # ymm11 = ymm10[1,1,3,3,5,5,7,7]+        vmovshdup %ymm12, %ymm13  # ymm13 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm6, %ymm6+        vpmuldq %ymm11, %ymm7, %ymm7+        vpmuldq %ymm12, %ymm8, %ymm8+        vpmuldq %ymm13, %ymm9, %ymm9+        vmovdqa %ymm6, %ymm2+        vmovdqa %ymm7, %ymm3+        vmovdqa %ymm8, %ymm4+        vmovdqa %ymm9, %ymm5+        vmovdqa 0x400(%rsi), %ymm6+        vmovdqa 0x420(%rsi), %ymm8+        vmovdqa 0x400(%rdx), %ymm10+        vmovdqa 0x420(%rdx), %ymm12+        vpsrlq $0x20, %ymm6, %ymm7+        vpsrlq $0x20, %ymm8, %ymm9+        vmovshdup %ymm10, %ymm11  # ymm11 = ymm10[1,1,3,3,5,5,7,7]+        vmovshdup %ymm12, %ymm13  # ymm13 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm6, %ymm6+        vpmuldq %ymm11, %ymm7, %ymm7+        vpmuldq %ymm12, %ymm8, %ymm8+        vpmuldq %ymm13, %ymm9, %ymm9+        vpaddq %ymm2, %ymm6, %ymm2+        vpaddq %ymm3, %ymm7, %ymm3+        vpaddq %ymm4, %ymm8, %ymm4+        vpaddq %ymm5, %ymm9, %ymm5+        vmovdqa 0x800(%rsi), %ymm6+        vmovdqa 0x820(%rsi), %ymm8+        vmovdqa 0x800(%rdx), %ymm10+        vmovdqa 0x820(%rdx), %ymm12+        vpsrlq $0x20, %ymm6, %ymm7+        vpsrlq $0x20, %ymm8, %ymm9+        vmovshdup %ymm10, %ymm11  # ymm11 = ymm10[1,1,3,3,5,5,7,7]+        vmovshdup %ymm12, %ymm13  # ymm13 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm6, %ymm6+        vpmuldq %ymm11, %ymm7, %ymm7+        vpmuldq %ymm12, %ymm8, %ymm8+        vpmuldq %ymm13, %ymm9, %ymm9+        vpaddq %ymm2, %ymm6, %ymm2+        vpaddq %ymm3, %ymm7, %ymm3+        vpaddq %ymm4, %ymm8, %ymm4+        vpaddq %ymm5, %ymm9, %ymm5+        vmovdqa 0xc00(%rsi), %ymm6+        vmovdqa 0xc20(%rsi), %ymm8+        vmovdqa 0xc00(%rdx), %ymm10+        vmovdqa 0xc20(%rdx), %ymm12+        vpsrlq $0x20, %ymm6, %ymm7+        vpsrlq $0x20, %ymm8, %ymm9+        vmovshdup %ymm10, %ymm11  # ymm11 = ymm10[1,1,3,3,5,5,7,7]+        vmovshdup %ymm12, %ymm13  # ymm13 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm6, %ymm6+        vpmuldq %ymm11, %ymm7, %ymm7+        vpmuldq %ymm12, %ymm8, %ymm8+        vpmuldq %ymm13, %ymm9, %ymm9+        vpaddq %ymm2, %ymm6, %ymm2+        vpaddq %ymm3, %ymm7, %ymm3+        vpaddq %ymm4, %ymm8, %ymm4+        vpaddq %ymm5, %ymm9, %ymm5+        vpmuldq %ymm2, %ymm0, %ymm6+        vpmuldq %ymm3, %ymm0, %ymm7+        vpmuldq %ymm4, %ymm0, %ymm8+        vpmuldq %ymm5, %ymm0, %ymm9+        vpmuldq %ymm6, %ymm1, %ymm6+        vpmuldq %ymm7, %ymm1, %ymm7+        vpmuldq %ymm8, %ymm1, %ymm8+        vpmuldq %ymm9, %ymm1, %ymm9+        vpsubq %ymm6, %ymm2, %ymm2+        vpsubq %ymm7, %ymm3, %ymm3+        vpsubq %ymm8, %ymm4, %ymm4+        vpsubq %ymm9, %ymm5, %ymm5+        vpsrlq $0x20, %ymm2, %ymm2+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm3, %ymm2, %ymm2 # ymm2 = ymm2[0],ymm3[1],ymm2[2],ymm3[3],ymm2[4],ymm3[5],ymm2[6],ymm3[7]+        vpblendd $0xaa, %ymm5, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7]+        vmovdqa %ymm2, (%rdi)+        vmovdqa %ymm4, 0x20(%rdi)+        addq $0x40, %rsi+        addq $0x40, %rdx+        addq $0x40, %rdi+        addl $0x1, %eax+        cmpl $0x10, %eax+        jb Lmld_pointwise_acc_l4_avx2_looptop2+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(pointwise_acc_l4_avx2_asm)++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \+          && (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 4) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/mldsa_pointwise_acc_l5_avx2_asm.S view
@@ -0,0 +1,189 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Dilithium implementation @[REF_AVX2].+ */++/*yaml+  Name: pointwise_acc_l5_avx2_asm+  Description: x86_64 AVX2 pointwise multiply-accumulate of length-5 polynomial vectors+  Signature: void mld_pointwise_acc_l5_avx2_asm(int32_t *c, const int32_t a[5][256], const int32_t b[5][256], const int32_t *qdata)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t *c+      description: Output polynomial (256 x int32_t)+    rsi:+      type: buffer+      size_bytes: 5120+      permissions: read-only+      c_parameter: const int32_t a[5][256]+      description: Input polynomial vector a (5 x 256 x int32_t)+    rdx:+      type: buffer+      size_bytes: 5120+      permissions: read-only+      c_parameter: const int32_t b[5][256]+      description: Input polynomial vector b (5 x 256 x int32_t)+    rcx:+      type: buffer+      size_bytes: 2496+      permissions: read-only+      c_parameter: const int32_t *qdata+      description: Precomputed constants (624 x int32_t)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \+    (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 5)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_pointwise_acc_l5_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(pointwise_acc_l5_avx2_asm)+MLD_ASM_FN_SYMBOL(pointwise_acc_l5_avx2_asm)++        .cfi_startproc+        vmovdqa 0x20(%rcx), %ymm0+        vmovdqa (%rcx), %ymm1+        xorl %eax, %eax++Lmld_pointwise_acc_l5_avx2_looptop2:+        vmovdqa (%rsi), %ymm6+        vmovdqa 0x20(%rsi), %ymm8+        vmovdqa (%rdx), %ymm10+        vmovdqa 0x20(%rdx), %ymm12+        vpsrlq $0x20, %ymm6, %ymm7+        vpsrlq $0x20, %ymm8, %ymm9+        vmovshdup %ymm10, %ymm11  # ymm11 = ymm10[1,1,3,3,5,5,7,7]+        vmovshdup %ymm12, %ymm13  # ymm13 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm6, %ymm6+        vpmuldq %ymm11, %ymm7, %ymm7+        vpmuldq %ymm12, %ymm8, %ymm8+        vpmuldq %ymm13, %ymm9, %ymm9+        vmovdqa %ymm6, %ymm2+        vmovdqa %ymm7, %ymm3+        vmovdqa %ymm8, %ymm4+        vmovdqa %ymm9, %ymm5+        vmovdqa 0x400(%rsi), %ymm6+        vmovdqa 0x420(%rsi), %ymm8+        vmovdqa 0x400(%rdx), %ymm10+        vmovdqa 0x420(%rdx), %ymm12+        vpsrlq $0x20, %ymm6, %ymm7+        vpsrlq $0x20, %ymm8, %ymm9+        vmovshdup %ymm10, %ymm11  # ymm11 = ymm10[1,1,3,3,5,5,7,7]+        vmovshdup %ymm12, %ymm13  # ymm13 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm6, %ymm6+        vpmuldq %ymm11, %ymm7, %ymm7+        vpmuldq %ymm12, %ymm8, %ymm8+        vpmuldq %ymm13, %ymm9, %ymm9+        vpaddq %ymm2, %ymm6, %ymm2+        vpaddq %ymm3, %ymm7, %ymm3+        vpaddq %ymm4, %ymm8, %ymm4+        vpaddq %ymm5, %ymm9, %ymm5+        vmovdqa 0x800(%rsi), %ymm6+        vmovdqa 0x820(%rsi), %ymm8+        vmovdqa 0x800(%rdx), %ymm10+        vmovdqa 0x820(%rdx), %ymm12+        vpsrlq $0x20, %ymm6, %ymm7+        vpsrlq $0x20, %ymm8, %ymm9+        vmovshdup %ymm10, %ymm11  # ymm11 = ymm10[1,1,3,3,5,5,7,7]+        vmovshdup %ymm12, %ymm13  # ymm13 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm6, %ymm6+        vpmuldq %ymm11, %ymm7, %ymm7+        vpmuldq %ymm12, %ymm8, %ymm8+        vpmuldq %ymm13, %ymm9, %ymm9+        vpaddq %ymm2, %ymm6, %ymm2+        vpaddq %ymm3, %ymm7, %ymm3+        vpaddq %ymm4, %ymm8, %ymm4+        vpaddq %ymm5, %ymm9, %ymm5+        vmovdqa 0xc00(%rsi), %ymm6+        vmovdqa 0xc20(%rsi), %ymm8+        vmovdqa 0xc00(%rdx), %ymm10+        vmovdqa 0xc20(%rdx), %ymm12+        vpsrlq $0x20, %ymm6, %ymm7+        vpsrlq $0x20, %ymm8, %ymm9+        vmovshdup %ymm10, %ymm11  # ymm11 = ymm10[1,1,3,3,5,5,7,7]+        vmovshdup %ymm12, %ymm13  # ymm13 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm6, %ymm6+        vpmuldq %ymm11, %ymm7, %ymm7+        vpmuldq %ymm12, %ymm8, %ymm8+        vpmuldq %ymm13, %ymm9, %ymm9+        vpaddq %ymm2, %ymm6, %ymm2+        vpaddq %ymm3, %ymm7, %ymm3+        vpaddq %ymm4, %ymm8, %ymm4+        vpaddq %ymm5, %ymm9, %ymm5+        vmovdqa 0x1000(%rsi), %ymm6+        vmovdqa 0x1020(%rsi), %ymm8+        vmovdqa 0x1000(%rdx), %ymm10+        vmovdqa 0x1020(%rdx), %ymm12+        vpsrlq $0x20, %ymm6, %ymm7+        vpsrlq $0x20, %ymm8, %ymm9+        vmovshdup %ymm10, %ymm11  # ymm11 = ymm10[1,1,3,3,5,5,7,7]+        vmovshdup %ymm12, %ymm13  # ymm13 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm6, %ymm6+        vpmuldq %ymm11, %ymm7, %ymm7+        vpmuldq %ymm12, %ymm8, %ymm8+        vpmuldq %ymm13, %ymm9, %ymm9+        vpaddq %ymm2, %ymm6, %ymm2+        vpaddq %ymm3, %ymm7, %ymm3+        vpaddq %ymm4, %ymm8, %ymm4+        vpaddq %ymm5, %ymm9, %ymm5+        vpmuldq %ymm2, %ymm0, %ymm6+        vpmuldq %ymm3, %ymm0, %ymm7+        vpmuldq %ymm4, %ymm0, %ymm8+        vpmuldq %ymm5, %ymm0, %ymm9+        vpmuldq %ymm6, %ymm1, %ymm6+        vpmuldq %ymm7, %ymm1, %ymm7+        vpmuldq %ymm8, %ymm1, %ymm8+        vpmuldq %ymm9, %ymm1, %ymm9+        vpsubq %ymm6, %ymm2, %ymm2+        vpsubq %ymm7, %ymm3, %ymm3+        vpsubq %ymm8, %ymm4, %ymm4+        vpsubq %ymm9, %ymm5, %ymm5+        vpsrlq $0x20, %ymm2, %ymm2+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm3, %ymm2, %ymm2 # ymm2 = ymm2[0],ymm3[1],ymm2[2],ymm3[3],ymm2[4],ymm3[5],ymm2[6],ymm3[7]+        vpblendd $0xaa, %ymm5, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7]+        vmovdqa %ymm2, (%rdi)+        vmovdqa %ymm4, 0x20(%rdi)+        addq $0x40, %rsi+        addq $0x40, %rdx+        addq $0x40, %rdi+        addl $0x1, %eax+        cmpl $0x10, %eax+        jb Lmld_pointwise_acc_l5_avx2_looptop2+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(pointwise_acc_l5_avx2_asm)++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \+          && (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 5) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/mldsa_pointwise_acc_l7_avx2_asm.S view
@@ -0,0 +1,221 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Dilithium implementation @[REF_AVX2].+ */++/*yaml+  Name: pointwise_acc_l7_avx2_asm+  Description: x86_64 AVX2 pointwise multiply-accumulate of length-7 polynomial vectors+  Signature: void mld_pointwise_acc_l7_avx2_asm(int32_t *c, const int32_t a[7][256], const int32_t b[7][256], const int32_t *qdata)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t *c+      description: Output polynomial (256 x int32_t)+    rsi:+      type: buffer+      size_bytes: 7168+      permissions: read-only+      c_parameter: const int32_t a[7][256]+      description: Input polynomial vector a (7 x 256 x int32_t)+    rdx:+      type: buffer+      size_bytes: 7168+      permissions: read-only+      c_parameter: const int32_t b[7][256]+      description: Input polynomial vector b (7 x 256 x int32_t)+    rcx:+      type: buffer+      size_bytes: 2496+      permissions: read-only+      c_parameter: const int32_t *qdata+      description: Precomputed constants (624 x int32_t)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \+    (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 7)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_pointwise_acc_l7_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(pointwise_acc_l7_avx2_asm)+MLD_ASM_FN_SYMBOL(pointwise_acc_l7_avx2_asm)++        .cfi_startproc+        vmovdqa 0x20(%rcx), %ymm0+        vmovdqa (%rcx), %ymm1+        xorl %eax, %eax++Lmld_pointwise_acc_l7_avx2_looptop2:+        vmovdqa (%rsi), %ymm6+        vmovdqa 0x20(%rsi), %ymm8+        vmovdqa (%rdx), %ymm10+        vmovdqa 0x20(%rdx), %ymm12+        vpsrlq $0x20, %ymm6, %ymm7+        vpsrlq $0x20, %ymm8, %ymm9+        vmovshdup %ymm10, %ymm11  # ymm11 = ymm10[1,1,3,3,5,5,7,7]+        vmovshdup %ymm12, %ymm13  # ymm13 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm6, %ymm6+        vpmuldq %ymm11, %ymm7, %ymm7+        vpmuldq %ymm12, %ymm8, %ymm8+        vpmuldq %ymm13, %ymm9, %ymm9+        vmovdqa %ymm6, %ymm2+        vmovdqa %ymm7, %ymm3+        vmovdqa %ymm8, %ymm4+        vmovdqa %ymm9, %ymm5+        vmovdqa 0x400(%rsi), %ymm6+        vmovdqa 0x420(%rsi), %ymm8+        vmovdqa 0x400(%rdx), %ymm10+        vmovdqa 0x420(%rdx), %ymm12+        vpsrlq $0x20, %ymm6, %ymm7+        vpsrlq $0x20, %ymm8, %ymm9+        vmovshdup %ymm10, %ymm11  # ymm11 = ymm10[1,1,3,3,5,5,7,7]+        vmovshdup %ymm12, %ymm13  # ymm13 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm6, %ymm6+        vpmuldq %ymm11, %ymm7, %ymm7+        vpmuldq %ymm12, %ymm8, %ymm8+        vpmuldq %ymm13, %ymm9, %ymm9+        vpaddq %ymm2, %ymm6, %ymm2+        vpaddq %ymm3, %ymm7, %ymm3+        vpaddq %ymm4, %ymm8, %ymm4+        vpaddq %ymm5, %ymm9, %ymm5+        vmovdqa 0x800(%rsi), %ymm6+        vmovdqa 0x820(%rsi), %ymm8+        vmovdqa 0x800(%rdx), %ymm10+        vmovdqa 0x820(%rdx), %ymm12+        vpsrlq $0x20, %ymm6, %ymm7+        vpsrlq $0x20, %ymm8, %ymm9+        vmovshdup %ymm10, %ymm11  # ymm11 = ymm10[1,1,3,3,5,5,7,7]+        vmovshdup %ymm12, %ymm13  # ymm13 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm6, %ymm6+        vpmuldq %ymm11, %ymm7, %ymm7+        vpmuldq %ymm12, %ymm8, %ymm8+        vpmuldq %ymm13, %ymm9, %ymm9+        vpaddq %ymm2, %ymm6, %ymm2+        vpaddq %ymm3, %ymm7, %ymm3+        vpaddq %ymm4, %ymm8, %ymm4+        vpaddq %ymm5, %ymm9, %ymm5+        vmovdqa 0xc00(%rsi), %ymm6+        vmovdqa 0xc20(%rsi), %ymm8+        vmovdqa 0xc00(%rdx), %ymm10+        vmovdqa 0xc20(%rdx), %ymm12+        vpsrlq $0x20, %ymm6, %ymm7+        vpsrlq $0x20, %ymm8, %ymm9+        vmovshdup %ymm10, %ymm11  # ymm11 = ymm10[1,1,3,3,5,5,7,7]+        vmovshdup %ymm12, %ymm13  # ymm13 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm6, %ymm6+        vpmuldq %ymm11, %ymm7, %ymm7+        vpmuldq %ymm12, %ymm8, %ymm8+        vpmuldq %ymm13, %ymm9, %ymm9+        vpaddq %ymm2, %ymm6, %ymm2+        vpaddq %ymm3, %ymm7, %ymm3+        vpaddq %ymm4, %ymm8, %ymm4+        vpaddq %ymm5, %ymm9, %ymm5+        vmovdqa 0x1000(%rsi), %ymm6+        vmovdqa 0x1020(%rsi), %ymm8+        vmovdqa 0x1000(%rdx), %ymm10+        vmovdqa 0x1020(%rdx), %ymm12+        vpsrlq $0x20, %ymm6, %ymm7+        vpsrlq $0x20, %ymm8, %ymm9+        vmovshdup %ymm10, %ymm11  # ymm11 = ymm10[1,1,3,3,5,5,7,7]+        vmovshdup %ymm12, %ymm13  # ymm13 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm6, %ymm6+        vpmuldq %ymm11, %ymm7, %ymm7+        vpmuldq %ymm12, %ymm8, %ymm8+        vpmuldq %ymm13, %ymm9, %ymm9+        vpaddq %ymm2, %ymm6, %ymm2+        vpaddq %ymm3, %ymm7, %ymm3+        vpaddq %ymm4, %ymm8, %ymm4+        vpaddq %ymm5, %ymm9, %ymm5+        vmovdqa 0x1400(%rsi), %ymm6+        vmovdqa 0x1420(%rsi), %ymm8+        vmovdqa 0x1400(%rdx), %ymm10+        vmovdqa 0x1420(%rdx), %ymm12+        vpsrlq $0x20, %ymm6, %ymm7+        vpsrlq $0x20, %ymm8, %ymm9+        vmovshdup %ymm10, %ymm11  # ymm11 = ymm10[1,1,3,3,5,5,7,7]+        vmovshdup %ymm12, %ymm13  # ymm13 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm6, %ymm6+        vpmuldq %ymm11, %ymm7, %ymm7+        vpmuldq %ymm12, %ymm8, %ymm8+        vpmuldq %ymm13, %ymm9, %ymm9+        vpaddq %ymm2, %ymm6, %ymm2+        vpaddq %ymm3, %ymm7, %ymm3+        vpaddq %ymm4, %ymm8, %ymm4+        vpaddq %ymm5, %ymm9, %ymm5+        vmovdqa 0x1800(%rsi), %ymm6+        vmovdqa 0x1820(%rsi), %ymm8+        vmovdqa 0x1800(%rdx), %ymm10+        vmovdqa 0x1820(%rdx), %ymm12+        vpsrlq $0x20, %ymm6, %ymm7+        vpsrlq $0x20, %ymm8, %ymm9+        vmovshdup %ymm10, %ymm11  # ymm11 = ymm10[1,1,3,3,5,5,7,7]+        vmovshdup %ymm12, %ymm13  # ymm13 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm6, %ymm6+        vpmuldq %ymm11, %ymm7, %ymm7+        vpmuldq %ymm12, %ymm8, %ymm8+        vpmuldq %ymm13, %ymm9, %ymm9+        vpaddq %ymm2, %ymm6, %ymm2+        vpaddq %ymm3, %ymm7, %ymm3+        vpaddq %ymm4, %ymm8, %ymm4+        vpaddq %ymm5, %ymm9, %ymm5+        vpmuldq %ymm2, %ymm0, %ymm6+        vpmuldq %ymm3, %ymm0, %ymm7+        vpmuldq %ymm4, %ymm0, %ymm8+        vpmuldq %ymm5, %ymm0, %ymm9+        vpmuldq %ymm6, %ymm1, %ymm6+        vpmuldq %ymm7, %ymm1, %ymm7+        vpmuldq %ymm8, %ymm1, %ymm8+        vpmuldq %ymm9, %ymm1, %ymm9+        vpsubq %ymm6, %ymm2, %ymm2+        vpsubq %ymm7, %ymm3, %ymm3+        vpsubq %ymm8, %ymm4, %ymm4+        vpsubq %ymm9, %ymm5, %ymm5+        vpsrlq $0x20, %ymm2, %ymm2+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm3, %ymm2, %ymm2 # ymm2 = ymm2[0],ymm3[1],ymm2[2],ymm3[3],ymm2[4],ymm3[5],ymm2[6],ymm3[7]+        vpblendd $0xaa, %ymm5, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7]+        vmovdqa %ymm2, (%rdi)+        vmovdqa %ymm4, 0x20(%rdi)+        addq $0x40, %rsi+        addq $0x40, %rdx+        addq $0x40, %rdi+        addl $0x1, %eax+        cmpl $0x10, %eax+        jb Lmld_pointwise_acc_l7_avx2_looptop2+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(pointwise_acc_l7_avx2_asm)++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \+          && (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 7) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/mldsa_pointwise_avx2_asm.S view
@@ -0,0 +1,158 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Dilithium implementation @[REF_AVX2].+ */++/*yaml+  Name: pointwise_avx2_asm+  Description: x86_64 AVX2 pointwise Montgomery multiplication+  Signature: void mld_pointwise_avx2_asm(int32_t *a, const int32_t *b, const int32_t *qdata)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: read/write+      c_parameter: int32_t *a+      description: Input/output polynomial (256 x int32_t)+    rsi:+      type: buffer+      size_bytes: 1024+      permissions: read-only+      c_parameter: const int32_t *b+      description: Input polynomial (256 x int32_t)+    rdx:+      type: buffer+      size_bytes: 2496+      permissions: read-only+      c_parameter: const int32_t *qdata+      description: Precomputed constants (624 x int32_t)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_pointwise_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(pointwise_avx2_asm)+MLD_ASM_FN_SYMBOL(pointwise_avx2_asm)++        .cfi_startproc+        vmovdqa 0x20(%rdx), %ymm0+        vmovdqa (%rdx), %ymm1+        xorl %eax, %eax++Lmld_pointwise_avx2_looptop1:+        vmovdqa (%rdi), %ymm2+        vmovdqa 0x20(%rdi), %ymm4+        vmovdqa 0x40(%rdi), %ymm6+        vmovdqa (%rsi), %ymm10+        vmovdqa 0x20(%rsi), %ymm12+        vmovdqa 0x40(%rsi), %ymm14+        vpsrlq $0x20, %ymm2, %ymm3+        vpsrlq $0x20, %ymm4, %ymm5+        vmovshdup %ymm6, %ymm7    # ymm7 = ymm6[1,1,3,3,5,5,7,7]+        vpsrlq $0x20, %ymm10, %ymm11+        vpsrlq $0x20, %ymm12, %ymm13+        vmovshdup %ymm14, %ymm15  # ymm15 = ymm14[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm2, %ymm2+        vpmuldq %ymm11, %ymm3, %ymm3+        vpmuldq %ymm12, %ymm4, %ymm4+        vpmuldq %ymm13, %ymm5, %ymm5+        vpmuldq %ymm14, %ymm6, %ymm6+        vpmuldq %ymm15, %ymm7, %ymm7+        vpmuldq %ymm2, %ymm0, %ymm10+        vpmuldq %ymm3, %ymm0, %ymm11+        vpmuldq %ymm4, %ymm0, %ymm12+        vpmuldq %ymm5, %ymm0, %ymm13+        vpmuldq %ymm6, %ymm0, %ymm14+        vpmuldq %ymm7, %ymm0, %ymm15+        vpmuldq %ymm10, %ymm1, %ymm10+        vpmuldq %ymm11, %ymm1, %ymm11+        vpmuldq %ymm12, %ymm1, %ymm12+        vpmuldq %ymm13, %ymm1, %ymm13+        vpmuldq %ymm14, %ymm1, %ymm14+        vpmuldq %ymm15, %ymm1, %ymm15+        vpsubq %ymm10, %ymm2, %ymm2+        vpsubq %ymm11, %ymm3, %ymm3+        vpsubq %ymm12, %ymm4, %ymm4+        vpsubq %ymm13, %ymm5, %ymm5+        vpsubq %ymm14, %ymm6, %ymm6+        vpsubq %ymm15, %ymm7, %ymm7+        vpsrlq $0x20, %ymm2, %ymm2+        vpsrlq $0x20, %ymm4, %ymm4+        vmovshdup %ymm6, %ymm6    # ymm6 = ymm6[1,1,3,3,5,5,7,7]+        vpblendd $0xaa, %ymm3, %ymm2, %ymm2 # ymm2 = ymm2[0],ymm3[1],ymm2[2],ymm3[3],ymm2[4],ymm3[5],ymm2[6],ymm3[7]+        vpblendd $0xaa, %ymm5, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7]+        vpblendd $0xaa, %ymm7, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7]+        vmovdqa %ymm2, (%rdi)+        vmovdqa %ymm4, 0x20(%rdi)+        vmovdqa %ymm6, 0x40(%rdi)+        addq $0x60, %rdi+        addq $0x60, %rsi+        addl $0x1, %eax+        cmpl $0xa, %eax+        jb Lmld_pointwise_avx2_looptop1+        vmovdqa (%rdi), %ymm2+        vmovdqa 0x20(%rdi), %ymm4+        vmovdqa (%rsi), %ymm10+        vmovdqa 0x20(%rsi), %ymm12+        vpsrlq $0x20, %ymm2, %ymm3+        vpsrlq $0x20, %ymm4, %ymm5+        vmovshdup %ymm10, %ymm11  # ymm11 = ymm10[1,1,3,3,5,5,7,7]+        vmovshdup %ymm12, %ymm13  # ymm13 = ymm12[1,1,3,3,5,5,7,7]+        vpmuldq %ymm10, %ymm2, %ymm2+        vpmuldq %ymm11, %ymm3, %ymm3+        vpmuldq %ymm12, %ymm4, %ymm4+        vpmuldq %ymm13, %ymm5, %ymm5+        vpmuldq %ymm2, %ymm0, %ymm10+        vpmuldq %ymm3, %ymm0, %ymm11+        vpmuldq %ymm4, %ymm0, %ymm12+        vpmuldq %ymm5, %ymm0, %ymm13+        vpmuldq %ymm10, %ymm1, %ymm10+        vpmuldq %ymm11, %ymm1, %ymm11+        vpmuldq %ymm12, %ymm1, %ymm12+        vpmuldq %ymm13, %ymm1, %ymm13+        vpsubq %ymm10, %ymm2, %ymm2+        vpsubq %ymm11, %ymm3, %ymm3+        vpsubq %ymm12, %ymm4, %ymm4+        vpsubq %ymm13, %ymm5, %ymm5+        vpsrlq $0x20, %ymm2, %ymm2+        vmovshdup %ymm4, %ymm4    # ymm4 = ymm4[1,1,3,3,5,5,7,7]+        vpblendd $0x55, %ymm2, %ymm3, %ymm2 # ymm2 = ymm2[0],ymm3[1],ymm2[2],ymm3[3],ymm2[4],ymm3[5],ymm2[6],ymm3[7]+        vpblendd $0x55, %ymm4, %ymm5, %ymm4 # ymm4 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7]+        vmovdqa %ymm2, (%rdi)+        vmovdqa %ymm4, 0x20(%rdi)+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(pointwise_avx2_asm)++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/mldsa_poly_caddq_avx2_asm.S view
@@ -0,0 +1,199 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Dilithium implementation @[REF_AVX2].+ */+++/*yaml+  Name: poly_caddq_avx2_asm+  Description: x86_64 AVX2 conditional addition of q to each coefficient.+    For all coefficients of the in/out polynomial, add Q if the coefficient+    is negative.+  Signature: void mld_poly_caddq_avx2_asm(int32_t *r)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: read/write+      c_parameter: int32_t *r+      description: Input/output polynomial (256 x int32_t)+*/++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)+++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_poly_caddq_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(poly_caddq_avx2_asm)+MLD_ASM_FN_SYMBOL(poly_caddq_avx2_asm)++        .cfi_startproc+        vpxor %xmm2, %xmm2, %xmm2+        movl $0x7fe001, %eax         # imm = 0x7FE001+        vmovd %eax, %xmm1+        vpbroadcastd %xmm1, %ymm1+        vpcmpgtd (%rdi), %ymm2, %ymm0+        vpand %ymm1, %ymm0, %ymm0+        vpaddd (%rdi), %ymm0, %ymm0+        vmovdqa %ymm0, (%rdi)+        vpcmpgtd 0x20(%rdi), %ymm2, %ymm3+        vpand %ymm1, %ymm3, %ymm3+        vpaddd 0x20(%rdi), %ymm3, %ymm3+        vmovdqa %ymm3, 0x20(%rdi)+        vpcmpgtd 0x40(%rdi), %ymm2, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpaddd 0x40(%rdi), %ymm4, %ymm4+        vmovdqa %ymm4, 0x40(%rdi)+        vpcmpgtd 0x60(%rdi), %ymm2, %ymm5+        vpand %ymm1, %ymm5, %ymm5+        vpaddd 0x60(%rdi), %ymm5, %ymm5+        vmovdqa %ymm5, 0x60(%rdi)+        vpcmpgtd 0x80(%rdi), %ymm2, %ymm0+        vpand %ymm1, %ymm0, %ymm0+        vpaddd 0x80(%rdi), %ymm0, %ymm0+        vmovdqa %ymm0, 0x80(%rdi)+        vpcmpgtd 0xa0(%rdi), %ymm2, %ymm3+        vpand %ymm1, %ymm3, %ymm3+        vpaddd 0xa0(%rdi), %ymm3, %ymm3+        vmovdqa %ymm3, 0xa0(%rdi)+        vpcmpgtd 0xc0(%rdi), %ymm2, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpaddd 0xc0(%rdi), %ymm4, %ymm4+        vmovdqa %ymm4, 0xc0(%rdi)+        vpcmpgtd 0xe0(%rdi), %ymm2, %ymm5+        vpand %ymm1, %ymm5, %ymm5+        vpaddd 0xe0(%rdi), %ymm5, %ymm5+        vmovdqa %ymm5, 0xe0(%rdi)+        vpcmpgtd 0x100(%rdi), %ymm2, %ymm0+        vpand %ymm1, %ymm0, %ymm0+        vpaddd 0x100(%rdi), %ymm0, %ymm0+        vmovdqa %ymm0, 0x100(%rdi)+        vpcmpgtd 0x120(%rdi), %ymm2, %ymm3+        vpand %ymm1, %ymm3, %ymm3+        vpaddd 0x120(%rdi), %ymm3, %ymm3+        vmovdqa %ymm3, 0x120(%rdi)+        vpcmpgtd 0x140(%rdi), %ymm2, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpaddd 0x140(%rdi), %ymm4, %ymm4+        vmovdqa %ymm4, 0x140(%rdi)+        vpcmpgtd 0x160(%rdi), %ymm2, %ymm5+        vpand %ymm1, %ymm5, %ymm5+        vpaddd 0x160(%rdi), %ymm5, %ymm5+        vmovdqa %ymm5, 0x160(%rdi)+        vpcmpgtd 0x180(%rdi), %ymm2, %ymm0+        vpand %ymm1, %ymm0, %ymm0+        vpaddd 0x180(%rdi), %ymm0, %ymm0+        vmovdqa %ymm0, 0x180(%rdi)+        vpcmpgtd 0x1a0(%rdi), %ymm2, %ymm3+        vpand %ymm1, %ymm3, %ymm3+        vpaddd 0x1a0(%rdi), %ymm3, %ymm3+        vmovdqa %ymm3, 0x1a0(%rdi)+        vpcmpgtd 0x1c0(%rdi), %ymm2, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpaddd 0x1c0(%rdi), %ymm4, %ymm4+        vmovdqa %ymm4, 0x1c0(%rdi)+        vpcmpgtd 0x1e0(%rdi), %ymm2, %ymm5+        vpand %ymm1, %ymm5, %ymm5+        vpaddd 0x1e0(%rdi), %ymm5, %ymm5+        vmovdqa %ymm5, 0x1e0(%rdi)+        vpcmpgtd 0x200(%rdi), %ymm2, %ymm0+        vpand %ymm1, %ymm0, %ymm0+        vpaddd 0x200(%rdi), %ymm0, %ymm0+        vmovdqa %ymm0, 0x200(%rdi)+        vpcmpgtd 0x220(%rdi), %ymm2, %ymm3+        vpand %ymm1, %ymm3, %ymm3+        vpaddd 0x220(%rdi), %ymm3, %ymm3+        vmovdqa %ymm3, 0x220(%rdi)+        vpcmpgtd 0x240(%rdi), %ymm2, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpaddd 0x240(%rdi), %ymm4, %ymm4+        vmovdqa %ymm4, 0x240(%rdi)+        vpcmpgtd 0x260(%rdi), %ymm2, %ymm5+        vpand %ymm1, %ymm5, %ymm5+        vpaddd 0x260(%rdi), %ymm5, %ymm5+        vmovdqa %ymm5, 0x260(%rdi)+        vpcmpgtd 0x280(%rdi), %ymm2, %ymm0+        vpand %ymm1, %ymm0, %ymm0+        vpaddd 0x280(%rdi), %ymm0, %ymm0+        vmovdqa %ymm0, 0x280(%rdi)+        vpcmpgtd 0x2a0(%rdi), %ymm2, %ymm3+        vpand %ymm1, %ymm3, %ymm3+        vpaddd 0x2a0(%rdi), %ymm3, %ymm3+        vmovdqa %ymm3, 0x2a0(%rdi)+        vpcmpgtd 0x2c0(%rdi), %ymm2, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpaddd 0x2c0(%rdi), %ymm4, %ymm4+        vmovdqa %ymm4, 0x2c0(%rdi)+        vpcmpgtd 0x2e0(%rdi), %ymm2, %ymm5+        vpand %ymm1, %ymm5, %ymm5+        vpaddd 0x2e0(%rdi), %ymm5, %ymm5+        vmovdqa %ymm5, 0x2e0(%rdi)+        vpcmpgtd 0x300(%rdi), %ymm2, %ymm0+        vpand %ymm1, %ymm0, %ymm0+        vpaddd 0x300(%rdi), %ymm0, %ymm0+        vmovdqa %ymm0, 0x300(%rdi)+        vpcmpgtd 0x320(%rdi), %ymm2, %ymm3+        vpand %ymm1, %ymm3, %ymm3+        vpaddd 0x320(%rdi), %ymm3, %ymm3+        vmovdqa %ymm3, 0x320(%rdi)+        vpcmpgtd 0x340(%rdi), %ymm2, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpaddd 0x340(%rdi), %ymm4, %ymm4+        vmovdqa %ymm4, 0x340(%rdi)+        vpcmpgtd 0x360(%rdi), %ymm2, %ymm5+        vpand %ymm1, %ymm5, %ymm5+        vpaddd 0x360(%rdi), %ymm5, %ymm5+        vmovdqa %ymm5, 0x360(%rdi)+        vpcmpgtd 0x380(%rdi), %ymm2, %ymm0+        vpand %ymm1, %ymm0, %ymm0+        vpaddd 0x380(%rdi), %ymm0, %ymm0+        vmovdqa %ymm0, 0x380(%rdi)+        vpcmpgtd 0x3a0(%rdi), %ymm2, %ymm3+        vpand %ymm1, %ymm3, %ymm3+        vpaddd 0x3a0(%rdi), %ymm3, %ymm3+        vmovdqa %ymm3, 0x3a0(%rdi)+        vpcmpgtd 0x3c0(%rdi), %ymm2, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpaddd 0x3c0(%rdi), %ymm4, %ymm4+        vmovdqa %ymm4, 0x3c0(%rdi)+        vpcmpgtd 0x3e0(%rdi), %ymm2, %ymm5+        vpand %ymm1, %ymm5, %ymm5+        vpaddd 0x3e0(%rdi), %ymm5, %ymm5+        vmovdqa %ymm5, 0x3e0(%rdi)+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(poly_caddq_avx2_asm)+++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/mldsa_poly_chknorm_avx2_asm.S view
@@ -0,0 +1,176 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Dilithium implementation @[REF_AVX2].+ */+++/*yaml+  Name: poly_chknorm_avx2_asm+  Description: x86_64 AVX2 infinity-norm bound check on polynomial coefficients.+    Check the infinity norm of the polynomial against the given bound B.+    Returns 0 if the norm is strictly smaller than B; otherwise returns 1+    (i.e. returns 1 if any |coefficient| >= B, 0 otherwise).+  Signature: int mld_poly_chknorm_avx2_asm(const int32_t *a, int32_t B)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: read-only+      c_parameter: const int32_t *a+      description: Input polynomial (256 x int32_t)+    rsi:+      type: scalar+      c_parameter: int32_t B+      description: Norm bound (must be non-negative)+      test_with: 131072  # representative non-negative bound (1 << 17)+*/++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)+++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_poly_chknorm_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(poly_chknorm_avx2_asm)+MLD_ASM_FN_SYMBOL(poly_chknorm_avx2_asm)++        .cfi_startproc+        subl $0x1, %esi+        vpxor %xmm1, %xmm1, %xmm1+        vmovd %esi, %xmm2+        vpbroadcastd %xmm2, %ymm2+        vpabsd (%rdi), %ymm0+        vpcmpgtd %ymm2, %ymm0, %ymm0+        vpor %ymm0, %ymm1, %ymm1+        vpabsd 0x20(%rdi), %ymm3+        vpcmpgtd %ymm2, %ymm3, %ymm3+        vpor %ymm3, %ymm1, %ymm1+        vpabsd 0x40(%rdi), %ymm4+        vpcmpgtd %ymm2, %ymm4, %ymm4+        vpor %ymm4, %ymm1, %ymm1+        vpabsd 0x60(%rdi), %ymm5+        vpcmpgtd %ymm2, %ymm5, %ymm5+        vpor %ymm5, %ymm1, %ymm1+        vpabsd 0x80(%rdi), %ymm0+        vpcmpgtd %ymm2, %ymm0, %ymm0+        vpor %ymm0, %ymm1, %ymm1+        vpabsd 0xa0(%rdi), %ymm3+        vpcmpgtd %ymm2, %ymm3, %ymm3+        vpor %ymm3, %ymm1, %ymm1+        vpabsd 0xc0(%rdi), %ymm4+        vpcmpgtd %ymm2, %ymm4, %ymm4+        vpor %ymm4, %ymm1, %ymm1+        vpabsd 0xe0(%rdi), %ymm5+        vpcmpgtd %ymm2, %ymm5, %ymm5+        vpor %ymm5, %ymm1, %ymm1+        vpabsd 0x100(%rdi), %ymm0+        vpcmpgtd %ymm2, %ymm0, %ymm0+        vpor %ymm0, %ymm1, %ymm1+        vpabsd 0x120(%rdi), %ymm3+        vpcmpgtd %ymm2, %ymm3, %ymm3+        vpor %ymm3, %ymm1, %ymm1+        vpabsd 0x140(%rdi), %ymm4+        vpcmpgtd %ymm2, %ymm4, %ymm4+        vpor %ymm4, %ymm1, %ymm1+        vpabsd 0x160(%rdi), %ymm5+        vpcmpgtd %ymm2, %ymm5, %ymm5+        vpor %ymm5, %ymm1, %ymm1+        vpabsd 0x180(%rdi), %ymm0+        vpcmpgtd %ymm2, %ymm0, %ymm0+        vpor %ymm0, %ymm1, %ymm1+        vpabsd 0x1a0(%rdi), %ymm3+        vpcmpgtd %ymm2, %ymm3, %ymm3+        vpor %ymm3, %ymm1, %ymm1+        vpabsd 0x1c0(%rdi), %ymm4+        vpcmpgtd %ymm2, %ymm4, %ymm4+        vpor %ymm4, %ymm1, %ymm1+        vpabsd 0x1e0(%rdi), %ymm5+        vpcmpgtd %ymm2, %ymm5, %ymm5+        vpor %ymm5, %ymm1, %ymm1+        vpabsd 0x200(%rdi), %ymm0+        vpcmpgtd %ymm2, %ymm0, %ymm0+        vpor %ymm0, %ymm1, %ymm1+        vpabsd 0x220(%rdi), %ymm3+        vpcmpgtd %ymm2, %ymm3, %ymm3+        vpor %ymm3, %ymm1, %ymm1+        vpabsd 0x240(%rdi), %ymm4+        vpcmpgtd %ymm2, %ymm4, %ymm4+        vpor %ymm4, %ymm1, %ymm1+        vpabsd 0x260(%rdi), %ymm5+        vpcmpgtd %ymm2, %ymm5, %ymm5+        vpor %ymm5, %ymm1, %ymm1+        vpabsd 0x280(%rdi), %ymm0+        vpcmpgtd %ymm2, %ymm0, %ymm0+        vpor %ymm0, %ymm1, %ymm1+        vpabsd 0x2a0(%rdi), %ymm3+        vpcmpgtd %ymm2, %ymm3, %ymm3+        vpor %ymm3, %ymm1, %ymm1+        vpabsd 0x2c0(%rdi), %ymm4+        vpcmpgtd %ymm2, %ymm4, %ymm4+        vpor %ymm4, %ymm1, %ymm1+        vpabsd 0x2e0(%rdi), %ymm5+        vpcmpgtd %ymm2, %ymm5, %ymm5+        vpor %ymm5, %ymm1, %ymm1+        vpabsd 0x300(%rdi), %ymm0+        vpcmpgtd %ymm2, %ymm0, %ymm0+        vpor %ymm0, %ymm1, %ymm1+        vpabsd 0x320(%rdi), %ymm3+        vpcmpgtd %ymm2, %ymm3, %ymm3+        vpor %ymm3, %ymm1, %ymm1+        vpabsd 0x340(%rdi), %ymm4+        vpcmpgtd %ymm2, %ymm4, %ymm4+        vpor %ymm4, %ymm1, %ymm1+        vpabsd 0x360(%rdi), %ymm5+        vpcmpgtd %ymm2, %ymm5, %ymm5+        vpor %ymm5, %ymm1, %ymm1+        vpabsd 0x380(%rdi), %ymm0+        vpcmpgtd %ymm2, %ymm0, %ymm0+        vpor %ymm0, %ymm1, %ymm1+        vpabsd 0x3a0(%rdi), %ymm3+        vpcmpgtd %ymm2, %ymm3, %ymm3+        vpor %ymm3, %ymm1, %ymm1+        vpabsd 0x3c0(%rdi), %ymm4+        vpcmpgtd %ymm2, %ymm4, %ymm4+        vpor %ymm4, %ymm1, %ymm1+        vpabsd 0x3e0(%rdi), %ymm5+        vpcmpgtd %ymm2, %ymm5, %ymm5+        vpor %ymm5, %ymm1, %ymm1+        xorl %eax, %eax+        vptest %ymm1, %ymm1+        setne %al+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(poly_chknorm_avx2_asm)+++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/mldsa_poly_decompose_32_avx2_asm.S view
@@ -0,0 +1,490 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Dilithium implementation @[REF_AVX2].+ *+ * The algorithm for Decompose(r) (more specifically the handling for the+ * wrap-around cases) is modified. See the AVX2 intrinsics version+ * (poly_decompose_32_avx2.c, predecessor of this file) for a more detailed+ * comparison.+ */+++/*yaml+  Name: poly_decompose_32_avx2_asm+  Description: x86_64 AVX2 coefficient decomposition (alpha = 2*(Q-1)/32).+    For all coefficients c of the input polynomial, compute high and low bits+    c0, c1 such that c = c1*ALPHA + c0 with -ALPHA/2 < c0 <= ALPHA/2, except if+    c1 = (Q-1)/ALPHA where we set c1 = 0 and -ALPHA/2 <= c0 = c - Q < 0.+    Assumes coefficients to be standard (unsigned canonical) representatives,+    i.e. 0 <= c < Q. For ML-DSA-65 / ML-DSA-87 (gamma2 = (Q-1)/32,+    alpha = 2*gamma2).+  Signature: void mld_poly_decompose_32_avx2_asm(int32_t *a1, int32_t *a0)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t *a1+      description: Output high-part polynomial (256 x int32_t)+    rsi:+      type: buffer+      size_bytes: 1024+      permissions: read/write+      c_parameter: int32_t *a0+      description: Input polynomial / output low-part (256 x int32_t)+*/++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) &&   \+    !defined(MLD_CONFIG_NO_SIGN_API) &&            \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) &&   \+    (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+     (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87))+++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_poly_decompose_32_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(poly_decompose_32_avx2_asm)+MLD_ASM_FN_SYMBOL(poly_decompose_32_avx2_asm)++        .cfi_startproc+        movl $0x7f, %eax+        vmovd %eax, %xmm10+        vpbroadcastd %xmm10, %ymm10+        movl $0x401, %eax            # imm = 0x401+        vmovd %eax, %xmm11+        vpbroadcastd %xmm11, %ymm11+        movl $0x200, %eax            # imm = 0x200+        vmovd %eax, %xmm12+        vpbroadcastd %xmm12, %ymm12+        movl $0x7be100, %eax         # imm = 0x7BE100+        vmovd %eax, %xmm13+        vpbroadcastd %xmm13, %ymm13+        movl $0x7fe00, %eax          # imm = 0x7FE00+        vmovd %eax, %xmm14+        vpbroadcastd %xmm14, %ymm14+        vmovdqa (%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, (%rdi)+        vmovdqa %ymm2, (%rsi)+        vmovdqa 0x20(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x20(%rdi)+        vmovdqa %ymm2, 0x20(%rsi)+        vmovdqa 0x40(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x40(%rdi)+        vmovdqa %ymm2, 0x40(%rsi)+        vmovdqa 0x60(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x60(%rdi)+        vmovdqa %ymm2, 0x60(%rsi)+        vmovdqa 0x80(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x80(%rdi)+        vmovdqa %ymm2, 0x80(%rsi)+        vmovdqa 0xa0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0xa0(%rdi)+        vmovdqa %ymm2, 0xa0(%rsi)+        vmovdqa 0xc0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0xc0(%rdi)+        vmovdqa %ymm2, 0xc0(%rsi)+        vmovdqa 0xe0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0xe0(%rdi)+        vmovdqa %ymm2, 0xe0(%rsi)+        vmovdqa 0x100(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x100(%rdi)+        vmovdqa %ymm2, 0x100(%rsi)+        vmovdqa 0x120(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x120(%rdi)+        vmovdqa %ymm2, 0x120(%rsi)+        vmovdqa 0x140(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x140(%rdi)+        vmovdqa %ymm2, 0x140(%rsi)+        vmovdqa 0x160(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x160(%rdi)+        vmovdqa %ymm2, 0x160(%rsi)+        vmovdqa 0x180(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x180(%rdi)+        vmovdqa %ymm2, 0x180(%rsi)+        vmovdqa 0x1a0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x1a0(%rdi)+        vmovdqa %ymm2, 0x1a0(%rsi)+        vmovdqa 0x1c0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x1c0(%rdi)+        vmovdqa %ymm2, 0x1c0(%rsi)+        vmovdqa 0x1e0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x1e0(%rdi)+        vmovdqa %ymm2, 0x1e0(%rsi)+        vmovdqa 0x200(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x200(%rdi)+        vmovdqa %ymm2, 0x200(%rsi)+        vmovdqa 0x220(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x220(%rdi)+        vmovdqa %ymm2, 0x220(%rsi)+        vmovdqa 0x240(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x240(%rdi)+        vmovdqa %ymm2, 0x240(%rsi)+        vmovdqa 0x260(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x260(%rdi)+        vmovdqa %ymm2, 0x260(%rsi)+        vmovdqa 0x280(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x280(%rdi)+        vmovdqa %ymm2, 0x280(%rsi)+        vmovdqa 0x2a0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x2a0(%rdi)+        vmovdqa %ymm2, 0x2a0(%rsi)+        vmovdqa 0x2c0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x2c0(%rdi)+        vmovdqa %ymm2, 0x2c0(%rsi)+        vmovdqa 0x2e0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x2e0(%rdi)+        vmovdqa %ymm2, 0x2e0(%rsi)+        vmovdqa 0x300(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x300(%rdi)+        vmovdqa %ymm2, 0x300(%rsi)+        vmovdqa 0x320(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x320(%rdi)+        vmovdqa %ymm2, 0x320(%rsi)+        vmovdqa 0x340(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x340(%rdi)+        vmovdqa %ymm2, 0x340(%rsi)+        vmovdqa 0x360(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x360(%rdi)+        vmovdqa %ymm2, 0x360(%rsi)+        vmovdqa 0x380(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x380(%rdi)+        vmovdqa %ymm2, 0x380(%rsi)+        vmovdqa 0x3a0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x3a0(%rdi)+        vmovdqa %ymm2, 0x3a0(%rsi)+        vmovdqa 0x3c0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x3c0(%rdi)+        vmovdqa %ymm2, 0x3c0(%rsi)+        vmovdqa 0x3e0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x3e0(%rdi)+        vmovdqa %ymm2, 0x3e0(%rsi)+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(poly_decompose_32_avx2_asm)+++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_NO_SIGN_API && \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/mldsa_poly_decompose_88_avx2_asm.S view
@@ -0,0 +1,489 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Dilithium implementation @[REF_AVX2].+ *+ * The algorithm for Decompose(r) (more specifically the handling for the+ * wrap-around cases) is modified. See the AVX2 intrinsics version+ * (poly_decompose_88_avx2.c, predecessor of this file) for a more detailed+ * comparison.+ */+++/*yaml+  Name: poly_decompose_88_avx2_asm+  Description: x86_64 AVX2 coefficient decomposition (alpha = 2*(Q-1)/88).+    For all coefficients c of the input polynomial, compute high and low bits+    c0, c1 such that c = c1*ALPHA + c0 with -ALPHA/2 < c0 <= ALPHA/2, except if+    c1 = (Q-1)/ALPHA where we set c1 = 0 and -ALPHA/2 <= c0 = c - Q < 0.+    Assumes coefficients to be standard (unsigned canonical) representatives,+    i.e. 0 <= c < Q. For ML-DSA-44 (gamma2 = (Q-1)/88, alpha = 2*gamma2).+  Signature: void mld_poly_decompose_88_avx2_asm(int32_t *a1, int32_t *a0)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t *a1+      description: Output high-part polynomial (256 x int32_t)+    rsi:+      type: buffer+      size_bytes: 1024+      permissions: read/write+      c_parameter: int32_t *a0+      description: Input polynomial / output low-part (256 x int32_t)+*/++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) &&   \+    !defined(MLD_CONFIG_NO_SIGN_API) &&            \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) &&   \+    (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+     MLD_CONFIG_PARAMETER_SET == 44)+++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_poly_decompose_88_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(poly_decompose_88_avx2_asm)+MLD_ASM_FN_SYMBOL(poly_decompose_88_avx2_asm)++        .cfi_startproc+        movl $0x7f, %eax+        vmovd %eax, %xmm10+        vpbroadcastd %xmm10, %ymm10+        movl $0x2c0b, %eax           # imm = 0x2C0B+        vmovd %eax, %xmm11+        vpbroadcastd %xmm11, %ymm11+        movl $0x80, %eax+        vmovd %eax, %xmm12+        vpbroadcastd %xmm12, %ymm12+        movl $0x7e6c00, %eax         # imm = 0x7E6C00+        vmovd %eax, %xmm13+        vpbroadcastd %xmm13, %ymm13+        movl $0x2e800, %eax          # imm = 0x2E800+        vmovd %eax, %xmm14+        vpbroadcastd %xmm14, %ymm14+        vmovdqa (%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, (%rdi)+        vmovdqa %ymm2, (%rsi)+        vmovdqa 0x20(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x20(%rdi)+        vmovdqa %ymm2, 0x20(%rsi)+        vmovdqa 0x40(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x40(%rdi)+        vmovdqa %ymm2, 0x40(%rsi)+        vmovdqa 0x60(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x60(%rdi)+        vmovdqa %ymm2, 0x60(%rsi)+        vmovdqa 0x80(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x80(%rdi)+        vmovdqa %ymm2, 0x80(%rsi)+        vmovdqa 0xa0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0xa0(%rdi)+        vmovdqa %ymm2, 0xa0(%rsi)+        vmovdqa 0xc0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0xc0(%rdi)+        vmovdqa %ymm2, 0xc0(%rsi)+        vmovdqa 0xe0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0xe0(%rdi)+        vmovdqa %ymm2, 0xe0(%rsi)+        vmovdqa 0x100(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x100(%rdi)+        vmovdqa %ymm2, 0x100(%rsi)+        vmovdqa 0x120(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x120(%rdi)+        vmovdqa %ymm2, 0x120(%rsi)+        vmovdqa 0x140(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x140(%rdi)+        vmovdqa %ymm2, 0x140(%rsi)+        vmovdqa 0x160(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x160(%rdi)+        vmovdqa %ymm2, 0x160(%rsi)+        vmovdqa 0x180(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x180(%rdi)+        vmovdqa %ymm2, 0x180(%rsi)+        vmovdqa 0x1a0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x1a0(%rdi)+        vmovdqa %ymm2, 0x1a0(%rsi)+        vmovdqa 0x1c0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x1c0(%rdi)+        vmovdqa %ymm2, 0x1c0(%rsi)+        vmovdqa 0x1e0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x1e0(%rdi)+        vmovdqa %ymm2, 0x1e0(%rsi)+        vmovdqa 0x200(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x200(%rdi)+        vmovdqa %ymm2, 0x200(%rsi)+        vmovdqa 0x220(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x220(%rdi)+        vmovdqa %ymm2, 0x220(%rsi)+        vmovdqa 0x240(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x240(%rdi)+        vmovdqa %ymm2, 0x240(%rsi)+        vmovdqa 0x260(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x260(%rdi)+        vmovdqa %ymm2, 0x260(%rsi)+        vmovdqa 0x280(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x280(%rdi)+        vmovdqa %ymm2, 0x280(%rsi)+        vmovdqa 0x2a0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x2a0(%rdi)+        vmovdqa %ymm2, 0x2a0(%rsi)+        vmovdqa 0x2c0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x2c0(%rdi)+        vmovdqa %ymm2, 0x2c0(%rsi)+        vmovdqa 0x2e0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x2e0(%rdi)+        vmovdqa %ymm2, 0x2e0(%rsi)+        vmovdqa 0x300(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x300(%rdi)+        vmovdqa %ymm2, 0x300(%rsi)+        vmovdqa 0x320(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x320(%rdi)+        vmovdqa %ymm2, 0x320(%rsi)+        vmovdqa 0x340(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x340(%rdi)+        vmovdqa %ymm2, 0x340(%rsi)+        vmovdqa 0x360(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x360(%rdi)+        vmovdqa %ymm2, 0x360(%rsi)+        vmovdqa 0x380(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x380(%rdi)+        vmovdqa %ymm2, 0x380(%rsi)+        vmovdqa 0x3a0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x3a0(%rdi)+        vmovdqa %ymm2, 0x3a0(%rsi)+        vmovdqa 0x3c0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x3c0(%rdi)+        vmovdqa %ymm2, 0x3c0(%rsi)+        vmovdqa 0x3e0(%rsi), %ymm0+        vpaddd %ymm10, %ymm0, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm11, %ymm1, %ymm1+        vpmulhrsw %ymm12, %ymm1, %ymm1+        vpcmpgtd %ymm13, %ymm0, %ymm3+        vpmulld %ymm14, %ymm1, %ymm2+        vpsubd %ymm2, %ymm0, %ymm2+        vpandn %ymm1, %ymm3, %ymm1+        vpaddd %ymm3, %ymm2, %ymm2+        vmovdqa %ymm1, 0x3e0(%rdi)+        vmovdqa %ymm2, 0x3e0(%rsi)+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(poly_decompose_88_avx2_asm)+++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_NO_SIGN_API && \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44) \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/mldsa_poly_use_hint_32_avx2_asm.S view
@@ -0,0 +1,123 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Dilithium implementation @[REF_AVX2].+ */+++/*yaml+  Name: poly_use_hint_32_avx2_asm+  Description: x86_64 AVX2 hint application (alpha = (Q-1)/32).+    Use the hint polynomial h to correct the high bits of the polynomial a,+    in place. Variant for parameter sets ML-DSA-65 and ML-DSA-87+    (GAMMA2 = (Q-1)/32).+  Signature: void mld_poly_use_hint_32_avx2_asm(int32_t *a, const int32_t *h)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: read/write+      c_parameter: int32_t *a+      description: Input/output polynomial (256 x int32_t)+    rsi:+      type: buffer+      size_bytes: 1024+      permissions: read-only+      c_parameter: const int32_t *h+      description: Hint polynomial (256 x int32_t)+*/++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) &&   \+    !defined(MLD_CONFIG_NO_VERIFY_API) &&          \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) &&   \+    (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87))+++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_poly_use_hint_32_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(poly_use_hint_32_avx2_asm)+MLD_ASM_FN_SYMBOL(poly_use_hint_32_avx2_asm)++        .cfi_startproc+        movl $0x7f, %ecx+        movl $0x401, %r8d            # imm = 0x401+        vmovd %r8d, %xmm8+        vpbroadcastd %xmm8, %ymm8+        xorl %eax, %eax+        vpxor %xmm6, %xmm6, %xmm6+        vmovd %ecx, %xmm5+        movl $0x7be100, %ecx         # imm = 0x7BE100+        movl $0x200, %r9d            # imm = 0x200+        vmovd %r9d, %xmm7+        vpbroadcastd %xmm7, %ymm7+        vmovd %ecx, %xmm4+        movl $0xf, %ecx+        vpbroadcastd %xmm5, %ymm5+        vmovd %ecx, %xmm3+        vpbroadcastd %xmm4, %ymm4+        vpbroadcastd %xmm3, %ymm3++Lmld_poly_use_hint_32_avx2_asm_loop:+        vmovdqa (%rdi), %ymm0+        vmovdqa (%rsi), %ymm2+        vpaddd %ymm0, %ymm5, %ymm1+        vpsrld $0x7, %ymm1, %ymm1+        vpmulhuw %ymm8, %ymm1, %ymm1+        vpmulhrsw %ymm7, %ymm1, %ymm1+        vpcmpgtd %ymm4, %ymm0, %ymm11+        vpandn %ymm1, %ymm11, %ymm9+        vpslld $0xa, %ymm1, %ymm10+        vpsubd %ymm1, %ymm10, %ymm1+        vpslld $0x9, %ymm1, %ymm1+        vpsubd %ymm1, %ymm0, %ymm0+        vpaddd %ymm11, %ymm0, %ymm0+        vpcmpgtd %ymm6, %ymm0, %ymm0+        vpandn %ymm2, %ymm0, %ymm0+        vpslld $0x1, %ymm0, %ymm0+        vpsubd %ymm0, %ymm2, %ymm2+        vpaddd %ymm9, %ymm2, %ymm2+        vpand %ymm3, %ymm2, %ymm2+        vmovdqa %ymm2, (%rdi)+        addq $0x20, %rdi+        addq $0x20, %rsi+        addq $0x20, %rax+        cmpq $0x400, %rax            # imm = 0x400+        jne Lmld_poly_use_hint_32_avx2_asm_loop+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(poly_use_hint_32_avx2_asm)+++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_NO_VERIFY_API && \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/mldsa_poly_use_hint_88_avx2_asm.S view
@@ -0,0 +1,125 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Dilithium implementation @[REF_AVX2].+ */+++/*yaml+  Name: poly_use_hint_88_avx2_asm+  Description: x86_64 AVX2 hint application (alpha = (Q-1)/88).+    Use the hint polynomial h to correct the high bits of the polynomial a,+    in place. Variant for parameter set ML-DSA-44 (GAMMA2 = (Q-1)/88).+  Signature: void mld_poly_use_hint_88_avx2_asm(int32_t *a, const int32_t *h)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: read/write+      c_parameter: int32_t *a+      description: Input/output polynomial (256 x int32_t)+    rsi:+      type: buffer+      size_bytes: 1024+      permissions: read-only+      c_parameter: const int32_t *h+      description: Hint polynomial (256 x int32_t)+*/++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) &&   \+    !defined(MLD_CONFIG_NO_VERIFY_API) &&          \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) &&   \+    (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+    MLD_CONFIG_PARAMETER_SET == 44)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_poly_use_hint_88_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(poly_use_hint_88_avx2_asm)+MLD_ASM_FN_SYMBOL(poly_use_hint_88_avx2_asm)++        .cfi_startproc+        movl $0x7f, %ecx+        xorl %eax, %eax+        vpxor %xmm5, %xmm5, %xmm5+        movl $0x2c0b, %r8d           # imm = 0x2C0B+        vmovd %r8d, %xmm8+        vpbroadcastd %xmm8, %ymm8+        vmovd %ecx, %xmm4+        movl $0x7e6c00, %ecx         # imm = 0x7E6C00+        movl $0x80, %r9d+        vmovd %r9d, %xmm7+        vpbroadcastd %xmm7, %ymm7+        movl $0x2b, %r10d+        vmovd %r10d, %xmm6+        vpbroadcastd %xmm6, %ymm6+        vmovd %ecx, %xmm3+        vpbroadcastd %xmm4, %ymm4+        vpbroadcastd %xmm3, %ymm3++Lmld_poly_use_hint_88_avx2_asm_loop:+        vmovdqa (%rdi), %ymm0+        vmovdqa (%rsi), %ymm1+        vpaddd %ymm0, %ymm4, %ymm10+        vpsrld $0x7, %ymm10, %ymm10+        vpmulhuw %ymm8, %ymm10, %ymm10+        vpmulhrsw %ymm7, %ymm10, %ymm10+        vpcmpgtd %ymm3, %ymm0, %ymm11+        vpandn %ymm10, %ymm11, %ymm9+        vpslld $0x1, %ymm10, %ymm12+        vpaddd %ymm10, %ymm12, %ymm12+        vpslld $0x5, %ymm12, %ymm10+        vpsubd %ymm12, %ymm10, %ymm10+        vpslld $0xb, %ymm10, %ymm10+        vpsubd %ymm10, %ymm0, %ymm0+        vpaddd %ymm11, %ymm0, %ymm0+        vpcmpgtd %ymm5, %ymm0, %ymm0+        vpandn %ymm1, %ymm0, %ymm0+        vpslld $0x1, %ymm0, %ymm0+        vpsubd %ymm0, %ymm1, %ymm0+        vpaddd %ymm9, %ymm0, %ymm0+        vpblendvb %ymm0, %ymm6, %ymm0, %ymm0+        vpcmpgtd %ymm6, %ymm0, %ymm1+        vpandn %ymm0, %ymm1, %ymm0+        vmovdqa %ymm0, (%rdi)+        addq $0x20, %rdi+        addq $0x20, %rsi+        addq $0x20, %rax+        cmpq $0x400, %rax            # imm = 0x400+        jne Lmld_poly_use_hint_88_avx2_asm_loop+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(poly_use_hint_88_avx2_asm)+++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_NO_VERIFY_API && \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44) \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/mldsa_polyz_unpack_17_avx2_asm.S view
@@ -0,0 +1,355 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Dilithium implementation @[REF_AVX2].+ */+++/*yaml+  Name: polyz_unpack_17_avx2_asm+  Description: x86_64 AVX2 unpacking of 17-bit packed coefficients.+    Unpack polynomial z with 18-bit packed coefficients (GAMMA1 = 2^17),+    mapping packed [0, 2^18-1] to signed [-(2^17-1), 2^17] via GAMMA1 - x.+  Signature: void mld_polyz_unpack_17_avx2_asm(int32_t *r, const uint8_t *a)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t *r+      description: Output polynomial (256 x int32_t)+    rsi:+      type: buffer+      size_bytes: 576+      permissions: read-only+      c_parameter: const uint8_t *a+      description: Packed input bytes+*/++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) &&   \+    (!defined(MLD_CONFIG_NO_SIGN_API) ||           \+     !defined(MLD_CONFIG_NO_VERIFY_API)) &&        \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) &&   \+    (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+     MLD_CONFIG_PARAMETER_SET == 44)+++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_polyz_unpack_17_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(polyz_unpack_17_avx2_asm)+MLD_ASM_FN_SYMBOL(polyz_unpack_17_avx2_asm)++        .cfi_startproc+        movabsq $-0xfbfcfd00fdff00, %rax # imm = 0xFF040302FF020100+        vmovq %rax, %xmm1+        movabsq $-0xf7f8f900f9fafc, %rax # imm = 0xFF080706FF060504+        vpinsrq $0x1, %rax, %xmm1, %xmm1+        movabsq $-0xe4e5e600e6e7e9, %rax # imm = 0xFF1B1A19FF191817+        vmovq %rax, %xmm5+        movabsq $-0xe0e1e200e2e3e5, %rax # imm = 0xFF1F1E1DFF1D1C1B+        vpinsrq $0x1, %rax, %xmm5, %xmm5+        vinserti128 $0x1, %xmm5, %ymm1, %ymm1+        movabsq $0x200000000, %rax      # imm = 0x200000000+        vmovq %rax, %xmm2+        movabsq $0x600000004, %rax      # imm = 0x600000004+        vpinsrq $0x1, %rax, %xmm2, %xmm2+        vinserti128 $0x1, %xmm2, %ymm2, %ymm2+        movl $0x3ffff, %eax          # imm = 0x3FFFF+        vmovd %eax, %xmm3+        vpbroadcastd %xmm3, %ymm3+        movl $0x20000, %eax          # imm = 0x20000+        vmovd %eax, %xmm4+        vpbroadcastd %xmm4, %ymm4+        vmovdqu (%rsi), %xmm0+        vmovdqu 0x2(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, (%rdi)+        vmovdqu 0x12(%rsi), %xmm0+        vmovdqu 0x14(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x20(%rdi)+        vmovdqu 0x24(%rsi), %xmm0+        vmovdqu 0x26(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x40(%rdi)+        vmovdqu 0x36(%rsi), %xmm0+        vmovdqu 0x38(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x60(%rdi)+        vmovdqu 0x48(%rsi), %xmm0+        vmovdqu 0x4a(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x80(%rdi)+        vmovdqu 0x5a(%rsi), %xmm0+        vmovdqu 0x5c(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0xa0(%rdi)+        vmovdqu 0x6c(%rsi), %xmm0+        vmovdqu 0x6e(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0xc0(%rdi)+        vmovdqu 0x7e(%rsi), %xmm0+        vmovdqu 0x80(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0xe0(%rdi)+        vmovdqu 0x90(%rsi), %xmm0+        vmovdqu 0x92(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x100(%rdi)+        vmovdqu 0xa2(%rsi), %xmm0+        vmovdqu 0xa4(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x120(%rdi)+        vmovdqu 0xb4(%rsi), %xmm0+        vmovdqu 0xb6(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x140(%rdi)+        vmovdqu 0xc6(%rsi), %xmm0+        vmovdqu 0xc8(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x160(%rdi)+        vmovdqu 0xd8(%rsi), %xmm0+        vmovdqu 0xda(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x180(%rdi)+        vmovdqu 0xea(%rsi), %xmm0+        vmovdqu 0xec(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x1a0(%rdi)+        vmovdqu 0xfc(%rsi), %xmm0+        vmovdqu 0xfe(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x1c0(%rdi)+        vmovdqu 0x10e(%rsi), %xmm0+        vmovdqu 0x110(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x1e0(%rdi)+        vmovdqu 0x120(%rsi), %xmm0+        vmovdqu 0x122(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x200(%rdi)+        vmovdqu 0x132(%rsi), %xmm0+        vmovdqu 0x134(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x220(%rdi)+        vmovdqu 0x144(%rsi), %xmm0+        vmovdqu 0x146(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x240(%rdi)+        vmovdqu 0x156(%rsi), %xmm0+        vmovdqu 0x158(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x260(%rdi)+        vmovdqu 0x168(%rsi), %xmm0+        vmovdqu 0x16a(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x280(%rdi)+        vmovdqu 0x17a(%rsi), %xmm0+        vmovdqu 0x17c(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x2a0(%rdi)+        vmovdqu 0x18c(%rsi), %xmm0+        vmovdqu 0x18e(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x2c0(%rdi)+        vmovdqu 0x19e(%rsi), %xmm0+        vmovdqu 0x1a0(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x2e0(%rdi)+        vmovdqu 0x1b0(%rsi), %xmm0+        vmovdqu 0x1b2(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x300(%rdi)+        vmovdqu 0x1c2(%rsi), %xmm0+        vmovdqu 0x1c4(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x320(%rdi)+        vmovdqu 0x1d4(%rsi), %xmm0+        vmovdqu 0x1d6(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x340(%rdi)+        vmovdqu 0x1e6(%rsi), %xmm0+        vmovdqu 0x1e8(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x360(%rdi)+        vmovdqu 0x1f8(%rsi), %xmm0+        vmovdqu 0x1fa(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x380(%rdi)+        vmovdqu 0x20a(%rsi), %xmm0+        vmovdqu 0x20c(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x3a0(%rdi)+        vmovdqu 0x21c(%rsi), %xmm0+        vmovdqu 0x21e(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x3c0(%rdi)+        vmovdqu 0x22e(%rsi), %xmm0+        vmovdqu 0x230(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x3e0(%rdi)+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(polyz_unpack_17_avx2_asm)+++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && (!MLD_CONFIG_NO_SIGN_API || \+          !MLD_CONFIG_NO_VERIFY_API) && !MLD_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44) \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/mldsa_polyz_unpack_19_avx2_asm.S view
@@ -0,0 +1,355 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Dilithium implementation @[REF_AVX2].+ */+++/*yaml+  Name: polyz_unpack_19_avx2_asm+  Description: x86_64 AVX2 unpacking of 19-bit packed coefficients.+    Unpack polynomial z with 20-bit packed coefficients (GAMMA1 = 2^19),+    mapping packed [0, 2^20-1] to signed [-(2^19-1), 2^19] via GAMMA1 - x.+  Signature: void mld_polyz_unpack_19_avx2_asm(int32_t *r, const uint8_t *a)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t *r+      description: Output polynomial (256 x int32_t)+    rsi:+      type: buffer+      size_bytes: 640+      permissions: read-only+      c_parameter: const uint8_t *a+      description: Packed input bytes+*/++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) &&   \+    (!defined(MLD_CONFIG_NO_SIGN_API) ||           \+     !defined(MLD_CONFIG_NO_VERIFY_API)) &&        \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) &&   \+    (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+     (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87))+++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_polyz_unpack_19_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(polyz_unpack_19_avx2_asm)+MLD_ASM_FN_SYMBOL(polyz_unpack_19_avx2_asm)++        .cfi_startproc+        movabsq $-0xfbfcfd00fdff00, %rax # imm = 0xFF040302FF020100+        vmovq %rax, %xmm1+        movabsq $-0xf6f7f800f8f9fb, %rax # imm = 0xFF090807FF070605+        vpinsrq $0x1, %rax, %xmm1, %xmm1+        movabsq $-0xe5e6e700e7e8ea, %rax # imm = 0xFF1A1918FF181716+        vmovq %rax, %xmm5+        movabsq $-0xe0e1e200e2e3e5, %rax # imm = 0xFF1F1E1DFF1D1C1B+        vpinsrq $0x1, %rax, %xmm5, %xmm5+        vinserti128 $0x1, %xmm5, %ymm1, %ymm1+        movabsq $0x400000000, %rax      # imm = 0x400000000+        vmovq %rax, %xmm2+        movabsq $0x400000000, %rax      # imm = 0x400000000+        vpinsrq $0x1, %rax, %xmm2, %xmm2+        vinserti128 $0x1, %xmm2, %ymm2, %ymm2+        movl $0xfffff, %eax          # imm = 0xFFFFF+        vmovd %eax, %xmm3+        vpbroadcastd %xmm3, %ymm3+        movl $0x80000, %eax          # imm = 0x80000+        vmovd %eax, %xmm4+        vpbroadcastd %xmm4, %ymm4+        vmovdqu (%rsi), %xmm0+        vmovdqu 0x4(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, (%rdi)+        vmovdqu 0x14(%rsi), %xmm0+        vmovdqu 0x18(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x20(%rdi)+        vmovdqu 0x28(%rsi), %xmm0+        vmovdqu 0x2c(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x40(%rdi)+        vmovdqu 0x3c(%rsi), %xmm0+        vmovdqu 0x40(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x60(%rdi)+        vmovdqu 0x50(%rsi), %xmm0+        vmovdqu 0x54(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x80(%rdi)+        vmovdqu 0x64(%rsi), %xmm0+        vmovdqu 0x68(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0xa0(%rdi)+        vmovdqu 0x78(%rsi), %xmm0+        vmovdqu 0x7c(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0xc0(%rdi)+        vmovdqu 0x8c(%rsi), %xmm0+        vmovdqu 0x90(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0xe0(%rdi)+        vmovdqu 0xa0(%rsi), %xmm0+        vmovdqu 0xa4(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x100(%rdi)+        vmovdqu 0xb4(%rsi), %xmm0+        vmovdqu 0xb8(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x120(%rdi)+        vmovdqu 0xc8(%rsi), %xmm0+        vmovdqu 0xcc(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x140(%rdi)+        vmovdqu 0xdc(%rsi), %xmm0+        vmovdqu 0xe0(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x160(%rdi)+        vmovdqu 0xf0(%rsi), %xmm0+        vmovdqu 0xf4(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x180(%rdi)+        vmovdqu 0x104(%rsi), %xmm0+        vmovdqu 0x108(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x1a0(%rdi)+        vmovdqu 0x118(%rsi), %xmm0+        vmovdqu 0x11c(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x1c0(%rdi)+        vmovdqu 0x12c(%rsi), %xmm0+        vmovdqu 0x130(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x1e0(%rdi)+        vmovdqu 0x140(%rsi), %xmm0+        vmovdqu 0x144(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x200(%rdi)+        vmovdqu 0x154(%rsi), %xmm0+        vmovdqu 0x158(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x220(%rdi)+        vmovdqu 0x168(%rsi), %xmm0+        vmovdqu 0x16c(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x240(%rdi)+        vmovdqu 0x17c(%rsi), %xmm0+        vmovdqu 0x180(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x260(%rdi)+        vmovdqu 0x190(%rsi), %xmm0+        vmovdqu 0x194(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x280(%rdi)+        vmovdqu 0x1a4(%rsi), %xmm0+        vmovdqu 0x1a8(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x2a0(%rdi)+        vmovdqu 0x1b8(%rsi), %xmm0+        vmovdqu 0x1bc(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x2c0(%rdi)+        vmovdqu 0x1cc(%rsi), %xmm0+        vmovdqu 0x1d0(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x2e0(%rdi)+        vmovdqu 0x1e0(%rsi), %xmm0+        vmovdqu 0x1e4(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x300(%rdi)+        vmovdqu 0x1f4(%rsi), %xmm0+        vmovdqu 0x1f8(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x320(%rdi)+        vmovdqu 0x208(%rsi), %xmm0+        vmovdqu 0x20c(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x340(%rdi)+        vmovdqu 0x21c(%rsi), %xmm0+        vmovdqu 0x220(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x360(%rdi)+        vmovdqu 0x230(%rsi), %xmm0+        vmovdqu 0x234(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x380(%rdi)+        vmovdqu 0x244(%rsi), %xmm0+        vmovdqu 0x248(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x3a0(%rdi)+        vmovdqu 0x258(%rsi), %xmm0+        vmovdqu 0x25c(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x3c0(%rdi)+        vmovdqu 0x26c(%rsi), %xmm0+        vmovdqu 0x270(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm0, %ymm0+        vpshufb %ymm1, %ymm0, %ymm0+        vpsrlvd %ymm2, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubd %ymm0, %ymm4, %ymm0+        vmovdqa %ymm0, 0x3e0(%rdi)+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(polyz_unpack_19_avx2_asm)+++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && (!MLD_CONFIG_NO_SIGN_API || \+          !MLD_CONFIG_NO_VERIFY_API) && !MLD_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/mldsa_rej_uniform_avx2_asm.S view
@@ -0,0 +1,132 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Dilithium implementation @[REF_AVX2].+ */++/*yaml+  Name: rej_uniform_avx2_asm+  Description: x86_64 AVX2 rejection sampling of uniform coefficients mod q.+    Extract 23-bit values from the input byte buffer and accept those that are+    < MLDSA_Q, writing accepted coefficients to the output buffer. Returns the+    number of valid coefficients written (at most 256).+  Signature: unsigned mld_rej_uniform_avx2_asm(int32_t *r, const uint8_t buf[840], const uint8_t table[256][8])+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t *r+      description: Output buffer (256 x int32_t)+    rsi:+      type: buffer+      size_bytes: 840+      permissions: read-only+      c_parameter: const uint8_t buf[840]+      description: Input buffer (MLD_POLY_UNIFORM_NBLOCKS * SHAKE128_RATE = 5 * 168)+    rdx:+      type: buffer+      size_bytes: 2048+      permissions: read-only+      c_parameter: const uint8_t table[256][8]+      description: Lookup table (256 x 8 x uint8_t)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_rej_uniform_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(rej_uniform_avx2_asm)+MLD_ASM_FN_SYMBOL(rej_uniform_avx2_asm)++        .cfi_startproc+        movabsq $-0xfafbfc00fdff00, %r10 # imm = 0xFF050403FF020100+        vmovq %r10, %xmm0+        movabsq $-0xf4f5f600f7f8fa, %r10 # imm = 0xFF0B0A09FF080706+        vpinsrq $0x1, %r10, %xmm0, %xmm0+        movabsq $-0xf6f7f800f9fafc, %r10 # imm = 0xFF090807FF060504+        vmovq %r10, %xmm3+        movabsq $-0xf0f1f200f3f4f6, %r10 # imm = 0xFF0F0E0DFF0C0B0A+        vpinsrq $0x1, %r10, %xmm3, %xmm3+        vinserti128 $0x1, %xmm3, %ymm0, %ymm0+        movl $0x7fffff, %r8d         # imm = 0x7FFFFF+        vmovd %r8d, %xmm1+        vpbroadcastd %xmm1, %ymm1+        movl $0x7fe001, %r8d         # imm = 0x7FE001+        vmovd %r8d, %xmm2+        vpbroadcastd %xmm2, %ymm2+        xorl %eax, %eax+        xorl %ecx, %ecx++Lmld_rej_uniform_avx2_asm_loop:+        cmpl $0xf8, %eax+        ja Lmld_rej_uniform_avx2_asm_scalar+        cmpl $0x328, %ecx            # imm = 0x328+        ja Lmld_rej_uniform_avx2_asm_scalar+        vmovdqu (%rsi,%rcx), %ymm3+        addl $0x18, %ecx+        vpermq $0x94, %ymm3, %ymm3     # ymm3 = ymm3[0,1,1,2]+        vpshufb %ymm0, %ymm3, %ymm3+        vpand %ymm1, %ymm3, %ymm3+        vpsubd %ymm2, %ymm3, %ymm4+        vmovmskps %ymm4, %r8d+        popcntl %r8d, %r9d+        vmovq (%rdx,%r8,8), %xmm4+        vpmovzxbd %xmm4, %ymm4    # ymm4 = xmm4[0],zero,zero,zero,xmm4[1],zero,zero,zero,xmm4[2],zero,zero,zero,xmm4[3],zero,zero,zero,xmm4[4],zero,zero,zero,xmm4[5],zero,zero,zero,xmm4[6],zero,zero,zero,xmm4[7],zero,zero,zero+        vpermd %ymm3, %ymm4, %ymm3+        vmovdqu %ymm3, (%rdi,%rax,4)+        addl %r9d, %eax+        jmp Lmld_rej_uniform_avx2_asm_loop++Lmld_rej_uniform_avx2_asm_scalar:+        cmpl $0x100, %eax            # imm = 0x100+        jae Lmld_rej_uniform_avx2_asm_done+        cmpl $0x345, %ecx            # imm = 0x345+        ja Lmld_rej_uniform_avx2_asm_done+        movzwl (%rsi,%rcx), %r8d+        movzbl 0x2(%rsi,%rcx), %r9d+        shll $0x10, %r9d+        orl %r9d, %r8d+        andl $0x7fffff, %r8d         # imm = 0x7FFFFF+        addl $0x3, %ecx+        cmpl $0x7fe001, %r8d         # imm = 0x7FE001+        jae Lmld_rej_uniform_avx2_asm_scalar+        movl %r8d, (%rdi,%rax,4)+        addl $0x1, %eax+        jmp Lmld_rej_uniform_avx2_asm_scalar++Lmld_rej_uniform_avx2_asm_done:+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(rej_uniform_avx2_asm)++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/mldsa_rej_uniform_eta2_avx2_asm.S view
@@ -0,0 +1,205 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Dilithium implementation @[REF_AVX2].+ */++/*yaml+  Name: rej_uniform_eta2_avx2_asm+  Description: x86_64 AVX2 rejection sampling of eta=2 secret coefficients.+    Extracts 4-bit nibbles from the input byte buffer, accepts those < 15,+    and maps an accepted nibble n to the coefficient 2 - (n mod 5) via a+    centered modulo-5 reduction, producing values in [-2, 2].+  Signature: unsigned mld_rej_uniform_eta2_avx2_asm(int32_t *r, const uint8_t *buf, const uint8_t table[256][8])+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t *r+      description: Output coefficient buffer (256 x int32_t)+    rsi:+      type: buffer+      size_bytes: 136+      permissions: read-only+      c_parameter: const uint8_t *buf+      description: Input byte buffer (MLD_AVX2_REJ_UNIFORM_ETA2_BUFLEN = 136)+    rdx:+      type: buffer+      size_bytes: 2048+      permissions: read-only+      c_parameter: const uint8_t *table+      description: Lookup table (256 x 8 uint8_t = mld_rej_uniform_table)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_rej_uniform_eta2_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(rej_uniform_eta2_avx2_asm)+MLD_ASM_FN_SYMBOL(rej_uniform_eta2_avx2_asm)++        .cfi_startproc+        movl $0xf0f0f0f, %r8d        # imm = 0xF0F0F0F+        vmovd %r8d, %xmm3+        vpbroadcastd %xmm3, %ymm3+        movl $0x2020202, %r8d        # imm = 0x2020202+        vmovd %r8d, %xmm4+        vpbroadcastd %xmm4, %ymm4+        movl $0xf0f0f0f, %r8d        # imm = 0xF0F0F0F+        vmovd %r8d, %xmm5+        vpbroadcastd %xmm5, %ymm5+        movl $0xffffe660, %r8d       # imm = 0xFFFFE660+        vpinsrw $0x0, %r8d, %xmm6, %xmm6+        vpbroadcastw %xmm6, %ymm6+        movl $0x5, %r8d+        vpinsrw $0x0, %r8d, %xmm7, %xmm7+        vpbroadcastw %xmm7, %ymm7+        xorl %eax, %eax+        xorl %ecx, %ecx++Lmld_rej_uniform_eta2_avx2_asm_loop:+        cmpl $0xf8, %eax+        ja Lmld_rej_uniform_eta2_avx2_asm_scalar+        cmpl $0x78, %ecx+        ja Lmld_rej_uniform_eta2_avx2_asm_scalar+        vpmovzxbw (%rsi,%rcx), %ymm0+        vpsllw $0x4, %ymm0, %ymm1+        vpor %ymm1, %ymm0, %ymm0+        vpand %ymm3, %ymm0, %ymm0+        vpsubb %ymm5, %ymm0, %ymm1+        vpsubb %ymm0, %ymm4, %ymm0+        vpmovmskb %ymm1, %r8d+        vextracti128 $0x0, %ymm0, %xmm8+        movzbl %r8b, %r10d+        vmovq (%rdx,%r10,8), %xmm9+        vpshufb %xmm9, %xmm8, %xmm9+        vpmovsxbd %xmm9, %ymm1+        vpmulhrsw %ymm6, %ymm1, %ymm2+        vpmullw %ymm7, %ymm2, %ymm2+        vpaddd %ymm2, %ymm1, %ymm1+        vmovdqu %ymm1, (%rdi,%rax,4)+        popcntl %r10d, %r9d+        addl %r9d, %eax+        shrl $0x8, %r8d+        addl $0x4, %ecx+        cmpl $0xf8, %eax+        ja Lmld_rej_uniform_eta2_avx2_asm_scalar+        vpsrldq $0x8, %xmm8, %xmm8      # xmm8 = xmm8[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero+        movzbl %r8b, %r10d+        vmovq (%rdx,%r10,8), %xmm9+        vpshufb %xmm9, %xmm8, %xmm9+        vpmovsxbd %xmm9, %ymm1+        vpmulhrsw %ymm6, %ymm1, %ymm2+        vpmullw %ymm7, %ymm2, %ymm2+        vpaddd %ymm2, %ymm1, %ymm1+        vmovdqu %ymm1, (%rdi,%rax,4)+        popcntl %r10d, %r9d+        addl %r9d, %eax+        shrl $0x8, %r8d+        addl $0x4, %ecx+        cmpl $0xf8, %eax+        ja Lmld_rej_uniform_eta2_avx2_asm_scalar+        vextracti128 $0x1, %ymm0, %xmm8+        movzbl %r8b, %r10d+        vmovq (%rdx,%r10,8), %xmm9+        vpshufb %xmm9, %xmm8, %xmm9+        vpmovsxbd %xmm9, %ymm1+        vpmulhrsw %ymm6, %ymm1, %ymm2+        vpmullw %ymm7, %ymm2, %ymm2+        vpaddd %ymm2, %ymm1, %ymm1+        vmovdqu %ymm1, (%rdi,%rax,4)+        popcntl %r10d, %r9d+        addl %r9d, %eax+        shrl $0x8, %r8d+        addl $0x4, %ecx+        cmpl $0xf8, %eax+        ja Lmld_rej_uniform_eta2_avx2_asm_scalar+        vpsrldq $0x8, %xmm8, %xmm8      # xmm8 = xmm8[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero+        movzbl %r8b, %r10d+        vmovq (%rdx,%r10,8), %xmm9+        vpshufb %xmm9, %xmm8, %xmm9+        vpmovsxbd %xmm9, %ymm1+        vpmulhrsw %ymm6, %ymm1, %ymm2+        vpmullw %ymm7, %ymm2, %ymm2+        vpaddd %ymm2, %ymm1, %ymm1+        vmovdqu %ymm1, (%rdi,%rax,4)+        popcntl %r10d, %r9d+        addl %r9d, %eax+        addl $0x4, %ecx+        jmp Lmld_rej_uniform_eta2_avx2_asm_loop++Lmld_rej_uniform_eta2_avx2_asm_scalar:+        cmpl $0x100, %eax            # imm = 0x100+        jae Lmld_rej_uniform_eta2_avx2_asm_done+        cmpl $0x88, %ecx+        jae Lmld_rej_uniform_eta2_avx2_asm_done+        movzbl (%rsi,%rcx), %r11d+        incl %ecx+        movl %r11d, %r10d+        andl $0xf, %r10d+        cmpl $0xf, %r10d+        jae Lmld_rej_uniform_eta2_avx2_asm_high_nibble+        movl %r10d, %r11d+        imull $0xcd, %r11d, %r11d+        shrl $0xa, %r11d+        imull $0x5, %r11d, %r11d+        subl %r11d, %r10d+        movl $0x2, %r11d+        subl %r10d, %r11d+        movl %r11d, (%rdi,%rax,4)+        incl %eax+        cmpl $0x100, %eax            # imm = 0x100+        jae Lmld_rej_uniform_eta2_avx2_asm_done++Lmld_rej_uniform_eta2_avx2_asm_high_nibble:+        movzbl -0x1(%rsi,%rcx), %r11d+        shrl $0x4, %r11d+        andl $0xf, %r11d+        cmpl $0xf, %r11d+        jae Lmld_rej_uniform_eta2_avx2_asm_scalar+        movl %r11d, %r10d+        imull $0xcd, %r10d, %r10d+        shrl $0xa, %r10d+        imull $0x5, %r10d, %r10d+        subl %r10d, %r11d+        movl $0x2, %r10d+        subl %r11d, %r10d+        movl %r10d, (%rdi,%rax,4)+        incl %eax+        jmp Lmld_rej_uniform_eta2_avx2_asm_scalar++Lmld_rej_uniform_eta2_avx2_asm_done:+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(rej_uniform_eta2_avx2_asm)++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/mldsa_rej_uniform_eta4_avx2_asm.S view
@@ -0,0 +1,176 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Dilithium optimized AVX2 implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Dilithium implementation @[REF_AVX2].+ */++/*yaml+  Name: rej_uniform_eta4_avx2_asm+  Description: x86_64 AVX2 rejection sampling of eta=4 secret coefficients.+    Extracts 4-bit nibbles from the input byte buffer, accepts those < 9,+    and maps an accepted nibble n to the coefficient 4 - n, producing+    values in [-4, 4].+  Signature: unsigned mld_rej_uniform_eta4_avx2_asm(int32_t *r, const uint8_t *buf, const uint8_t table[256][8])+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 1024+      permissions: write-only+      c_parameter: int32_t *r+      description: Output coefficient buffer (256 x int32_t)+    rsi:+      type: buffer+      size_bytes: 272+      permissions: read-only+      c_parameter: const uint8_t *buf+      description: Input byte buffer (MLD_AVX2_REJ_UNIFORM_ETA4_BUFLEN = 272)+    rdx:+      type: buffer+      size_bytes: 2048+      permissions: read-only+      c_parameter: const uint8_t *table+      description: Lookup table (256 x 8 uint8_t = mld_rej_uniform_table)+*/++#include "../../../common.h"+#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mldsa-native source file+ *   dev/x86_64/src/mldsa_rej_uniform_eta4_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLD_ASM_NAMESPACE(rej_uniform_eta4_avx2_asm)+MLD_ASM_FN_SYMBOL(rej_uniform_eta4_avx2_asm)++        .cfi_startproc+        movl $0xf0f0f0f, %r8d        # imm = 0xF0F0F0F+        vmovd %r8d, %xmm2+        vpbroadcastd %xmm2, %ymm2+        movl $0x4040404, %r8d        # imm = 0x4040404+        vmovd %r8d, %xmm3+        vpbroadcastd %xmm3, %ymm3+        movl $0x9090909, %r8d        # imm = 0x9090909+        vmovd %r8d, %xmm4+        vpbroadcastd %xmm4, %ymm4+        xorl %eax, %eax+        xorl %ecx, %ecx++Lmld_rej_uniform_eta4_avx2_asm_loop:+        cmpl $0xf8, %eax+        ja Lmld_rej_uniform_eta4_avx2_asm_scalar+        cmpl $0x100, %ecx            # imm = 0x100+        ja Lmld_rej_uniform_eta4_avx2_asm_scalar+        vpmovzxbw (%rsi,%rcx), %ymm0+        vpsllw $0x4, %ymm0, %ymm1+        vpor %ymm1, %ymm0, %ymm0+        vpand %ymm2, %ymm0, %ymm0+        vpsubb %ymm4, %ymm0, %ymm1+        vpsubb %ymm0, %ymm3, %ymm0+        vpmovmskb %ymm1, %r8d+        vextracti128 $0x0, %ymm0, %xmm5+        movzbl %r8b, %r10d+        vmovq (%rdx,%r10,8), %xmm6+        vpshufb %xmm6, %xmm5, %xmm6+        vpmovsxbd %xmm6, %ymm1+        vmovdqu %ymm1, (%rdi,%rax,4)+        popcntl %r10d, %r9d+        addl %r9d, %eax+        shrl $0x8, %r8d+        addl $0x4, %ecx+        cmpl $0xf8, %eax+        ja Lmld_rej_uniform_eta4_avx2_asm_scalar+        vpsrldq $0x8, %xmm5, %xmm5      # xmm5 = xmm5[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero+        movzbl %r8b, %r10d+        vmovq (%rdx,%r10,8), %xmm6+        vpshufb %xmm6, %xmm5, %xmm6+        vpmovsxbd %xmm6, %ymm1+        vmovdqu %ymm1, (%rdi,%rax,4)+        popcntl %r10d, %r9d+        addl %r9d, %eax+        shrl $0x8, %r8d+        addl $0x4, %ecx+        cmpl $0xf8, %eax+        ja Lmld_rej_uniform_eta4_avx2_asm_scalar+        vextracti128 $0x1, %ymm0, %xmm5+        movzbl %r8b, %r10d+        vmovq (%rdx,%r10,8), %xmm6+        vpshufb %xmm6, %xmm5, %xmm6+        vpmovsxbd %xmm6, %ymm1+        vmovdqu %ymm1, (%rdi,%rax,4)+        popcntl %r10d, %r9d+        addl %r9d, %eax+        shrl $0x8, %r8d+        addl $0x4, %ecx+        cmpl $0xf8, %eax+        ja Lmld_rej_uniform_eta4_avx2_asm_scalar+        vpsrldq $0x8, %xmm5, %xmm5      # xmm5 = xmm5[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero+        movzbl %r8b, %r10d+        vmovq (%rdx,%r10,8), %xmm6+        vpshufb %xmm6, %xmm5, %xmm6+        vpmovsxbd %xmm6, %ymm1+        vmovdqu %ymm1, (%rdi,%rax,4)+        popcntl %r10d, %r9d+        addl %r9d, %eax+        addl $0x4, %ecx+        jmp Lmld_rej_uniform_eta4_avx2_asm_loop++Lmld_rej_uniform_eta4_avx2_asm_scalar:+        cmpl $0x100, %eax            # imm = 0x100+        jae Lmld_rej_uniform_eta4_avx2_asm_done+        cmpl $0x110, %ecx            # imm = 0x110+        jae Lmld_rej_uniform_eta4_avx2_asm_done+        movzbl (%rsi,%rcx), %r11d+        incl %ecx+        movl %r11d, %r10d+        andl $0xf, %r10d+        cmpl $0x9, %r10d+        jae Lmld_rej_uniform_eta4_avx2_asm_high_nibble+        movl $0x4, %r9d+        subl %r10d, %r9d+        movl %r9d, (%rdi,%rax,4)+        incl %eax+        cmpl $0x100, %eax            # imm = 0x100+        jae Lmld_rej_uniform_eta4_avx2_asm_done++Lmld_rej_uniform_eta4_avx2_asm_high_nibble:+        shrl $0x4, %r11d+        andl $0xf, %r11d+        cmpl $0x9, %r11d+        jae Lmld_rej_uniform_eta4_avx2_asm_scalar+        movl $0x4, %r10d+        subl %r11d, %r10d+        movl %r10d, (%rdi,%rax,4)+        incl %eax+        jmp Lmld_rej_uniform_eta4_avx2_asm_scalar++Lmld_rej_uniform_eta4_avx2_asm_done:+        retq+        .cfi_endproc++MLD_ASM_FN_SIZE(rej_uniform_eta4_avx2_asm)++#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mldsa/src/native/x86_64/src/rej_uniform_table.c view
@@ -0,0 +1,161 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mldsa-native repository.+ *          Do not modify it directly.+ */++#include "../../../common.h"++#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)++#include "arith_native_x86_64.h"++/*+ * Lookup table used by rejection sampling.+ * See autogen for details.+ */+MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint8_t+    mld_rej_uniform_table[256][8] = {+        {0, 0, 0, 0, 0, 0, 0, 0}, {0, 0, 0, 0, 0, 0, 0, 0},+        {1, 0, 0, 0, 0, 0, 0, 0}, {0, 1, 0, 0, 0, 0, 0, 0},+        {2, 0, 0, 0, 0, 0, 0, 0}, {0, 2, 0, 0, 0, 0, 0, 0},+        {1, 2, 0, 0, 0, 0, 0, 0}, {0, 1, 2, 0, 0, 0, 0, 0},+        {3, 0, 0, 0, 0, 0, 0, 0}, {0, 3, 0, 0, 0, 0, 0, 0},+        {1, 3, 0, 0, 0, 0, 0, 0}, {0, 1, 3, 0, 0, 0, 0, 0},+        {2, 3, 0, 0, 0, 0, 0, 0}, {0, 2, 3, 0, 0, 0, 0, 0},+        {1, 2, 3, 0, 0, 0, 0, 0}, {0, 1, 2, 3, 0, 0, 0, 0},+        {4, 0, 0, 0, 0, 0, 0, 0}, {0, 4, 0, 0, 0, 0, 0, 0},+        {1, 4, 0, 0, 0, 0, 0, 0}, {0, 1, 4, 0, 0, 0, 0, 0},+        {2, 4, 0, 0, 0, 0, 0, 0}, {0, 2, 4, 0, 0, 0, 0, 0},+        {1, 2, 4, 0, 0, 0, 0, 0}, {0, 1, 2, 4, 0, 0, 0, 0},+        {3, 4, 0, 0, 0, 0, 0, 0}, {0, 3, 4, 0, 0, 0, 0, 0},+        {1, 3, 4, 0, 0, 0, 0, 0}, {0, 1, 3, 4, 0, 0, 0, 0},+        {2, 3, 4, 0, 0, 0, 0, 0}, {0, 2, 3, 4, 0, 0, 0, 0},+        {1, 2, 3, 4, 0, 0, 0, 0}, {0, 1, 2, 3, 4, 0, 0, 0},+        {5, 0, 0, 0, 0, 0, 0, 0}, {0, 5, 0, 0, 0, 0, 0, 0},+        {1, 5, 0, 0, 0, 0, 0, 0}, {0, 1, 5, 0, 0, 0, 0, 0},+        {2, 5, 0, 0, 0, 0, 0, 0}, {0, 2, 5, 0, 0, 0, 0, 0},+        {1, 2, 5, 0, 0, 0, 0, 0}, {0, 1, 2, 5, 0, 0, 0, 0},+        {3, 5, 0, 0, 0, 0, 0, 0}, {0, 3, 5, 0, 0, 0, 0, 0},+        {1, 3, 5, 0, 0, 0, 0, 0}, {0, 1, 3, 5, 0, 0, 0, 0},+        {2, 3, 5, 0, 0, 0, 0, 0}, {0, 2, 3, 5, 0, 0, 0, 0},+        {1, 2, 3, 5, 0, 0, 0, 0}, {0, 1, 2, 3, 5, 0, 0, 0},+        {4, 5, 0, 0, 0, 0, 0, 0}, {0, 4, 5, 0, 0, 0, 0, 0},+        {1, 4, 5, 0, 0, 0, 0, 0}, {0, 1, 4, 5, 0, 0, 0, 0},+        {2, 4, 5, 0, 0, 0, 0, 0}, {0, 2, 4, 5, 0, 0, 0, 0},+        {1, 2, 4, 5, 0, 0, 0, 0}, {0, 1, 2, 4, 5, 0, 0, 0},+        {3, 4, 5, 0, 0, 0, 0, 0}, {0, 3, 4, 5, 0, 0, 0, 0},+        {1, 3, 4, 5, 0, 0, 0, 0}, {0, 1, 3, 4, 5, 0, 0, 0},+        {2, 3, 4, 5, 0, 0, 0, 0}, {0, 2, 3, 4, 5, 0, 0, 0},+        {1, 2, 3, 4, 5, 0, 0, 0}, {0, 1, 2, 3, 4, 5, 0, 0},+        {6, 0, 0, 0, 0, 0, 0, 0}, {0, 6, 0, 0, 0, 0, 0, 0},+        {1, 6, 0, 0, 0, 0, 0, 0}, {0, 1, 6, 0, 0, 0, 0, 0},+        {2, 6, 0, 0, 0, 0, 0, 0}, {0, 2, 6, 0, 0, 0, 0, 0},+        {1, 2, 6, 0, 0, 0, 0, 0}, {0, 1, 2, 6, 0, 0, 0, 0},+        {3, 6, 0, 0, 0, 0, 0, 0}, {0, 3, 6, 0, 0, 0, 0, 0},+        {1, 3, 6, 0, 0, 0, 0, 0}, {0, 1, 3, 6, 0, 0, 0, 0},+        {2, 3, 6, 0, 0, 0, 0, 0}, {0, 2, 3, 6, 0, 0, 0, 0},+        {1, 2, 3, 6, 0, 0, 0, 0}, {0, 1, 2, 3, 6, 0, 0, 0},+        {4, 6, 0, 0, 0, 0, 0, 0}, {0, 4, 6, 0, 0, 0, 0, 0},+        {1, 4, 6, 0, 0, 0, 0, 0}, {0, 1, 4, 6, 0, 0, 0, 0},+        {2, 4, 6, 0, 0, 0, 0, 0}, {0, 2, 4, 6, 0, 0, 0, 0},+        {1, 2, 4, 6, 0, 0, 0, 0}, {0, 1, 2, 4, 6, 0, 0, 0},+        {3, 4, 6, 0, 0, 0, 0, 0}, {0, 3, 4, 6, 0, 0, 0, 0},+        {1, 3, 4, 6, 0, 0, 0, 0}, {0, 1, 3, 4, 6, 0, 0, 0},+        {2, 3, 4, 6, 0, 0, 0, 0}, {0, 2, 3, 4, 6, 0, 0, 0},+        {1, 2, 3, 4, 6, 0, 0, 0}, {0, 1, 2, 3, 4, 6, 0, 0},+        {5, 6, 0, 0, 0, 0, 0, 0}, {0, 5, 6, 0, 0, 0, 0, 0},+        {1, 5, 6, 0, 0, 0, 0, 0}, {0, 1, 5, 6, 0, 0, 0, 0},+        {2, 5, 6, 0, 0, 0, 0, 0}, {0, 2, 5, 6, 0, 0, 0, 0},+        {1, 2, 5, 6, 0, 0, 0, 0}, {0, 1, 2, 5, 6, 0, 0, 0},+        {3, 5, 6, 0, 0, 0, 0, 0}, {0, 3, 5, 6, 0, 0, 0, 0},+        {1, 3, 5, 6, 0, 0, 0, 0}, {0, 1, 3, 5, 6, 0, 0, 0},+        {2, 3, 5, 6, 0, 0, 0, 0}, {0, 2, 3, 5, 6, 0, 0, 0},+        {1, 2, 3, 5, 6, 0, 0, 0}, {0, 1, 2, 3, 5, 6, 0, 0},+        {4, 5, 6, 0, 0, 0, 0, 0}, {0, 4, 5, 6, 0, 0, 0, 0},+        {1, 4, 5, 6, 0, 0, 0, 0}, {0, 1, 4, 5, 6, 0, 0, 0},+        {2, 4, 5, 6, 0, 0, 0, 0}, {0, 2, 4, 5, 6, 0, 0, 0},+        {1, 2, 4, 5, 6, 0, 0, 0}, {0, 1, 2, 4, 5, 6, 0, 0},+        {3, 4, 5, 6, 0, 0, 0, 0}, {0, 3, 4, 5, 6, 0, 0, 0},+        {1, 3, 4, 5, 6, 0, 0, 0}, {0, 1, 3, 4, 5, 6, 0, 0},+        {2, 3, 4, 5, 6, 0, 0, 0}, {0, 2, 3, 4, 5, 6, 0, 0},+        {1, 2, 3, 4, 5, 6, 0, 0}, {0, 1, 2, 3, 4, 5, 6, 0},+        {7, 0, 0, 0, 0, 0, 0, 0}, {0, 7, 0, 0, 0, 0, 0, 0},+        {1, 7, 0, 0, 0, 0, 0, 0}, {0, 1, 7, 0, 0, 0, 0, 0},+        {2, 7, 0, 0, 0, 0, 0, 0}, {0, 2, 7, 0, 0, 0, 0, 0},+        {1, 2, 7, 0, 0, 0, 0, 0}, {0, 1, 2, 7, 0, 0, 0, 0},+        {3, 7, 0, 0, 0, 0, 0, 0}, {0, 3, 7, 0, 0, 0, 0, 0},+        {1, 3, 7, 0, 0, 0, 0, 0}, {0, 1, 3, 7, 0, 0, 0, 0},+        {2, 3, 7, 0, 0, 0, 0, 0}, {0, 2, 3, 7, 0, 0, 0, 0},+        {1, 2, 3, 7, 0, 0, 0, 0}, {0, 1, 2, 3, 7, 0, 0, 0},+        {4, 7, 0, 0, 0, 0, 0, 0}, {0, 4, 7, 0, 0, 0, 0, 0},+        {1, 4, 7, 0, 0, 0, 0, 0}, {0, 1, 4, 7, 0, 0, 0, 0},+        {2, 4, 7, 0, 0, 0, 0, 0}, {0, 2, 4, 7, 0, 0, 0, 0},+        {1, 2, 4, 7, 0, 0, 0, 0}, {0, 1, 2, 4, 7, 0, 0, 0},+        {3, 4, 7, 0, 0, 0, 0, 0}, {0, 3, 4, 7, 0, 0, 0, 0},+        {1, 3, 4, 7, 0, 0, 0, 0}, {0, 1, 3, 4, 7, 0, 0, 0},+        {2, 3, 4, 7, 0, 0, 0, 0}, {0, 2, 3, 4, 7, 0, 0, 0},+        {1, 2, 3, 4, 7, 0, 0, 0}, {0, 1, 2, 3, 4, 7, 0, 0},+        {5, 7, 0, 0, 0, 0, 0, 0}, {0, 5, 7, 0, 0, 0, 0, 0},+        {1, 5, 7, 0, 0, 0, 0, 0}, {0, 1, 5, 7, 0, 0, 0, 0},+        {2, 5, 7, 0, 0, 0, 0, 0}, {0, 2, 5, 7, 0, 0, 0, 0},+        {1, 2, 5, 7, 0, 0, 0, 0}, {0, 1, 2, 5, 7, 0, 0, 0},+        {3, 5, 7, 0, 0, 0, 0, 0}, {0, 3, 5, 7, 0, 0, 0, 0},+        {1, 3, 5, 7, 0, 0, 0, 0}, {0, 1, 3, 5, 7, 0, 0, 0},+        {2, 3, 5, 7, 0, 0, 0, 0}, {0, 2, 3, 5, 7, 0, 0, 0},+        {1, 2, 3, 5, 7, 0, 0, 0}, {0, 1, 2, 3, 5, 7, 0, 0},+        {4, 5, 7, 0, 0, 0, 0, 0}, {0, 4, 5, 7, 0, 0, 0, 0},+        {1, 4, 5, 7, 0, 0, 0, 0}, {0, 1, 4, 5, 7, 0, 0, 0},+        {2, 4, 5, 7, 0, 0, 0, 0}, {0, 2, 4, 5, 7, 0, 0, 0},+        {1, 2, 4, 5, 7, 0, 0, 0}, {0, 1, 2, 4, 5, 7, 0, 0},+        {3, 4, 5, 7, 0, 0, 0, 0}, {0, 3, 4, 5, 7, 0, 0, 0},+        {1, 3, 4, 5, 7, 0, 0, 0}, {0, 1, 3, 4, 5, 7, 0, 0},+        {2, 3, 4, 5, 7, 0, 0, 0}, {0, 2, 3, 4, 5, 7, 0, 0},+        {1, 2, 3, 4, 5, 7, 0, 0}, {0, 1, 2, 3, 4, 5, 7, 0},+        {6, 7, 0, 0, 0, 0, 0, 0}, {0, 6, 7, 0, 0, 0, 0, 0},+        {1, 6, 7, 0, 0, 0, 0, 0}, {0, 1, 6, 7, 0, 0, 0, 0},+        {2, 6, 7, 0, 0, 0, 0, 0}, {0, 2, 6, 7, 0, 0, 0, 0},+        {1, 2, 6, 7, 0, 0, 0, 0}, {0, 1, 2, 6, 7, 0, 0, 0},+        {3, 6, 7, 0, 0, 0, 0, 0}, {0, 3, 6, 7, 0, 0, 0, 0},+        {1, 3, 6, 7, 0, 0, 0, 0}, {0, 1, 3, 6, 7, 0, 0, 0},+        {2, 3, 6, 7, 0, 0, 0, 0}, {0, 2, 3, 6, 7, 0, 0, 0},+        {1, 2, 3, 6, 7, 0, 0, 0}, {0, 1, 2, 3, 6, 7, 0, 0},+        {4, 6, 7, 0, 0, 0, 0, 0}, {0, 4, 6, 7, 0, 0, 0, 0},+        {1, 4, 6, 7, 0, 0, 0, 0}, {0, 1, 4, 6, 7, 0, 0, 0},+        {2, 4, 6, 7, 0, 0, 0, 0}, {0, 2, 4, 6, 7, 0, 0, 0},+        {1, 2, 4, 6, 7, 0, 0, 0}, {0, 1, 2, 4, 6, 7, 0, 0},+        {3, 4, 6, 7, 0, 0, 0, 0}, {0, 3, 4, 6, 7, 0, 0, 0},+        {1, 3, 4, 6, 7, 0, 0, 0}, {0, 1, 3, 4, 6, 7, 0, 0},+        {2, 3, 4, 6, 7, 0, 0, 0}, {0, 2, 3, 4, 6, 7, 0, 0},+        {1, 2, 3, 4, 6, 7, 0, 0}, {0, 1, 2, 3, 4, 6, 7, 0},+        {5, 6, 7, 0, 0, 0, 0, 0}, {0, 5, 6, 7, 0, 0, 0, 0},+        {1, 5, 6, 7, 0, 0, 0, 0}, {0, 1, 5, 6, 7, 0, 0, 0},+        {2, 5, 6, 7, 0, 0, 0, 0}, {0, 2, 5, 6, 7, 0, 0, 0},+        {1, 2, 5, 6, 7, 0, 0, 0}, {0, 1, 2, 5, 6, 7, 0, 0},+        {3, 5, 6, 7, 0, 0, 0, 0}, {0, 3, 5, 6, 7, 0, 0, 0},+        {1, 3, 5, 6, 7, 0, 0, 0}, {0, 1, 3, 5, 6, 7, 0, 0},+        {2, 3, 5, 6, 7, 0, 0, 0}, {0, 2, 3, 5, 6, 7, 0, 0},+        {1, 2, 3, 5, 6, 7, 0, 0}, {0, 1, 2, 3, 5, 6, 7, 0},+        {4, 5, 6, 7, 0, 0, 0, 0}, {0, 4, 5, 6, 7, 0, 0, 0},+        {1, 4, 5, 6, 7, 0, 0, 0}, {0, 1, 4, 5, 6, 7, 0, 0},+        {2, 4, 5, 6, 7, 0, 0, 0}, {0, 2, 4, 5, 6, 7, 0, 0},+        {1, 2, 4, 5, 6, 7, 0, 0}, {0, 1, 2, 4, 5, 6, 7, 0},+        {3, 4, 5, 6, 7, 0, 0, 0}, {0, 3, 4, 5, 6, 7, 0, 0},+        {1, 3, 4, 5, 6, 7, 0, 0}, {0, 1, 3, 4, 5, 6, 7, 0},+        {2, 3, 4, 5, 6, 7, 0, 0}, {0, 2, 3, 4, 5, 6, 7, 0},+        {1, 2, 3, 4, 5, 6, 7, 0}, {0, 1, 2, 3, 4, 5, 6, 7},+};++#else /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \+       */++MLD_EMPTY_CU(avx2_rej_uniform_table)++#endif /* !(MLD_ARITH_BACKEND_X86_64_DEFAULT && \+          !MLD_CONFIG_MULTILEVEL_NO_SHARED) */
+ cbits/mldsa/src/packing.c view
@@ -0,0 +1,213 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#include <string.h>++#include "common.h"+#include "packing.h"+#include "poly.h"+#include "polyvec.h"+#include "rounding.h"++/* Parameter set namespacing+ * This is to facilitate building multiple instances+ * of mldsa-native (e.g. with varying parameter sets)+ * within a single compilation unit. */+/* End of parameter set namespacing */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+MLD_INTERNAL_API+void mld_unpack_pk_t1(mld_poly *t1,+                      const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+                      unsigned int i)+{+  mld_polyt1_unpack(t1, pk + MLDSA_PK_T1_OFFSET + i * MLDSA_POLYT1_PACKEDBYTES);+}+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+MLD_INTERNAL_API+void mld_pack_sk_s1(uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                    const mld_polyvecl *s1)+{+  mld_polyvecl_pack_eta(sk + MLDSA_SK_S1_OFFSET, s1);+}++MLD_INTERNAL_API+void mld_pack_sk_rho_key_tr_s2(uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                               const uint8_t rho[MLDSA_SEEDBYTES],+                               const uint8_t tr[MLDSA_TRBYTES],+                               const uint8_t key[MLDSA_SEEDBYTES],+                               const mld_polyveck *s2)+{+  mld_memcpy(sk + MLDSA_SK_RHO_OFFSET, rho, MLDSA_SEEDBYTES);+  mld_memcpy(sk + MLDSA_SK_KEY_OFFSET, key, MLDSA_SEEDBYTES);+  mld_memcpy(sk + MLDSA_SK_TR_OFFSET, tr, MLDSA_TRBYTES);+  /* s1 already packed via mld_pack_sk_s1 */+  mld_polyveck_pack_eta(sk + MLDSA_SK_S2_OFFSET, s2);+  /* t0 already packed via mld_compute_pack_t0_t1 */+}+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+MLD_INTERNAL_API+void mld_unpack_sk(uint8_t rho[MLDSA_SEEDBYTES], uint8_t tr[MLDSA_TRBYTES],+                   uint8_t key[MLDSA_SEEDBYTES], mld_sk_t0hat *t0,+                   mld_sk_s1hat *s1, mld_sk_s2hat *s2,+                   const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES])+{+  mld_memcpy(rho, sk + MLDSA_SK_RHO_OFFSET, MLDSA_SEEDBYTES);+  mld_memcpy(key, sk + MLDSA_SK_KEY_OFFSET, MLDSA_SEEDBYTES);+  mld_memcpy(tr, sk + MLDSA_SK_TR_OFFSET, MLDSA_TRBYTES);+  mld_unpack_sk_s1hat(s1, sk + MLDSA_SK_S1_OFFSET);+  mld_unpack_sk_s2hat(s2, sk + MLDSA_SK_S2_OFFSET);+  mld_unpack_sk_t0hat(t0, sk + MLDSA_SK_T0_OFFSET);+}++MLD_INTERNAL_API+void mld_pack_sig_c(uint8_t sig[MLDSA_CRYPTO_BYTES],+                    const uint8_t c[MLDSA_CTILDEBYTES])+{+  mld_memcpy(sig, c, MLDSA_CTILDEBYTES);+}++MLD_INTERNAL_API+int mld_pack_sig_h(uint8_t sig[MLDSA_CRYPTO_BYTES], const mld_polyveck *w0,+                   const mld_polyveck *w1)+{+  unsigned int j, k, n;++  /* The hint section of sig[] is MLDSA_POLYVECH_PACKEDBYTES long, where+   * MLDSA_POLYVECH_PACKEDBYTES = MLDSA_OMEGA + MLDSA_K.+   *+   * The first OMEGA bytes record the index numbers of the coefficients+   * that are not equal to 0.+   *+   * The final K bytes record a running tally of the number of hints+   * coming from each of the K polynomials. */+  uint8_t *sig_h = sig + MLDSA_SIG_H_OFFSET;++  mld_memset(sig_h, 0, MLDSA_POLYVECH_PACKEDBYTES);+  n = 0;++  /* For each coefficient of each polynomial, compute its hint bit and, if+   * non-zero, record the index in the hint section of sig. If recording the+   * hint would overflow the OMEGA-sized index array, abort early and return+   * MLD_ERR_FAIL. The caller is expected to reject the signature in that case.+   *+   * Constant time: At this point w0/w1 are public (see comment in sign.c+   * before the call), so a data-dependent early return is fine. */+  for (k = 0; k < MLDSA_K; k++)+  __loop__(+    assigns(k, j, n, memory_slice(sig_h, MLDSA_POLYVECH_PACKEDBYTES))+    invariant(k <= MLDSA_K && n <= MLDSA_OMEGA)+    decreases(MLDSA_K - k)+  )+  {+    for (j = 0; j < MLDSA_N; j++)+    __loop__(+      assigns(j, n, memory_slice(sig_h, MLDSA_POLYVECH_PACKEDBYTES))+      invariant(j <= MLDSA_N && n <= MLDSA_OMEGA)+      decreases(MLDSA_N - j)+    )+    {+      const unsigned int hint_bit =+          mld_make_hint(w0->vec[k].coeffs[j], w1->vec[k].coeffs[j]);+      if (hint_bit)+      {+        if (n == MLDSA_OMEGA)+        {+          return MLD_ERR_FAIL;+        }+        /* Safety: branch above ensures n < MLDSA_OMEGA so n is a valid index+         * into the OMEGA-sized index array; j < MLDSA_N <= 256 fits in+         * uint8_t. */+        sig_h[n] = (uint8_t)j;+        n++;+      }+    }+    /* Record the running tally into the correct slot for this polynomial.+     * Safety: k < MLDSA_K, so MLDSA_OMEGA + k is a valid index into the+     * K-byte tally tail; n <= MLDSA_OMEGA fits in uint8_t. */+    sig_h[MLDSA_OMEGA + k] = (uint8_t)n;+  }+  return 0;+}++MLD_INTERNAL_API+void mld_pack_sig_z(uint8_t sig[MLDSA_CRYPTO_BYTES], const mld_poly *zi,+                    unsigned i)+{+  mld_polyz_pack(sig + MLDSA_SIG_Z_OFFSET + i * MLDSA_POLYZ_PACKEDBYTES, zi);+}+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+MLD_INTERNAL_API+int mld_sig_unpack_hints(mld_poly *h, const uint8_t sig[MLDSA_CRYPTO_BYTES],+                         unsigned int i)+{+  const uint8_t *packed_hints = sig + MLDSA_SIG_H_OFFSET;+  const unsigned int old_hint_count =+      (i == 0) ? 0 : packed_hints[MLDSA_OMEGA + i - 1];+  const unsigned int new_hint_count = packed_hints[MLDSA_OMEGA + i];+  unsigned int j;++  if (new_hint_count < old_hint_count || new_hint_count > MLDSA_OMEGA)+  {+    return MLD_ERR_FAIL;+  }++  mld_memset(h, 0, sizeof(mld_poly));++  for (j = old_hint_count; j < new_hint_count; ++j)+  __loop__(+    invariant(j >= old_hint_count && j <= new_hint_count &&+              new_hint_count <= MLDSA_OMEGA)+    invariant(array_bound(h->coeffs, 0, MLDSA_N, 0, 2))+    invariant(forall(p, 0, MLDSA_N,+      (h->coeffs[p] == 1) ==+      exists(hj, old_hint_count, j, packed_hints[hj] == p)))+    decreases(new_hint_count - j)+  )+  {+    if (j > old_hint_count && packed_hints[j] <= packed_hints[j - 1])+    {+      return MLD_ERR_FAIL;+    }+    /* Safety: packed_hints[j] is uint8_t (<= 255) and MLDSA_N == 256. */+    h->coeffs[packed_hints[j]] = 1;+  }++  /* On the last row, also verify that the trailing index slots are zero. */+  if (i == MLDSA_K - 1)+  {+    for (j = new_hint_count; j < MLDSA_OMEGA; ++j)+    __loop__(+      invariant(j <= MLDSA_OMEGA)+      decreases(MLDSA_OMEGA - j)+    )+    {+      if (packed_hints[j] != 0)+      {+        return MLD_ERR_FAIL;+      }+    }+  }++  /* On success, h->coeffs[p] is 1 exactly for the hint indices decoded for this+   * row, i.e. packed_hints[old_hint_count, new_hint_count). Asserted here+   * rather than posted as a contract to not unnecessarily increase proof+   * complexity at callers that do not need the functional description. */+  cassert(forall(+      p, 0, MLDSA_N,+      (h->coeffs[p] == 1) ==+          exists(hj, old_hint_count, new_hint_count, packed_hints[hj] == p)));++  return 0;+}+#endif /* !MLD_CONFIG_NO_VERIFY_API */++/* To facilitate single-compilation-unit (SCU) builds, undefine all macros.+ * Don't modify by hand -- this is auto-generated by scripts/autogen. */
+ cbits/mldsa/src/packing.h view
@@ -0,0 +1,277 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ */++#ifndef MLD_PACKING_H+#define MLD_PACKING_H++#include "polyvec.h"+#include "polyvec_lazy.h"++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+#define mld_pack_sk_s1 MLD_NAMESPACE_KL(pack_sk_s1)+/**+ * Bit-pack the s1 component into the secret key.+ *+ * @spec{Partially implements @[FIPS204, Algorithm 24, skEncode] (s1+ * component).}+ *+ * @param[out] sk Output byte array.+ * @param[in]  s1 Pointer to vector s1.+ */+MLD_INTERNAL_API+void mld_pack_sk_s1(uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                    const mld_polyvecl *s1)+__contract__(+  requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES))+  requires(memory_no_alias(s1, sizeof(mld_polyvecl)))+  requires(forall(k1, 0, MLDSA_L,+    array_abs_bound(s1->vec[k1].coeffs, 0, MLDSA_N, MLDSA_ETA + 1)))+  assigns(memory_slice(sk, MLDSA_CRYPTO_SECRETKEYBYTES))+);++#define mld_pack_sk_rho_key_tr_s2 MLD_NAMESPACE_KL(pack_sk_rho_key_tr_s2)+/**+ * Bit-pack rho, key, tr, s2 into the secret key.+ *+ * s1 must already be packed via mld_pack_sk_s1, and t0 via+ * mld_compute_pack_t0_t1.+ *+ * @spec{Partially implements @[FIPS204, Algorithm 24, skEncode] (rho, key, tr,+ * s2 components).}+ *+ * @param[out] sk  Output byte array.+ * @param[in]  rho Byte array containing rho.+ * @param[in]  tr  Byte array containing tr.+ * @param[in]  key Byte array containing key.+ * @param[in]  s2  Pointer to vector s2.+ */+MLD_INTERNAL_API+void mld_pack_sk_rho_key_tr_s2(uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                               const uint8_t rho[MLDSA_SEEDBYTES],+                               const uint8_t tr[MLDSA_TRBYTES],+                               const uint8_t key[MLDSA_SEEDBYTES],+                               const mld_polyveck *s2)+__contract__(+  requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES))+  requires(memory_no_alias(rho, MLDSA_SEEDBYTES))+  requires(memory_no_alias(tr, MLDSA_TRBYTES))+  requires(memory_no_alias(key, MLDSA_SEEDBYTES))+  requires(memory_no_alias(s2, sizeof(mld_polyveck)))+  requires(forall(k2, 0, MLDSA_K,+    array_abs_bound(s2->vec[k2].coeffs, 0, MLDSA_N, MLDSA_ETA + 1)))+  assigns(memory_slice(sk, MLDSA_CRYPTO_SECRETKEYBYTES))+);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */+++#if !defined(MLD_CONFIG_NO_SIGN_API)+#define mld_pack_sig_c MLD_NAMESPACE_KL(pack_sig_c)+/**+ * Bit-pack challenge c into sig = (c, z, h).+ *+ * @spec{Partially implements @[FIPS204, Algorithm 26, sigEncode] (c+ * component).}+ *+ * @param[out] sig Output byte array.+ * @param[in]  c   Pointer to challenge hash.+ */+MLD_INTERNAL_API+void mld_pack_sig_c(uint8_t sig[MLDSA_CRYPTO_BYTES],+                    const uint8_t c[MLDSA_CTILDEBYTES])+__contract__(+  requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES))+  requires(memory_no_alias(c, MLDSA_CTILDEBYTES))+  assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES))+);++#define mld_pack_sig_h MLD_NAMESPACE_KL(pack_sig_h)+/**+ * Compute hints from (w0, w1) and pack them into the hint section of sig.+ *+ * @spec{Combines the hint computation @[FIPS204, Algorithm 39, MakeHint] with+ * the packing @[FIPS204, Algorithm 20, HintBitPack] (the h component of+ * @[FIPS204, Algorithm 26, sigEncode]): it computes the hint vector h from+ * (w0, w1) and packs it, rather than receiving a ready-made h as HintBitPack+ * does. The hints are computed via mld_make_hint (rounding.h), a specialized+ * MakeHint valid only for the values arising during signing; see the block+ * comment in mld_attempt_signature_generation (sign.c).}+ *+ * @param[in,out] sig Byte array containing signature.+ * @param[in]     w0  Pointer to low part of input vector.+ * @param[in]     w1  Pointer to high part of input vector.+ *+ * @retval 0            Success.+ * @retval MLD_ERR_FAIL The total number of hints exceeds MLDSA_OMEGA. In this+ *                      case the hint section of sig is left in a+ *                      partially-written state and the caller must reject the+ *                      signature.+ */+MLD_INTERNAL_API+MLD_MUST_CHECK_RETURN_VALUE+int mld_pack_sig_h(uint8_t sig[MLDSA_CRYPTO_BYTES], const mld_polyveck *w0,+                   const mld_polyveck *w1)+__contract__(+  requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES))+  requires(memory_no_alias(w0, sizeof(mld_polyveck)))+  requires(memory_no_alias(w1, sizeof(mld_polyveck)))+  assigns(memory_slice(sig + MLDSA_SIG_H_OFFSET, MLDSA_POLYVECH_PACKEDBYTES))+  ensures(return_value == 0 || return_value == MLD_ERR_FAIL)+);++#define mld_pack_sig_z MLD_NAMESPACE_KL(pack_sig_z)+/**+ * Bit-pack single polynomial of z component of sig = (c, z, h).+ *+ * The c and h components are packed separately using mld_pack_sig_c and+ * mld_pack_sig_h.+ *+ * @spec{Partially implements @[FIPS204, Algorithm 26, sigEncode] (one+ * polynomial of the z component).}+ *+ * @param[in,out] sig Output byte array.+ * @param[in]     zi  Pointer to a single polynomial in z.+ * @param         i   Index of zi in vector z.+ */+MLD_INTERNAL_API+void mld_pack_sig_z(uint8_t sig[MLDSA_CRYPTO_BYTES], const mld_poly *zi,+                    unsigned i)+__contract__(+  requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES))+  requires(memory_no_alias(zi, sizeof(mld_poly)))+  requires(i < MLDSA_L)+  requires(array_bound(zi->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))+  assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES))+);+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+#define mld_unpack_pk_t1 MLD_NAMESPACE_KL(unpack_pk_t1)+/**+ * Unpack a single polynomial of the t1 component of a public key+ * pk = (rho, t1).+ *+ * @spec{Partially implements @[FIPS204, Algorithm 23, pkDecode] (one polynomial+ * of t1).}+ *+ * @param[out] t1 Pointer to output polynomial t1[i].+ * @param[in]  pk Byte array containing bit-packed pk.+ * @param      i  Row index, must be < MLDSA_K.+ */+MLD_INTERNAL_API+void mld_unpack_pk_t1(mld_poly *t1,+                      const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+                      unsigned int i)+__contract__(+  requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES))+  requires(memory_no_alias(t1, sizeof(mld_poly)))+  requires(i < MLDSA_K)+  assigns(memory_slice(t1, sizeof(mld_poly)))+  ensures(array_bound(t1->coeffs, 0, MLDSA_N, 0, 1 << 10))+);+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+#define mld_unpack_sk MLD_NAMESPACE_KL(unpack_sk)+/**+ * Unpack secret key sk = (rho, tr, key, t0, s1, s2).+ *+ * NOTE: In REDUCE_RAM mode, s1/s2/t0 borrow from sk rather than copying.+ *+ * @spec{Implements @[FIPS204, Algorithm 25, skDecode].}+ *+ * @param[out] rho Output byte array for rho.+ * @param[out] tr  Output byte array for tr.+ * @param[out] key Output byte array for key.+ * @param[out] t0  Pointer to output vector t0.+ * @param[out] s1  Pointer to output vector s1.+ * @param[out] s2  Pointer to output vector s2.+ * @param[in]  sk  Byte array containing bit-packed sk.+ */+MLD_INTERNAL_API+void mld_unpack_sk(uint8_t rho[MLDSA_SEEDBYTES], uint8_t tr[MLDSA_TRBYTES],+                   uint8_t key[MLDSA_SEEDBYTES], mld_sk_t0hat *t0,+                   mld_sk_s1hat *s1, mld_sk_s2hat *s2,+                   const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES])+__contract__(+  requires(memory_no_alias(rho, MLDSA_SEEDBYTES))+  requires(memory_no_alias(tr, MLDSA_TRBYTES))+  requires(memory_no_alias(key, MLDSA_SEEDBYTES))+  requires(memory_no_alias(t0, sizeof(mld_sk_t0hat)))+  requires(memory_no_alias(s1, sizeof(mld_sk_s1hat)))+  requires(memory_no_alias(s2, sizeof(mld_sk_s2hat)))+  requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES))+  assigns(memory_slice(rho, MLDSA_SEEDBYTES))+  assigns(memory_slice(tr, MLDSA_TRBYTES))+  assigns(memory_slice(key, MLDSA_SEEDBYTES))+  assigns(memory_slice(t0, sizeof(mld_sk_t0hat)))+  assigns(memory_slice(s1, sizeof(mld_sk_s1hat)))+  assigns(memory_slice(s2, sizeof(mld_sk_s2hat)))+  MLD_IF_NOT_REDUCE_RAM(+    ensures(forall(k0, 0, MLDSA_K,+      array_abs_bound(t0->vec.vec[k0].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+    ensures(forall(k1, 0, MLDSA_L,+      array_abs_bound(s1->vec.vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+    ensures(forall(k2, 0, MLDSA_K,+      array_abs_bound(s2->vec.vec[k2].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+  )+  MLD_IF_REDUCE_RAM(+    ensures(s1->packed == old(sk) + MLDSA_SK_S1_OFFSET)+    ensures(s2->packed == old(sk) + MLDSA_SK_S2_OFFSET)+    ensures(t0->packed == old(sk) + MLDSA_SK_T0_OFFSET)+  )+);+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+#define mld_sig_unpack_hints MLD_NAMESPACE_KL(sig_unpack_hints)+/**+ * Decode and validate a single row of the hint vector h from a signature+ * buffer.+ *+ * The hint encoding is shared across all rows (a count array followed by a+ * single index list), so this function performs the validation relevant to+ * row i:+ *   - the i'th hint count is non-decreasing and bounded by MLDSA_OMEGA;+ *   - the indices for row i are strictly ascending;+ *   - on i == MLDSA_K - 1, the trailing index slots are zero.+ *+ * Callers must invoke this for every i in [0, 1, .., MLDSA_K - 1]; if any+ * call returns MLD_ERR_FAIL the encoding is malformed and the signature must+ * be rejected.+ *+ * @spec{Implements @[FIPS204, Algorithm 21, HintBitUnpack] (one row; part of+ * @[FIPS204, Algorithm 27, sigDecode]).}+ *+ * @param[out] h   Pointer to output polynomial h[i].+ * @param[in]  sig Signature buffer.+ * @param      i   Row index, must be < MLDSA_K.+ *+ * @retval 0            Hints were decoded successfully.+ * @retval MLD_ERR_FAIL Hints are malformed.+ */+MLD_INTERNAL_API+MLD_MUST_CHECK_RETURN_VALUE+int mld_sig_unpack_hints(mld_poly *h, const uint8_t sig[MLDSA_CRYPTO_BYTES],+                         unsigned int i)+__contract__(+  requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES))+  requires(memory_no_alias(h, sizeof(mld_poly)))+  requires(i < MLDSA_K)+  assigns(memory_slice(h, sizeof(mld_poly)))+  ensures(return_value == 0 || return_value == MLD_ERR_FAIL)+  ensures(return_value == 0 ==> array_bound(h->coeffs, 0, MLDSA_N, 0, 2))+);+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#endif /* !MLD_PACKING_H */
+ cbits/mldsa/src/params.h view
@@ -0,0 +1,153 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLD_PARAMS_H+#define MLD_PARAMS_H++#define MLDSA_SEEDBYTES 32+#define MLDSA_CRHBYTES 64+#define MLDSA_TRBYTES 64+#define MLDSA_RNDBYTES 32+#define MLDSA_N 256+#define MLDSA_Q 8380417+#define MLDSA_Q_HALF ((MLDSA_Q + 1) / 2)+#define MLDSA_D 13++#define MLDSA_GAMMA2_88 ((MLDSA_Q - 1) / 88)+#define MLDSA_GAMMA2_32 ((MLDSA_Q - 1) / 32)+#define MLDSA_POLYW1_PACKEDBYTES_88 192+#define MLDSA_POLYW1_PACKEDBYTES_32 128++#if MLD_CONFIG_PARAMETER_SET == 44++#define MLDSA_K 4+#define MLDSA_L 4+#define MLDSA_ETA 2+#define MLDSA_TAU 39+#define MLDSA_BETA 78+#define MLDSA_GAMMA1 ((int32_t)1 << 17)+#define MLDSA_GAMMA2 MLDSA_GAMMA2_88+#define MLDSA_OMEGA 80+#define MLDSA_CTILDEBYTES 32+#define MLDSA_POLYZ_PACKEDBYTES 576+#define MLDSA_POLYW1_PACKEDBYTES MLDSA_POLYW1_PACKEDBYTES_88+#define MLDSA_POLYETA_PACKEDBYTES 96++#elif MLD_CONFIG_PARAMETER_SET == 65++#define MLDSA_K 6+#define MLDSA_L 5+#define MLDSA_ETA 4+#define MLDSA_TAU 49+#define MLDSA_BETA 196+#define MLDSA_GAMMA1 ((int32_t)1 << 19)+#define MLDSA_GAMMA2 MLDSA_GAMMA2_32+#define MLDSA_OMEGA 55+#define MLDSA_CTILDEBYTES 48+#define MLDSA_POLYZ_PACKEDBYTES 640+#define MLDSA_POLYW1_PACKEDBYTES MLDSA_POLYW1_PACKEDBYTES_32+#define MLDSA_POLYETA_PACKEDBYTES 128++#elif MLD_CONFIG_PARAMETER_SET == 87++#define MLDSA_K 8+#define MLDSA_L 7+#define MLDSA_ETA 2+#define MLDSA_TAU 60+#define MLDSA_BETA 120+#define MLDSA_GAMMA1 ((int32_t)1 << 19)+#define MLDSA_GAMMA2 MLDSA_GAMMA2_32+#define MLDSA_OMEGA 75+#define MLDSA_CTILDEBYTES 64+#define MLDSA_POLYZ_PACKEDBYTES 640+#define MLDSA_POLYW1_PACKEDBYTES MLDSA_POLYW1_PACKEDBYTES_32+#define MLDSA_POLYETA_PACKEDBYTES 96++#endif /* MLD_CONFIG_PARAMETER_SET == 87 */++#define MLDSA_POLYT1_PACKEDBYTES 320+#define MLDSA_POLYT0_PACKEDBYTES 416+#define MLDSA_POLYVECH_PACKEDBYTES (MLDSA_OMEGA + MLDSA_K)++/* Sampling y from counter kappa uses nonces kappa, ..., kappa+L-1, which fit in+ * uint16_t iff kappa <= UINT16_MAX - MLDSA_L. With kappa = attempt*MLDSA_L this+ * bounds the number of signing attempts by MLD_MAX_KAPPA / MLDSA_L; see+ * MLD_MAX_SIGNING_ATTEMPTS in sign.c. */+#define MLD_MAX_KAPPA (UINT16_MAX - MLDSA_L)++/* Layout of the packed public key pk[MLDSA_CRYPTO_PUBLICKEYBYTES] = (rho, t1):+ *+ *   +-------------+--------------------------++ *   | rho         | t1                       |+ *   +-------------+--------------------------++ *   | SEEDBYTES   | K * POLYT1_PACKEDBYTES   |+ *   +-------------+--------------------------++ */+#define MLDSA_PK_RHO_OFFSET 0+#define MLDSA_PK_RHO_BYTES MLDSA_SEEDBYTES++#define MLDSA_PK_T1_OFFSET (MLDSA_PK_RHO_OFFSET + MLDSA_PK_RHO_BYTES)+#define MLDSA_PK_T1_BYTES (MLDSA_K * MLDSA_POLYT1_PACKEDBYTES)++#define MLDSA_PK_END (MLDSA_PK_T1_OFFSET + MLDSA_PK_T1_BYTES)++#define MLDSA_CRYPTO_PUBLICKEYBYTES MLDSA_PK_END++/* Layout of the packed secret key+ *  sk[MLDSA_CRYPTO_SECRETKEYBYTES] = (rho, key, tr, s1, s2, t0):+ *+ *   +-----------+-----------+-----------+-----------+-----------+-----------++ *   | rho       | key       | tr        | s1        | s2        | t0        |+ *   +-----------+-----------+-----------+-----------+-----------+-----------++ *   | SEEDBYTES | SEEDBYTES | TRBYTES   | L *       | K *       | K *       |+ *   |           |           |           | POLYETA_  | POLYETA_  | POLYT0_   |+ *   |           |           |           | PACKED-   | PACKED-   | PACKED-   |+ *   |           |           |           | BYTES     | BYTES     | BYTES     |+ *   +-----------+-----------+-----------+-----------+-----------+-----------++ */+#define MLDSA_SK_RHO_OFFSET 0+#define MLDSA_SK_RHO_BYTES MLDSA_SEEDBYTES++#define MLDSA_SK_KEY_OFFSET (MLDSA_SK_RHO_OFFSET + MLDSA_SK_RHO_BYTES)+#define MLDSA_SK_KEY_BYTES MLDSA_SEEDBYTES++#define MLDSA_SK_TR_OFFSET (MLDSA_SK_KEY_OFFSET + MLDSA_SK_KEY_BYTES)+#define MLDSA_SK_TR_BYTES MLDSA_TRBYTES++#define MLDSA_SK_S1_OFFSET (MLDSA_SK_TR_OFFSET + MLDSA_SK_TR_BYTES)+#define MLDSA_SK_S1_BYTES (MLDSA_L * MLDSA_POLYETA_PACKEDBYTES)++#define MLDSA_SK_S2_OFFSET (MLDSA_SK_S1_OFFSET + MLDSA_SK_S1_BYTES)+#define MLDSA_SK_S2_BYTES (MLDSA_K * MLDSA_POLYETA_PACKEDBYTES)++#define MLDSA_SK_T0_OFFSET (MLDSA_SK_S2_OFFSET + MLDSA_SK_S2_BYTES)+#define MLDSA_SK_T0_BYTES (MLDSA_K * MLDSA_POLYT0_PACKEDBYTES)++#define MLDSA_SK_END (MLDSA_SK_T0_OFFSET + MLDSA_SK_T0_BYTES)++#define MLDSA_CRYPTO_SECRETKEYBYTES MLDSA_SK_END++/* Layout of the packed signature sig[MLDSA_CRYPTO_BYTES] = (c, z, h):+ *+ *   +----------------+-------------------+----------------------++ *   | c (challenge)  | z                 | h (hints)            |+ *   +----------------+-------------------+----------------------++ *   | CTILDEBYTES    | L *               | POLYVECH_PACKEDBYTES |+ *   |                | POLYZ_PACKEDBYTES | (= OMEGA + K)        |+ *   +----------------+-------------------+----------------------++ */+#define MLDSA_SIG_C_OFFSET 0+#define MLDSA_SIG_C_BYTES MLDSA_CTILDEBYTES++#define MLDSA_SIG_Z_OFFSET (MLDSA_SIG_C_OFFSET + MLDSA_SIG_C_BYTES)+#define MLDSA_SIG_Z_BYTES (MLDSA_L * MLDSA_POLYZ_PACKEDBYTES)++#define MLDSA_SIG_H_OFFSET (MLDSA_SIG_Z_OFFSET + MLDSA_SIG_Z_BYTES)+#define MLDSA_SIG_H_BYTES MLDSA_POLYVECH_PACKEDBYTES++#define MLDSA_SIG_END (MLDSA_SIG_H_OFFSET + MLDSA_SIG_H_BYTES)++#define MLDSA_CRYPTO_BYTES MLDSA_SIG_END++#endif /* !MLD_PARAMS_H */
+ cbits/mldsa/src/poly.c view
@@ -0,0 +1,1066 @@+/*+ * Copyright (c) The mldsa-native project authors+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ *+ * - [REF]+ *   CRYSTALS-Dilithium reference implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/ref+ */++#include "poly.h"++#include "common.h"+#include "ct.h"+#include "debug.h"+#include "reduce.h"+#include "rounding.h"+#include "symmetric.h"++#if !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED)+#include "zetas.inc"++MLD_INTERNAL_API+void mld_poly_reduce(mld_poly *a)+{+  unsigned int i;+  mld_assert_bound(a->coeffs, MLDSA_N, INT32_MIN, MLD_REDUCE32_DOMAIN_MAX);++  for (i = 0; i < MLDSA_N; ++i)+  __loop__(+    invariant(i <= MLDSA_N)+    invariant(forall(k0, i, MLDSA_N, a->coeffs[k0] == loop_entry(*a).coeffs[k0]))+    invariant(array_bound(a->coeffs, 0, i, -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX))+    decreases(MLDSA_N - i))+  {+    a->coeffs[i] = mld_reduce32(a->coeffs[i]);+  }++  mld_assert_bound(a->coeffs, MLDSA_N, -MLD_REDUCE32_RANGE_MAX,+                   MLD_REDUCE32_RANGE_MAX);+}++MLD_STATIC_TESTABLE void mld_poly_caddq_c(mld_poly *a)+__contract__(+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_Q))+  assigns(memory_slice(a, sizeof(mld_poly)))+  ensures(array_bound(a->coeffs, 0, MLDSA_N, 0, MLDSA_Q))+)+{+  unsigned int i;+  mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_Q);++  for (i = 0; i < MLDSA_N; ++i)+  __loop__(+    invariant(i <= MLDSA_N)+    invariant(forall(k0, i, MLDSA_N, a->coeffs[k0] == loop_entry(*a).coeffs[k0]))+    invariant(array_bound(a->coeffs, 0, i, 0, MLDSA_Q))+    decreases(MLDSA_N - i)+    )+  {+    a->coeffs[i] = mld_caddq(a->coeffs[i]);+  }++  mld_assert_bound(a->coeffs, MLDSA_N, 0, MLDSA_Q);+}++MLD_INTERNAL_API+void mld_poly_caddq(mld_poly *a)+{+#if defined(MLD_USE_NATIVE_POLY_CADDQ)+  int ret;+  mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_Q);+  ret = mld_poly_caddq_native(a->coeffs);+  if (ret == MLD_NATIVE_FUNC_SUCCESS)+  {+    mld_assert_bound(a->coeffs, MLDSA_N, 0, MLDSA_Q);+    return;+  }+#endif /* MLD_USE_NATIVE_POLY_CADDQ */+  mld_poly_caddq_c(a);+}++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_SIGN_API) || \+    defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+/* Reference: We use destructive version (output=first input) to avoid+ *            reasoning about aliasing in the CBMC specification */+MLD_INTERNAL_API+void mld_poly_add(mld_poly *r, const mld_poly *b)+{+  unsigned int i;+  for (i = 0; i < MLDSA_N; ++i)+  __loop__(+    assigns(i, memory_slice(r, sizeof(mld_poly)))+    invariant(i <= MLDSA_N)+    invariant(forall(k0, i, MLDSA_N, r->coeffs[k0] == loop_entry(*r).coeffs[k0]))+    invariant(forall(k1, 0, i, r->coeffs[k1] == loop_entry(*r).coeffs[k1] + b->coeffs[k1]))+    invariant(forall(k2, 0, i, r->coeffs[k2] < MLD_REDUCE32_DOMAIN_MAX))+    invariant(forall(k2, 0, i, r->coeffs[k2] >= INT32_MIN))+    decreases(MLDSA_N - i)+  )+  {+    r->coeffs[i] = r->coeffs[i] + b->coeffs[i];+  }+}+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_SIGN_API || \+          MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+/* Reference: We use destructive version (output=first input) to avoid+ *            reasoning about aliasing in the CBMC specification */+MLD_INTERNAL_API+void mld_poly_sub(mld_poly *r, const mld_poly *b)+{+  unsigned int i;+  mld_assert_abs_bound(b->coeffs, MLDSA_N, MLDSA_Q);+  mld_assert_abs_bound(r->coeffs, MLDSA_N, MLDSA_Q);++  for (i = 0; i < MLDSA_N; ++i)+  __loop__(+    invariant(i <= MLDSA_N)+    invariant(array_bound(r->coeffs, 0, i, INT32_MIN, MLD_REDUCE32_DOMAIN_MAX))+    invariant(forall(k0, i, MLDSA_N, r->coeffs[k0] == loop_entry(*r).coeffs[k0]))+    decreases(MLDSA_N - i)+  )+  {+    r->coeffs[i] = r->coeffs[i] - b->coeffs[i];+  }++  mld_assert_bound(r->coeffs, MLDSA_N, INT32_MIN, MLD_REDUCE32_DOMAIN_MAX);+}+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+MLD_INTERNAL_API+void mld_poly_shiftl(mld_poly *a)+{+  unsigned int i;+  mld_assert_bound(a->coeffs, MLDSA_N, 0, 1 << 10);++  for (i = 0; i < MLDSA_N; i++)+  __loop__(+    invariant(i <= MLDSA_N)+    invariant(array_bound(a->coeffs, 0, i, 0, MLDSA_Q))+    invariant(forall(k0, i, MLDSA_N, a->coeffs[k0] == loop_entry(*a).coeffs[k0]))+    decreases(MLDSA_N - i))+  {+    /* Reference: uses a left shift by MLDSA_D which is undefined behaviour in+     * C90/C99+     */+    a->coeffs[i] *= (1 << MLDSA_D);+  }+  mld_assert_bound(a->coeffs, MLDSA_N, 0, MLDSA_Q);+}+#endif /* !MLD_CONFIG_NO_VERIFY_API */++static MLD_INLINE int32_t mld_fqmul(int32_t a, int32_t b)+__contract__(+  requires(b > -MLDSA_Q_HALF && b < MLDSA_Q_HALF)+  ensures(return_value > -MLD_FQMUL_BOUND && return_value < MLD_FQMUL_BOUND)+)+{+  /* Bounds: We argue in mld_montgomery_reduce() that the result+   * of Montgomery reduction is < MLDSA_Q if the input is smaller+   * than 2^31 * MLDSA_Q in absolute value. Indeed, we have:+   *+   *    |a * b|   = |a| * |b|+   *              < 2^31 * MLDSA_Q_HALF+   *              < 2^31 * MLDSA_Q+   *+   * So the output is < MLDSA_Q < MLD_FQMUL_BOUND.+   */+  return mld_montgomery_reduce((int64_t)a * (int64_t)b);+}++/* mld_ntt_butterfly_block()+ *+ * Computes a block CT butterflies with a fixed twiddle factor,+ * using Montgomery multiplication.+ *+ * Parameters:+ * - r: Pointer to base of polynomial (_not_ the base of butterfly block)+ * - zeta: Twiddle factor to use for the butterfly. This must be in+ *         Montgomery form and signed canonical.+ * - start: Offset to the beginning of the butterfly block+ * - len: Index difference between coefficients subject to a butterfly+ * - bound: Ghost variable describing coefficient bound: Prior to `start`,+ *          coefficients must be bound by `bound + MLDSA_Q`. Post `start`,+ *          they must be bound by `bound`.+ * When this function returns, output coefficients in the index range+ * [start, start+2*len) have bound bumped to `bound + MLDSA_Q`.+ * Example:+ * - start=8, len=4+ *   This would compute the following four butterflies+ *          8     --    12+ *             9    --     13+ *                10   --     14+ *                   11   --     15+ * - start=4, len=2+ *   This would compute the following two butterflies+ *          4 -- 6+ *             5 -- 7+ */++/* Reference: Embedded in `ntt()` in the reference implementation @[REF]. */+static MLD_INLINE void mld_ntt_butterfly_block(int32_t r[MLDSA_N],+                                               const int32_t zeta,+                                               const unsigned start,+                                               const unsigned len,+                                               const uint32_t bound)+__contract__(+  requires(start < MLDSA_N)+  requires(1 <= len && len <= MLDSA_N / 2 && start + 2 * len <= MLDSA_N)+  requires(0 <= bound && bound < INT32_MAX - MLD_FQMUL_BOUND)+  requires(-MLDSA_Q_HALF < zeta && zeta < MLDSA_Q_HALF)+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(array_abs_bound(r, 0, start, bound + MLD_FQMUL_BOUND))+  requires(array_abs_bound(r, start, MLDSA_N, bound))+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(array_abs_bound(r, 0, start + 2*len, bound + MLD_FQMUL_BOUND))+  ensures(array_abs_bound(r, start + 2 * len, MLDSA_N, bound)))+{+  /* `bound` is a ghost variable only needed in the CBMC specification */+  unsigned j;+  ((void)bound);+  for (j = start; j < start + len; j++)+  __loop__(+    invariant(start <= j && j <= start + len)+    /*+     * Coefficients are updated in strided pairs, so the bounds for the+     * intermediate states alternate twice between the old and new bound+     */+    invariant(array_abs_bound(r, 0,           j,           bound + MLD_FQMUL_BOUND))+    invariant(array_abs_bound(r, j,           start + len, bound))+    invariant(array_abs_bound(r, start + len, j + len,     bound + MLD_FQMUL_BOUND))+    invariant(array_abs_bound(r, j + len,     MLDSA_N,     bound))+    decreases(start + len - j))+  {+    int32_t t;+    t = mld_fqmul(r[j + len], zeta);+    r[j + len] = r[j] - t;+    r[j] = r[j] + t;+  }+}++/* mld_ntt_layer()+ *+ * Compute one layer of forward NTT+ *+ * Parameters:+ * - r:     Pointer to base of polynomial+ * - layer: Indicates which layer is being applied.+ */++/* Reference: Embedded in `ntt()` in the reference implementation @[REF]. */+static MLD_INLINE void mld_ntt_layer(int32_t r[MLDSA_N], const unsigned layer)+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(1 <= layer && layer <= 8)+  requires(array_abs_bound(r, 0, MLDSA_N, layer * MLD_FQMUL_BOUND))+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(array_abs_bound(r, 0, MLDSA_N, (layer + 1) * MLD_FQMUL_BOUND)))+{+  unsigned start, k, len;+  /* Twiddle factors for layer n are at indices 2^(n-1)..2^n-1. */+  k = 1u << (layer - 1);+  len = (unsigned)MLDSA_N >> layer;+  for (start = 0; start < MLDSA_N; start += 2 * len)+  __loop__(+    invariant(start < MLDSA_N + 2 * len)+    invariant(k <= MLDSA_N)+    invariant(2 * len * k == start + MLDSA_N)+    invariant(array_abs_bound(r, 0, start, layer * MLD_FQMUL_BOUND + MLD_FQMUL_BOUND))+    invariant(array_abs_bound(r, start, MLDSA_N, layer * MLD_FQMUL_BOUND))+    decreases(MLDSA_N - start))+  {+    int32_t zeta = mld_zetas[k++];+    mld_ntt_butterfly_block(r, zeta, start, len, layer * MLD_FQMUL_BOUND);+  }+}++MLD_STATIC_TESTABLE void mld_poly_ntt_c(mld_poly *a)+__contract__(+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_Q))+  assigns(memory_slice(a, sizeof(mld_poly)))+  ensures(array_abs_bound(a->coeffs, 0, MLDSA_N, MLD_NTT_BOUND))+)+{+  unsigned int layer;+  int32_t *r;+++  mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_Q);+  r = a->coeffs;++  for (layer = 1; layer < 9; layer++)+  __loop__(+    invariant(1 <= layer && layer <= 9)+    invariant(array_abs_bound(r, 0, MLDSA_N, layer * MLD_FQMUL_BOUND))+    decreases(9 - layer)+  )+  {+    mld_ntt_layer(r, layer);+  }++  mld_assert_abs_bound(a->coeffs, MLDSA_N, MLD_NTT_BOUND);+}++MLD_INTERNAL_API+void mld_poly_ntt(mld_poly *a)+{+#if defined(MLD_USE_NATIVE_NTT)+  int ret;+  mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_Q);+  ret = mld_ntt_native(a->coeffs);+  if (ret == MLD_NATIVE_FUNC_SUCCESS)+  {+    mld_assert_abs_bound(a->coeffs, MLDSA_N, MLD_NTT_BOUND);+    return;+  }+#endif /* MLD_USE_NATIVE_NTT */+  mld_poly_ntt_c(a);+}++/**+ * Scale a field element by mont/256, i.e., perform Montgomery multiplication+ * by mont^2/256.+ *+ * Input is expected to have absolute value smaller than 256 * MLDSA_Q. Output+ * has absolute value smaller than MLD_INTT_BOUND.+ *+ * @param a Field element to be scaled.+ */+static MLD_INLINE int32_t mld_fqscale(int32_t a)+__contract__(+  requires(a > -256*MLDSA_Q && a < 256*MLDSA_Q)+  ensures(return_value > -MLD_INTT_BOUND && return_value < MLD_INTT_BOUND)+)+{+  /* check-magic: 41978 == pow(2,64-8,MLDSA_Q) */+  const int32_t f = 41978;+  /* Bounds: MLD_INTT_BOUND is MLDSA_Q, so the bounds reasoning is just+   * a special case of that in mld_fqmul(). */+  return mld_montgomery_reduce((int64_t)a * f);+}++/* Reference: Embedded into `invntt_tomont()` in the reference implementation+ * @[REF] */+static MLD_INLINE void mld_invntt_layer(int32_t r[MLDSA_N], unsigned layer)+__contract__(+  requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N))+  requires(1 <= layer && layer <= 8)+  requires(array_abs_bound(r, 0, MLDSA_N, (MLDSA_N >> layer) * MLDSA_Q))+  assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N))+  ensures(array_abs_bound(r, 0, MLDSA_N, (MLDSA_N >> (layer - 1)) * MLDSA_Q)))+{+  unsigned start, k, len;+  len = (unsigned)MLDSA_N >> layer;+  k = (1u << layer) - 1;+  for (start = 0; start < MLDSA_N; start += 2 * len)+  __loop__(+    invariant(start <= MLDSA_N && k <= 255)+    invariant(2 * len * k + start == 2 * MLDSA_N - 2 * len)+    invariant(array_abs_bound(r, 0, start, (MLDSA_N >> (layer - 1)) * MLDSA_Q))+    invariant(array_abs_bound(r, start, MLDSA_N, (MLDSA_N >> layer) * MLDSA_Q))+    decreases(MLDSA_N - start))+  {+    unsigned j;+    int32_t zeta = -mld_zetas[k--];++    /* The bound `(MLDSA_N >> (layer - 1)) * MLDSA_Q` is loose enough to+     * cover both the input bound `(MLDSA_N >> layer) * MLDSA_Q`+     * (for layers >= 1) and the fqmul output bound `MLD_FQMUL_BOUND`+     * (which is < 2 * MLDSA_Q <= (MLDSA_N >> (layer - 1)) * MLDSA_Q). */+    for (j = start; j < start + len; j++)+    __loop__(+      invariant(start <= j && j <= start + len)+      invariant(array_abs_bound(r, 0, start, (MLDSA_N >> (layer - 1)) * MLDSA_Q))+      invariant(array_abs_bound(r, start, j, (MLDSA_N >> (layer - 1)) * MLDSA_Q))+      invariant(array_abs_bound(r, j, start + len, (MLDSA_N >> layer) * MLDSA_Q))+      invariant(array_abs_bound(r, start + len, j + len, (MLDSA_N >> (layer - 1)) * MLDSA_Q))+      invariant(array_abs_bound(r, j + len, MLDSA_N, (MLDSA_N >> layer) * MLDSA_Q))+      decreases(start + len - j))+    {+      int32_t t = r[j];+      r[j] = t + r[j + len];+      r[j + len] = t - r[j + len];+      r[j + len] = mld_fqmul(r[j + len], zeta);+    }+  }+}++MLD_STATIC_TESTABLE void mld_poly_invntt_tomont_c(mld_poly *a)+__contract__(+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_Q))+  assigns(memory_slice(a, sizeof(mld_poly)))+  ensures(array_abs_bound(a->coeffs, 0, MLDSA_N, MLD_INTT_BOUND))+)+{+  unsigned int layer, j;+  int32_t *r;++  mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_Q);++  r = a->coeffs;+  for (layer = 8; layer >= 1; layer--)+  __loop__(+    invariant(layer <= 8)+    /* Absolute bounds increase from 1Q before layer 8 */+    /* up to 256Q after layer 1                        */+    invariant(array_abs_bound(r, 0, MLDSA_N, (MLDSA_N >> layer) * MLDSA_Q))+    decreases(layer))+  {+    mld_invntt_layer(r, layer);+  }++  /* Coefficient bounds are now at 256Q. We now scale by mont / 256,+   * i.e., compute the Montgomery multiplication by mont^2 / 256.+   * mont corrects the mont^-1  factor introduced in the basemul.+   * 1/256 performs that scaling of the inverse NTT.+   * The reduced value is bounded by  MLD_INTT_BOUND in absolute+   * value.*/+  for (j = 0; j < MLDSA_N; ++j)+  __loop__(+    invariant(j <= MLDSA_N)+    invariant(array_abs_bound(r, 0, j, MLD_INTT_BOUND))+    invariant(array_abs_bound(r, j, MLDSA_N, MLDSA_N * MLDSA_Q))+    decreases(MLDSA_N - j)+  )+  {+    r[j] = mld_fqscale(r[j]);+  }++  mld_assert_abs_bound(a->coeffs, MLDSA_N, MLD_INTT_BOUND);+}+++MLD_INTERNAL_API+void mld_poly_invntt_tomont(mld_poly *a)+{+#if defined(MLD_USE_NATIVE_INTT)+  int ret;+  mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_Q);+  ret = mld_intt_native(a->coeffs);+  if (ret == MLD_NATIVE_FUNC_SUCCESS)+  {+    mld_assert_abs_bound(a->coeffs, MLDSA_N, MLD_INTT_BOUND);+    return;+  }+#endif /* MLD_USE_NATIVE_INTT */+  mld_poly_invntt_tomont_c(a);+}++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) || \+    defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+MLD_STATIC_TESTABLE void mld_poly_pointwise_montgomery_c(mld_poly *a,+                                                         const mld_poly *b)+__contract__(+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(memory_no_alias(b, sizeof(mld_poly)))+  requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLD_NTT_BOUND))+  requires(array_abs_bound(b->coeffs, 0, MLDSA_N, MLD_NTT_BOUND))+  assigns(memory_slice(a, sizeof(mld_poly)))+  ensures(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_Q))+)+{+  unsigned int i;+  mld_assert_abs_bound(a->coeffs, MLDSA_N, MLD_NTT_BOUND);+  mld_assert_abs_bound(b->coeffs, MLDSA_N, MLD_NTT_BOUND);++  for (i = 0; i < MLDSA_N; ++i)+  __loop__(+    invariant(i <= MLDSA_N)+    invariant(array_abs_bound(a->coeffs, 0, i, MLDSA_Q))+    invariant(array_abs_bound(a->coeffs, i, MLDSA_N, MLD_NTT_BOUND))+    decreases(MLDSA_N - i)+  )+  {+    a->coeffs[i] = mld_montgomery_reduce((int64_t)a->coeffs[i] * b->coeffs[i]);+  }+  mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_Q);+}++MLD_INTERNAL_API+void mld_poly_pointwise_montgomery(mld_poly *a, const mld_poly *b)+{+#if defined(MLD_USE_NATIVE_POINTWISE_MONTGOMERY)+  int ret;+  mld_assert_abs_bound(a->coeffs, MLDSA_N, MLD_NTT_BOUND);+  mld_assert_abs_bound(b->coeffs, MLDSA_N, MLD_NTT_BOUND);+  ret = mld_poly_pointwise_montgomery_native(a->coeffs, b->coeffs);+  if (ret == MLD_NATIVE_FUNC_SUCCESS)+  {+    mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_Q);+    return;+  }+#endif /* MLD_USE_NATIVE_POINTWISE_MONTGOMERY */+  mld_poly_pointwise_montgomery_c(a, b);+}+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API || \+          MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+MLD_INTERNAL_API+void mld_poly_power2round(mld_poly *a1, mld_poly *a0, const mld_poly *a)+{+  unsigned int i;+  mld_assert_bound(a->coeffs, MLDSA_N, 0, MLDSA_Q);++  for (i = 0; i < MLDSA_N; ++i)+  __loop__(+    assigns(i, memory_slice(a0, sizeof(mld_poly)), memory_slice(a1, sizeof(mld_poly)))+    invariant(i <= MLDSA_N)+    invariant(forall(k0, i, MLDSA_N, a->coeffs[k0] == loop_entry(*a).coeffs[k0]))+    invariant(array_bound(a0->coeffs, 0, i, -(MLD_2_POW_D/2)+1, (MLD_2_POW_D/2)+1))+    invariant(array_bound(a1->coeffs, 0, i, 0, ((MLDSA_Q - 1) / MLD_2_POW_D) + 1))+    decreases(MLDSA_N - i)+  )+  {+    mld_power2round(&a0->coeffs[i], &a1->coeffs[i], a->coeffs[i]);+  }++  mld_assert_bound(a0->coeffs, MLDSA_N, -(MLD_2_POW_D / 2) + 1,+                   (MLD_2_POW_D / 2) + 1);+  mld_assert_bound(a1->coeffs, MLDSA_N, 0, ((MLDSA_Q - 1) / MLD_2_POW_D) + 1);+}+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#ifndef MLD_POLY_UNIFORM_NBLOCKS+#define MLD_POLY_UNIFORM_NBLOCKS \+  ((768 + MLD_STREAM128_BLOCKBYTES - 1) / MLD_STREAM128_BLOCKBYTES)+#endif+/* Reference: `mld_rej_uniform()` in the reference implementation @[REF].+ *            - Our signature differs from the reference implementation+ *              in that it adds the offset and always expects the base of the+ *              target buffer. This avoids shifting the buffer base in the+ *              caller, which appears tricky to reason about. */+MLD_STATIC_TESTABLE unsigned int mld_rej_uniform_c(int32_t *a,+                                                   unsigned int target,+                                                   unsigned int offset,+                                                   const uint8_t *buf,+                                                   unsigned int buflen)+__contract__(+  requires(offset <= target && target <= MLDSA_N)+  requires(buflen <= (MLD_POLY_UNIFORM_NBLOCKS * MLD_STREAM128_BLOCKBYTES) && buflen % 3 == 0)+  requires(memory_no_alias(a, sizeof(int32_t) * target))+  requires(memory_no_alias(buf, buflen))+  requires(array_bound(a, 0, offset, 0, MLDSA_Q))+  assigns(memory_slice(a, sizeof(int32_t) * target))+  ensures(offset <= return_value && return_value <= target)+  ensures(array_bound(a, 0, return_value, 0, MLDSA_Q))+)+{+  unsigned int ctr, pos;+  uint32_t t;+  mld_assert_bound(a, offset, 0, MLDSA_Q);++  ctr = offset;+  pos = 0;+  /* pos + 3 cannot overflow due to the assumption+  buflen <= (MLD_POLY_UNIFORM_NBLOCKS * MLD_STREAM128_BLOCKBYTES) */+  while (ctr < target && pos + 3 <= buflen)+  __loop__(+    invariant(offset <= ctr && ctr <= target && pos <= buflen)+    invariant(array_bound(a, 0, ctr, 0, MLDSA_Q))+    decreases(buflen - pos))+  {+    t = buf[pos++];+    t |= (uint32_t)buf[pos++] << 8;+    t |= (uint32_t)buf[pos++] << 16;+    t &= 0x7FFFFF;++    if (t < MLDSA_Q)+    {+      a[ctr++] = (int32_t)t;+    }+  }++  mld_assert_bound(a, ctr, 0, MLDSA_Q);++  return ctr;+}+/**+ * Sample uniformly random coefficients in [0, MLDSA_Q-1] by performing+ * rejection sampling on an array of random bytes.+ *+ * @param[out] a      Pointer to output array (allocated).+ * @param      target Requested number of coefficients to sample.+ * @param      offset Number of coefficients already sampled.+ * @param[in]  buf    Array of random bytes to sample from.+ * @param      buflen Length of array of random bytes (must be multiple of 3).+ *+ * @return Number of sampled coefficients. Can be smaller than len if not+ *         enough random bytes were given.+ */++/* Reference: `mld_rej_uniform()` in the reference implementation @[REF].+ *            - Our signature differs from the reference implementation+ *              in that it adds the offset and always expects the base of the+ *              target buffer. This avoids shifting the buffer base in the+ *              caller, which appears tricky to reason about. */+static unsigned int mld_rej_uniform(int32_t *a, unsigned int target,+                                    unsigned int offset, const uint8_t *buf,+                                    unsigned int buflen)+__contract__(+  requires(offset <= target && target <= MLDSA_N)+  requires(buflen <= (MLD_POLY_UNIFORM_NBLOCKS * MLD_STREAM128_BLOCKBYTES) && buflen % 3 == 0)+  requires(memory_no_alias(a, sizeof(int32_t) * target))+  requires(memory_no_alias(buf, buflen))+  requires(array_bound(a, 0, offset, 0, MLDSA_Q))+  assigns(memory_slice(a, sizeof(int32_t) * target))+  ensures(offset <= return_value && return_value <= target)+  ensures(array_bound(a, 0, return_value, 0, MLDSA_Q))+)+{+#if defined(MLD_USE_NATIVE_REJ_UNIFORM)+  int ret;+  mld_assert_bound(a, offset, 0, MLDSA_Q);+  if (offset == 0)+  {+    ret = mld_rej_uniform_native(a, target, buf, buflen);+    if (ret != MLD_NATIVE_FUNC_FALLBACK)+    {+      unsigned res = (unsigned)ret;+      mld_assert_bound(a, res, 0, MLDSA_Q);+      return res;+    }+  }+#endif /* MLD_USE_NATIVE_REJ_UNIFORM */++  return mld_rej_uniform_c(a, target, offset, buf, buflen);+}++/* Reference: poly_uniform() in the reference implementation @[REF].+ *           - Simplified from reference by removing buffer tail handling+ *             since buflen % 3 = 0 always holds true (MLD_STREAM128_BLOCKBYTES+ * = 168).+ *           - Modified rej_uniform interface to track offset directly.+ *           - Pass nonce packed in the extended seed array instead of a third+ *             argument.+ * */+MLD_INTERNAL_API+void mld_poly_uniform(mld_poly *a, const uint8_t seed[MLDSA_SEEDBYTES + 2])+{+  unsigned int ctr;+  unsigned int buflen = MLD_POLY_UNIFORM_NBLOCKS * MLD_STREAM128_BLOCKBYTES;+  MLD_ALIGN uint8_t buf[MLD_POLY_UNIFORM_NBLOCKS * MLD_STREAM128_BLOCKBYTES];+  mld_xof128_ctx state;++  mld_xof128_init(&state);+  mld_xof128_absorb_once(&state, seed, MLDSA_SEEDBYTES + 2);+  mld_xof128_squeezeblocks(buf, MLD_POLY_UNIFORM_NBLOCKS, &state);++  ctr = mld_rej_uniform(a->coeffs, MLDSA_N, 0, buf, buflen);+  buflen = MLD_STREAM128_BLOCKBYTES;+  while (ctr < MLDSA_N)+  __loop__(+    assigns(ctr, state, memory_slice(a, sizeof(mld_poly)), object_whole(buf))+    invariant(ctr <= MLDSA_N)+    invariant(array_bound(a->coeffs, 0, ctr, 0, MLDSA_Q))+    invariant(state.pos <= SHAKE128_RATE)+  )+  {+    mld_xof128_squeezeblocks(buf, 1, &state);+    ctr = mld_rej_uniform(a->coeffs, MLDSA_N, ctr, buf, buflen);+  }+  mld_xof128_release(&state);+  mld_assert_bound(a->coeffs, MLDSA_N, 0, MLDSA_Q);++  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(buf, sizeof(buf));+}++#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) && \+    (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))+MLD_INTERNAL_API+void mld_poly_uniform_4x(mld_poly *vec0, mld_poly *vec1, mld_poly *vec2,+                         mld_poly *vec3,+                         uint8_t seed[4][MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2)])+{+  /* Temporary buffers for XOF output before rejection sampling */+  MLD_ALIGN uint8_t+      buf[4][MLD_ALIGN_UP(MLD_POLY_UNIFORM_NBLOCKS * MLD_STREAM128_BLOCKBYTES)];++  /* Tracks the number of coefficients we have already sampled */+  unsigned ctr[4];+  mld_xof128_x4_ctx state;+  unsigned buflen;++  mld_xof128_x4_init(&state);+  mld_xof128_x4_absorb(&state, seed, MLDSA_SEEDBYTES + 2);++  /*+   * Initially, squeeze heuristic number of MLD_POLY_UNIFORM_NBLOCKS.+   * This should generate the matrix entries with high probability.+   */++  mld_xof128_x4_squeezeblocks(buf, MLD_POLY_UNIFORM_NBLOCKS, &state);+  buflen = MLD_POLY_UNIFORM_NBLOCKS * MLD_STREAM128_BLOCKBYTES;+  ctr[0] = mld_rej_uniform(vec0->coeffs, MLDSA_N, 0, buf[0], buflen);+  ctr[1] = mld_rej_uniform(vec1->coeffs, MLDSA_N, 0, buf[1], buflen);+  ctr[2] = mld_rej_uniform(vec2->coeffs, MLDSA_N, 0, buf[2], buflen);+  ctr[3] = mld_rej_uniform(vec3->coeffs, MLDSA_N, 0, buf[3], buflen);++  /*+   * So long as not all matrix entries have been generated, squeeze+   * one more block a time until we're done.+   */+  buflen = MLD_STREAM128_BLOCKBYTES;+  while (ctr[0] < MLDSA_N || ctr[1] < MLDSA_N || ctr[2] < MLDSA_N ||+         ctr[3] < MLDSA_N)+  __loop__(+    assigns(ctr, state, object_whole(buf),+            memory_slice(vec0, sizeof(mld_poly)), memory_slice(vec1, sizeof(mld_poly)),+            memory_slice(vec2, sizeof(mld_poly)), memory_slice(vec3, sizeof(mld_poly)))+    invariant(ctr[0] <= MLDSA_N && ctr[1] <= MLDSA_N)+    invariant(ctr[2] <= MLDSA_N && ctr[3] <= MLDSA_N)+    invariant(array_bound(vec0->coeffs, 0, ctr[0], 0, MLDSA_Q))+    invariant(array_bound(vec1->coeffs, 0, ctr[1], 0, MLDSA_Q))+    invariant(array_bound(vec2->coeffs, 0, ctr[2], 0, MLDSA_Q))+    invariant(array_bound(vec3->coeffs, 0, ctr[3], 0, MLDSA_Q)))+  {+    mld_xof128_x4_squeezeblocks(buf, 1, &state);+    ctr[0] = mld_rej_uniform(vec0->coeffs, MLDSA_N, ctr[0], buf[0], buflen);+    ctr[1] = mld_rej_uniform(vec1->coeffs, MLDSA_N, ctr[1], buf[1], buflen);+    ctr[2] = mld_rej_uniform(vec2->coeffs, MLDSA_N, ctr[2], buf[2], buflen);+    ctr[3] = mld_rej_uniform(vec3->coeffs, MLDSA_N, ctr[3], buf[3], buflen);+  }+  mld_xof128_x4_release(&state);++  mld_assert_bound(vec0->coeffs, MLDSA_N, 0, MLDSA_Q);+  mld_assert_bound(vec1->coeffs, MLDSA_N, 0, MLDSA_Q);+  mld_assert_bound(vec2->coeffs, MLDSA_N, 0, MLDSA_Q);+  mld_assert_bound(vec3->coeffs, MLDSA_N, 0, MLDSA_Q);++  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(buf, sizeof(buf));+}++#endif /* !MLD_CONFIG_SERIAL_FIPS202_ONLY && (!MLD_CONFIG_REDUCE_RAM || \+          MLD_UNIT_TEST) */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+MLD_INTERNAL_API+void mld_polyt1_pack(uint8_t r[MLDSA_POLYT1_PACKEDBYTES], const mld_poly *a)+{+  unsigned int i;+  mld_assert_bound(a->coeffs, MLDSA_N, 0, 1 << 10);++  for (i = 0; i < MLDSA_N / 4; ++i)+  __loop__(+    invariant(i <= MLDSA_N/4)+    decreases(MLDSA_N / 4 - i))+  {+    r[5 * i + 0] = (uint8_t)((a->coeffs[4 * i + 0] >> 0) & 0xFF);+    r[5 * i + 1] =+        (uint8_t)(((a->coeffs[4 * i + 0] >> 8) | (a->coeffs[4 * i + 1] << 2)) &+                  0xFF);+    r[5 * i + 2] =+        (uint8_t)(((a->coeffs[4 * i + 1] >> 6) | (a->coeffs[4 * i + 2] << 4)) &+                  0xFF);+    r[5 * i + 3] =+        (uint8_t)(((a->coeffs[4 * i + 2] >> 4) | (a->coeffs[4 * i + 3] << 6)) &+                  0xFF);+    r[5 * i + 4] = (uint8_t)((a->coeffs[4 * i + 3] >> 2) & 0xFF);+  }+}+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+MLD_INTERNAL_API+void mld_polyt1_unpack(mld_poly *r, const uint8_t a[MLDSA_POLYT1_PACKEDBYTES])+{+  unsigned int i;++  for (i = 0; i < MLDSA_N / 4; ++i)+  __loop__(+    invariant(i <= MLDSA_N/4)+    invariant(array_bound(r->coeffs, 0, i*4, 0, 1 << 10))+    decreases(MLDSA_N / 4 - i))+  {+    r->coeffs[4 * i + 0] =+        ((a[5 * i + 0] >> 0) | ((int32_t)a[5 * i + 1] << 8)) & 0x3FF;+    r->coeffs[4 * i + 1] =+        ((a[5 * i + 1] >> 2) | ((int32_t)a[5 * i + 2] << 6)) & 0x3FF;+    r->coeffs[4 * i + 2] =+        ((a[5 * i + 2] >> 4) | ((int32_t)a[5 * i + 3] << 4)) & 0x3FF;+    r->coeffs[4 * i + 3] =+        ((a[5 * i + 3] >> 6) | ((int32_t)a[5 * i + 4] << 2)) & 0x3FF;+  }++  mld_assert_bound(r->coeffs, MLDSA_N, 0, 1 << 10);+}+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+MLD_INTERNAL_API+void mld_polyt0_pack(uint8_t r[MLDSA_POLYT0_PACKEDBYTES], const mld_poly *a)+{+  unsigned int i;+  uint32_t t[8];++  mld_assert_bound(a->coeffs, MLDSA_N, -(1 << (MLDSA_D - 1)) + 1,+                   (1 << (MLDSA_D - 1)) + 1);++  for (i = 0; i < MLDSA_N / 8; ++i)+  __loop__(+    invariant(i <= MLDSA_N/8)+    decreases(MLDSA_N / 8 - i))+  {+    /* Safety: a->coeffs[i] <= (1 << (MLDSA_D - 1) as they are output of+     * power2round, hence, these casts are safe. */+    t[0] = (uint32_t)((1 << (MLDSA_D - 1)) - a->coeffs[8 * i + 0]);+    t[1] = (uint32_t)((1 << (MLDSA_D - 1)) - a->coeffs[8 * i + 1]);+    t[2] = (uint32_t)((1 << (MLDSA_D - 1)) - a->coeffs[8 * i + 2]);+    t[3] = (uint32_t)((1 << (MLDSA_D - 1)) - a->coeffs[8 * i + 3]);+    t[4] = (uint32_t)((1 << (MLDSA_D - 1)) - a->coeffs[8 * i + 4]);+    t[5] = (uint32_t)((1 << (MLDSA_D - 1)) - a->coeffs[8 * i + 5]);+    t[6] = (uint32_t)((1 << (MLDSA_D - 1)) - a->coeffs[8 * i + 6]);+    t[7] = (uint32_t)((1 << (MLDSA_D - 1)) - a->coeffs[8 * i + 7]);++    r[13 * i + 0] = (uint8_t)((t[0]) & 0xFF);+    r[13 * i + 1] = (uint8_t)((t[0] >> 8) & 0xFF);+    r[13 * i + 1] |= (uint8_t)((t[1] << 5) & 0xFF);+    r[13 * i + 2] = (uint8_t)((t[1] >> 3) & 0xFF);+    r[13 * i + 3] = (uint8_t)((t[1] >> 11) & 0xFF);+    r[13 * i + 3] |= (uint8_t)((t[2] << 2) & 0xFF);+    r[13 * i + 4] = (uint8_t)((t[2] >> 6) & 0xFF);+    r[13 * i + 4] |= (uint8_t)((t[3] << 7) & 0xFF);+    r[13 * i + 5] = (uint8_t)((t[3] >> 1) & 0xFF);+    r[13 * i + 6] = (uint8_t)((t[3] >> 9) & 0xFF);+    r[13 * i + 6] |= (uint8_t)((t[4] << 4) & 0xFF);+    r[13 * i + 7] = (uint8_t)((t[4] >> 4) & 0xFF);+    r[13 * i + 8] = (uint8_t)((t[4] >> 12) & 0xFF);+    r[13 * i + 8] |= (uint8_t)((t[5] << 1) & 0xFF);+    r[13 * i + 9] = (uint8_t)((t[5] >> 7) & 0xFF);+    r[13 * i + 9] |= (uint8_t)((t[6] << 6) & 0xFF);+    r[13 * i + 10] = (uint8_t)((t[6] >> 2) & 0xFF);+    r[13 * i + 11] = (uint8_t)((t[6] >> 10) & 0xFF);+    r[13 * i + 11] |= (uint8_t)((t[7] << 3) & 0xFF);+    r[13 * i + 12] = (uint8_t)((t[7] >> 5) & 0xFF);+  }+}+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST)+MLD_INTERNAL_API+void mld_polyt0_unpack(mld_poly *r, const uint8_t a[MLDSA_POLYT0_PACKEDBYTES])+{+  unsigned int i;++  for (i = 0; i < MLDSA_N / 8; ++i)+  __loop__(+    invariant(i <= MLDSA_N/8)+    invariant(array_bound(r->coeffs, 0, i*8, -(1<<(MLDSA_D-1)) + 1, (1<<(MLDSA_D-1)) + 1))+    decreases(MLDSA_N / 8 - i))+  {+    r->coeffs[8 * i + 0] = a[13 * i + 0];+    r->coeffs[8 * i + 0] |= (int32_t)a[13 * i + 1] << 8;+    r->coeffs[8 * i + 0] &= 0x1FFF;++    r->coeffs[8 * i + 1] = a[13 * i + 1] >> 5;+    r->coeffs[8 * i + 1] |= (int32_t)a[13 * i + 2] << 3;+    r->coeffs[8 * i + 1] |= (int32_t)a[13 * i + 3] << 11;+    r->coeffs[8 * i + 1] &= 0x1FFF;++    r->coeffs[8 * i + 2] = a[13 * i + 3] >> 2;+    r->coeffs[8 * i + 2] |= (int32_t)a[13 * i + 4] << 6;+    r->coeffs[8 * i + 2] &= 0x1FFF;++    r->coeffs[8 * i + 3] = a[13 * i + 4] >> 7;+    r->coeffs[8 * i + 3] |= (int32_t)a[13 * i + 5] << 1;+    r->coeffs[8 * i + 3] |= (int32_t)a[13 * i + 6] << 9;+    r->coeffs[8 * i + 3] &= 0x1FFF;++    r->coeffs[8 * i + 4] = a[13 * i + 6] >> 4;+    r->coeffs[8 * i + 4] |= (int32_t)a[13 * i + 7] << 4;+    r->coeffs[8 * i + 4] |= (int32_t)a[13 * i + 8] << 12;+    r->coeffs[8 * i + 4] &= 0x1FFF;++    r->coeffs[8 * i + 5] = a[13 * i + 8] >> 1;+    r->coeffs[8 * i + 5] |= (int32_t)a[13 * i + 9] << 7;+    r->coeffs[8 * i + 5] &= 0x1FFF;++    r->coeffs[8 * i + 6] = a[13 * i + 9] >> 6;+    r->coeffs[8 * i + 6] |= (int32_t)a[13 * i + 10] << 2;+    r->coeffs[8 * i + 6] |= (int32_t)a[13 * i + 11] << 10;+    r->coeffs[8 * i + 6] &= 0x1FFF;++    r->coeffs[8 * i + 7] = a[13 * i + 11] >> 3;+    r->coeffs[8 * i + 7] |= (int32_t)a[13 * i + 12] << 5;+    r->coeffs[8 * i + 7] &= 0x1FFF;++    r->coeffs[8 * i + 0] = (1 << (MLDSA_D - 1)) - r->coeffs[8 * i + 0];+    r->coeffs[8 * i + 1] = (1 << (MLDSA_D - 1)) - r->coeffs[8 * i + 1];+    r->coeffs[8 * i + 2] = (1 << (MLDSA_D - 1)) - r->coeffs[8 * i + 2];+    r->coeffs[8 * i + 3] = (1 << (MLDSA_D - 1)) - r->coeffs[8 * i + 3];+    r->coeffs[8 * i + 4] = (1 << (MLDSA_D - 1)) - r->coeffs[8 * i + 4];+    r->coeffs[8 * i + 5] = (1 << (MLDSA_D - 1)) - r->coeffs[8 * i + 5];+    r->coeffs[8 * i + 6] = (1 << (MLDSA_D - 1)) - r->coeffs[8 * i + 6];+    r->coeffs[8 * i + 7] = (1 << (MLDSA_D - 1)) - r->coeffs[8 * i + 7];+  }++  mld_assert_bound(r->coeffs, MLDSA_N, -(1 << (MLDSA_D - 1)) + 1,+                   (1 << (MLDSA_D - 1)) + 1);+}+#endif /* !MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST */++MLD_STATIC_TESTABLE uint32_t mld_poly_chknorm_c(const mld_poly *a, int32_t B)+__contract__(+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(0 <= B && B <= MLDSA_Q - MLD_REDUCE32_RANGE_MAX)+  requires(array_bound(a->coeffs, 0, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX))+  ensures(return_value == 0 || return_value == 0xFFFFFFFF)+  ensures((return_value == 0) == array_abs_bound(a->coeffs, 0, MLDSA_N, B))+)+{+  unsigned int i;+  uint32_t t = 0;+  mld_assert_bound(a->coeffs, MLDSA_N, -MLD_REDUCE32_RANGE_MAX,+                   MLD_REDUCE32_RANGE_MAX);+  for (i = 0; i < MLDSA_N; ++i)+  __loop__(+    invariant(i <= MLDSA_N)+    invariant(t == 0 || t == 0xFFFFFFFF)+    invariant((t == 0) == array_abs_bound(a->coeffs, 0, i, B))+    decreases(MLDSA_N - i)+  )+  {+    /*+     * Since we know that -MLD_REDUCE32_RANGE_MAX <= a < MLD_REDUCE32_RANGE_MAX,+     * and B <= MLDSA_Q - MLD_REDUCE32_RANGE_MAX, to check if+     * -B < (a mod± MLDSA_Q) < B, it suffices to check if -B < a < B.+     *+     * We prove this to be true using the following CBMC assertions.+     * a ==> b expressed as !a || b to also allow run-time assertion.+     */+    mld_assert(a->coeffs[i] < B || a->coeffs[i] - MLDSA_Q <= -B);+    mld_assert(a->coeffs[i] > -B || a->coeffs[i] + MLDSA_Q >= B);++    /* Reference: Leaks which coefficient violates the bound via a conditional.+     * We are more conservative to reduce the number of declassifications in+     * constant-time testing.+     */++    /* if (abs(a[i]) >= B) */+    t |= mld_ct_cmask_neg_i32(B - 1 - mld_ct_abs_i32(a->coeffs[i]));+  }++  return t;+}++/* Reference: explicitly checks the bound B to be <= (MLDSA_Q - 1) / 8).+ * This is unnecessary as it's always a compile-time constant.+ * We instead model it as a precondition.+ * Checking the bound is performed using a conditional arguing+ * that it is okay to leak which coefficient violates the bound (while the+ * coefficient itself must remain secret).+ * We instead perform everything in constant-time.+ * Also it is sufficient to check that it is smaller than+ * MLDSA_Q - MLD_REDUCE32_RANGE_MAX > (MLDSA_Q - 1) / 8).+ */+MLD_INTERNAL_API+uint32_t mld_poly_chknorm(const mld_poly *a, int32_t B)+{+#if defined(MLD_USE_NATIVE_POLY_CHKNORM)+  int ret;+  int success;+  mld_assert_bound(a->coeffs, MLDSA_N, -MLD_REDUCE32_RANGE_MAX,+                   MLD_REDUCE32_RANGE_MAX);+  /* The native backend returns 0 if all coefficients are within the bound,+   * 1 if at least one coefficient exceeds the bound, and+   * -1 (MLD_NATIVE_FUNC_FALLBACK) if the platform does not have the+   * required capabilities to run the native function.+   */+  ret = mld_poly_chknorm_native(a->coeffs, B);++  success = (ret != MLD_NATIVE_FUNC_FALLBACK);+  /* Constant-time: It would be fine to leak the return value of chknorm+   * entirely (as it is fine to leak if any coefficient exceeded the bound or+   * not). However, it is cleaner to perform declassification in sign.c.+   * Hence, here we only declassify if the native function returned+   * MLD_NATIVE_FUNC_FALLBACK or not (which solely depends on system+   * capabilities).+   */+  MLD_CT_TESTING_DECLASSIFY(&success, sizeof(int));+  if (success)+  {+    /* Convert 0 / 1 to 0 / 0xFFFFFFFF here */+    return mld_ct_cmask_nonzero_u32((uint32_t)ret);+  }+#endif /* MLD_USE_NATIVE_POLY_CHKNORM */+  return mld_poly_chknorm_c(a, B);+}++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+    MLD_CONFIG_PARAMETER_SET == 44+MLD_INTERNAL_API+void mld_polyw1_pack_88(uint8_t r[MLDSA_POLYW1_PACKEDBYTES_88],+                        const mld_poly *a)+{+  unsigned int i;++  mld_assert_bound(a->coeffs, MLDSA_N, 0,+                   (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2_88));++  for (i = 0; i < MLDSA_N / 4; ++i)+  __loop__(+    invariant(i <= MLDSA_N/4)+    decreases(MLDSA_N / 4 - i))+  {+    r[3 * i + 0] = (uint8_t)((a->coeffs[4 * i + 0]) & 0xFF);+    r[3 * i + 0] |= (uint8_t)((a->coeffs[4 * i + 1] << 6) & 0xFF);+    r[3 * i + 1] = (uint8_t)((a->coeffs[4 * i + 1] >> 2) & 0xFF);+    r[3 * i + 1] |= (uint8_t)((a->coeffs[4 * i + 2] << 4) & 0xFF);+    r[3 * i + 2] = (uint8_t)((a->coeffs[4 * i + 2] >> 4) & 0xFF);+    r[3 * i + 2] |= (uint8_t)((a->coeffs[4 * i + 3] << 2) & 0xFF);+  }+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \+        */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)+MLD_INTERNAL_API+void mld_polyw1_pack_32(uint8_t r[MLDSA_POLYW1_PACKEDBYTES_32],+                        const mld_poly *a)+{+  unsigned int i;++  mld_assert_bound(a->coeffs, MLDSA_N, 0,+                   (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2_32));++  for (i = 0; i < MLDSA_N / 2; ++i)+  __loop__(+    invariant(i <= MLDSA_N/2)+    decreases(MLDSA_N / 2 - i))+  {+    r[i] =+        (uint8_t)((a->coeffs[2 * i + 0] | (a->coeffs[2 * i + 1] << 4)) & 0xFF);+  }+}+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87 */+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */++#else  /* !MLD_CONFIG_MULTILEVEL_NO_SHARED */+MLD_EMPTY_CU(mld_poly)+#endif /* MLD_CONFIG_MULTILEVEL_NO_SHARED */++/* To facilitate single-compilation-unit (SCU) builds, undefine all macros.+ * Don't modify by hand -- this is auto-generated by scripts/autogen. */+#undef MLD_POLY_UNIFORM_NBLOCKS
+ cbits/mldsa/src/poly.h view
@@ -0,0 +1,464 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ */++#ifndef MLD_POLY_H+#define MLD_POLY_H++#include "cbmc.h"+#include "common.h"+#include "reduce.h"+#include "rounding.h"++/* Absolute exclusive upper bound for the output of fqmul */+#define MLD_FQMUL_BOUND ((5 * MLDSA_Q + 3) / 4)+/* Absolute exclusive upper bound for the output of the forward NTT */+#define MLD_NTT_BOUND (9 * MLD_FQMUL_BOUND)+/* Absolute exclusive upper bound for the output of the inverse NTT*/+#define MLD_INTT_BOUND MLDSA_Q++/**+ * Element of R_q = Z_q[X]/(X^n + 1). Represents polynomial+ * coeffs[0] + X*coeffs[1] + X^2*coeffs[2] + ... + X^{n-1}*coeffs[n-1].+ */+typedef struct+{+  int32_t coeffs[MLDSA_N]; /**< Polynomial coefficients. */+} MLD_ALIGN mld_poly;++#define mld_poly_reduce MLD_NAMESPACE(poly_reduce)+/**+ * In-place reduction of all coefficients of polynomial to representative in+ * [-MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX].+ *+ * @param[in,out] a Pointer to input/output polynomial.+ */+MLD_INTERNAL_API+void mld_poly_reduce(mld_poly *a)+__contract__(+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(array_bound(a->coeffs, 0, MLDSA_N, INT32_MIN, MLD_REDUCE32_DOMAIN_MAX))+  assigns(memory_slice(a, sizeof(mld_poly)))+  ensures(array_bound(a->coeffs, 0, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX))+);++#define mld_poly_caddq MLD_NAMESPACE(poly_caddq)+/**+ * For all coefficients of in/out polynomial add MLDSA_Q if coefficient is+ * negative.+ *+ * @param[in,out] a Pointer to input/output polynomial.+ */+MLD_INTERNAL_API+void mld_poly_caddq(mld_poly *a)+__contract__(+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_Q))+  assigns(memory_slice(a, sizeof(mld_poly)))+  ensures(array_bound(a->coeffs, 0, MLDSA_N, 0, MLDSA_Q))+);++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_SIGN_API) || \+    defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+#define mld_poly_add MLD_NAMESPACE(poly_add)+/**+ * Add polynomials. No modular reduction is performed.+ *+ * @spec{Implements @[FIPS204, Algorithm 44, AddNTT] (coefficientwise+ * polynomial addition; also used for addition in the normal domain).}+ *+ * @param[in,out] r Pointer to input-output polynomial to be added to.+ * @param[in]     b Pointer to input polynomial that should be added to r.+ *                  Must be disjoint from r.+ */++/*+ * NOTE: The reference implementation uses a 3-argument poly_add.+ * We specialize to the accumulator form to avoid reasoning about aliasing.+ */+MLD_INTERNAL_API+void mld_poly_add(mld_poly *r, const mld_poly *b)+__contract__(+  requires(memory_no_alias(b, sizeof(mld_poly)))+  requires(memory_no_alias(r, sizeof(mld_poly)))+  requires(forall(k0, 0, MLDSA_N, (int64_t) r->coeffs[k0] + b->coeffs[k0] < MLD_REDUCE32_DOMAIN_MAX))+  requires(forall(k1, 0, MLDSA_N, (int64_t) r->coeffs[k1] + b->coeffs[k1] >= INT32_MIN))+  assigns(memory_slice(r, sizeof(mld_poly)))+  ensures(forall(k2, 0, MLDSA_N, r->coeffs[k2] == old(*r).coeffs[k2] + b->coeffs[k2]))+  ensures(forall(k3, 0, MLDSA_N, r->coeffs[k3] < MLD_REDUCE32_DOMAIN_MAX))+  ensures(forall(k4, 0, MLDSA_N, r->coeffs[k4] >= INT32_MIN))+);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_SIGN_API || \+          MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+#define mld_poly_sub MLD_NAMESPACE(poly_sub)+/**+ * Subtract polynomials. No modular reduction is performed.+ *+ * @param[in,out] r Pointer to input-output polynomial.+ * @param[in]     b Pointer to input polynomial that should be subtracted from+ *                  r. Must be disjoint from r.+ */+/*+ * NOTE: The reference implementation uses a 3-argument poly_sub.+ * We specialize to the accumulator form to avoid reasoning about aliasing.+ */+MLD_INTERNAL_API+void mld_poly_sub(mld_poly *r, const mld_poly *b)+__contract__(+  requires(memory_no_alias(b, sizeof(mld_poly)))+  requires(memory_no_alias(r, sizeof(mld_poly)))+  requires(array_abs_bound(r->coeffs, 0, MLDSA_N, MLDSA_Q))+  requires(array_abs_bound(b->coeffs, 0, MLDSA_N, MLDSA_Q))+  assigns(memory_slice(r, sizeof(mld_poly)))+  ensures(array_bound(r->coeffs, 0, MLDSA_N, INT32_MIN, MLD_REDUCE32_DOMAIN_MAX))+);+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+#define mld_poly_shiftl MLD_NAMESPACE(poly_shiftl)+/**+ * Multiply polynomial by 2^MLDSA_D without modular reduction. Assumes input+ * coefficients to be less than 2^{31-MLDSA_D} in absolute value.+ *+ * @param[in,out] a Pointer to input/output polynomial.+ */+MLD_INTERNAL_API+void mld_poly_shiftl(mld_poly *a)+__contract__(+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(array_bound(a->coeffs, 0, MLDSA_N, 0, 1 << 10))+  assigns(memory_slice(a, sizeof(mld_poly)))+  ensures(array_bound(a->coeffs, 0, MLDSA_N, 0, MLDSA_Q))+);+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#define mld_poly_ntt MLD_NAMESPACE(poly_ntt)+/**+ * In-place forward NTT. Output coefficients are bounded by MLD_NTT_BOUND in+ * absolute value.+ *+ * @spec{Implements @[FIPS204, Algorithm 41, NTT].}+ *+ * @param[in,out] a Pointer to input/output polynomial.+ */+MLD_INTERNAL_API+void mld_poly_ntt(mld_poly *a)+__contract__(+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_Q))+  assigns(memory_slice(a, sizeof(mld_poly)))+  ensures(array_abs_bound(a->coeffs, 0, MLDSA_N, MLD_NTT_BOUND))+);+++#define mld_poly_invntt_tomont MLD_NAMESPACE(poly_invntt_tomont)+/**+ * In-place inverse NTT.+ *+ * Input coefficients need to be less than MLDSA_Q in absolute value and+ * output coefficients are bounded by MLD_INTT_BOUND.+ *+ * @spec{Implements @[FIPS204, Algorithm 42, NTT^{-1}] up to scaling:+ * The input is scaled by 2^{-32} as a result of the Montgomery base+ * multiplication. The output is in normal domain. In other words, this+ * function implements `NTT^{-1} o mult(2^32)`.}+ *+ * @param[in,out] a Pointer to input/output polynomial.+ */+MLD_INTERNAL_API+void mld_poly_invntt_tomont(mld_poly *a)+__contract__(+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_Q))+  assigns(memory_slice(a, sizeof(mld_poly)))+  ensures(array_abs_bound(a->coeffs, 0, MLDSA_N, MLD_INTT_BOUND))+);++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) || \+    defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+#define mld_poly_pointwise_montgomery MLD_NAMESPACE(poly_pointwise_montgomery)+/**+ * Pointwise multiplication of polynomials. Destructive in the first argument.+ *+ * @spec{Implements @[FIPS204, Algorithm 45, MultiplyNTT], up to scaling: The+ * input is in normal domain, the output is scaled by 2^{-32} as a result of+ * the use of Montgomery multiplication. In other words, this function+ * implements `mult(2^{-32}) o MultiplyNTT`.}+ *+ * @param[in,out] a Pointer to first input/output polynomial. On entry, holds+ *                  the first multiplicand; on exit, holds the product+ *                  a * b * 2^{-32}.+ * @param[in]     b Pointer to second input polynomial.+ */+MLD_INTERNAL_API+void mld_poly_pointwise_montgomery(mld_poly *a, const mld_poly *b)+__contract__(+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(memory_no_alias(b, sizeof(mld_poly)))+  requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLD_NTT_BOUND))+  requires(array_abs_bound(b->coeffs, 0, MLDSA_N, MLD_NTT_BOUND))+  assigns(memory_slice(a, sizeof(mld_poly)))+  ensures(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_Q))+);+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API || \+          MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+#define mld_poly_power2round MLD_NAMESPACE(poly_power2round)+/**+ * For all coefficients c of the input polynomial, compute c0, c1 such that+ * c mod MLDSA_Q = c1*2^MLDSA_D + c0 with -2^{MLDSA_D-1} < c0 <= 2^{MLDSA_D-1}.+ * Assumes coefficients to be standard representatives.+ *+ * @param[out] a1 Pointer to output polynomial with coefficients c1.+ * @param[out] a0 Pointer to output polynomial with coefficients c0; may alias+ *                the input polynomial a.+ * @param[in]  a  Pointer to input polynomial.+ */+MLD_INTERNAL_API+void mld_poly_power2round(mld_poly *a1, mld_poly *a0, const mld_poly *a)+__contract__(+  requires(memory_no_alias(a0, sizeof(mld_poly)))+  requires(memory_no_alias(a1, sizeof(mld_poly)))+  /* The implementation does not require a0 == a, but the single call site+   * aliases them and asserting equality simplifies the proof. */+  requires(a0 == a)+  requires(array_bound(a->coeffs, 0, MLDSA_N, 0, MLDSA_Q))+  assigns(memory_slice(a1, sizeof(mld_poly)))+  assigns(memory_slice(a0, sizeof(mld_poly)))+  ensures(array_bound(a0->coeffs, 0, MLDSA_N, -(MLD_2_POW_D/2)+1, (MLD_2_POW_D/2)+1))+  ensures(array_bound(a1->coeffs, 0, MLDSA_N, 0, ((MLDSA_Q - 1) / MLD_2_POW_D) + 1))+);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#define mld_poly_uniform MLD_NAMESPACE(poly_uniform)+/**+ * Sample polynomial with uniformly random coefficients in [0, MLDSA_Q-1] by+ * performing rejection sampling on the output stream of SHAKE128(seed|nonce).+ *+ * @spec{Implements @[FIPS204, Algorithm 30, RejNTTPoly].}+ *+ * @param[out] a    Pointer to output polynomial.+ * @param[in]  seed Byte array with seed of length MLDSA_SEEDBYTES and the+ *                  packed 2-byte nonce.+ */+MLD_INTERNAL_API+void mld_poly_uniform(mld_poly *a, const uint8_t seed[MLDSA_SEEDBYTES + 2])+__contract__(+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(memory_no_alias(seed, MLDSA_SEEDBYTES + 2))+  assigns(memory_slice(a, sizeof(mld_poly)))+  ensures(array_bound(a->coeffs, 0, MLDSA_N, 0, MLDSA_Q))+);++#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) && \+    (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))+#define mld_poly_uniform_4x MLD_NAMESPACE(poly_uniform_4x)+/**+ * Generate four polynomials using rejection sampling on (pseudo-)uniformly+ * random bytes sampled from a seed.+ *+ * @spec{Implements @[FIPS204, Algorithm 30, RejNTTPoly] (four-way batched).}+ *+ * @param[out] vec0 Pointer to first polynomial to be sampled.+ * @param[out] vec1 Pointer to second polynomial to be sampled.+ * @param[out] vec2 Pointer to third polynomial to be sampled.+ * @param[out] vec3 Pointer to fourth polynomial to be sampled.+ * @param[in]  seed Pointer to consecutive array of seed buffers of size+ *                  MLDSA_SEEDBYTES + 2 each, plus padding for alignment.+ */+MLD_INTERNAL_API+void mld_poly_uniform_4x(mld_poly *vec0, mld_poly *vec1, mld_poly *vec2,+                         mld_poly *vec3,+                         uint8_t seed[4][MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2)])+__contract__(+  requires(memory_no_alias(vec0, sizeof(mld_poly)))+  requires(memory_no_alias(vec1, sizeof(mld_poly)))+  requires(memory_no_alias(vec2, sizeof(mld_poly)))+  requires(memory_no_alias(vec3, sizeof(mld_poly)))+  requires(memory_no_alias(seed,  4 * MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2)))+  assigns(memory_slice(vec0, sizeof(mld_poly)))+  assigns(memory_slice(vec1, sizeof(mld_poly)))+  assigns(memory_slice(vec2, sizeof(mld_poly)))+  assigns(memory_slice(vec3, sizeof(mld_poly)))+  ensures(array_bound(vec0->coeffs, 0, MLDSA_N, 0, MLDSA_Q))+  ensures(array_bound(vec1->coeffs, 0, MLDSA_N, 0, MLDSA_Q))+  ensures(array_bound(vec2->coeffs, 0, MLDSA_N, 0, MLDSA_Q))+  ensures(array_bound(vec3->coeffs, 0, MLDSA_N, 0, MLDSA_Q))+);+#endif /* !MLD_CONFIG_SERIAL_FIPS202_ONLY && (!MLD_CONFIG_REDUCE_RAM || \+          MLD_UNIT_TEST) */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+#define mld_polyt1_pack MLD_NAMESPACE(polyt1_pack)+/**+ * Bit-pack polynomial t1 with coefficients fitting in 10 bits. Input+ * coefficients are assumed to be standard representatives.+ *+ * @spec{Implements @[FIPS204, Algorithm 16, SimpleBitPack].}+ *+ * @param[out] r Pointer to output byte array with at least+ *               MLDSA_POLYT1_PACKEDBYTES bytes.+ * @param[in]  a Pointer to input polynomial.+ */+MLD_INTERNAL_API+void mld_polyt1_pack(uint8_t r[MLDSA_POLYT1_PACKEDBYTES], const mld_poly *a)+__contract__(+  requires(memory_no_alias(r, MLDSA_POLYT1_PACKEDBYTES))+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(array_bound(a->coeffs, 0, MLDSA_N, 0, 1 << 10))+  assigns(memory_slice(r, MLDSA_POLYT1_PACKEDBYTES))+);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+#define mld_polyt1_unpack MLD_NAMESPACE(polyt1_unpack)+/**+ * Unpack polynomial t1 with 10-bit coefficients. Output coefficients are+ * standard representatives.+ *+ * @spec{Implements @[FIPS204, Algorithm 18, SimpleBitUnpack].}+ *+ * @param[out] r Pointer to output polynomial.+ * @param[in]  a Byte array with bit-packed polynomial.+ */+MLD_INTERNAL_API+void mld_polyt1_unpack(mld_poly *r, const uint8_t a[MLDSA_POLYT1_PACKEDBYTES])+__contract__(+  requires(memory_no_alias(r, sizeof(mld_poly)))+  requires(memory_no_alias(a, MLDSA_POLYT1_PACKEDBYTES))+  assigns(memory_slice(r, sizeof(mld_poly)))+  ensures(array_bound(r->coeffs, 0, MLDSA_N, 0, 1 << 10))+);+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+#define mld_polyt0_pack MLD_NAMESPACE(polyt0_pack)+/**+ * Bit-pack polynomial t0 with coefficients in ]-2^{MLDSA_D-1}, 2^{MLDSA_D-1}].+ *+ * @spec{Implements @[FIPS204, Algorithm 17, BitPack].}+ *+ * @param[out] r Pointer to output byte array with at least+ *               MLDSA_POLYT0_PACKEDBYTES bytes.+ * @param[in]  a Pointer to input polynomial.+ */+MLD_INTERNAL_API+void mld_polyt0_pack(uint8_t r[MLDSA_POLYT0_PACKEDBYTES], const mld_poly *a)+__contract__(+  requires(memory_no_alias(r, MLDSA_POLYT0_PACKEDBYTES))+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(array_bound(a->coeffs, 0, MLDSA_N, -(1<<(MLDSA_D-1)) + 1, (1<<(MLDSA_D-1)) + 1))+  assigns(memory_slice(r, MLDSA_POLYT0_PACKEDBYTES))+);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST)+#define mld_polyt0_unpack MLD_NAMESPACE(polyt0_unpack)+/**+ * Unpack polynomial t0 with coefficients in ]-2^{MLDSA_D-1}, 2^{MLDSA_D-1}].+ *+ * @spec{Implements @[FIPS204, Algorithm 19, BitUnpack].}+ *+ * @param[out] r Pointer to output polynomial.+ * @param[in]  a Byte array with bit-packed polynomial.+ */+MLD_INTERNAL_API+void mld_polyt0_unpack(mld_poly *r, const uint8_t a[MLDSA_POLYT0_PACKEDBYTES])+__contract__(+  requires(memory_no_alias(r, sizeof(mld_poly)))+  requires(memory_no_alias(a, MLDSA_POLYT0_PACKEDBYTES))+  assigns(memory_slice(r, sizeof(mld_poly)))+  ensures(array_bound(r->coeffs, 0, MLDSA_N, -(1<<(MLDSA_D-1)) + 1, (1<<(MLDSA_D-1)) + 1))+);+#endif /* !MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST */++#define mld_poly_chknorm MLD_NAMESPACE(poly_chknorm)+/**+ * Check infinity norm of polynomial against given bound. Assumes input+ * coefficients were reduced by mld_reduce32().+ *+ * @spec{@[FIPS204] defines the infinity norm via signed canonical reduction+ * (mod± MLDSA_Q) prior to applying the bounds check. However,+ * `-B < (a mod± MLDSA_Q) < B` is equivalent to `-B < a < B` under the+ * assumption that `B <= MLDSA_Q - MLD_REDUCE32_RANGE_MAX` (cf. the assertion in+ * the code). Hence, this contract and implementation are correct without+ * reduction.}+ *+ * @param[in] a Pointer to polynomial.+ * @param     B Norm bound.+ *+ * @return 0 if norm is strictly smaller than+ *         B <= (MLDSA_Q - MLD_REDUCE32_RANGE_MAX) and 0xFFFFFFFF otherwise.+ */+MLD_INTERNAL_API+MLD_MUST_CHECK_RETURN_VALUE+uint32_t mld_poly_chknorm(const mld_poly *a, int32_t B)+__contract__(+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(0 <= B && B <= MLDSA_Q - MLD_REDUCE32_RANGE_MAX)+  requires(array_bound(a->coeffs, 0, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX))+  ensures(return_value == 0 || return_value == 0xFFFFFFFF)+  ensures((return_value == 0) == array_abs_bound(a->coeffs, 0, MLDSA_N, B))+);++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44+#define mld_polyw1_pack_88 MLD_NAMESPACE(polyw1_pack_88)+/**+ * Bit-pack polynomial w1, using 6 bits per coefficient.+ * This is the variant for parameter sets with MLDSA_GAMMA2 = (MLDSA_Q-1)/88+ * (ML-DSA-44), for which w1 coefficients lie in [0, 43].+ *+ * @param[out] r Pointer to output byte array (MLDSA_POLYW1_PACKEDBYTES_88).+ * @param[in]  a Pointer to input polynomial.+ */+MLD_INTERNAL_API+void mld_polyw1_pack_88(uint8_t r[MLDSA_POLYW1_PACKEDBYTES_88],+                        const mld_poly *a)+__contract__(+  requires(memory_no_alias(r, MLDSA_POLYW1_PACKEDBYTES_88))+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(array_bound(a->coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2_88)))+  assigns(memory_slice(r, MLDSA_POLYW1_PACKEDBYTES_88))+);+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \+        */++#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)+#define mld_polyw1_pack_32 MLD_NAMESPACE(polyw1_pack_32)+/**+ * Bit-pack polynomial w1, using 4 bits per coefficient.+ * This is the variant for parameter sets with MLDSA_GAMMA2 = (MLDSA_Q-1)/32+ * (ML-DSA-65 and ML-DSA-87), for which w1 coefficients lie in [0, 15].+ *+ * @param[out] r Pointer to output byte array (MLDSA_POLYW1_PACKEDBYTES_32).+ * @param[in]  a Pointer to input polynomial.+ */+MLD_INTERNAL_API+void mld_polyw1_pack_32(uint8_t r[MLDSA_POLYW1_PACKEDBYTES_32],+                        const mld_poly *a)+__contract__(+  requires(memory_no_alias(r, MLDSA_POLYW1_PACKEDBYTES_32))+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(array_bound(a->coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2_32)))+  assigns(memory_slice(r, MLDSA_POLYW1_PACKEDBYTES_32))+);+#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87 */+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */++#endif /* !MLD_POLY_H */
+ cbits/mldsa/src/poly_kl.c view
@@ -0,0 +1,910 @@+/*+ * Copyright (c) The mldsa-native project authors+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ *+ * - [REF]+ *   CRYSTALS-Dilithium reference implementation+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/dilithium/tree/master/ref+ */++#include "poly_kl.h"++#include "ct.h"+#include "debug.h"+#include "rounding.h"+#include "symmetric.h"++/* Parameter set namespacing+ * This is to facilitate building multiple instances+ * of mldsa-native (e.g. with varying parameter sets)+ * within a single compilation unit. */+#define mld_rej_eta MLD_ADD_PARAM_SET(mld_rej_eta)+#define mld_rej_eta_c MLD_ADD_PARAM_SET(mld_rej_eta_c)+#define mld_poly_decompose_c MLD_ADD_PARAM_SET(mld_poly_decompose_c)+#define mld_poly_use_hint_c MLD_ADD_PARAM_SET(mld_poly_use_hint_c)+#define mld_polyz_unpack_c MLD_ADD_PARAM_SET(mld_polyz_unpack_c)+/* End of parameter set namespacing */+++#if !defined(MLD_CONFIG_NO_SIGN_API)+MLD_STATIC_TESTABLE+void mld_poly_decompose_c(mld_poly *a1, mld_poly *a0)+__contract__(+  requires(memory_no_alias(a1,  sizeof(mld_poly)))+  requires(memory_no_alias(a0, sizeof(mld_poly)))+  requires(array_bound(a0->coeffs, 0, MLDSA_N, 0, MLDSA_Q))+  assigns(memory_slice(a1, sizeof(mld_poly)))+  assigns(memory_slice(a0, sizeof(mld_poly)))+  ensures(array_bound(a1->coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2)))+  ensures(array_abs_bound(a0->coeffs, 0, MLDSA_N, MLDSA_GAMMA2+1))+)+{+  unsigned int i;+  mld_assert_bound(a0->coeffs, MLDSA_N, 0, MLDSA_Q);+  for (i = 0; i < MLDSA_N; ++i)+  __loop__(+    assigns(i, memory_slice(a0, sizeof(mld_poly)), memory_slice(a1, sizeof(mld_poly)))+    invariant(i <= MLDSA_N)+    invariant(array_bound(a0->coeffs, i, MLDSA_N, 0, MLDSA_Q))+    invariant(array_bound(a1->coeffs, 0, i, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2)))+    invariant(array_abs_bound(a0->coeffs, 0, i, MLDSA_GAMMA2+1))+    decreases(MLDSA_N - i)+  )+  {+    mld_decompose(&a0->coeffs[i], &a1->coeffs[i], a0->coeffs[i]);+  }++  mld_assert_abs_bound(a0->coeffs, MLDSA_N, MLDSA_GAMMA2 + 1);+  mld_assert_bound(a1->coeffs, MLDSA_N, 0, (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2));+}++MLD_INTERNAL_API+void mld_poly_decompose(mld_poly *a1, mld_poly *a0)+{+#if defined(MLD_USE_NATIVE_POLY_DECOMPOSE_88) && MLD_CONFIG_PARAMETER_SET == 44+  int ret;+  mld_assert_bound(a0->coeffs, MLDSA_N, 0, MLDSA_Q);+  ret = mld_poly_decompose_88_native(a1->coeffs, a0->coeffs);+  if (ret == MLD_NATIVE_FUNC_SUCCESS)+  {+    mld_assert_abs_bound(a0->coeffs, MLDSA_N, MLDSA_GAMMA2 + 1);+    mld_assert_bound(a1->coeffs, MLDSA_N, 0,+                     (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2));+    return;+  }+#elif defined(MLD_USE_NATIVE_POLY_DECOMPOSE_32) && \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)+  int ret;+  mld_assert_bound(a0->coeffs, MLDSA_N, 0, MLDSA_Q);+  ret = mld_poly_decompose_32_native(a1->coeffs, a0->coeffs);+  if (ret == MLD_NATIVE_FUNC_SUCCESS)+  {+    mld_assert_abs_bound(a0->coeffs, MLDSA_N, MLDSA_GAMMA2 + 1);+    mld_assert_bound(a1->coeffs, MLDSA_N, 0,+                     (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2));+    return;+  }+#endif /* !(MLD_USE_NATIVE_POLY_DECOMPOSE_88 && MLD_CONFIG_PARAMETER_SET ==    \+          44) && MLD_USE_NATIVE_POLY_DECOMPOSE_32 && (MLD_CONFIG_PARAMETER_SET \+          == 65 || MLD_CONFIG_PARAMETER_SET == 87) */+  mld_poly_decompose_c(a1, a0);+}++#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+MLD_STATIC_TESTABLE void mld_poly_use_hint_c(mld_poly *a, const mld_poly *h)+__contract__(+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(memory_no_alias(h, sizeof(mld_poly)))+  requires(array_bound(a->coeffs, 0, MLDSA_N, 0, MLDSA_Q))+  requires(array_bound(h->coeffs, 0, MLDSA_N, 0, 2))+  assigns(memory_slice(a, sizeof(mld_poly)))+  ensures(array_bound(a->coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2)))+)+{+  unsigned int i;+  mld_assert_bound(a->coeffs, MLDSA_N, 0, MLDSA_Q);+  mld_assert_bound(h->coeffs, MLDSA_N, 0, 2);++  for (i = 0; i < MLDSA_N; ++i)+  __loop__(+    invariant(i <= MLDSA_N)+    invariant(array_bound(a->coeffs, 0, i, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2)))+    invariant(array_bound(a->coeffs, i, MLDSA_N, 0, MLDSA_Q))+    decreases(MLDSA_N - i)+  )+  {+    a->coeffs[i] = mld_use_hint(a->coeffs[i], h->coeffs[i]);+  }+  mld_assert_bound(a->coeffs, MLDSA_N, 0, (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2));+}++MLD_INTERNAL_API+void mld_poly_use_hint(mld_poly *a, const mld_poly *h)+{+#if defined(MLD_USE_NATIVE_POLY_USE_HINT_88) && MLD_CONFIG_PARAMETER_SET == 44+  int ret;+  mld_assert_bound(a->coeffs, MLDSA_N, 0, MLDSA_Q);+  mld_assert_bound(h->coeffs, MLDSA_N, 0, 2);+  ret = mld_poly_use_hint_88_native(a->coeffs, h->coeffs);+  if (ret == MLD_NATIVE_FUNC_SUCCESS)+  {+    mld_assert_bound(a->coeffs, MLDSA_N, 0, (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2));+    return;+  }+#elif defined(MLD_USE_NATIVE_POLY_USE_HINT_32) && \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)+  int ret;+  mld_assert_bound(a->coeffs, MLDSA_N, 0, MLDSA_Q);+  mld_assert_bound(h->coeffs, MLDSA_N, 0, 2);+  ret = mld_poly_use_hint_32_native(a->coeffs, h->coeffs);+  if (ret == MLD_NATIVE_FUNC_SUCCESS)+  {+    mld_assert_bound(a->coeffs, MLDSA_N, 0, (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2));+    return;+  }+#endif /* !(MLD_USE_NATIVE_POLY_USE_HINT_88 && MLD_CONFIG_PARAMETER_SET == 44) \+          && MLD_USE_NATIVE_POLY_USE_HINT_32 && (MLD_CONFIG_PARAMETER_SET ==   \+          65 || MLD_CONFIG_PARAMETER_SET == 87) */+  mld_poly_use_hint_c(a, h);+}+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+/**+ * Sample uniformly random coefficients in [-MLDSA_ETA, MLDSA_ETA] by+ * performing rejection sampling on an array of random bytes.+ *+ * @param[out] a      Pointer to output array (allocated).+ * @param      target Requested number of coefficients to sample.+ * @param      offset Number of coefficients already sampled.+ * @param[in]  buf    Array of random bytes to sample from.+ * @param      buflen Length of array of random bytes.+ *+ * @return Number of sampled coefficients. Can be smaller than target if not+ *         enough random bytes were given.+ */++/* Reference: `mld_rej_eta()` in the reference implementation @[REF].+ *            - Our signature differs from the reference implementation+ *              in that it adds the offset and always expects the base of the+ *              target buffer. This avoids shifting the buffer base in the+ *              caller, which appears tricky to reason about. */+#if MLDSA_ETA == 2+/*+ * Sampling 256 coefficients mod 15 using rejection sampling from 4 bits.+ * Expected number of required bytes: (256 * (16/15))/2 = 136.5 bytes.+ * We sample 1 block (=136 bytes) of SHAKE256_RATE output initially.+ * Sampling 2 blocks initially results in slightly worse performance.+ */+#define MLD_POLY_UNIFORM_ETA_NBLOCKS 1+#elif MLDSA_ETA == 4+/*+ * Sampling 256 coefficients mod 9 using rejection sampling from 4 bits.+ * Expected number of required bytes: (256 * (16/9))/2 = 227.5 bytes.+ * We sample 2 blocks (=272 bytes) of SHAKE256_RATE output initially.+ */+#define MLD_POLY_UNIFORM_ETA_NBLOCKS 2+#else /* MLDSA_ETA == 4 */+#error "Invalid value of MLDSA_ETA"+#endif /* MLDSA_ETA != 2 && MLDSA_ETA != 4 */++MLD_STATIC_TESTABLE unsigned int mld_rej_eta_c(int32_t *a, unsigned int target,+                                               unsigned int offset,+                                               const uint8_t *buf,+                                               unsigned int buflen)+__contract__(+  requires(offset <= target && target <= MLDSA_N)+  requires(buflen <= (MLD_POLY_UNIFORM_ETA_NBLOCKS * MLD_STREAM256_BLOCKBYTES))+  requires(memory_no_alias(a, sizeof(int32_t) * target))+  requires(memory_no_alias(buf, buflen))+  requires(array_abs_bound(a, 0, offset, MLDSA_ETA + 1))+  assigns(memory_slice(a, sizeof(int32_t) * target))+  ensures(offset <= return_value && return_value <= target)+  ensures(array_abs_bound(a, 0, return_value, MLDSA_ETA + 1))+)+{+  unsigned int ctr, pos;+  int t_valid;+  uint32_t t0, t1;+  mld_assert_abs_bound(a, offset, MLDSA_ETA + 1);+  ctr = offset;+  pos = 0;+  while (ctr < target && pos < buflen)+  __loop__(+    invariant(offset <= ctr && ctr <= target && pos <= buflen)+    invariant(array_abs_bound(a, 0, ctr, MLDSA_ETA + 1))+    decreases(buflen - pos)+  )+  {+    t0 = buf[pos] & 0x0F;+    t1 = buf[pos++] >> 4;++    /* Constant time: The inputs and outputs to the rejection sampling are+     * secret. However, it is fine to leak which coefficients have been+     * rejected. For constant-time testing, we declassify the result of+     * the comparison.+     */+#if MLDSA_ETA == 2+    t_valid = t0 < 15;+    MLD_CT_TESTING_DECLASSIFY(&t_valid, sizeof(int));+    if (t_valid) /* t0 < 15 */+    {+      t0 = t0 - (205 * t0 >> 10) * 5;+      a[ctr++] = 2 - (int32_t)t0;+    }+    t_valid = t1 < 15;+    MLD_CT_TESTING_DECLASSIFY(&t_valid, sizeof(int));+    if (t_valid && ctr < target) /* t1 < 15 */+    {+      t1 = t1 - (205 * t1 >> 10) * 5;+      a[ctr++] = 2 - (int32_t)t1;+    }+#elif MLDSA_ETA == 4+    t_valid = t0 < 9;+    MLD_CT_TESTING_DECLASSIFY(&t_valid, sizeof(int));+    if (t_valid) /* t0 < 9 */+    {+      a[ctr++] = 4 - (int32_t)t0;+    }+    t_valid = t1 < 9; /* t1 < 9 */+    MLD_CT_TESTING_DECLASSIFY(&t_valid, sizeof(int));+    if (t_valid && ctr < target)+    {+      a[ctr++] = 4 - (int32_t)t1;+    }+#else /* MLDSA_ETA == 4 */+#error "Invalid value of MLDSA_ETA"+#endif /* MLDSA_ETA != 2 && MLDSA_ETA != 4 */+  }++  mld_assert_abs_bound(a, ctr, MLDSA_ETA + 1);++  return ctr;+}++static unsigned int mld_rej_eta(int32_t *a, unsigned int target,+                                unsigned int offset, const uint8_t *buf,+                                unsigned int buflen)+__contract__(+  requires(offset <= target && target <= MLDSA_N)+  requires(buflen <= (MLD_POLY_UNIFORM_ETA_NBLOCKS * MLD_STREAM256_BLOCKBYTES))+  requires(memory_no_alias(a, sizeof(int32_t) * target))+  requires(memory_no_alias(buf, buflen))+  requires(array_abs_bound(a, 0, offset, MLDSA_ETA + 1))+  assigns(memory_slice(a, sizeof(int32_t) * target))+  ensures(offset <= return_value && return_value <= target)+  ensures(array_abs_bound(a, 0, return_value, MLDSA_ETA + 1))+)+{+#if MLDSA_ETA == 2 && defined(MLD_USE_NATIVE_REJ_UNIFORM_ETA2)+  int ret;+  mld_assert_abs_bound(a, offset, MLDSA_ETA + 1);+  if (offset == 0)+  {+    ret = mld_rej_uniform_eta2_native(a, target, buf, buflen);+    if (ret != MLD_NATIVE_FUNC_FALLBACK)+    {+      unsigned res = (unsigned)ret;+      mld_assert_abs_bound(a, res, MLDSA_ETA + 1);+      return res;+    }+  }+#elif MLDSA_ETA == 4 && defined(MLD_USE_NATIVE_REJ_UNIFORM_ETA4)+  int ret;+  mld_assert_abs_bound(a, offset, MLDSA_ETA + 1);+  if (offset == 0)+  {+    ret = mld_rej_uniform_eta4_native(a, target, buf, buflen);+    if (ret != MLD_NATIVE_FUNC_FALLBACK)+    {+      unsigned res = (unsigned)ret;+      mld_assert_abs_bound(a, res, MLDSA_ETA + 1);+      return res;+    }+  }+#endif /* !(MLDSA_ETA == 2 && MLD_USE_NATIVE_REJ_UNIFORM_ETA2) && MLDSA_ETA == \+          4 && MLD_USE_NATIVE_REJ_UNIFORM_ETA4 */++  return mld_rej_eta_c(a, target, offset, buf, buflen);+}++#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY)+MLD_INTERNAL_API+void mld_poly_uniform_eta_4x(mld_poly *r0, mld_poly *r1, mld_poly *r2,+                             mld_poly *r3, const uint8_t seed[MLDSA_CRHBYTES],+                             uint8_t nonce0, uint8_t nonce1, uint8_t nonce2,+                             uint8_t nonce3)+{+  /* Temporary buffers for XOF output before rejection sampling */+  MLD_ALIGN uint8_t buf[4][MLD_ALIGN_UP(MLD_POLY_UNIFORM_ETA_NBLOCKS *+                                        MLD_STREAM256_BLOCKBYTES)];++  MLD_ALIGN uint8_t extseed[4][MLD_ALIGN_UP(MLDSA_CRHBYTES + 2)];++  /* Tracks the number of coefficients we have already sampled */+  unsigned ctr[4];+  mld_xof256_x4_ctx state;+  unsigned buflen;++  mld_memcpy(extseed[0], seed, MLDSA_CRHBYTES);+  mld_memcpy(extseed[1], seed, MLDSA_CRHBYTES);+  mld_memcpy(extseed[2], seed, MLDSA_CRHBYTES);+  mld_memcpy(extseed[3], seed, MLDSA_CRHBYTES);+  extseed[0][MLDSA_CRHBYTES] = nonce0;+  extseed[1][MLDSA_CRHBYTES] = nonce1;+  extseed[2][MLDSA_CRHBYTES] = nonce2;+  extseed[3][MLDSA_CRHBYTES] = nonce3;+  extseed[0][MLDSA_CRHBYTES + 1] = 0;+  extseed[1][MLDSA_CRHBYTES + 1] = 0;+  extseed[2][MLDSA_CRHBYTES + 1] = 0;+  extseed[3][MLDSA_CRHBYTES + 1] = 0;++  mld_xof256_x4_init(&state);+  mld_xof256_x4_absorb(&state, extseed, MLDSA_CRHBYTES + 2);++  /*+   * Initially, squeeze heuristic number of MLD_POLY_UNIFORM_ETA_NBLOCKS.+   * This should generate the coefficients with high probability.+   */+  mld_xof256_x4_squeezeblocks(buf, MLD_POLY_UNIFORM_ETA_NBLOCKS, &state);+  buflen = MLD_POLY_UNIFORM_ETA_NBLOCKS * MLD_STREAM256_BLOCKBYTES;++  ctr[0] = mld_rej_eta(r0->coeffs, MLDSA_N, 0, buf[0], buflen);+  ctr[1] = mld_rej_eta(r1->coeffs, MLDSA_N, 0, buf[1], buflen);+  ctr[2] = mld_rej_eta(r2->coeffs, MLDSA_N, 0, buf[2], buflen);+  ctr[3] = mld_rej_eta(r3->coeffs, MLDSA_N, 0, buf[3], buflen);++  /*+   * So long as not all entries have been generated, squeeze+   * one more block at a time until we're done.+   */+  buflen = MLD_STREAM256_BLOCKBYTES;+  while (ctr[0] < MLDSA_N || ctr[1] < MLDSA_N || ctr[2] < MLDSA_N ||+         ctr[3] < MLDSA_N)+  __loop__(+    assigns(ctr, state, memory_slice(r0, sizeof(mld_poly)),+            memory_slice(r1, sizeof(mld_poly)), memory_slice(r2, sizeof(mld_poly)),+            memory_slice(r3, sizeof(mld_poly)), object_whole(buf[0]),+            object_whole(buf[1]), object_whole(buf[2]),+            object_whole(buf[3]))+    invariant(ctr[0] <= MLDSA_N && ctr[1] <= MLDSA_N)+    invariant(ctr[2] <= MLDSA_N && ctr[3] <= MLDSA_N)+    invariant(array_abs_bound(r0->coeffs, 0, ctr[0], MLDSA_ETA + 1))+    invariant(array_abs_bound(r1->coeffs, 0, ctr[1], MLDSA_ETA + 1))+    invariant(array_abs_bound(r2->coeffs, 0, ctr[2], MLDSA_ETA + 1))+    invariant(array_abs_bound(r3->coeffs, 0, ctr[3], MLDSA_ETA + 1)))+  {+    mld_xof256_x4_squeezeblocks(buf, 1, &state);+    ctr[0] = mld_rej_eta(r0->coeffs, MLDSA_N, ctr[0], buf[0], buflen);+    ctr[1] = mld_rej_eta(r1->coeffs, MLDSA_N, ctr[1], buf[1], buflen);+    ctr[2] = mld_rej_eta(r2->coeffs, MLDSA_N, ctr[2], buf[2], buflen);+    ctr[3] = mld_rej_eta(r3->coeffs, MLDSA_N, ctr[3], buf[3], buflen);+  }++  mld_xof256_x4_release(&state);++  mld_assert_abs_bound(r0->coeffs, MLDSA_N, MLDSA_ETA + 1);+  mld_assert_abs_bound(r1->coeffs, MLDSA_N, MLDSA_ETA + 1);+  mld_assert_abs_bound(r2->coeffs, MLDSA_N, MLDSA_ETA + 1);+  mld_assert_abs_bound(r3->coeffs, MLDSA_N, MLDSA_ETA + 1);++  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(buf, sizeof(buf));+  mld_zeroize(extseed, sizeof(extseed));+}+#else  /* !MLD_CONFIG_SERIAL_FIPS202_ONLY */++MLD_INTERNAL_API+void mld_poly_uniform_eta(mld_poly *r, const uint8_t seed[MLDSA_CRHBYTES],+                          uint8_t nonce)+{+  /* Temporary buffer for XOF output before rejection sampling */+  MLD_ALIGN uint8_t+      buf[MLD_POLY_UNIFORM_ETA_NBLOCKS * MLD_STREAM256_BLOCKBYTES];+  MLD_ALIGN uint8_t extseed[MLDSA_CRHBYTES + 2];++  /* Tracks the number of coefficients we have already sampled */+  unsigned ctr;+  mld_xof256_ctx state;+  unsigned buflen;++  mld_memcpy(extseed, seed, MLDSA_CRHBYTES);+  extseed[MLDSA_CRHBYTES] = nonce;+  extseed[MLDSA_CRHBYTES + 1] = 0;++  mld_xof256_init(&state);+  mld_xof256_absorb_once(&state, extseed, MLDSA_CRHBYTES + 2);++  /*+   * Initially, squeeze heuristic number of MLD_POLY_UNIFORM_ETA_NBLOCKS.+   * This should generate the coefficients with high probability.+   */+  mld_xof256_squeezeblocks(buf, MLD_POLY_UNIFORM_ETA_NBLOCKS, &state);+  buflen = MLD_POLY_UNIFORM_ETA_NBLOCKS * MLD_STREAM256_BLOCKBYTES;++  ctr = mld_rej_eta(r->coeffs, MLDSA_N, 0, buf, buflen);++  /*+   * So long as not all entries have been generated, squeeze+   * one more block at a time until we're done.+   */+  buflen = MLD_STREAM256_BLOCKBYTES;+  while (ctr < MLDSA_N)+  __loop__(+    assigns(ctr, object_whole(&state),+      object_whole(buf), memory_slice(r, sizeof(mld_poly)))+    invariant(ctr <= MLDSA_N)+    invariant(state.pos <= SHAKE256_RATE)+    invariant(array_abs_bound(r->coeffs, 0, ctr, MLDSA_ETA + 1)))+  {+    mld_xof256_squeezeblocks(buf, 1, &state);+    ctr = mld_rej_eta(r->coeffs, MLDSA_N, ctr, buf, buflen);+  }++  mld_xof256_release(&state);++  mld_assert_abs_bound(r->coeffs, MLDSA_N, MLDSA_ETA + 1);++  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(buf, sizeof(buf));+  mld_zeroize(extseed, sizeof(extseed));+}+#endif /* MLD_CONFIG_SERIAL_FIPS202_ONLY */+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+#define MLD_POLY_UNIFORM_GAMMA1_NBLOCKS                       \+  ((MLDSA_POLYZ_PACKEDBYTES + MLD_STREAM256_BLOCKBYTES - 1) / \+   MLD_STREAM256_BLOCKBYTES)++#if MLD_CONFIG_PARAMETER_SET == 65 ||              \+    defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) ||     \+    defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+MLD_INTERNAL_API+void mld_poly_uniform_gamma1(mld_poly *a, const uint8_t seed[MLDSA_CRHBYTES],+                             uint16_t nonce)+{+  MLD_ALIGN uint8_t+      buf[MLD_POLY_UNIFORM_GAMMA1_NBLOCKS * MLD_STREAM256_BLOCKBYTES];+  MLD_ALIGN uint8_t extseed[MLDSA_CRHBYTES + 2];+  mld_xof256_ctx state;++  mld_memcpy(extseed, seed, MLDSA_CRHBYTES);+  extseed[MLDSA_CRHBYTES] = (uint8_t)(nonce & 0xFF);+  extseed[MLDSA_CRHBYTES + 1] = (uint8_t)(nonce >> 8);++  mld_xof256_init(&state);+  mld_xof256_absorb_once(&state, extseed, MLDSA_CRHBYTES + 2);++  mld_xof256_squeezeblocks(buf, MLD_POLY_UNIFORM_GAMMA1_NBLOCKS, &state);+  mld_polyz_unpack(a, buf);++  mld_xof256_release(&state);++  mld_assert_bound(a->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1);++  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(buf, sizeof(buf));+  mld_zeroize(extseed, sizeof(extseed));+}+#endif /* MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_SERIAL_FIPS202_ONLY || \+          MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */+++#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) && \+    (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))+MLD_INTERNAL_API+void mld_poly_uniform_gamma1_4x(mld_poly *r0, mld_poly *r1, mld_poly *r2,+                                mld_poly *r3,+                                const uint8_t seed[MLDSA_CRHBYTES],+                                uint16_t nonce0, uint16_t nonce1,+                                uint16_t nonce2, uint16_t nonce3)+{+  /* Temporary buffers for XOF output before rejection sampling */+  MLD_ALIGN uint8_t buf[4][MLD_ALIGN_UP(MLD_POLY_UNIFORM_GAMMA1_NBLOCKS *+                                        MLD_STREAM256_BLOCKBYTES)];++  MLD_ALIGN uint8_t extseed[4][MLD_ALIGN_UP(MLDSA_CRHBYTES + 2)];++  /* Tracks the number of coefficients we have already sampled */+  mld_xof256_x4_ctx state;++  mld_memcpy(extseed[0], seed, MLDSA_CRHBYTES);+  mld_memcpy(extseed[1], seed, MLDSA_CRHBYTES);+  mld_memcpy(extseed[2], seed, MLDSA_CRHBYTES);+  mld_memcpy(extseed[3], seed, MLDSA_CRHBYTES);+  extseed[0][MLDSA_CRHBYTES] = (uint8_t)(nonce0 & 0xFF);+  extseed[1][MLDSA_CRHBYTES] = (uint8_t)(nonce1 & 0xFF);+  extseed[2][MLDSA_CRHBYTES] = (uint8_t)(nonce2 & 0xFF);+  extseed[3][MLDSA_CRHBYTES] = (uint8_t)(nonce3 & 0xFF);+  extseed[0][MLDSA_CRHBYTES + 1] = (uint8_t)(nonce0 >> 8);+  extseed[1][MLDSA_CRHBYTES + 1] = (uint8_t)(nonce1 >> 8);+  extseed[2][MLDSA_CRHBYTES + 1] = (uint8_t)(nonce2 >> 8);+  extseed[3][MLDSA_CRHBYTES + 1] = (uint8_t)(nonce3 >> 8);++  mld_xof256_x4_init(&state);+  mld_xof256_x4_absorb(&state, extseed, MLDSA_CRHBYTES + 2);+  mld_xof256_x4_squeezeblocks(buf, MLD_POLY_UNIFORM_GAMMA1_NBLOCKS, &state);++  mld_polyz_unpack(r0, buf[0]);+  mld_polyz_unpack(r1, buf[1]);+  mld_polyz_unpack(r2, buf[2]);+  mld_polyz_unpack(r3, buf[3]);+  mld_xof256_x4_release(&state);++  mld_assert_bound(r0->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1);+  mld_assert_bound(r1->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1);+  mld_assert_bound(r2->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1);+  mld_assert_bound(r3->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1);++  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(buf, sizeof(buf));+  mld_zeroize(extseed, sizeof(extseed));+}+#endif /* !MLD_CONFIG_SERIAL_FIPS202_ONLY && (!MLD_CONFIG_REDUCE_RAM || \+          MLD_UNIT_TEST) */+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+MLD_INTERNAL_API+void mld_poly_challenge(mld_poly *c, const uint8_t seed[MLDSA_CTILDEBYTES])+{+  unsigned int i, j, pos;+  uint64_t signs;+  uint64_t offset;+  MLD_ALIGN uint8_t buf[SHAKE256_RATE];+  mld_shake256ctx state;++  mld_shake256_init(&state);+  mld_shake256_absorb(&state, seed, MLDSA_CTILDEBYTES);+  mld_shake256_finalize(&state);+  mld_shake256_squeeze(buf, SHAKE256_RATE, &state);++  /* Convert the first 8 bytes of buf[] into an unsigned 64-bit value.   */+  /* Each bit of that dictates the sign of the resulting challenge value */+  signs = 0;+  for (i = 0; i < 8; ++i)+  __loop__(+    assigns(i, signs)+    invariant(i <= 8)+    decreases(8 - i)+  )+  {+    signs |= (uint64_t)buf[i] << 8 * i;+  }+  pos = 8;++  mld_memset(c, 0, sizeof(mld_poly));++  for (i = MLDSA_N - MLDSA_TAU; i < MLDSA_N; ++i)+  __loop__(+    assigns(i, j, object_whole(buf), state, pos, memory_slice(c, sizeof(mld_poly)), signs)+    invariant(i >= MLDSA_N - MLDSA_TAU)+    invariant(i <= MLDSA_N)+    invariant(pos >= 1)+    invariant(pos <= SHAKE256_RATE)+    invariant(array_bound(c->coeffs, 0, MLDSA_N, -1, 2))+    invariant(state.pos <= SHAKE256_RATE)+    decreases(MLDSA_N - i)+  )+  {+    /* This loop terminates only probabilistically, hence no decreases+     * clause. */+    do+    __loop__(+      assigns(j, object_whole(buf), state, pos)+      invariant(state.pos <= SHAKE256_RATE)+    )+    {+      if (pos >= SHAKE256_RATE)+      {+        mld_shake256_squeeze(buf, SHAKE256_RATE, &state);+        pos = 0;+      }+      j = buf[pos++];+    } while (j > i);++    c->coeffs[i] = c->coeffs[j];++    /* Reference: Compute coefficient value here in two steps to */+    /* avoid mixing unsigned and signed arithmetic with implicit */+    /* conversions, and so that CBMC can keep track of ranges    */+    /* to complete type-safety proof here.                       */++    /* The least-significant bit of signs tells us if we want -1 or +1 */+    offset = 2 * (signs & 1);++    /* offset has value 0 or 2 here, so (1 - (int32_t) offset) has+     * value -1 or +1 */+    c->coeffs[j] = 1 - (int32_t)offset;++    /* Move to the next bit of signs for next time */+    signs >>= 1;+  }++  mld_assert_bound(c->coeffs, MLDSA_N, -1, 2);+  mld_shake256_release(&state);++  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(buf, sizeof(buf));+  mld_zeroize(&signs, sizeof(signs));+}+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+MLD_INTERNAL_API+void mld_polyeta_pack(uint8_t r[MLDSA_POLYETA_PACKEDBYTES], const mld_poly *a)+{+  unsigned int i;+  uint8_t t[8];++  mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_ETA + 1);++#if MLDSA_ETA == 2+  for (i = 0; i < MLDSA_N / 8; ++i)+  __loop__(+    invariant(i <= MLDSA_N/8)+    decreases(MLDSA_N / 8 - i))+  {+    /* The casts are safe since we assume that the coefficients+     * of a are <= MLDSA_ETA in absolute value. */+    t[0] = (uint8_t)(MLDSA_ETA - a->coeffs[8 * i + 0]);+    t[1] = (uint8_t)(MLDSA_ETA - a->coeffs[8 * i + 1]);+    t[2] = (uint8_t)(MLDSA_ETA - a->coeffs[8 * i + 2]);+    t[3] = (uint8_t)(MLDSA_ETA - a->coeffs[8 * i + 3]);+    t[4] = (uint8_t)(MLDSA_ETA - a->coeffs[8 * i + 4]);+    t[5] = (uint8_t)(MLDSA_ETA - a->coeffs[8 * i + 5]);+    t[6] = (uint8_t)(MLDSA_ETA - a->coeffs[8 * i + 6]);+    t[7] = (uint8_t)(MLDSA_ETA - a->coeffs[8 * i + 7]);++    r[3 * i + 0] = (uint8_t)(((t[0] >> 0) | (t[1] << 3) | (t[2] << 6)) & 0xFF);+    r[3 * i + 1] =+        (uint8_t)(((t[2] >> 2) | (t[3] << 1) | (t[4] << 4) | (t[5] << 7)) &+                  0xFF);+    r[3 * i + 2] = (uint8_t)(((t[5] >> 1) | (t[6] << 2) | (t[7] << 5)) & 0xFF);+  }+#elif MLDSA_ETA == 4+  for (i = 0; i < MLDSA_N / 2; ++i)+  __loop__(+    invariant(i <= MLDSA_N/2)+    decreases(MLDSA_N / 2 - i))+  {+    /* The casts are safe since we assume that the coefficients+     * of a are <= MLDSA_ETA in absolute value. */+    t[0] = (uint8_t)(MLDSA_ETA - a->coeffs[2 * i + 0]);+    t[1] = (uint8_t)(MLDSA_ETA - a->coeffs[2 * i + 1]);+    r[i] = (uint8_t)(t[0] | (t[1] << 4));+  }+#else /* MLDSA_ETA == 4 */+#error "Invalid value of MLDSA_ETA"+#endif /* MLDSA_ETA != 2 && MLDSA_ETA != 4 */+}+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_SIGN_API)+MLD_INTERNAL_API+void mld_polyeta_unpack(mld_poly *r, const uint8_t a[MLDSA_POLYETA_PACKEDBYTES])+{+  unsigned int i;++#if MLDSA_ETA == 2+  for (i = 0; i < MLDSA_N / 8; ++i)+  __loop__(+    invariant(i <= MLDSA_N/8)+    invariant(array_bound(r->coeffs, 0, i*8, -5, MLDSA_ETA + 1))+    decreases(MLDSA_N / 8 - i))+  {+    r->coeffs[8 * i + 0] = (a[3 * i + 0] >> 0) & 7;+    r->coeffs[8 * i + 1] = (a[3 * i + 0] >> 3) & 7;+    r->coeffs[8 * i + 2] = ((a[3 * i + 0] >> 6) | (a[3 * i + 1] << 2)) & 7;+    r->coeffs[8 * i + 3] = (a[3 * i + 1] >> 1) & 7;+    r->coeffs[8 * i + 4] = (a[3 * i + 1] >> 4) & 7;+    r->coeffs[8 * i + 5] = ((a[3 * i + 1] >> 7) | (a[3 * i + 2] << 1)) & 7;+    r->coeffs[8 * i + 6] = (a[3 * i + 2] >> 2) & 7;+    r->coeffs[8 * i + 7] = (a[3 * i + 2] >> 5) & 7;++    r->coeffs[8 * i + 0] = MLDSA_ETA - r->coeffs[8 * i + 0];+    r->coeffs[8 * i + 1] = MLDSA_ETA - r->coeffs[8 * i + 1];+    r->coeffs[8 * i + 2] = MLDSA_ETA - r->coeffs[8 * i + 2];+    r->coeffs[8 * i + 3] = MLDSA_ETA - r->coeffs[8 * i + 3];+    r->coeffs[8 * i + 4] = MLDSA_ETA - r->coeffs[8 * i + 4];+    r->coeffs[8 * i + 5] = MLDSA_ETA - r->coeffs[8 * i + 5];+    r->coeffs[8 * i + 6] = MLDSA_ETA - r->coeffs[8 * i + 6];+    r->coeffs[8 * i + 7] = MLDSA_ETA - r->coeffs[8 * i + 7];+  }+#elif MLDSA_ETA == 4+  for (i = 0; i < MLDSA_N / 2; ++i)+  __loop__(+    invariant(i <= MLDSA_N/2)+    invariant(array_bound(r->coeffs, 0, i*2, -11, MLDSA_ETA + 1))+    decreases(MLDSA_N / 2 - i))+  {+    r->coeffs[2 * i + 0] = a[i] & 0x0F;+    r->coeffs[2 * i + 1] = a[i] >> 4;+    r->coeffs[2 * i + 0] = MLDSA_ETA - r->coeffs[2 * i + 0];+    r->coeffs[2 * i + 1] = MLDSA_ETA - r->coeffs[2 * i + 1];+  }+#else /* MLDSA_ETA == 4 */+#error "Invalid value of MLDSA_ETA"+#endif /* MLDSA_ETA != 2 && MLDSA_ETA != 4 */++  mld_assert_bound(r->coeffs, MLDSA_N, MLD_POLYETA_UNPACK_LOWER_BOUND,+                   MLDSA_ETA + 1);+}+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+MLD_INTERNAL_API+void mld_polyz_pack(uint8_t r[MLDSA_POLYZ_PACKEDBYTES], const mld_poly *a)+{+  unsigned int i;+  uint32_t t[4];++  mld_assert_bound(a->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1);++#if MLD_CONFIG_PARAMETER_SET == 44+  for (i = 0; i < MLDSA_N / 4; ++i)+  __loop__(+    invariant(i <= MLDSA_N/4)+    decreases(MLDSA_N / 4 - i))+  {+    /* Safety: a->coeffs[i] <= MLDSA_GAMMA1, hence, these casts are safe. */+    t[0] = (uint32_t)(MLDSA_GAMMA1 - a->coeffs[4 * i + 0]);+    t[1] = (uint32_t)(MLDSA_GAMMA1 - a->coeffs[4 * i + 1]);+    t[2] = (uint32_t)(MLDSA_GAMMA1 - a->coeffs[4 * i + 2]);+    t[3] = (uint32_t)(MLDSA_GAMMA1 - a->coeffs[4 * i + 3]);++    r[9 * i + 0] = (uint8_t)((t[0]) & 0xFF);+    r[9 * i + 1] = (uint8_t)((t[0] >> 8) & 0xFF);+    r[9 * i + 2] = (uint8_t)((t[0] >> 16) & 0xFF);+    r[9 * i + 2] |= (uint8_t)((t[1] << 2) & 0xFF);+    r[9 * i + 3] = (uint8_t)((t[1] >> 6) & 0xFF);+    r[9 * i + 4] = (uint8_t)((t[1] >> 14) & 0xFF);+    r[9 * i + 4] |= (uint8_t)((t[2] << 4) & 0xFF);+    r[9 * i + 5] = (uint8_t)((t[2] >> 4) & 0xFF);+    r[9 * i + 6] = (uint8_t)((t[2] >> 12) & 0xFF);+    r[9 * i + 6] |= (uint8_t)((t[3] << 6) & 0xFF);+    r[9 * i + 7] = (uint8_t)((t[3] >> 2) & 0xFF);+    r[9 * i + 8] = (uint8_t)((t[3] >> 10) & 0xFF);+  }+#else  /* MLD_CONFIG_PARAMETER_SET == 44 */+  for (i = 0; i < MLDSA_N / 2; ++i)+  __loop__(+    invariant(i <= MLDSA_N/2)+    decreases(MLDSA_N / 2 - i))+  {+    /* Safety: a->coeffs[i] <= MLDSA_GAMMA1, hence, these casts are safe. */+    t[0] = (uint32_t)(MLDSA_GAMMA1 - a->coeffs[2 * i + 0]);+    t[1] = (uint32_t)(MLDSA_GAMMA1 - a->coeffs[2 * i + 1]);++    r[5 * i + 0] = (uint8_t)((t[0]) & 0xFF);+    r[5 * i + 1] = (uint8_t)((t[0] >> 8) & 0xFF);+    r[5 * i + 2] = (uint8_t)((t[0] >> 16) & 0xFF);+    r[5 * i + 2] |= (uint8_t)((t[1] << 4) & 0xFF);+    r[5 * i + 3] = (uint8_t)((t[1] >> 4) & 0xFF);+    r[5 * i + 4] = (uint8_t)((t[1] >> 12) & 0xFF);+  }+#endif /* MLD_CONFIG_PARAMETER_SET != 44 */+}+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+MLD_STATIC_TESTABLE void mld_polyz_unpack_c(+    mld_poly *r, const uint8_t a[MLDSA_POLYZ_PACKEDBYTES])+__contract__(+  requires(memory_no_alias(r, sizeof(mld_poly)))+  requires(memory_no_alias(a, MLDSA_POLYZ_PACKEDBYTES))+  assigns(memory_slice(r, sizeof(mld_poly)))+  ensures(array_bound(r->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))+)+{+  unsigned int i;+#if MLD_CONFIG_PARAMETER_SET == 44+  for (i = 0; i < MLDSA_N / 4; ++i)+  __loop__(+    invariant(i <= MLDSA_N/4)+    invariant(array_bound(r->coeffs, 0, i*4, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))+    decreases(MLDSA_N / 4 - i))+  {+    r->coeffs[4 * i + 0] = a[9 * i + 0];+    r->coeffs[4 * i + 0] |= (int32_t)a[9 * i + 1] << 8;+    r->coeffs[4 * i + 0] |= (int32_t)a[9 * i + 2] << 16;+    r->coeffs[4 * i + 0] &= 0x3FFFF;++    r->coeffs[4 * i + 1] = a[9 * i + 2] >> 2;+    r->coeffs[4 * i + 1] |= (int32_t)a[9 * i + 3] << 6;+    r->coeffs[4 * i + 1] |= (int32_t)a[9 * i + 4] << 14;+    r->coeffs[4 * i + 1] &= 0x3FFFF;++    r->coeffs[4 * i + 2] = a[9 * i + 4] >> 4;+    r->coeffs[4 * i + 2] |= (int32_t)a[9 * i + 5] << 4;+    r->coeffs[4 * i + 2] |= (int32_t)a[9 * i + 6] << 12;+    r->coeffs[4 * i + 2] &= 0x3FFFF;++    r->coeffs[4 * i + 3] = a[9 * i + 6] >> 6;+    r->coeffs[4 * i + 3] |= (int32_t)a[9 * i + 7] << 2;+    r->coeffs[4 * i + 3] |= (int32_t)a[9 * i + 8] << 10;+    r->coeffs[4 * i + 3] &= 0x3FFFF;++    r->coeffs[4 * i + 0] = MLDSA_GAMMA1 - r->coeffs[4 * i + 0];+    r->coeffs[4 * i + 1] = MLDSA_GAMMA1 - r->coeffs[4 * i + 1];+    r->coeffs[4 * i + 2] = MLDSA_GAMMA1 - r->coeffs[4 * i + 2];+    r->coeffs[4 * i + 3] = MLDSA_GAMMA1 - r->coeffs[4 * i + 3];+  }+#else  /* MLD_CONFIG_PARAMETER_SET == 44 */+  for (i = 0; i < MLDSA_N / 2; ++i)+  __loop__(+    invariant(i <= MLDSA_N/2)+    invariant(array_bound(r->coeffs, 0, i*2, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))+    decreases(MLDSA_N / 2 - i))+  {+    r->coeffs[2 * i + 0] = a[5 * i + 0];+    r->coeffs[2 * i + 0] |= (int32_t)a[5 * i + 1] << 8;+    r->coeffs[2 * i + 0] |= (int32_t)a[5 * i + 2] << 16;+    r->coeffs[2 * i + 0] &= 0xFFFFF;++    r->coeffs[2 * i + 1] = a[5 * i + 2] >> 4;+    r->coeffs[2 * i + 1] |= (int32_t)a[5 * i + 3] << 4;+    r->coeffs[2 * i + 1] |= (int32_t)a[5 * i + 4] << 12;+    /* r->coeffs[2*i+1] &= 0xFFFFF; */ /* No effect, since we're anyway at 20+                                          bits */++    r->coeffs[2 * i + 0] = MLDSA_GAMMA1 - r->coeffs[2 * i + 0];+    r->coeffs[2 * i + 1] = MLDSA_GAMMA1 - r->coeffs[2 * i + 1];+  }+#endif /* MLD_CONFIG_PARAMETER_SET != 44 */+  mld_assert_bound(r->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1);+}++MLD_INTERNAL_API+void mld_polyz_unpack(mld_poly *r, const uint8_t a[MLDSA_POLYZ_PACKEDBYTES])+{+#if defined(MLD_USE_NATIVE_POLYZ_UNPACK_17) && MLD_CONFIG_PARAMETER_SET == 44+  int ret;+  ret = mld_polyz_unpack_17_native(r->coeffs, a);+  if (ret == MLD_NATIVE_FUNC_SUCCESS)+  {+    mld_assert_bound(r->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1);+    return;+  }+#elif defined(MLD_USE_NATIVE_POLYZ_UNPACK_19) && \+    (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)+  int ret;+  ret = mld_polyz_unpack_19_native(r->coeffs, a);+  if (ret == MLD_NATIVE_FUNC_SUCCESS)+  {+    mld_assert_bound(r->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1);+    return;+  }+#endif /* !(MLD_USE_NATIVE_POLYZ_UNPACK_17 && MLD_CONFIG_PARAMETER_SET == 44)  \+          && MLD_USE_NATIVE_POLYZ_UNPACK_19 && (MLD_CONFIG_PARAMETER_SET == 65 \+          || MLD_CONFIG_PARAMETER_SET == 87) */++  mld_polyz_unpack_c(r, a);+}+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */++/* To facilitate single-compilation-unit (SCU) builds, undefine all macros. */++/* To facilitate single-compilation-unit (SCU) builds, undefine all macros.+ * Don't modify by hand -- this is auto-generated by scripts/autogen. */+#undef mld_rej_eta+#undef mld_rej_eta_c+#undef mld_poly_decompose_c+#undef mld_poly_use_hint_c+#undef mld_polyz_unpack_c+#undef MLD_POLY_UNIFORM_ETA_NBLOCKS+#undef MLD_POLY_UNIFORM_GAMMA1_NBLOCKS
+ cbits/mldsa/src/poly_kl.h view
@@ -0,0 +1,367 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ */++#ifndef MLD_POLY_KL_H+#define MLD_POLY_KL_H++#include "cbmc.h"+#include "common.h"+#include "poly.h"++#if !defined(MLD_CONFIG_NO_SIGN_API)+#define mld_poly_decompose MLD_NAMESPACE_KL(poly_decompose)+/**+ * For all coefficients c of the input polynomial, compute high and low bits+ * c0, c1 such c mod MLDSA_Q = c1*ALPHA + c0 with -ALPHA/2 < c0 <= ALPHA/2+ * except c1 = (MLDSA_Q-1)/ALPHA where we set c1 = 0 and+ * -ALPHA/2 <= c0 = c mod MLDSA_Q - MLDSA_Q < 0. Assumes coefficients to be+ * standard representatives.+ *+ * @reference{The reference implementation has the input polynomial as a+ * separate argument that may be aliased with either of the outputs. Removing+ * the aliasing eases CBMC proofs.}+ *+ * @param[out]    a1 Pointer to output polynomial with coefficients c1.+ * @param[in,out] a0 Pointer to input/output polynomial. Output polynomial has+ *                   coefficients c0.+ */+MLD_INTERNAL_API+void mld_poly_decompose(mld_poly *a1, mld_poly *a0)+__contract__(+  requires(memory_no_alias(a1,  sizeof(mld_poly)))+  requires(memory_no_alias(a0, sizeof(mld_poly)))+  requires(array_bound(a0->coeffs, 0, MLDSA_N, 0, MLDSA_Q))+  assigns(memory_slice(a1, sizeof(mld_poly)))+  assigns(memory_slice(a0, sizeof(mld_poly)))+  ensures(array_bound(a1->coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2)))+  ensures(array_abs_bound(a0->coeffs, 0, MLDSA_N, MLDSA_GAMMA2+1))+);++#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+#define mld_poly_use_hint MLD_NAMESPACE_KL(poly_use_hint)+/**+ * Use hint polynomial h to correct the high bits of a in-place.+ *+ * @param[in,out] a Input/output polynomial.+ * @param[in]     h Hint polynomial.+ */+MLD_INTERNAL_API+void mld_poly_use_hint(mld_poly *a, const mld_poly *h)+__contract__(+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(memory_no_alias(h, sizeof(mld_poly)))+  requires(array_bound(a->coeffs, 0, MLDSA_N, 0, MLDSA_Q))+  requires(array_bound(h->coeffs, 0, MLDSA_N, 0, 2))+  assigns(memory_slice(a, sizeof(mld_poly)))+  ensures(array_bound(a->coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2)))+);+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY)+#define mld_poly_uniform_eta_4x MLD_NAMESPACE_KL(poly_uniform_eta_4x)+/**+ * Sample four polynomials with uniformly random coefficients in+ * [-MLDSA_ETA, MLDSA_ETA] by performing rejection sampling on the output+ * stream from SHAKE256(seed|nonce_i).+ *+ * @spec{Implements @[FIPS204, Algorithm 31, RejBoundedPoly] (four-way+ * batched).}+ *+ * @param[out] r0     Pointer to first output polynomial.+ * @param[out] r1     Pointer to second output polynomial.+ * @param[out] r2     Pointer to third output polynomial.+ * @param[out] r3     Pointer to fourth output polynomial.+ * @param[in]  seed   Byte array with seed of length MLDSA_CRHBYTES.+ * @param      nonce0 First nonce.+ * @param      nonce1 Second nonce.+ * @param      nonce2 Third nonce.+ * @param      nonce3 Fourth nonce.+ */+MLD_INTERNAL_API+void mld_poly_uniform_eta_4x(mld_poly *r0, mld_poly *r1, mld_poly *r2,+                             mld_poly *r3, const uint8_t seed[MLDSA_CRHBYTES],+                             uint8_t nonce0, uint8_t nonce1, uint8_t nonce2,+                             uint8_t nonce3)+__contract__(+  requires(memory_no_alias(r0, sizeof(mld_poly)))+  requires(memory_no_alias(r1, sizeof(mld_poly)))+  requires(memory_no_alias(r2, sizeof(mld_poly)))+  requires(memory_no_alias(r3, sizeof(mld_poly)))+  requires(memory_no_alias(seed, MLDSA_CRHBYTES))+  assigns(memory_slice(r0, sizeof(mld_poly)))+  assigns(memory_slice(r1, sizeof(mld_poly)))+  assigns(memory_slice(r2, sizeof(mld_poly)))+  assigns(memory_slice(r3, sizeof(mld_poly)))+  ensures(array_abs_bound(r0->coeffs, 0, MLDSA_N, MLDSA_ETA + 1))+  ensures(array_abs_bound(r1->coeffs, 0, MLDSA_N, MLDSA_ETA + 1))+  ensures(array_abs_bound(r2->coeffs, 0, MLDSA_N, MLDSA_ETA + 1))+  ensures(array_abs_bound(r3->coeffs, 0, MLDSA_N, MLDSA_ETA + 1))+);+#endif /* !MLD_CONFIG_SERIAL_FIPS202_ONLY */++#if defined(MLD_CONFIG_SERIAL_FIPS202_ONLY)+#define mld_poly_uniform_eta MLD_NAMESPACE_KL(poly_uniform_eta)+/**+ * Sample polynomial with uniformly random coefficients in+ * [-MLDSA_ETA, MLDSA_ETA] by performing rejection sampling on the output+ * stream from SHAKE256(seed|nonce).+ *+ * @spec{Implements @[FIPS204, Algorithm 31, RejBoundedPoly].}+ *+ * @param[out] r     Pointer to output polynomial.+ * @param[in]  seed  Byte array with seed of length MLDSA_CRHBYTES.+ * @param      nonce Nonce.+ */+MLD_INTERNAL_API+void mld_poly_uniform_eta(mld_poly *r, const uint8_t seed[MLDSA_CRHBYTES],+                          uint8_t nonce)+__contract__(+  requires(memory_no_alias(r, sizeof(mld_poly)))+  requires(memory_no_alias(seed, MLDSA_CRHBYTES))+  assigns(memory_slice(r, sizeof(mld_poly)))+  ensures(array_abs_bound(r->coeffs, 0, MLDSA_N, MLDSA_ETA + 1))+);+#endif /* MLD_CONFIG_SERIAL_FIPS202_ONLY */+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+#if MLD_CONFIG_PARAMETER_SET == 65 ||          \+    defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) || \+    defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+#define mld_poly_uniform_gamma1 MLD_NAMESPACE_KL(poly_uniform_gamma1)+/**+ * Sample polynomial with uniformly random coefficients in+ * [-(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1] by unpacking output stream of+ * SHAKE256(seed|nonce).+ *+ * @spec{Partially implements @[FIPS204, Algorithm 34, ExpandMask] (one+ * polynomial, i.e. the loop body of lines 3-5).}+ *+ * @param[out] a     Pointer to output polynomial.+ * @param[in]  seed  Byte array with seed of length MLDSA_CRHBYTES.+ * @param      nonce 16-bit nonce.+ */+MLD_INTERNAL_API+void mld_poly_uniform_gamma1(mld_poly *a, const uint8_t seed[MLDSA_CRHBYTES],+                             uint16_t nonce)+__contract__(+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(memory_no_alias(seed, MLDSA_CRHBYTES))+  assigns(memory_slice(a, sizeof(mld_poly)))+  ensures(array_bound(a->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))+);+#endif /* MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_SERIAL_FIPS202_ONLY || \+          MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) && \+    (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))+#define mld_poly_uniform_gamma1_4x MLD_NAMESPACE_KL(poly_uniform_gamma1_4x)+/**+ * Sample four polynomials with uniformly random coefficients in+ * [-(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1] by unpacking output streams of+ * SHAKE256(seed|nonce_i).+ *+ * @spec{Partially implements @[FIPS204, Algorithm 34, ExpandMask] (four-way+ * batched, i.e. four iterations of the loop body of lines 3-5).}+ *+ * @param[out] r0     Pointer to first output polynomial.+ * @param[out] r1     Pointer to second output polynomial.+ * @param[out] r2     Pointer to third output polynomial.+ * @param[out] r3     Pointer to fourth output polynomial.+ * @param[in]  seed   Byte array with seed of length MLDSA_CRHBYTES.+ * @param      nonce0 First 16-bit nonce.+ * @param      nonce1 Second 16-bit nonce.+ * @param      nonce2 Third 16-bit nonce.+ * @param      nonce3 Fourth 16-bit nonce.+ */+MLD_INTERNAL_API+void mld_poly_uniform_gamma1_4x(mld_poly *r0, mld_poly *r1, mld_poly *r2,+                                mld_poly *r3,+                                const uint8_t seed[MLDSA_CRHBYTES],+                                uint16_t nonce0, uint16_t nonce1,+                                uint16_t nonce2, uint16_t nonce3)+__contract__(+  requires(memory_no_alias(r0, sizeof(mld_poly)))+  requires(memory_no_alias(r1, sizeof(mld_poly)))+  requires(memory_no_alias(r2, sizeof(mld_poly)))+  requires(memory_no_alias(r3, sizeof(mld_poly)))+  requires(memory_no_alias(seed, MLDSA_CRHBYTES))+  assigns(memory_slice(r0, sizeof(mld_poly)))+  assigns(memory_slice(r1, sizeof(mld_poly)))+  assigns(memory_slice(r2, sizeof(mld_poly)))+  assigns(memory_slice(r3, sizeof(mld_poly)))+  ensures(array_bound(r0->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))+  ensures(array_bound(r1->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))+  ensures(array_bound(r2->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))+  ensures(array_bound(r3->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))+);+#endif /* !MLD_CONFIG_SERIAL_FIPS202_ONLY && (!MLD_CONFIG_REDUCE_RAM || \+          MLD_UNIT_TEST) */+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+#define mld_poly_challenge MLD_NAMESPACE_KL(poly_challenge)+/**+ * Samples polynomial with MLDSA_TAU nonzero coefficients in {-1, 1} using the+ * output stream of SHAKE256(seed).+ *+ * @spec{Implements @[FIPS204, Algorithm 29, SampleInBall].}+ *+ * @param[out] c    Pointer to output polynomial.+ * @param[in]  seed Byte array containing seed of length MLDSA_CTILDEBYTES.+ */+MLD_INTERNAL_API+void mld_poly_challenge(mld_poly *c, const uint8_t seed[MLDSA_CTILDEBYTES])+__contract__(+  requires(memory_no_alias(c, sizeof(mld_poly)))+  requires(memory_no_alias(seed, MLDSA_CTILDEBYTES))+  assigns(memory_slice(c, sizeof(mld_poly)))+  /* All coefficients of c are -1, 0 or +1 */+  ensures(array_bound(c->coeffs, 0, MLDSA_N, -1, 2))+);+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+#define mld_polyeta_pack MLD_NAMESPACE_KL(polyeta_pack)+/**+ * Bit-pack polynomial with coefficients in [-MLDSA_ETA, MLDSA_ETA].+ *+ * @spec{Implements @[FIPS204, Algorithm 17, BitPack].}+ *+ * @param[out] r Pointer to output byte array with at least+ *               MLDSA_POLYETA_PACKEDBYTES bytes.+ * @param[in]  a Pointer to input polynomial.+ */+MLD_INTERNAL_API+void mld_polyeta_pack(uint8_t r[MLDSA_POLYETA_PACKEDBYTES], const mld_poly *a)+__contract__(+  requires(memory_no_alias(r, MLDSA_POLYETA_PACKEDBYTES))+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_ETA + 1))+  assigns(memory_slice(r, MLDSA_POLYETA_PACKEDBYTES))+);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_SIGN_API)+/*+ * polyeta_unpack produces coefficients in [-MLDSA_ETA, MLDSA_ETA] for+ * well-formed inputs (i.e., those produced by polyeta_pack).+ * However, when passed an arbitrary byte array, it may produce smaller values,+ * i.e., values in [MLD_POLYETA_UNPACK_LOWER_BOUND, MLDSA_ETA].+ * Even though this should never happen, we use use the bound for arbitrary+ * inputs in the CBMC proofs.+ */+#if MLDSA_ETA == 2+#define MLD_POLYETA_UNPACK_LOWER_BOUND (-5)+#elif MLDSA_ETA == 4+#define MLD_POLYETA_UNPACK_LOWER_BOUND (-11)+#else+#error "Invalid value of MLDSA_ETA"+#endif++#define mld_polyeta_unpack MLD_NAMESPACE_KL(polyeta_unpack)+/**+ * Unpack polynomial with coefficients in [-MLDSA_ETA, MLDSA_ETA].+ *+ * @spec{Implements @[FIPS204, Algorithm 19, BitUnpack].}+ *+ * @param[out] r Pointer to output polynomial.+ * @param[in]  a Byte array with bit-packed polynomial.+ */+MLD_INTERNAL_API+void mld_polyeta_unpack(mld_poly *r, const uint8_t a[MLDSA_POLYETA_PACKEDBYTES])+__contract__(+  requires(memory_no_alias(r, sizeof(mld_poly)))+  requires(memory_no_alias(a, MLDSA_POLYETA_PACKEDBYTES))+  assigns(memory_slice(r, sizeof(mld_poly)))+  ensures(array_bound(r->coeffs, 0, MLDSA_N, MLD_POLYETA_UNPACK_LOWER_BOUND, MLDSA_ETA + 1))+);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+#define mld_polyz_pack MLD_NAMESPACE_KL(polyz_pack)+/**+ * Bit-pack polynomial with coefficients in+ * [-(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1].+ *+ * @spec{Implements @[FIPS204, Algorithm 17, BitPack].}+ *+ * @param[out] r Pointer to output byte array with at least+ *               MLDSA_POLYZ_PACKEDBYTES bytes.+ * @param[in]  a Pointer to input polynomial.+ */+MLD_INTERNAL_API+void mld_polyz_pack(uint8_t r[MLDSA_POLYZ_PACKEDBYTES], const mld_poly *a)+__contract__(+  requires(memory_no_alias(r, MLDSA_POLYZ_PACKEDBYTES))+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(array_bound(a->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))+  assigns(memory_slice(r, MLDSA_POLYZ_PACKEDBYTES))+);+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+#define mld_polyz_unpack MLD_NAMESPACE_KL(polyz_unpack)+/**+ * Unpack polynomial z with coefficients in+ * [-(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1].+ *+ * @spec{Implements @[FIPS204, Algorithm 19, BitUnpack].}+ *+ * @param[out] r Pointer to output polynomial.+ * @param[in]  a Byte array with bit-packed polynomial.+ */+MLD_INTERNAL_API+void mld_polyz_unpack(mld_poly *r, const uint8_t a[MLDSA_POLYZ_PACKEDBYTES])+__contract__(+  requires(memory_no_alias(r, sizeof(mld_poly)))+  requires(memory_no_alias(a, MLDSA_POLYZ_PACKEDBYTES))+  assigns(memory_slice(r, sizeof(mld_poly)))+  ensures(array_bound(r->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))+);++#define mld_polyw1_pack MLD_NAMESPACE_KL(polyw1_pack)+/**+ * Bit-pack polynomial w1. Input coefficients must be in+ * [0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2)), i.e. [0, 43] for ML-DSA-44 and [0, 15]+ * for ML-DSA-65/87. Dispatches to the value-specialized variant for the+ * selected parameter set.+ *+ * @spec{Implements @[FIPS204, Algorithm 16, SimpleBitPack].}+ *+ * @param[out] r Pointer to output byte array with at least+ *               MLDSA_POLYW1_PACKEDBYTES bytes.+ * @param[in]  a Pointer to input polynomial.+ */+static MLD_INLINE void mld_polyw1_pack(uint8_t r[MLDSA_POLYW1_PACKEDBYTES],+                                       const mld_poly *a)+__contract__(+  requires(memory_no_alias(r, MLDSA_POLYW1_PACKEDBYTES))+  requires(memory_no_alias(a, sizeof(mld_poly)))+  requires(array_bound(a->coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2)))+  assigns(memory_slice(r, MLDSA_POLYW1_PACKEDBYTES))+)+{+#if MLD_CONFIG_PARAMETER_SET == 44+  mld_polyw1_pack_88(r, a);+#else+  mld_polyw1_pack_32(r, a);+#endif+}+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */++#endif /* !MLD_POLY_KL_H */
+ cbits/mldsa/src/polyvec.c view
@@ -0,0 +1,509 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ */++#include "polyvec.h"++#include "debug.h"+#include "polyvec_lazy.h"++/* This namespacing is not done at the top to avoid a naming conflict+ * with native backends, which are currently not yet namespaced. */+#define mld_polyvecl_pointwise_acc_montgomery_c \+  MLD_ADD_PARAM_SET(mld_polyvecl_pointwise_acc_montgomery_c)++/**************************************************************/+/************ Vectors of polynomials of length MLDSA_L **************/+/**************************************************************/+#if !defined(MLD_CONFIG_NO_SIGN_API) && \+    (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))+MLD_INTERNAL_API+void mld_polyvecl_uniform_gamma1(mld_polyvecl *v,+                                 const uint8_t seed[MLDSA_CRHBYTES],+                                 uint16_t kappa)+{+#if defined(MLD_CONFIG_SERIAL_FIPS202_ONLY)+  int i;+#endif++  /* The caller passes the base counter kappa; component i is sampled from+   * kappa + i. Safety: kappa <= MLD_MAX_KAPPA and i < MLDSA_L, so the+   * casts below are safe. See MLD_MAX_KAPPA comment in params.h. */+#if defined(MLD_CONFIG_SERIAL_FIPS202_ONLY)+  for (i = 0; i < MLDSA_L; i++)+  {+    mld_poly_uniform_gamma1(&v->vec[i], seed, (uint16_t)(kappa + i));+  }+#else /* MLD_CONFIG_SERIAL_FIPS202_ONLY */+#if MLDSA_L == 4+  mld_poly_uniform_gamma1_4x(&v->vec[0], &v->vec[1], &v->vec[2], &v->vec[3],+                             seed, kappa, (uint16_t)(kappa + 1),+                             (uint16_t)(kappa + 2), (uint16_t)(kappa + 3));+#elif MLDSA_L == 5+  mld_poly_uniform_gamma1_4x(&v->vec[0], &v->vec[1], &v->vec[2], &v->vec[3],+                             seed, kappa, (uint16_t)(kappa + 1),+                             (uint16_t)(kappa + 2), (uint16_t)(kappa + 3));+  mld_poly_uniform_gamma1(&v->vec[4], seed, (uint16_t)(kappa + 4));+#elif MLDSA_L == 7+  mld_poly_uniform_gamma1_4x(&v->vec[0], &v->vec[1], &v->vec[2],+                             &v->vec[3 /* irrelevant */], seed, kappa,+                             (uint16_t)(kappa + 1), (uint16_t)(kappa + 2),+                             0xFF /* irrelevant */);+  mld_poly_uniform_gamma1_4x(&v->vec[3], &v->vec[4], &v->vec[5], &v->vec[6],+                             seed, (uint16_t)(kappa + 3), (uint16_t)(kappa + 4),+                             (uint16_t)(kappa + 5), (uint16_t)(kappa + 6));+#endif /* MLDSA_L == 7 */+#endif /* !MLD_CONFIG_SERIAL_FIPS202_ONLY */++  mld_assert_bound_2d(v->vec, MLDSA_L, MLDSA_N, -(MLDSA_GAMMA1 - 1),+                      MLDSA_GAMMA1 + 1);+}+#endif /* !MLD_CONFIG_NO_SIGN_API && (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) \+        */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) ||                                  \+    !defined(MLD_CONFIG_NO_VERIFY_API) ||                                   \+    (!defined(MLD_CONFIG_NO_SIGN_API) &&                                    \+     (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)))+MLD_INTERNAL_API+void mld_polyvecl_ntt(mld_polyvecl *v)+{+  unsigned int i;+  mld_assert_abs_bound_2d(v->vec, MLDSA_L, MLDSA_N, MLDSA_Q);++  for (i = 0; i < MLDSA_L; ++i)+  __loop__(+    assigns(i, memory_slice(v, sizeof(mld_polyvecl)))+    invariant(i <= MLDSA_L)+    invariant(forall(k0, i, MLDSA_L, forall(k1, 0, MLDSA_N, v->vec[k0].coeffs[k1] == loop_entry(*v).vec[k0].coeffs[k1])))+    invariant(forall(k1, 0, i, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+    decreases(MLDSA_L - i))+  {+    mld_poly_ntt(&v->vec[i]);+  }++  mld_assert_abs_bound_2d(v->vec, MLDSA_L, MLDSA_N, MLD_NTT_BOUND);+}+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_VERIFY_API || \+          (!MLD_CONFIG_NO_SIGN_API && (!MLD_CONFIG_REDUCE_RAM ||     \+          MLD_UNIT_TEST)) */++#if !defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+MLD_STATIC_TESTABLE void mld_polyvecl_pointwise_acc_montgomery_c(+    mld_poly *w, const mld_polyvecl *u, const mld_polyvecl *v)+__contract__(+  requires(memory_no_alias(w, sizeof(mld_poly)))+  requires(memory_no_alias(u, sizeof(mld_polyvecl)))+  requires(memory_no_alias(v, sizeof(mld_polyvecl)))+  requires(forall(l0, 0, MLDSA_L,+                  array_bound(u->vec[l0].coeffs, 0, MLDSA_N, 0, MLDSA_Q)))+  requires(forall(l1, 0, MLDSA_L,+    array_abs_bound(v->vec[l1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+  assigns(memory_slice(w, sizeof(mld_poly)))+  ensures(array_abs_bound(w->coeffs, 0, MLDSA_N, MLDSA_Q))+)+{+  unsigned int i, j;+  mld_assert_bound_2d(u->vec, MLDSA_L, MLDSA_N, 0, MLDSA_Q);+  mld_assert_abs_bound_2d(v->vec, MLDSA_L, MLDSA_N, MLD_NTT_BOUND);+  for (i = 0; i < MLDSA_N; i++)+  __loop__(+    assigns(i, j, memory_slice(w, sizeof(mld_poly)))+    invariant(i <= MLDSA_N)+    invariant(array_abs_bound(w->coeffs, 0, i, MLDSA_Q))+    decreases(MLDSA_N - i)+  )+  {+    int64_t t = 0;+    int32_t r;+    for (j = 0; j < MLDSA_L; j++)+    __loop__(+      assigns(j, t)+      invariant(j <= MLDSA_L)+      invariant(t >= -(int64_t)j*(MLDSA_Q - 1)*(MLD_NTT_BOUND - 1))+      invariant(t <= (int64_t)j*(MLDSA_Q - 1)*(MLD_NTT_BOUND - 1))+      decreases(MLDSA_L - j)+    )+    {+      t += (int64_t)u->vec[j].coeffs[i] * v->vec[j].coeffs[i];+    }++    r = mld_montgomery_reduce(t);+    w->coeffs[i] = r;+  }++  mld_assert_abs_bound(w->coeffs, MLDSA_N, MLDSA_Q);+}++MLD_INTERNAL_API+void mld_polyvecl_pointwise_acc_montgomery(mld_poly *w, const mld_polyvecl *u,+                                           const mld_polyvecl *v)+{+#if defined(MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4) && \+    MLD_CONFIG_PARAMETER_SET == 44+  int ret;+  mld_assert_bound_2d(u->vec, MLDSA_L, MLDSA_N, 0, MLDSA_Q);+  mld_assert_abs_bound_2d(v->vec, MLDSA_L, MLDSA_N, MLD_NTT_BOUND);+  ret = mld_polyvecl_pointwise_acc_montgomery_l4_native(+      w->coeffs, (const int32_t (*)[MLDSA_N])u->vec,+      (const int32_t (*)[MLDSA_N])v->vec);+  if (ret == MLD_NATIVE_FUNC_SUCCESS)+  {+    mld_assert_abs_bound(w->coeffs, MLDSA_N, MLDSA_Q);+    return;+  }+#elif defined(MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5) && \+    MLD_CONFIG_PARAMETER_SET == 65+  int ret;+  mld_assert_bound_2d(u->vec, MLDSA_L, MLDSA_N, 0, MLDSA_Q);+  mld_assert_abs_bound_2d(v->vec, MLDSA_L, MLDSA_N, MLD_NTT_BOUND);+  ret = mld_polyvecl_pointwise_acc_montgomery_l5_native(+      w->coeffs, (const int32_t (*)[MLDSA_N])u->vec,+      (const int32_t (*)[MLDSA_N])v->vec);+  if (ret == MLD_NATIVE_FUNC_SUCCESS)+  {+    mld_assert_abs_bound(w->coeffs, MLDSA_N, MLDSA_Q);+    return;+  }+#elif defined(MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7) && \+    MLD_CONFIG_PARAMETER_SET == 87+  int ret;+  mld_assert_bound_2d(u->vec, MLDSA_L, MLDSA_N, 0, MLDSA_Q);+  mld_assert_abs_bound_2d(v->vec, MLDSA_L, MLDSA_N, MLD_NTT_BOUND);+  ret = mld_polyvecl_pointwise_acc_montgomery_l7_native(+      w->coeffs, (const int32_t (*)[MLDSA_N])u->vec,+      (const int32_t (*)[MLDSA_N])v->vec);+  if (ret == MLD_NATIVE_FUNC_SUCCESS)+  {+    mld_assert_abs_bound(w->coeffs, MLDSA_N, MLDSA_Q);+    return;+  }+#endif /* !(MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4 && \+          MLD_CONFIG_PARAMETER_SET == 44) &&                       \+          !(MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5 && \+          MLD_CONFIG_PARAMETER_SET == 65) &&                       \+          MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7 &&   \+          MLD_CONFIG_PARAMETER_SET == 87 */+  /* The first input is bounded by [0, MLDSA_Q-1] inclusive.+   * The second input is bounded by [-(MLD_NTT_BOUND-1), MLD_NTT_BOUND-1].+   * Hence, we can safely accumulate in 64-bits without intermediate reductions+   * as MLDSA_L * (MLD_NTT_BOUND-1) * (MLDSA_Q-1) < INT64_MAX.+   *+   * The worst case is ML-DSA-87: 7 * (MLD_NTT_BOUND-1) * (MLDSA_Q-1) < 2**53+   * (and likewise for negative values).+   */+  mld_polyvecl_pointwise_acc_montgomery_c(w, u, v);+}+#endif /* !MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_VERIFY_API) || \+    defined(MLD_UNIT_TEST)+MLD_INTERNAL_API+uint32_t mld_polyvecl_chknorm(const mld_polyvecl *v, int32_t bound)+{+  unsigned int i;+  uint32_t t = 0;+  mld_assert_bound_2d(v->vec, MLDSA_L, MLDSA_N, -MLD_REDUCE32_RANGE_MAX,+                      MLD_REDUCE32_RANGE_MAX);++  for (i = 0; i < MLDSA_L; ++i)+  __loop__(+    invariant(i <= MLDSA_L)+    invariant(t == 0 || t == 0xFFFFFFFF)+    invariant((t == 0) == forall(k1, 0, i, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, bound)))+    decreases(MLDSA_L - i)+  )+  {+    /* Reference: Leaks which polynomial violates the bound via a conditional.+     * We are more conservative to reduce the number of declassifications in+     * constant-time testing.+     */+    t |= mld_poly_chknorm(&v->vec[i], bound);+  }+  return t;+}+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_VERIFY_API || \+          MLD_UNIT_TEST */++/**************************************************************/+/************ Vectors of polynomials of length MLDSA_K **************/+/**************************************************************/+#if (!defined(MLD_CONFIG_NO_SIGN_API) &&                       \+     defined(MLD_CONFIG_REDUCE_RAM)) ||                        \+    defined(MLD_UNIT_TEST)+MLD_INTERNAL_API+void mld_polyveck_reduce(mld_polyveck *v)+{+  unsigned int i;+  mld_assert_bound_2d(v->vec, MLDSA_K, MLDSA_N, INT32_MIN,+                      MLD_REDUCE32_DOMAIN_MAX);++  for (i = 0; i < MLDSA_K; ++i)+  __loop__(+    assigns(i, memory_slice(v, sizeof(mld_polyveck)))+    invariant(i <= MLDSA_K)+    invariant(forall(k0, i, MLDSA_K, forall(k1, 0, MLDSA_N, v->vec[k0].coeffs[k1] == loop_entry(*v).vec[k0].coeffs[k1])))+    invariant(forall(k2, 0, i,+      array_bound(v->vec[k2].coeffs, 0, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX)))+    decreases(MLDSA_K - i)+  )+  {+    mld_poly_reduce(&v->vec[i]);+  }++  mld_assert_bound_2d(v->vec, MLDSA_K, MLDSA_N, -MLD_REDUCE32_RANGE_MAX,+                      MLD_REDUCE32_RANGE_MAX);+}+#endif /* (!MLD_CONFIG_NO_SIGN_API && MLD_CONFIG_REDUCE_RAM) || MLD_UNIT_TEST \+        */++#if !defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST)+MLD_INTERNAL_API+void mld_polyveck_caddq(mld_polyveck *v)+{+  unsigned int i;+  mld_assert_abs_bound_2d(v->vec, MLDSA_K, MLDSA_N, MLDSA_Q);++  for (i = 0; i < MLDSA_K; ++i)+  __loop__(+    assigns(i, memory_slice(v, sizeof(mld_polyveck)))+    invariant(i <= MLDSA_K)+    invariant(forall(k0, i, MLDSA_K, forall(k1, 0, MLDSA_N, v->vec[k0].coeffs[k1] == loop_entry(*v).vec[k0].coeffs[k1])))+    invariant(forall(k1, 0, i, array_bound(v->vec[k1].coeffs, 0, MLDSA_N, 0, MLDSA_Q)))+    decreases(MLDSA_K - i))+  {+    mld_poly_caddq(&v->vec[i]);+  }++  mld_assert_bound_2d(v->vec, MLDSA_K, MLDSA_N, 0, MLDSA_Q);+}+#endif /* !MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST */++#if (!defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST)) && \+    (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))+MLD_INTERNAL_API+void mld_polyveck_ntt(mld_polyveck *v)+{+  unsigned int i;+  mld_assert_abs_bound_2d(v->vec, MLDSA_K, MLDSA_N, MLDSA_Q);++  for (i = 0; i < MLDSA_K; ++i)+  __loop__(+    assigns(i, memory_slice(v, sizeof(mld_polyveck)))+    invariant(i <= MLDSA_K)+    invariant(forall(k0, i, MLDSA_K, forall(k1, 0, MLDSA_N, v->vec[k0].coeffs[k1] == loop_entry(*v).vec[k0].coeffs[k1])))+    invariant(forall(k1, 0, i, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+    decreases(MLDSA_K - i))+  {+    mld_poly_ntt(&v->vec[i]);+  }+  mld_assert_abs_bound_2d(v->vec, MLDSA_K, MLDSA_N, MLD_NTT_BOUND);+}+#endif /* (!MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST) && \+          (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) */++#if !defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST)+MLD_INTERNAL_API+void mld_polyveck_invntt_tomont(mld_polyveck *v)+{+  unsigned int i;+  mld_assert_abs_bound_2d(v->vec, MLDSA_K, MLDSA_N, MLDSA_Q);++  for (i = 0; i < MLDSA_K; ++i)+  __loop__(+    assigns(i, memory_slice(v, sizeof(mld_polyveck)))+    invariant(i <= MLDSA_K)+    invariant(forall(k0, i, MLDSA_K, forall(k1, 0, MLDSA_N, v->vec[k0].coeffs[k1] == loop_entry(*v).vec[k0].coeffs[k1])))+    invariant(forall(k1, 0, i, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, MLD_INTT_BOUND)))+    decreases(MLDSA_K - i))+  {+    mld_poly_invntt_tomont(&v->vec[i]);+  }++  mld_assert_abs_bound_2d(v->vec, MLDSA_K, MLDSA_N, MLD_INTT_BOUND);+}+#endif /* !MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+MLD_INTERNAL_API+uint32_t mld_polyveck_chknorm(const mld_polyveck *v, int32_t bound)+{+  unsigned int i;+  uint32_t t = 0;+  mld_assert_bound_2d(v->vec, MLDSA_K, MLDSA_N, -MLD_REDUCE32_RANGE_MAX,+                      MLD_REDUCE32_RANGE_MAX);++  for (i = 0; i < MLDSA_K; ++i)+  __loop__(+    invariant(i <= MLDSA_K)+    invariant(t == 0 || t == 0xFFFFFFFF)+    invariant((t == 0) == forall(k1, 0, i, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, bound)))+    decreases(MLDSA_K - i)+  )+  {+    /* Reference: Leaks which polynomial violates the bound via a conditional.+     * We are more conservative to reduce the number of declassifications in+     * constant-time testing.+     */+    t |= mld_poly_chknorm(&v->vec[i], bound);+  }++  return t;+}++#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+MLD_INTERNAL_API+void mld_polyveck_decompose(mld_polyveck *v1, mld_polyveck *v0)+{+  unsigned int i;+  mld_assert_bound_2d(v0->vec, MLDSA_K, MLDSA_N, 0, MLDSA_Q);++  for (i = 0; i < MLDSA_K; ++i)+  __loop__(+    assigns(i, memory_slice(v0, sizeof(mld_polyveck)), memory_slice(v1, sizeof(mld_polyveck)))+    invariant(i <= MLDSA_K)+    invariant(forall(k1, 0, i,+                     array_bound(v1->vec[k1].coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2))))+    invariant(forall(k2, 0, i,+                     array_abs_bound(v0->vec[k2].coeffs, 0, MLDSA_N, MLDSA_GAMMA2+1)))+    invariant(forall(k3, i, MLDSA_K,+                     array_bound(v0->vec[k3].coeffs, 0, MLDSA_N, 0, MLDSA_Q)))+    decreases(MLDSA_K - i)+  )+  {+    mld_poly_decompose(&v1->vec[i], &v0->vec[i]);+  }++  mld_assert_bound_2d(v1->vec, MLDSA_K, MLDSA_N, 0,+                      (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2));+  mld_assert_abs_bound_2d(v0->vec, MLDSA_K, MLDSA_N, MLDSA_GAMMA2 + 1);+}+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+MLD_INTERNAL_API+void mld_polyveck_pack_w1(uint8_t r[MLDSA_K * MLDSA_POLYW1_PACKEDBYTES],+                          const mld_polyveck *w1)+{+  unsigned int i;+  mld_assert_bound_2d(w1->vec, MLDSA_K, MLDSA_N, 0,+                      (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2));++  for (i = 0; i < MLDSA_K; ++i)+  __loop__(+    assigns(i, memory_slice(r, MLDSA_K * MLDSA_POLYW1_PACKEDBYTES))+    invariant(i <= MLDSA_K)+    decreases(MLDSA_K - i)+  )+  {+    mld_polyw1_pack(&r[i * MLDSA_POLYW1_PACKEDBYTES], &w1->vec[i]);+  }+}+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+MLD_INTERNAL_API+void mld_polyveck_pack_eta(uint8_t r[MLDSA_K * MLDSA_POLYETA_PACKEDBYTES],+                           const mld_polyveck *p)+{+  unsigned int i;+  mld_assert_abs_bound_2d(p->vec, MLDSA_K, MLDSA_N, MLDSA_ETA + 1);+  for (i = 0; i < MLDSA_K; ++i)+  __loop__(+    assigns(i, memory_slice(r, MLDSA_K * MLDSA_POLYETA_PACKEDBYTES))+    invariant(i <= MLDSA_K)+    decreases(MLDSA_K - i)+  )+  {+    mld_polyeta_pack(&r[i * MLDSA_POLYETA_PACKEDBYTES], &p->vec[i]);+  }+}++MLD_INTERNAL_API+void mld_polyvecl_pack_eta(uint8_t r[MLDSA_L * MLDSA_POLYETA_PACKEDBYTES],+                           const mld_polyvecl *p)+{+  unsigned int i;+  mld_assert_abs_bound_2d(p->vec, MLDSA_L, MLDSA_N, MLDSA_ETA + 1);+  for (i = 0; i < MLDSA_L; ++i)+  __loop__(+    assigns(i, memory_slice(r, MLDSA_L * MLDSA_POLYETA_PACKEDBYTES))+    invariant(i <= MLDSA_L)+    decreases(MLDSA_L - i)+  )+  {+    mld_polyeta_pack(&r[i * MLDSA_POLYETA_PACKEDBYTES], &p->vec[i]);+  }+}++#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) ||                                  \+    (!defined(MLD_CONFIG_NO_SIGN_API) &&                                    \+     (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)))+MLD_INTERNAL_API+void mld_polyvecl_unpack_eta(+    mld_polyvecl *p, const uint8_t r[MLDSA_L * MLDSA_POLYETA_PACKEDBYTES])+{+  unsigned int i;+  for (i = 0; i < MLDSA_L; ++i)+  {+    mld_polyeta_unpack(&p->vec[i], r + i * MLDSA_POLYETA_PACKEDBYTES);+  }++  mld_assert_bound_2d(p->vec, MLDSA_L, MLDSA_N, MLD_POLYETA_UNPACK_LOWER_BOUND,+                      MLDSA_ETA + 1);+}+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || (!MLD_CONFIG_NO_SIGN_API && \+          (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST)) */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+MLD_INTERNAL_API+void mld_polyvecl_unpack_z(mld_polyvecl *z,+                           const uint8_t r[MLDSA_L * MLDSA_POLYZ_PACKEDBYTES])+{+  unsigned int i;+  for (i = 0; i < MLDSA_L; ++i)+  {+    mld_polyz_unpack(&z->vec[i], r + i * MLDSA_POLYZ_PACKEDBYTES);+  }++  mld_assert_bound_2d(z->vec, MLDSA_L, MLDSA_N, -(MLDSA_GAMMA1 - 1),+                      MLDSA_GAMMA1 + 1);+}+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) ||                                  \+    (!defined(MLD_CONFIG_NO_SIGN_API) &&                                    \+     (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)))+MLD_INTERNAL_API+void mld_polyveck_unpack_eta(+    mld_polyveck *p, const uint8_t r[MLDSA_K * MLDSA_POLYETA_PACKEDBYTES])+{+  unsigned int i;+  for (i = 0; i < MLDSA_K; ++i)+  {+    mld_polyeta_unpack(&p->vec[i], r + i * MLDSA_POLYETA_PACKEDBYTES);+  }++  mld_assert_bound_2d(p->vec, MLDSA_K, MLDSA_N, MLD_POLYETA_UNPACK_LOWER_BOUND,+                      MLDSA_ETA + 1);+}+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || (!MLD_CONFIG_NO_SIGN_API && \+          (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST)) */++/* To facilitate single-compilation-unit (SCU) builds, undefine all macros.+ * Don't modify by hand -- this is auto-generated by scripts/autogen. */+#undef mld_polyvecl_pointwise_acc_montgomery_c
+ cbits/mldsa/src/polyvec.h view
@@ -0,0 +1,435 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ */++#ifndef MLD_POLYVEC_H+#define MLD_POLYVEC_H++#include "cbmc.h"+#include "common.h"+#include "poly.h"+#include "poly_kl.h"++/* Parameter set namespacing+ * This is to facilitate building multiple instances+ * of mldsa-native (e.g. with varying parameter sets)+ * within a single compilation unit. */+#define mld_polyvecl MLD_ADD_PARAM_SET(mld_polyvecl)+#define mld_polyveck MLD_ADD_PARAM_SET(mld_polyveck)+/* End of parameter set namespacing */++/** Vector of MLDSA_L polynomials. */+typedef struct+{+  mld_poly vec[MLDSA_L]; /**< Component polynomials. */+} mld_polyvecl;+++#if !defined(MLD_CONFIG_NO_SIGN_API) && \+    (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))+#define mld_polyvecl_uniform_gamma1 MLD_NAMESPACE_KL(polyvecl_uniform_gamma1)+/**+ * Sample vector of polynomials with uniformly random coefficients in+ * [-(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1] by unpacking output stream of+ * SHAKE256(seed|kappa+i) for component i.+ *+ * @spec{Implements @[FIPS204, Algorithm 34, ExpandMask].}+ *+ * @param[out] v     Pointer to output vector.+ * @param[in]  seed  Byte array with seed of length MLDSA_CRHBYTES.+ * @param      kappa Base counter; component i uses kappa + i.+ */+MLD_INTERNAL_API+void mld_polyvecl_uniform_gamma1(mld_polyvecl *v,+                                 const uint8_t seed[MLDSA_CRHBYTES],+                                 uint16_t kappa)+__contract__(+  requires(memory_no_alias(v, sizeof(mld_polyvecl)))+  requires(memory_no_alias(seed, MLDSA_CRHBYTES))+  requires(kappa <= MLD_MAX_KAPPA)+  assigns(memory_slice(v, sizeof(mld_polyvecl)))+  ensures(forall(k0, 0, MLDSA_L,+    array_bound(v->vec[k0].coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)))+);+#endif /* !MLD_CONFIG_NO_SIGN_API && (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) \+        */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || \+    !defined(MLD_CONFIG_NO_VERIFY_API) ||  \+    (!defined(MLD_CONFIG_NO_SIGN_API) &&   \+     (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)))+#define mld_polyvecl_ntt MLD_NAMESPACE_KL(polyvecl_ntt)+/**+ * Forward NTT of all polynomials in vector of length MLDSA_L. Output+ * coefficients are bounded by MLD_NTT_BOUND in absolute value.+ *+ * @param[in,out] v Pointer to input/output vector.+ */+MLD_INTERNAL_API+void mld_polyvecl_ntt(mld_polyvecl *v)+__contract__(+  requires(memory_no_alias(v, sizeof(mld_polyvecl)))+  requires(forall(k0, 0, MLDSA_L, array_abs_bound(v->vec[k0].coeffs, 0, MLDSA_N, MLDSA_Q)))+  assigns(memory_slice(v, sizeof(mld_polyvecl)))+  ensures(forall(k1, 0, MLDSA_L, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_VERIFY_API || \+          (!MLD_CONFIG_NO_SIGN_API && (!MLD_CONFIG_REDUCE_RAM ||     \+          MLD_UNIT_TEST)) */++#if !defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+#define mld_polyvecl_pointwise_acc_montgomery \+  MLD_NAMESPACE_KL(polyvecl_pointwise_acc_montgomery)+/**+ * Pointwise multiply vectors of polynomials of length MLDSA_L, multiply+ * resulting vector by 2^{-32} and add (accumulate) polynomials in it.+ * Input/output vectors are in NTT domain representation.+ *+ * The first input "u" must be the output of polyvec_matrix_expand() and so+ * have coefficients in [0, MLDSA_Q-1] inclusive.+ *+ * The second input "v" is assumed to be output of an NTT, and hence must have+ * coefficients bounded by [-(MLD_NTT_BOUND-1), MLD_NTT_BOUND-1] inclusive.+ *+ * @spec{Partially implements @[FIPS204, Algorithm 48, MatrixVectorNTT]+ * (one output polynomial; multiply-accumulate of two NTT-domain vectors).}+ *+ * @param[out] w Output polynomial.+ * @param[in]  u Pointer to first input vector.+ * @param[in]  v Pointer to second input vector.+ */+MLD_INTERNAL_API+void mld_polyvecl_pointwise_acc_montgomery(mld_poly *w, const mld_polyvecl *u,+                                           const mld_polyvecl *v)+__contract__(+  requires(memory_no_alias(w, sizeof(mld_poly)))+  requires(memory_no_alias(u, sizeof(mld_polyvecl)))+  requires(memory_no_alias(v, sizeof(mld_polyvecl)))+  requires(forall(l0, 0, MLDSA_L,+                  array_bound(u->vec[l0].coeffs, 0, MLDSA_N, 0, MLDSA_Q)))+  requires(forall(l1, 0, MLDSA_L,+    array_abs_bound(v->vec[l1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+  assigns(memory_slice(w, sizeof(mld_poly)))+  ensures(array_abs_bound(w->coeffs, 0, MLDSA_N, MLDSA_Q))+);+#endif /* !MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+#define mld_polyvecl_chknorm MLD_NAMESPACE_KL(polyvecl_chknorm)+/**+ * Check infinity norm of polynomials in vector of length MLDSA_L. Assumes+ * input mld_polyvecl to be reduced by polyvecl_reduce().+ *+ * @param[in] v Pointer to vector.+ * @param     B Norm bound.+ *+ * @return 0 if norm of all polynomials is strictly smaller than+ *         B <= (MLDSA_Q-1)/8 and 0xFFFFFFFF otherwise.+ */+MLD_INTERNAL_API+MLD_MUST_CHECK_RETURN_VALUE+uint32_t mld_polyvecl_chknorm(const mld_polyvecl *v, int32_t B)+__contract__(+  requires(memory_no_alias(v, sizeof(mld_polyvecl)))+  requires(0 <= B && B <= (MLDSA_Q - 1) / 8)+  requires(forall(k0, 0, MLDSA_L,+    array_bound(v->vec[k0].coeffs, 0, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX)))+  ensures(return_value == 0 || return_value == 0xFFFFFFFF)+  ensures((return_value == 0) == forall(k1, 0, MLDSA_L, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, B)))+);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_VERIFY_API */++/** Vector of MLDSA_K polynomials. */+typedef struct+{+  mld_poly vec[MLDSA_K]; /**< Component polynomials. */+} mld_polyveck;++#if (!defined(MLD_CONFIG_NO_SIGN_API) && defined(MLD_CONFIG_REDUCE_RAM)) || \+    defined(MLD_UNIT_TEST)+#define mld_polyveck_reduce MLD_NAMESPACE_KL(polyveck_reduce)+/**+ * Reduce coefficients of polynomials in vector of length MLDSA_K to+ * representatives in [-MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX].+ *+ * @param[in,out] v Pointer to input/output vector.+ */+MLD_INTERNAL_API+void mld_polyveck_reduce(mld_polyveck *v)+__contract__(+  requires(memory_no_alias(v, sizeof(mld_polyveck)))+  requires(forall(k0, 0, MLDSA_K,+    array_bound(v->vec[k0].coeffs, 0, MLDSA_N, INT32_MIN, MLD_REDUCE32_DOMAIN_MAX)))+  assigns(memory_slice(v, sizeof(mld_polyveck)))+  ensures(forall(k1, 0, MLDSA_K,+    array_bound(v->vec[k1].coeffs, 0, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX)))+);+#endif /* (!MLD_CONFIG_NO_SIGN_API && MLD_CONFIG_REDUCE_RAM) || MLD_UNIT_TEST \+        */++#if !defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST)+#define mld_polyveck_caddq MLD_NAMESPACE_KL(polyveck_caddq)+/**+ * For all coefficients of polynomials in vector of length MLDSA_K add MLDSA_Q+ * if coefficient is negative.+ *+ * @param[in,out] v Pointer to input/output vector.+ */+MLD_INTERNAL_API+void mld_polyveck_caddq(mld_polyveck *v)+__contract__(+  requires(memory_no_alias(v, sizeof(mld_polyveck)))+  requires(forall(k0, 0, MLDSA_K,+    array_abs_bound(v->vec[k0].coeffs, 0, MLDSA_N, MLDSA_Q)))+  assigns(memory_slice(v, sizeof(mld_polyveck)))+  ensures(forall(k1, 0, MLDSA_K,+    array_bound(v->vec[k1].coeffs, 0, MLDSA_N, 0, MLDSA_Q)))+);+#endif /* !MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST */++#if (!defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST)) && \+    (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))+#define mld_polyveck_ntt MLD_NAMESPACE_KL(polyveck_ntt)+/**+ * Forward NTT of all polynomials in vector of length MLDSA_K. Output+ * coefficients are bounded by MLD_NTT_BOUND in absolute value.+ *+ * @param[in,out] v Pointer to input/output vector.+ */+MLD_INTERNAL_API+void mld_polyveck_ntt(mld_polyveck *v)+__contract__(+  requires(memory_no_alias(v, sizeof(mld_polyveck)))+  requires(forall(k0, 0, MLDSA_K, array_abs_bound(v->vec[k0].coeffs, 0, MLDSA_N, MLDSA_Q)))+  assigns(memory_slice(v, sizeof(mld_polyveck)))+  ensures(forall(k1, 0, MLDSA_K, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+);+#endif /* (!MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST) && \+          (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) */++#if !defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST)+#define mld_polyveck_invntt_tomont MLD_NAMESPACE_KL(polyveck_invntt_tomont)+/**+ * Inverse NTT and multiplication by 2^{32} of polynomials in vector of+ * length MLDSA_K.+ *+ * Input coefficients need to be less than MLDSA_Q, and output coefficients+ * are bounded by MLD_INTT_BOUND.+ *+ * @param[in,out] v Pointer to input/output vector.+ */+MLD_INTERNAL_API+void mld_polyveck_invntt_tomont(mld_polyveck *v)+__contract__(+  requires(memory_no_alias(v, sizeof(mld_polyveck)))+  requires(forall(k0, 0, MLDSA_K, array_abs_bound(v->vec[k0].coeffs, 0, MLDSA_N, MLDSA_Q)))+  assigns(memory_slice(v, sizeof(mld_polyveck)))+  ensures(forall(k1, 0, MLDSA_K, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, MLD_INTT_BOUND)))+);+#endif /* !MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+#define mld_polyveck_chknorm MLD_NAMESPACE_KL(polyveck_chknorm)+/**+ * Check infinity norm of polynomials in vector of length MLDSA_K. Assumes+ * input mld_polyveck to be reduced by polyveck_reduce().+ *+ * @param[in] v Pointer to vector.+ * @param     B Norm bound.+ *+ * @return 0 if norm of all polynomials are strictly smaller than+ *         B <= (MLDSA_Q-1)/8 and 0xFFFFFFFF otherwise.+ */+MLD_INTERNAL_API+MLD_MUST_CHECK_RETURN_VALUE+uint32_t mld_polyveck_chknorm(const mld_polyveck *v, int32_t B)+__contract__(+  requires(memory_no_alias(v, sizeof(mld_polyveck)))+  requires(0 <= B && B <= (MLDSA_Q - 1) / 8)+  requires(forall(k0, 0, MLDSA_K,+                  array_bound(v->vec[k0].coeffs, 0, MLDSA_N,+                              -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX)))+  ensures(return_value == 0 || return_value == 0xFFFFFFFF)+  ensures((return_value == 0) == forall(k1, 0, MLDSA_K, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, B)))+);++#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+#define mld_polyveck_decompose MLD_NAMESPACE_KL(polyveck_decompose)+/**+ * For all coefficients a of polynomials in vector of length MLDSA_K, compute+ * high and low bits a0, a1 such a mod^+ MLDSA_Q = a1*ALPHA + a0 with+ * -ALPHA/2 < a0 <= ALPHA/2 except a1 = (MLDSA_Q-1)/ALPHA where we set+ * a1 = 0 and -ALPHA/2 <= a0 = a mod MLDSA_Q - MLDSA_Q < 0. Assumes+ * coefficients to be standard representatives.+ *+ * @reference{The reference implementation has the input polynomial as a+ * separate argument that may be aliased with either of the outputs. Removing+ * the aliasing eases CBMC proofs.}+ *+ * @param[out]    v1 Pointer to output vector of polynomials with+ *                   coefficients a1.+ * @param[in,out] v0 Pointer to input/output vector of polynomials. Output+ *                   polynomial has coefficients a0.+ */+MLD_INTERNAL_API+void mld_polyveck_decompose(mld_polyveck *v1, mld_polyveck *v0)+__contract__(+  requires(memory_no_alias(v1,  sizeof(mld_polyveck)))+  requires(memory_no_alias(v0, sizeof(mld_polyveck)))+  requires(forall(k0, 0, MLDSA_K,+    array_bound(v0->vec[k0].coeffs, 0, MLDSA_N, 0, MLDSA_Q)))+  assigns(memory_slice(v1, sizeof(mld_polyveck)))+  assigns(memory_slice(v0, sizeof(mld_polyveck)))+  ensures(forall(k1, 0, MLDSA_K,+                 array_bound(v1->vec[k1].coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2))))+  ensures(forall(k2, 0, MLDSA_K,+                 array_abs_bound(v0->vec[k2].coeffs, 0, MLDSA_N, MLDSA_GAMMA2+1)))+);+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+#define mld_polyveck_pack_w1 MLD_NAMESPACE_KL(polyveck_pack_w1)+/**+ * Bit-pack polynomial vector w1 with coefficients in [0, 15] or [0, 43]. Input+ * coefficients are assumed to be standard representatives.+ *+ * @spec{Implements @[FIPS204, Algorithm 28, w1Encode].}+ *+ * @param[out] r  Pointer to output byte array with at least+ *                MLDSA_K * MLDSA_POLYW1_PACKEDBYTES bytes.+ * @param[in]  w1 Pointer to input polynomial vector.+ */+MLD_INTERNAL_API+void mld_polyveck_pack_w1(uint8_t r[MLDSA_K * MLDSA_POLYW1_PACKEDBYTES],+                          const mld_polyveck *w1)+__contract__(+  requires(memory_no_alias(r, MLDSA_K * MLDSA_POLYW1_PACKEDBYTES))+  requires(memory_no_alias(w1, sizeof(mld_polyveck)))+  requires(forall(k1, 0, MLDSA_K,+    array_bound(w1->vec[k1].coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2))))+  assigns(memory_slice(r, MLDSA_K * MLDSA_POLYW1_PACKEDBYTES))+);+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+#define mld_polyveck_pack_eta MLD_NAMESPACE_KL(polyveck_pack_eta)+/**+ * Bit-pack polynomial vector with coefficients in [-MLDSA_ETA, MLDSA_ETA].+ *+ * @param[out] r Pointer to output byte array with+ *               MLDSA_K * MLDSA_POLYETA_PACKEDBYTES bytes.+ * @param[in]  p Pointer to input polynomial vector.+ */+MLD_INTERNAL_API+void mld_polyveck_pack_eta(uint8_t r[MLDSA_K * MLDSA_POLYETA_PACKEDBYTES],+                           const mld_polyveck *p)+__contract__(+  requires(memory_no_alias(r,  MLDSA_K * MLDSA_POLYETA_PACKEDBYTES))+  requires(memory_no_alias(p, sizeof(mld_polyveck)))+  requires(forall(k1, 0, MLDSA_K,+    array_abs_bound(p->vec[k1].coeffs, 0, MLDSA_N, MLDSA_ETA + 1)))+  assigns(memory_slice(r, MLDSA_K * MLDSA_POLYETA_PACKEDBYTES))+);++#define mld_polyvecl_pack_eta MLD_NAMESPACE_KL(polyvecl_pack_eta)+/**+ * Bit-pack polynomial vector with coefficients in [-MLDSA_ETA, MLDSA_ETA].+ *+ * @param[out] r Pointer to output byte array with+ *               MLDSA_L * MLDSA_POLYETA_PACKEDBYTES bytes.+ * @param[in]  p Pointer to input polynomial vector.+ */+MLD_INTERNAL_API+void mld_polyvecl_pack_eta(uint8_t r[MLDSA_L * MLDSA_POLYETA_PACKEDBYTES],+                           const mld_polyvecl *p)+__contract__(+  requires(memory_no_alias(r,  MLDSA_L * MLDSA_POLYETA_PACKEDBYTES))+  requires(memory_no_alias(p, sizeof(mld_polyvecl)))+  requires(forall(k1, 0, MLDSA_L,+    array_abs_bound(p->vec[k1].coeffs, 0, MLDSA_N, MLDSA_ETA + 1)))+  assigns(memory_slice(r, MLDSA_L * MLDSA_POLYETA_PACKEDBYTES))+);++#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || \+    (!defined(MLD_CONFIG_NO_SIGN_API) &&   \+     (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)))+#define mld_polyvecl_unpack_eta MLD_NAMESPACE_KL(polyvecl_unpack_eta)+/**+ * Unpack polynomial vector with coefficients in [-MLDSA_ETA, MLDSA_ETA].+ *+ * @param[out] p Pointer to output polynomial vector.+ * @param[in]  r Input byte array with bit-packed polynomial vector.+ */+MLD_INTERNAL_API+void mld_polyvecl_unpack_eta(+    mld_polyvecl *p, const uint8_t r[MLDSA_L * MLDSA_POLYETA_PACKEDBYTES])+__contract__(+  requires(memory_no_alias(r,  MLDSA_L * MLDSA_POLYETA_PACKEDBYTES))+  requires(memory_no_alias(p, sizeof(mld_polyvecl)))+  assigns(memory_slice(p, sizeof(mld_polyvecl)))+  ensures(forall(k1, 0, MLDSA_L,+    array_bound(p->vec[k1].coeffs, 0, MLDSA_N, MLD_POLYETA_UNPACK_LOWER_BOUND, MLDSA_ETA + 1)))+);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || (!MLD_CONFIG_NO_SIGN_API && \+          (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST)) */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+#define mld_polyvecl_unpack_z MLD_NAMESPACE_KL(polyvecl_unpack_z)+/**+ * Unpack polynomial vector with coefficients in+ * [-(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1].+ *+ * @param[out] z Pointer to output polynomial vector.+ * @param[in]  r Input byte array with bit-packed polynomial vector.+ */+MLD_INTERNAL_API+void mld_polyvecl_unpack_z(mld_polyvecl *z,+                           const uint8_t r[MLDSA_L * MLDSA_POLYZ_PACKEDBYTES])+__contract__(+  requires(memory_no_alias(r,  MLDSA_L * MLDSA_POLYZ_PACKEDBYTES))+  requires(memory_no_alias(z, sizeof(mld_polyvecl)))+  assigns(memory_slice(z, sizeof(mld_polyvecl)))+  ensures(forall(k1, 0, MLDSA_L,+    array_bound(z->vec[k1].coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)))+);+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || \+    (!defined(MLD_CONFIG_NO_SIGN_API) &&   \+     (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)))+#define mld_polyveck_unpack_eta MLD_NAMESPACE_KL(polyveck_unpack_eta)+/**+ * Unpack polynomial vector with coefficients in [-MLDSA_ETA, MLDSA_ETA].+ *+ * @param[out] p Pointer to output polynomial vector.+ * @param[in]  r Input byte array with bit-packed polynomial vector.+ */+MLD_INTERNAL_API+void mld_polyveck_unpack_eta(+    mld_polyveck *p, const uint8_t r[MLDSA_K * MLDSA_POLYETA_PACKEDBYTES])+__contract__(+  requires(memory_no_alias(r,  MLDSA_K * MLDSA_POLYETA_PACKEDBYTES))+  requires(memory_no_alias(p, sizeof(mld_polyveck)))+  assigns(memory_slice(p, sizeof(mld_polyveck)))+  ensures(forall(k1, 0, MLDSA_K,+    array_bound(p->vec[k1].coeffs, 0, MLDSA_N, MLD_POLYETA_UNPACK_LOWER_BOUND, MLDSA_ETA + 1)))+);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || (!MLD_CONFIG_NO_SIGN_API && \+          (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST)) */+++#endif /* !MLD_POLYVEC_H */
+ cbits/mldsa/src/polyvec_lazy.c view
@@ -0,0 +1,311 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ */++#include "polyvec_lazy.h"++#include "debug.h"++/* This namespacing is not done at the top to avoid a naming conflict+ * with native backends, which are currently not yet namespaced. */+#define mld_polymat_expand_entry MLD_ADD_PARAM_SET(mld_polymat_expand_entry)++/**+ * Sample a single matrix entry A[k][l] of ExpandA(rho) by rejection sampling+ * from SHAKE128(rho|l|k), and apply the custom-order permutation when a+ * native NTT backend is in use.+ *+ * The caller is expected to have copied rho into the first MLDSA_SEEDBYTES+ * of seed_ext. This function writes the domain-separation bytes+ * seed_ext[SEEDBYTES..+2] = {l, k} before sampling.+ *+ * @spec{Partially implements @[FIPS204, Algorithm 32, ExpandA] (samples one+ * matrix entry via @[FIPS204, Algorithm 30, RejNTTPoly]).}+ *+ * @param[out]    p        Pointer to output polynomial.+ * @param[in,out] seed_ext Seed buffer pre-filled with rho in the first+ *                         MLDSA_SEEDBYTES; the final two bytes are+ *                         overwritten.+ * @param         l        Column index (inner, aka nonce low byte).+ * @param         k        Row index (outer, aka nonce high byte).+ */+static MLD_INLINE void mld_polymat_expand_entry(+    mld_poly *p, uint8_t seed_ext[MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2)], uint8_t l,+    uint8_t k)+__contract__(+  requires(memory_no_alias(p, sizeof(mld_poly)))+  requires(memory_no_alias(seed_ext, MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2)))+  assigns(memory_slice(p, sizeof(mld_poly)))+  assigns(memory_slice(seed_ext, MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2)))+  ensures(array_bound(p->coeffs, 0, MLDSA_N, 0, MLDSA_Q))+)+{+  seed_ext[MLDSA_SEEDBYTES + 0] = l;+  seed_ext[MLDSA_SEEDBYTES + 1] = k;+  mld_poly_uniform(p, seed_ext);+  mld_poly_permute_bitrev_to_custom_optional(p);+}++#if !defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)++MLD_INTERNAL_API+void mld_polyvec_matrix_expand_eager(mld_polymat_eager *mat,+                                     const uint8_t rho[MLDSA_SEEDBYTES])+{+  unsigned int i, j;+  MLD_ALIGN uint8_t seed_ext[4][MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2)];++  for (j = 0; j < 4; j++)+  __loop__(+    assigns(j, object_whole(seed_ext))+    invariant(j <= 4)+    decreases(4 - j)+  )+  {+    mld_memcpy(seed_ext[j], rho, MLDSA_SEEDBYTES);+  }++#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY)+  /* Sample 4 matrix entries a time. */+  for (i = 0; i < (MLDSA_K * MLDSA_L / 4) * 4; i += 4)+  __loop__(+    assigns(i, j, object_whole(seed_ext), memory_slice(mat, sizeof(mld_polymat_eager)))+    invariant(i <= (MLDSA_K * MLDSA_L / 4) * 4 && i % 4 == 0)+    /* vectors 0 .. i / MLDSA_L are completely sampled */+    invariant(forall(k1, 0, i / MLDSA_L, forall(l1, 0, MLDSA_L,+      array_bound(mat->vec[k1].vec[l1].coeffs, 0, MLDSA_N, 0, MLDSA_Q))))+    /* last vector is sampled up to i % MLDSA_L */+    invariant(forall(k2, i / MLDSA_L, i / MLDSA_L + 1, forall(l2, 0, i % MLDSA_L,+      array_bound(mat->vec[k2].vec[l2].coeffs, 0, MLDSA_N, 0, MLDSA_Q))))+    decreases((MLDSA_K * MLDSA_L / 4) * 4 - i)+  )+  {+    for (j = 0; j < 4; j++)+    __loop__(+      assigns(j, object_whole(seed_ext))+      invariant(j <= 4)+      decreases(4 - j)+    )+    {+      uint8_t x = (uint8_t)((i + j) / MLDSA_L);+      uint8_t y = (uint8_t)((i + j) % MLDSA_L);++      seed_ext[j][MLDSA_SEEDBYTES + 0] = y;+      seed_ext[j][MLDSA_SEEDBYTES + 1] = x;+    }++    mld_poly_uniform_4x(&mat->vec[i / MLDSA_L].vec[i % MLDSA_L],+                        &mat->vec[(i + 1) / MLDSA_L].vec[(i + 1) % MLDSA_L],+                        &mat->vec[(i + 2) / MLDSA_L].vec[(i + 2) % MLDSA_L],+                        &mat->vec[(i + 3) / MLDSA_L].vec[(i + 3) % MLDSA_L],+                        seed_ext);+    mld_poly_permute_bitrev_to_custom_optional(+        &mat->vec[i / MLDSA_L].vec[i % MLDSA_L]);+    mld_poly_permute_bitrev_to_custom_optional(+        &mat->vec[(i + 1) / MLDSA_L].vec[(i + 1) % MLDSA_L]);+    mld_poly_permute_bitrev_to_custom_optional(+        &mat->vec[(i + 2) / MLDSA_L].vec[(i + 2) % MLDSA_L]);+    mld_poly_permute_bitrev_to_custom_optional(+        &mat->vec[(i + 3) / MLDSA_L].vec[(i + 3) % MLDSA_L]);+  }+#else  /* !MLD_CONFIG_SERIAL_FIPS202_ONLY */+  i = 0;+#endif /* MLD_CONFIG_SERIAL_FIPS202_ONLY */++  /* Entries omitted by the batch-sampling are sampled individually. */+  while (i < MLDSA_K * MLDSA_L)+  __loop__(+    assigns(i, object_whole(seed_ext), memory_slice(mat, sizeof(mld_polymat_eager)))+    invariant(i <= MLDSA_K * MLDSA_L)+    /* vectors 0 .. i / MLDSA_L are completely sampled */+    invariant(forall(k1, 0, i / MLDSA_L, forall(l1, 0, MLDSA_L,+      array_bound(mat->vec[k1].vec[l1].coeffs, 0, MLDSA_N, 0, MLDSA_Q))))+    /* last vector is sampled up to i % MLDSA_L */+    invariant(forall(k2, i / MLDSA_L, i / MLDSA_L + 1, forall(l2, 0, i % MLDSA_L,+      array_bound(mat->vec[k2].vec[l2].coeffs, 0, MLDSA_N, 0, MLDSA_Q))))+    decreases(MLDSA_K * MLDSA_L - i)+  )+  {+    uint8_t x = (uint8_t)(i / MLDSA_L);+    uint8_t y = (uint8_t)(i % MLDSA_L);+    mld_polymat_expand_entry(&mat->vec[x].vec[y], seed_ext[0], y, x);+    i++;+  }++  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(seed_ext, sizeof(seed_ext));+}++MLD_INTERNAL_API+void mld_polyvec_matrix_pointwise_montgomery_row_eager(mld_poly *t_row,+                                                       mld_polymat_eager *mat,+                                                       const mld_polyvecl *v,+                                                       unsigned int i)+{+  mld_polyvecl_pointwise_acc_montgomery(t_row, &mat->vec[i], v);+}++#if !defined(MLD_CONFIG_NO_SIGN_API)+MLD_INTERNAL_API+void mld_polyvec_matrix_pointwise_montgomery_yvec_eager(mld_polyveck *w,+                                                        mld_polymat_eager *mat,+                                                        const mld_yvec_eager *y,+                                                        mld_polyvecl *scratch)+{+  unsigned int i;+  *scratch = y->vec;+  mld_polyvecl_ntt(scratch);++  for (i = 0; i < MLDSA_K; ++i)+  __loop__(+    assigns(i, memory_slice(w, sizeof(mld_polyveck)))+    invariant(i <= MLDSA_K)+    invariant(forall(k0, 0, i,+                     array_abs_bound(w->vec[k0].coeffs, 0, MLDSA_N, MLDSA_Q)))+    decreases(MLDSA_K - i)+  )+  {+    mld_polyvec_matrix_pointwise_montgomery_row_eager(&w->vec[i], mat, scratch,+                                                      i);+  }++  mld_polyveck_invntt_tomont(w);+}+#endif /* !MLD_CONFIG_NO_SIGN_API */++#endif /* !MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++#if defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)++MLD_INTERNAL_API+void mld_polyvec_matrix_expand_lazy(mld_polymat_lazy *mat,+                                    const uint8_t rho[MLDSA_SEEDBYTES])+{+  mld_memcpy(mat->rho, rho, MLDSA_SEEDBYTES);+}++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+MLD_INTERNAL_API+void mld_polyvec_matrix_pointwise_montgomery_row_lazy(mld_poly *t_row,+                                                      mld_polymat_lazy *mat,+                                                      const mld_polyvecl *v,+                                                      unsigned int i)+{+  unsigned int l;+  MLD_ALIGN uint8_t seed_ext[MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2)];+  mld_memcpy(seed_ext, mat->rho, MLDSA_SEEDBYTES);++  mld_polymat_expand_entry(t_row, seed_ext, 0, (uint8_t)i);+  mld_poly_pointwise_montgomery(t_row, &v->vec[0]);++  for (l = 1; l < MLDSA_L; ++l)+  __loop__(+    assigns(l, object_whole(seed_ext),+            memory_slice(t_row, sizeof(mld_poly)),+            memory_slice(mat, sizeof(mld_polymat_lazy)))+    invariant(l >= 1 && l <= MLDSA_L)+    invariant(array_abs_bound(t_row->coeffs, 0, MLDSA_N, l * MLDSA_Q))+    decreases(MLDSA_L - l)+  )+  {+    mld_polymat_expand_entry(&mat->cur, seed_ext, (uint8_t)l, (uint8_t)i);+    mld_poly_pointwise_montgomery(&mat->cur, &v->vec[l]);+    mld_poly_add(t_row, &mat->cur);+  }+  mld_poly_reduce(t_row);++  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(seed_ext, sizeof(seed_ext));+}+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+MLD_INTERNAL_API+void mld_polyvec_matrix_pointwise_montgomery_yvec_lazy(mld_polyveck *w,+                                                       mld_polymat_lazy *mat,+                                                       const mld_yvec_lazy *y,+                                                       mld_polyvecl *scratch)+{+  unsigned int k, l;+  MLD_ALIGN uint8_t seed_ext[MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2)];+  /* Only the first poly of the polyvecl scratch is used. The polyvecl type+   * matches the eager variant for API uniformity; in REDUCE_RAM mode the+   * polyvecl storage is provided "for free" by the caller's polyveck/polyvecl+   * union. */+  mld_poly *y_ntt = &scratch->vec[0];++  mld_memcpy(seed_ext, mat->rho, MLDSA_SEEDBYTES);++  /* Column-by-column: sample y[l], NTT, accumulate column l of A into w. */+  for (l = 0; l < MLDSA_L; l++)+  __loop__(+    assigns(k, l, object_whole(seed_ext),+            memory_slice(w, sizeof(mld_polyveck)),+            memory_slice(mat, sizeof(mld_polymat_lazy)),+            memory_slice(scratch, sizeof(mld_polyvecl)))+    invariant(l <= MLDSA_L)+    invariant(l == 0 ||+              forall(k0, 0, MLDSA_K,+                     array_abs_bound(w->vec[k0].coeffs, 0, MLDSA_N,+                                     (int)l * MLDSA_Q)))+    decreases(MLDSA_L - l)+  )+  {+    mld_yvec_get_poly_lazy(y_ntt, y, l);+    mld_poly_ntt(y_ntt);+    for (k = 0; k < MLDSA_K; k++)+    __loop__(+      assigns(k, object_whole(seed_ext),+              memory_slice(w, sizeof(mld_polyveck)),+              memory_slice(mat, sizeof(mld_polymat_lazy)))+      invariant(k <= MLDSA_K)+      invariant(l != 0 ||+                forall(k1, 0, k,+                       array_abs_bound(w->vec[k1].coeffs, 0, MLDSA_N, MLDSA_Q)))+      invariant(l == 0 ||+                forall(k2, 0, k,+                       array_abs_bound(w->vec[k2].coeffs, 0, MLDSA_N,+                                       ((int)l + 1) * MLDSA_Q)))+      invariant(l == 0 ||+                forall(k3, k, MLDSA_K,+                       array_abs_bound(w->vec[k3].coeffs, 0, MLDSA_N,+                                       (int)l * MLDSA_Q)))+      decreases(MLDSA_K - k)+    )+    {+      if (l == 0)+      {+        mld_polymat_expand_entry(&w->vec[k], seed_ext, 0, (uint8_t)k);+        mld_poly_pointwise_montgomery(&w->vec[k], y_ntt);+      }+      else+      {+        mld_polymat_expand_entry(&mat->cur, seed_ext, (uint8_t)l, (uint8_t)k);+        mld_poly_pointwise_montgomery(&mat->cur, y_ntt);+        mld_poly_add(&w->vec[k], &mat->cur);+      }+    }+  }++  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(seed_ext, sizeof(seed_ext));+  mld_polyveck_reduce(w);+  mld_polyveck_invntt_tomont(w);+}+#endif /* !MLD_CONFIG_NO_SIGN_API */++#endif /* MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++/* To facilitate single-compilation-unit (SCU) builds, undefine all macros.+ * Don't modify by hand -- this is auto-generated by scripts/autogen. */+#undef mld_polymat_expand_entry
+ cbits/mldsa/src/polyvec_lazy.h view
@@ -0,0 +1,652 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ */++/*+ * Eager and lazy variants of polynomial vector types.+ *+ * In eager mode, full vectors are precomputed and stored in memory.+ * In lazy mode, data is stored in packed form and expanded on demand,+ * trading computation for reduced memory usage.+ *+ * MLD_CONFIG_REDUCE_RAM selects which variant is used.+ */++#ifndef MLD_POLYVEC_LAZY_H+#define MLD_POLYVEC_LAZY_H++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_SIGN_API) || \+    !defined(MLD_CONFIG_NO_VERIFY_API)++#include "poly.h"+#include "poly_kl.h"+#include "polyvec.h"++/* Parameter set namespacing */+#define mld_sk_s1hat_eager MLD_ADD_PARAM_SET(mld_sk_s1hat_eager)+#define mld_sk_s1hat_lazy MLD_ADD_PARAM_SET(mld_sk_s1hat_lazy)+#define mld_sk_s1hat MLD_ADD_PARAM_SET(mld_sk_s1hat)+#define mld_unpack_sk_s1hat_eager MLD_ADD_PARAM_SET(mld_unpack_sk_s1hat_eager)+#define mld_unpack_sk_s1hat_lazy MLD_ADD_PARAM_SET(mld_unpack_sk_s1hat_lazy)+#define mld_sk_s1hat_get_poly_eager \+  MLD_ADD_PARAM_SET(mld_sk_s1hat_get_poly_eager)+#define mld_sk_s1hat_get_poly_lazy MLD_ADD_PARAM_SET(mld_sk_s1hat_get_poly_lazy)+#define mld_sk_s2hat_eager MLD_ADD_PARAM_SET(mld_sk_s2hat_eager)+#define mld_sk_s2hat_lazy MLD_ADD_PARAM_SET(mld_sk_s2hat_lazy)+#define mld_sk_s2hat MLD_ADD_PARAM_SET(mld_sk_s2hat)+#define mld_unpack_sk_s2hat_eager MLD_ADD_PARAM_SET(mld_unpack_sk_s2hat_eager)+#define mld_unpack_sk_s2hat_lazy MLD_ADD_PARAM_SET(mld_unpack_sk_s2hat_lazy)+#define mld_sk_s2hat_get_poly_eager \+  MLD_ADD_PARAM_SET(mld_sk_s2hat_get_poly_eager)+#define mld_sk_s2hat_get_poly_lazy MLD_ADD_PARAM_SET(mld_sk_s2hat_get_poly_lazy)+#define mld_sk_t0hat_eager MLD_ADD_PARAM_SET(mld_sk_t0hat_eager)+#define mld_sk_t0hat_lazy MLD_ADD_PARAM_SET(mld_sk_t0hat_lazy)+#define mld_sk_t0hat MLD_ADD_PARAM_SET(mld_sk_t0hat)+#define mld_unpack_sk_t0hat_eager MLD_ADD_PARAM_SET(mld_unpack_sk_t0hat_eager)+#define mld_unpack_sk_t0hat_lazy MLD_ADD_PARAM_SET(mld_unpack_sk_t0hat_lazy)+#define mld_sk_t0hat_get_poly_eager \+  MLD_ADD_PARAM_SET(mld_sk_t0hat_get_poly_eager)+#define mld_sk_t0hat_get_poly_lazy MLD_ADD_PARAM_SET(mld_sk_t0hat_get_poly_lazy)+#define mld_polymat MLD_ADD_PARAM_SET(mld_polymat)+#define mld_polymat_eager MLD_ADD_PARAM_SET(mld_polymat_eager)+#define mld_polymat_lazy MLD_ADD_PARAM_SET(mld_polymat_lazy)+#define mld_poly_permute_bitrev_to_custom_optional \+  MLD_ADD_PARAM_SET(mld_poly_permute_bitrev_to_custom_optional)+#define mld_polyvec_matrix_expand_eager \+  MLD_NAMESPACE_KL(polyvec_matrix_expand_eager)+#define mld_polyvec_matrix_expand_lazy \+  MLD_NAMESPACE_KL(polyvec_matrix_expand_lazy)+#define mld_polyvec_matrix_pointwise_montgomery \+  MLD_NAMESPACE_KL(polyvec_matrix_pointwise_montgomery)+#define mld_polyvec_matrix_pointwise_montgomery_row_eager \+  MLD_NAMESPACE_KL(polyvec_matrix_pointwise_montgomery_row_eager)+#define mld_polyvec_matrix_pointwise_montgomery_row_lazy \+  MLD_NAMESPACE_KL(polyvec_matrix_pointwise_montgomery_row_lazy)+#define mld_polyvec_matrix_pointwise_montgomery_yvec_eager \+  MLD_NAMESPACE_KL(polyvec_matrix_pointwise_montgomery_yvec_eager)+#define mld_polyvec_matrix_pointwise_montgomery_yvec_lazy \+  MLD_NAMESPACE_KL(polyvec_matrix_pointwise_montgomery_yvec_lazy)+#define mld_yvec_eager MLD_ADD_PARAM_SET(mld_yvec_eager)+#define mld_yvec_lazy MLD_ADD_PARAM_SET(mld_yvec_lazy)+#define mld_yvec MLD_ADD_PARAM_SET(mld_yvec)+#define mld_yvec_init_eager MLD_ADD_PARAM_SET(mld_yvec_init_eager)+#define mld_yvec_init_lazy MLD_ADD_PARAM_SET(mld_yvec_init_lazy)+#define mld_yvec_get_poly_eager MLD_ADD_PARAM_SET(mld_yvec_get_poly_eager)+#define mld_yvec_get_poly_lazy MLD_ADD_PARAM_SET(mld_yvec_get_poly_lazy)+/* End of parameter set namespacing */++/** Eager s1hat: precomputed s1 vector in NTT domain. */+typedef struct+{+  mld_polyvecl vec; /**< s1 vector in NTT domain. */+} mld_sk_s1hat_eager;++/** Eager s2hat: precomputed s2 vector in NTT domain. */+typedef struct+{+  mld_polyveck vec; /**< s2 vector in NTT domain. */+} mld_sk_s2hat_eager;++/** Eager t0hat: precomputed t0 vector in NTT domain. */+typedef struct+{+  mld_polyveck vec; /**< t0 vector in NTT domain. */+} mld_sk_t0hat_eager;++/** Lazy s1hat: borrow packed s1, unpack and convert to NTT domain on demand. */+typedef struct+{+  const uint8_t *packed; /**< Pointer to packed s1 in the secret key. */+} mld_sk_s1hat_lazy;++/** Lazy s2hat: borrow packed s2, unpack and convert to NTT domain on demand. */+typedef struct+{+  const uint8_t *packed; /**< Pointer to packed s2 in the secret key. */+} mld_sk_s2hat_lazy;++/** Lazy t0hat: borrow packed t0, unpack and convert to NTT domain on demand. */+typedef struct+{+  const uint8_t *packed; /**< Pointer to packed t0 in the secret key. */+} mld_sk_t0hat_lazy;++/** Eager yvec: precomputed and stored full signing masking vector y. */+typedef struct+{+  mld_polyvecl vec; /**< Masking vector y. */+} mld_yvec_eager;++/** Lazy yvec: store seed and base counter kappa, regenerate y[i] on demand. */+typedef struct+{+  const uint8_t *rhoprime; /**< Pointer to seed used to derive y. */+  uint16_t kappa;          /**< Base counter; component i uses kappa + i. */+} mld_yvec_lazy;++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_SIGN_API)+/* s1vec */++#if !defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+static MLD_INLINE void mld_unpack_sk_s1hat_eager(+    mld_sk_s1hat_eager *s1,+    const uint8_t packed_s1[MLDSA_L * MLDSA_POLYETA_PACKEDBYTES])+__contract__(+  requires(memory_no_alias(s1, sizeof(mld_sk_s1hat_eager)))+  requires(memory_no_alias(packed_s1, MLDSA_L * MLDSA_POLYETA_PACKEDBYTES))+  assigns(memory_slice(s1, sizeof(mld_sk_s1hat_eager)))+  ensures(forall(k1, 0, MLDSA_L,+    array_abs_bound(s1->vec.vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+)+{+  mld_polyvecl_unpack_eta(&s1->vec, packed_s1);+  mld_polyvecl_ntt(&s1->vec);+}++#if !defined(MLD_CONFIG_NO_SIGN_API)+static MLD_INLINE void mld_sk_s1hat_get_poly_eager(mld_poly *buf,+                                                   const mld_sk_s1hat_eager *s1,+                                                   unsigned int i)+__contract__(+  requires(memory_no_alias(buf, sizeof(mld_poly)))+  requires(memory_no_alias(s1, sizeof(mld_sk_s1hat_eager)))+  requires(i < MLDSA_L)+  requires(array_abs_bound(s1->vec.vec[i].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))+  assigns(memory_slice(buf, sizeof(mld_poly)))+  ensures(array_abs_bound(buf->coeffs, 0, MLDSA_N, MLD_NTT_BOUND))+) { *buf = s1->vec.vec[i]; }+#endif /* !MLD_CONFIG_NO_SIGN_API */+#endif /* !MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */+#if defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+static MLD_INLINE void mld_unpack_sk_s1hat_lazy(+    mld_sk_s1hat_lazy *s1,+    const uint8_t packed_s1[MLDSA_L * MLDSA_POLYETA_PACKEDBYTES])+__contract__(+  requires(memory_no_alias(s1, sizeof(mld_sk_s1hat_lazy)))+  assigns(memory_slice(s1, sizeof(mld_sk_s1hat_lazy)))+  ensures(s1->packed == old(packed_s1))+) { s1->packed = packed_s1; }++#if !defined(MLD_CONFIG_NO_SIGN_API)+static MLD_INLINE void mld_sk_s1hat_get_poly_lazy(mld_poly *buf,+                                                  const mld_sk_s1hat_lazy *s1,+                                                  unsigned int i)+__contract__(+  requires(memory_no_alias(buf, sizeof(mld_poly)))+  requires(memory_no_alias(s1, sizeof(mld_sk_s1hat_lazy)))+  requires(i < MLDSA_L)+  requires(memory_no_alias(s1->packed, MLDSA_L * MLDSA_POLYETA_PACKEDBYTES))+  assigns(memory_slice(buf, sizeof(mld_poly)))+  ensures(array_abs_bound(buf->coeffs, 0, MLDSA_N, MLD_NTT_BOUND))+)+{+  mld_polyeta_unpack(buf, s1->packed + i * MLDSA_POLYETA_PACKEDBYTES);+  mld_poly_ntt(buf);+}+#endif /* !MLD_CONFIG_NO_SIGN_API */+#endif /* MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++/* s2vec */++#if (!defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST)) && \+    (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))+static MLD_INLINE void mld_unpack_sk_s2hat_eager(+    mld_sk_s2hat_eager *s2,+    const uint8_t packed_s2[MLDSA_K * MLDSA_POLYETA_PACKEDBYTES])+__contract__(+  requires(memory_no_alias(s2, sizeof(mld_sk_s2hat_eager)))+  requires(memory_no_alias(packed_s2, MLDSA_K * MLDSA_POLYETA_PACKEDBYTES))+  assigns(memory_slice(s2, sizeof(mld_sk_s2hat_eager)))+  ensures(forall(k1, 0, MLDSA_K,+    array_abs_bound(s2->vec.vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+)+{+  mld_polyveck_unpack_eta(&s2->vec, packed_s2);+  mld_polyveck_ntt(&s2->vec);+}++#if !defined(MLD_CONFIG_NO_SIGN_API)+static MLD_INLINE void mld_sk_s2hat_get_poly_eager(mld_poly *buf,+                                                   const mld_sk_s2hat_eager *s2,+                                                   unsigned int i)+__contract__(+  requires(memory_no_alias(buf, sizeof(mld_poly)))+  requires(memory_no_alias(s2, sizeof(mld_sk_s2hat_eager)))+  requires(i < MLDSA_K)+  requires(array_abs_bound(s2->vec.vec[i].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))+  assigns(memory_slice(buf, sizeof(mld_poly)))+  ensures(array_abs_bound(buf->coeffs, 0, MLDSA_N, MLD_NTT_BOUND))+) { *buf = s2->vec.vec[i]; }+#endif /* !MLD_CONFIG_NO_SIGN_API */+#endif /* (!MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST) && \+          (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) */+#if defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+static MLD_INLINE void mld_unpack_sk_s2hat_lazy(+    mld_sk_s2hat_lazy *s2,+    const uint8_t packed_s2[MLDSA_K * MLDSA_POLYETA_PACKEDBYTES])+__contract__(+  requires(memory_no_alias(s2, sizeof(mld_sk_s2hat_lazy)))+  assigns(memory_slice(s2, sizeof(mld_sk_s2hat_lazy)))+  ensures(s2->packed == old(packed_s2))+) { s2->packed = packed_s2; }++#if !defined(MLD_CONFIG_NO_SIGN_API)+static MLD_INLINE void mld_sk_s2hat_get_poly_lazy(mld_poly *buf,+                                                  const mld_sk_s2hat_lazy *s2,+                                                  unsigned int i)+__contract__(+  requires(memory_no_alias(buf, sizeof(mld_poly)))+  requires(memory_no_alias(s2, sizeof(mld_sk_s2hat_lazy)))+  requires(i < MLDSA_K)+  requires(memory_no_alias(s2->packed, MLDSA_K * MLDSA_POLYETA_PACKEDBYTES))+  assigns(memory_slice(buf, sizeof(mld_poly)))+  ensures(array_abs_bound(buf->coeffs, 0, MLDSA_N, MLD_NTT_BOUND))+)+{+  mld_polyeta_unpack(buf, s2->packed + i * MLDSA_POLYETA_PACKEDBYTES);+  mld_poly_ntt(buf);+}+#endif /* !MLD_CONFIG_NO_SIGN_API */+#endif /* MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++/* t0vec */++#if (!defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST)) && \+    (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))+static MLD_INLINE void mld_unpack_sk_t0hat_eager(+    mld_sk_t0hat_eager *t0,+    const uint8_t packed_t0[MLDSA_K * MLDSA_POLYT0_PACKEDBYTES])+__contract__(+  requires(memory_no_alias(t0, sizeof(mld_sk_t0hat_eager)))+  requires(memory_no_alias(packed_t0, MLDSA_K * MLDSA_POLYT0_PACKEDBYTES))+  assigns(memory_slice(t0, sizeof(mld_sk_t0hat_eager)))+  ensures(forall(k1, 0, MLDSA_K,+    array_abs_bound(t0->vec.vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+)+{+  unsigned int i;+  for (i = 0; i < MLDSA_K; ++i)+  __loop__(+    assigns(i, memory_slice(t0, sizeof(mld_sk_t0hat_eager)))+    invariant(i <= MLDSA_K)+    invariant(forall(k0, 0, i,+      array_bound(t0->vec.vec[k0].coeffs, 0, MLDSA_N,+                  -(1 << (MLDSA_D - 1)) + 1, (1 << (MLDSA_D - 1)) + 1)))+    decreases(MLDSA_K - i)+  )+  {+    mld_polyt0_unpack(&t0->vec.vec[i],+                      packed_t0 + i * MLDSA_POLYT0_PACKEDBYTES);+  }+  mld_polyveck_ntt(&t0->vec);+}++#if !defined(MLD_CONFIG_NO_SIGN_API)+static MLD_INLINE void mld_sk_t0hat_get_poly_eager(mld_poly *buf,+                                                   const mld_sk_t0hat_eager *t0,+                                                   unsigned int i)+__contract__(+  requires(memory_no_alias(buf, sizeof(mld_poly)))+  requires(memory_no_alias(t0, sizeof(mld_sk_t0hat_eager)))+  requires(i < MLDSA_K)+  requires(array_abs_bound(t0->vec.vec[i].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))+  assigns(memory_slice(buf, sizeof(mld_poly)))+  ensures(array_abs_bound(buf->coeffs, 0, MLDSA_N, MLD_NTT_BOUND))+) { *buf = t0->vec.vec[i]; }+#endif /* !MLD_CONFIG_NO_SIGN_API */+#endif /* (!MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST) && \+          (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) */+#if defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+static MLD_INLINE void mld_unpack_sk_t0hat_lazy(+    mld_sk_t0hat_lazy *t0,+    const uint8_t packed_t0[MLDSA_K * MLDSA_POLYT0_PACKEDBYTES])+__contract__(+  requires(memory_no_alias(t0, sizeof(mld_sk_t0hat_lazy)))+  assigns(memory_slice(t0, sizeof(mld_sk_t0hat_lazy)))+  ensures(t0->packed == old(packed_t0))+) { t0->packed = packed_t0; }++#if !defined(MLD_CONFIG_NO_SIGN_API)+static MLD_INLINE void mld_sk_t0hat_get_poly_lazy(mld_poly *buf,+                                                  const mld_sk_t0hat_lazy *t0,+                                                  unsigned int i)+__contract__(+  requires(memory_no_alias(buf, sizeof(mld_poly)))+  requires(memory_no_alias(t0, sizeof(mld_sk_t0hat_lazy)))+  requires(i < MLDSA_K)+  requires(memory_no_alias(t0->packed, MLDSA_K * MLDSA_POLYT0_PACKEDBYTES))+  assigns(memory_slice(buf, sizeof(mld_poly)))+  ensures(array_abs_bound(buf->coeffs, 0, MLDSA_N, MLD_NTT_BOUND))+)+{+  mld_polyt0_unpack(buf, t0->packed + i * MLDSA_POLYT0_PACKEDBYTES);+  mld_poly_ntt(buf);+}+#endif /* !MLD_CONFIG_NO_SIGN_API */+#endif /* MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_SIGN_API */++/* yvec */++#if !defined(MLD_CONFIG_NO_SIGN_API) && \+    (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))+static MLD_INLINE void mld_yvec_init_eager(+    mld_yvec_eager *y, const uint8_t rhoprime[MLDSA_CRHBYTES], uint16_t kappa)+__contract__(+  requires(memory_no_alias(y, sizeof(mld_yvec_eager)))+  requires(memory_no_alias(rhoprime, MLDSA_CRHBYTES))+  requires(kappa <= MLD_MAX_KAPPA)+  assigns(memory_slice(y, sizeof(mld_yvec_eager)))+  ensures(forall(k1, 0, MLDSA_L,+    array_bound(y->vec.vec[k1].coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)))+)+{+  mld_polyvecl_uniform_gamma1(&y->vec, rhoprime, kappa);+}++static MLD_INLINE void mld_yvec_get_poly_eager(mld_poly *buf,+                                               const mld_yvec_eager *y,+                                               unsigned int i)+__contract__(+  requires(memory_no_alias(buf, sizeof(mld_poly)))+  requires(memory_no_alias(y, sizeof(mld_yvec_eager)))+  requires(i < MLDSA_L)+  requires(array_bound(y->vec.vec[i].coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))+  assigns(memory_slice(buf, sizeof(mld_poly)))+  ensures(array_bound(buf->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))+) { *buf = y->vec.vec[i]; }+#endif /* !MLD_CONFIG_NO_SIGN_API && (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) \+        */+#if !defined(MLD_CONFIG_NO_SIGN_API) && \+    (defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))+static MLD_INLINE void mld_yvec_init_lazy(+    mld_yvec_lazy *y, const uint8_t rhoprime[MLDSA_CRHBYTES], uint16_t kappa)+__contract__(+  requires(memory_no_alias(y, sizeof(mld_yvec_lazy)))+  assigns(memory_slice(y, sizeof(mld_yvec_lazy)))+  ensures(y->rhoprime == old(rhoprime))+  ensures(y->kappa == old(kappa))+)+{+  y->rhoprime = rhoprime;+  y->kappa = kappa;+}++static MLD_INLINE void mld_yvec_get_poly_lazy(mld_poly *buf,+                                              const mld_yvec_lazy *y,+                                              unsigned int i)+__contract__(+  requires(memory_no_alias(buf, sizeof(mld_poly)))+  requires(memory_no_alias(y, sizeof(mld_yvec_lazy)))+  requires(i < MLDSA_L)+  requires(memory_no_alias(y->rhoprime, MLDSA_CRHBYTES))+  requires(y->kappa <= MLD_MAX_KAPPA)+  assigns(memory_slice(buf, sizeof(mld_poly)))+  ensures(array_bound(buf->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))+)+{+  /* Safety: y->kappa <= MLD_MAX_KAPPA and i < MLDSA_L, so y->kappa + i+   * fits in uint16_t. See MLD_MAX_KAPPA comment in params.h. */+  mld_poly_uniform_gamma1(buf, y->rhoprime, (uint16_t)(y->kappa + i));+}+#endif /* !MLD_CONFIG_NO_SIGN_API && (MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) \+        */++/* polymat */++#if !defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+/** Eager polymat: precomputed and stored full MLDSA_K x MLDSA_L matrix. */+typedef struct+{+  mld_polyvecl vec[MLDSA_K]; /**< Rows of the matrix. */+} mld_polymat_eager;+#endif /* !MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++/** Lazy polymat: store seed, sample elements A[k][l] on demand. */+typedef struct+{+  mld_poly cur; /**< On-demand sampled matrix element A[k][l]. */+  uint8_t rho[MLDSA_SEEDBYTES]; /**< Public seed used to expand A. */+} mld_polymat_lazy;++static MLD_INLINE void mld_poly_permute_bitrev_to_custom_optional(mld_poly *p)+__contract__(+  /* We don't specify that this is a permutation, only that it preserves+   * the bounds.+   * When the native NTT backend does not use the custom order, this is a no-op. */+  requires(memory_no_alias(p, sizeof(mld_poly)))+  requires(array_bound(p->coeffs, 0, MLDSA_N, 0, MLDSA_Q))+  assigns(memory_slice(p, sizeof(mld_poly)))+  ensures(array_bound(p->coeffs, 0, MLDSA_N, 0, MLDSA_Q))+)+{+#if defined(MLD_USE_NATIVE_NTT_CUSTOM_ORDER)+  mld_poly_permute_bitrev_to_custom(p->coeffs);+#else+  (void)p;+#endif+}++#if !defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+/**+ * Generates matrix A with uniformly random coefficients a_{i,j} by performing+ * rejection sampling on the output stream of SHAKE128(rho|j|i).+ *+ * @spec{Implements @[FIPS204, Algorithm 32, ExpandA].}+ *+ * @param[out] mat Pointer to output matrix.+ * @param[in]  rho Byte array containing seed rho.+ */+MLD_INTERNAL_API+void mld_polyvec_matrix_expand_eager(mld_polymat_eager *mat,+                                     const uint8_t rho[MLDSA_SEEDBYTES])+__contract__(+  requires(memory_no_alias(mat, sizeof(mld_polymat_eager)))+  requires(memory_no_alias(rho, MLDSA_SEEDBYTES))+  assigns(memory_slice(mat, sizeof(mld_polymat_eager)))+  ensures(forall(k1, 0, MLDSA_K, forall(l1, 0, MLDSA_L,+    array_bound(mat->vec[k1].vec[l1].coeffs, 0, MLDSA_N, 0, MLDSA_Q))))+);++/**+ * Compute row i of matrix-vector multiplication in NTT domain with pointwise+ * multiplication and multiplication by 2^{-32}.+ *+ * Input matrix and vector must be in NTT domain representation. Output+ * coefficients are bounded by MLDSA_Q in absolute value.+ *+ * @param[out] t_row Pointer to output row polynomial.+ * @param[in]  mat   Pointer to input matrix.+ * @param[in]  v     Pointer to input vector v.+ * @param      i     Row index, 0 <= i < MLDSA_K.+ */+MLD_INTERNAL_API+void mld_polyvec_matrix_pointwise_montgomery_row_eager(mld_poly *t_row,+                                                       mld_polymat_eager *mat,+                                                       const mld_polyvecl *v,+                                                       unsigned int i)+__contract__(+  requires(memory_no_alias(t_row, sizeof(mld_poly)))+  requires(memory_no_alias(mat, sizeof(mld_polymat_eager)))+  requires(memory_no_alias(v, sizeof(mld_polyvecl)))+  requires(i < MLDSA_K)+  requires(forall(l1, 0, MLDSA_L,+                  array_bound(mat->vec[i].vec[l1].coeffs, 0, MLDSA_N, 0, MLDSA_Q)))+  requires(forall(l2, 0, MLDSA_L,+                  array_abs_bound(v->vec[l2].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+  assigns(memory_slice(t_row, sizeof(mld_poly)))+  ensures(array_abs_bound(t_row->coeffs, 0, MLDSA_N, MLDSA_Q))+);++#if !defined(MLD_CONFIG_NO_SIGN_API)+/**+ * Compute w = invNTT(A * NTT(y)) for the signing y vector.+ *+ * The eager variant copies y into the scratch polyvecl, NTTs it in place,+ * calls the standard matrix-vector multiply, and finally inverse-NTTs the+ * result into w.+ *+ * @param[out] w       Pointer to output vector.+ * @param[in]  mat     Pointer to input matrix.+ * @param[in]  y       Pointer to (non-NTT) y vector.+ * @param[out] scratch Scratch polyvecl for NTT'd copy of y.+ */+MLD_INTERNAL_API+void mld_polyvec_matrix_pointwise_montgomery_yvec_eager(mld_polyveck *w,+                                                        mld_polymat_eager *mat,+                                                        const mld_yvec_eager *y,+                                                        mld_polyvecl *scratch)+__contract__(+  requires(memory_no_alias(w, sizeof(mld_polyveck)))+  requires(memory_no_alias(mat, sizeof(mld_polymat_eager)))+  requires(memory_no_alias(y, sizeof(mld_yvec_eager)))+  requires(memory_no_alias(scratch, sizeof(mld_polyvecl)))+  requires(forall(k1, 0, MLDSA_K, forall(l1, 0, MLDSA_L,+    array_bound(mat->vec[k1].vec[l1].coeffs, 0, MLDSA_N, 0, MLDSA_Q))))+  requires(forall(l2, 0, MLDSA_L,+    array_bound(y->vec.vec[l2].coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)))+  assigns(memory_slice(w, sizeof(mld_polyveck)))+  assigns(memory_slice(scratch, sizeof(mld_polyvecl)))+  ensures(forall(k0, 0, MLDSA_K,+    array_abs_bound(w->vec[k0].coeffs, 0, MLDSA_N, MLD_INTT_BOUND)))+);+#endif /* !MLD_CONFIG_NO_SIGN_API */+#endif /* !MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++#if defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)+MLD_INTERNAL_API+void mld_polyvec_matrix_expand_lazy(mld_polymat_lazy *mat,+                                    const uint8_t rho[MLDSA_SEEDBYTES])+__contract__(+  requires(memory_no_alias(mat, sizeof(mld_polymat_lazy)))+  requires(memory_no_alias(rho, MLDSA_SEEDBYTES))+  assigns(memory_slice(mat, sizeof(mld_polymat_lazy)))+);++#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+/**+ * Compute row i of matrix-vector multiplication in NTT domain with pointwise+ * multiplication and multiplication by 2^{-32}.+ *+ * Input vector must be in NTT domain representation; the matrix entries are+ * sampled on demand from the seed stored in mat->rho, using mat->cur as+ * scratch. Output coefficients are bounded by MLDSA_Q in absolute value.+ *+ * @param[out]    t_row Pointer to output row polynomial.+ * @param[in,out] mat   Pointer to input matrix (seed + scratch).+ * @param[in]     v     Pointer to input vector v.+ * @param         i     Row index, 0 <= i < MLDSA_K.+ */+MLD_INTERNAL_API+void mld_polyvec_matrix_pointwise_montgomery_row_lazy(mld_poly *t_row,+                                                      mld_polymat_lazy *mat,+                                                      const mld_polyvecl *v,+                                                      unsigned int i)+__contract__(+  requires(memory_no_alias(t_row, sizeof(mld_poly)))+  requires(memory_no_alias(mat, sizeof(mld_polymat_lazy)))+  requires(memory_no_alias(v, sizeof(mld_polyvecl)))+  requires(i < MLDSA_K)+  requires(forall(l1, 0, MLDSA_L,+                  array_abs_bound(v->vec[l1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+  assigns(memory_slice(t_row, sizeof(mld_poly)))+  assigns(memory_slice(mat, sizeof(mld_polymat_lazy)))+  ensures(array_abs_bound(t_row->coeffs, 0, MLDSA_N, MLDSA_Q))+);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+/**+ * Compute w = invNTT(A * NTT(y)) for the signing y vector.+ *+ * The lazy variant samples one column of y at a time, NTTs it into+ * &scratch->vec[0], and accumulates the matrix-vector product+ * column-by-column with on-demand sampling of A[k][l]. Only the first poly of+ * the polyvecl scratch is used; the polyvecl type is shared with the eager+ * variant for API uniformity (the storage is provided "for free" by the+ * caller's polyveck/polyvecl union in REDUCE_RAM mode).+ *+ * @param[out]    w       Pointer to output vector.+ * @param[in,out] mat     Pointer to input matrix.+ * @param[in]     y       Pointer to y seed/kappa.+ * @param[out]    scratch Scratch (only &scratch->vec[0] used).+ */+MLD_INTERNAL_API+void mld_polyvec_matrix_pointwise_montgomery_yvec_lazy(mld_polyveck *w,+                                                       mld_polymat_lazy *mat,+                                                       const mld_yvec_lazy *y,+                                                       mld_polyvecl *scratch)+__contract__(+  requires(memory_no_alias(w, sizeof(mld_polyveck)))+  requires(memory_no_alias(mat, sizeof(mld_polymat_lazy)))+  requires(memory_no_alias(y, sizeof(mld_yvec_lazy)))+  requires(memory_no_alias(scratch, sizeof(mld_polyvecl)))+  requires(memory_no_alias(y->rhoprime, MLDSA_CRHBYTES))+  requires(y->kappa <= MLD_MAX_KAPPA)+  assigns(memory_slice(w, sizeof(mld_polyveck)))+  assigns(memory_slice(mat, sizeof(mld_polymat_lazy)))+  assigns(memory_slice(scratch, sizeof(mld_polyvecl)))+  ensures(forall(k0, 0, MLDSA_K,+    array_abs_bound(w->vec[k0].coeffs, 0, MLDSA_N, MLD_INTT_BOUND)))+);+#endif /* !MLD_CONFIG_NO_SIGN_API */+#endif /* MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */++/* Dispatch: typedef and define based on MLD_CONFIG_REDUCE_RAM */+#if defined(MLD_CONFIG_REDUCE_RAM)+typedef mld_sk_s1hat_lazy mld_sk_s1hat;+typedef mld_sk_s2hat_lazy mld_sk_s2hat;+typedef mld_sk_t0hat_lazy mld_sk_t0hat;+typedef mld_polymat_lazy mld_polymat;+typedef mld_yvec_lazy mld_yvec;+#define mld_unpack_sk_s1hat mld_unpack_sk_s1hat_lazy+#define mld_unpack_sk_s2hat mld_unpack_sk_s2hat_lazy+#define mld_unpack_sk_t0hat mld_unpack_sk_t0hat_lazy+#if !defined(MLD_CONFIG_NO_SIGN_API)+#define mld_sk_s1hat_get_poly mld_sk_s1hat_get_poly_lazy+#define mld_sk_s2hat_get_poly mld_sk_s2hat_get_poly_lazy+#define mld_sk_t0hat_get_poly mld_sk_t0hat_get_poly_lazy+#endif+#define mld_polyvec_matrix_expand mld_polyvec_matrix_expand_lazy+#define mld_polyvec_matrix_pointwise_montgomery_row \+  mld_polyvec_matrix_pointwise_montgomery_row_lazy+#define mld_yvec_init mld_yvec_init_lazy+#define mld_yvec_get_poly mld_yvec_get_poly_lazy+#define mld_polyvec_matrix_pointwise_montgomery_yvec \+  mld_polyvec_matrix_pointwise_montgomery_yvec_lazy+#else /* MLD_CONFIG_REDUCE_RAM */+typedef mld_sk_s1hat_eager mld_sk_s1hat;+typedef mld_sk_s2hat_eager mld_sk_s2hat;+typedef mld_sk_t0hat_eager mld_sk_t0hat;+typedef mld_polymat_eager mld_polymat;+typedef mld_yvec_eager mld_yvec;+#define mld_unpack_sk_s1hat mld_unpack_sk_s1hat_eager+#define mld_unpack_sk_s2hat mld_unpack_sk_s2hat_eager+#define mld_unpack_sk_t0hat mld_unpack_sk_t0hat_eager+#if !defined(MLD_CONFIG_NO_SIGN_API)+#define mld_sk_s2hat_get_poly mld_sk_s2hat_get_poly_eager+#define mld_sk_s1hat_get_poly mld_sk_s1hat_get_poly_eager+#define mld_sk_t0hat_get_poly mld_sk_t0hat_get_poly_eager+#endif+#define mld_polyvec_matrix_expand mld_polyvec_matrix_expand_eager+#define mld_polyvec_matrix_pointwise_montgomery_row \+  mld_polyvec_matrix_pointwise_montgomery_row_eager+#define mld_yvec_init mld_yvec_init_eager+#define mld_yvec_get_poly mld_yvec_get_poly_eager+#define mld_polyvec_matrix_pointwise_montgomery_yvec \+  mld_polyvec_matrix_pointwise_montgomery_yvec_eager+#endif /* !MLD_CONFIG_REDUCE_RAM */++#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_SIGN_API || \+          !MLD_CONFIG_NO_VERIFY_API */+#endif /* !MLD_POLYVEC_LAZY_H */
+ cbits/mldsa/src/randombytes.h view
@@ -0,0 +1,26 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLD_RANDOMBYTES_H+#define MLD_RANDOMBYTES_H++#include <stddef.h>++#include "cbmc.h"+#include "common.h"++#if !defined(MLD_CONFIG_NO_RANDOMIZED_API)+#if !defined(MLD_CONFIG_CUSTOM_RANDOMBYTES)+MLD_MUST_CHECK_RETURN_VALUE+int randombytes(uint8_t *out, size_t outlen);++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_randombytes(uint8_t *out, size_t outlen)+__contract__(+  requires(memory_no_alias(out, outlen))+  assigns(memory_slice(out, outlen))+) { return randombytes(out, outlen); }+#endif /* !MLD_CONFIG_CUSTOM_RANDOMBYTES */+#endif /* !MLD_CONFIG_NO_RANDOMIZED_API */+#endif /* !MLD_RANDOMBYTES_H */
+ cbits/mldsa/src/reduce.h view
@@ -0,0 +1,144 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ */++#ifndef MLD_REDUCE_H+#define MLD_REDUCE_H++#include "cbmc.h"+#include "common.h"+#include "ct.h"+#include "debug.h"++/* check-magic: -4186625 == pow(2,32,MLDSA_Q) */+#define MLD_MONT (-4186625)++/* Upper bound for domain of mld_reduce32() */+#define MLD_REDUCE32_DOMAIN_MAX (INT32_MAX - ((int32_t)1 << 22))++/* Absolute bound for range of mld_reduce32() */+/* check-magic: 6283009 == (MLD_REDUCE32_DOMAIN_MAX - 255 * MLDSA_Q + 1) */+#define MLD_REDUCE32_RANGE_MAX 6283009++/**+ * Generic Montgomery reduction; given a 64-bit integer a, computes a 32-bit+ * integer congruent to a * R^-1 mod MLDSA_Q, where R=2^32.+ *+ * @spec{Implements @[FIPS204, Algorithm 49, MontgomeryReduce].}+ *+ * @param a Input integer to be reduced, of absolute value smaller or equal+ *          to INT64_MAX - 2^31 * MLDSA_Q.+ *+ * @return Integer congruent to a * R^-1 modulo MLDSA_Q, with absolute value+ *         <= |a| / 2^32 + MLDSA_Q / 2.+ *         In particular, if |a| < 2^31 * MLDSA_Q, the absolute value of the+ *         return value is < MLDSA_Q.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int32_t mld_montgomery_reduce(int64_t a)+__contract__(+  /* We don't attempt to express an input-dependent output bound+   * as the post-condition here, as all call-sites satisfy the+   * absolute input bound 2^31 * MLDSA_Q and higher-level+   * reasoning can be conducted using |return_value| < MLDSA_Q. */+  requires(a > -(((int64_t)1 << 31) * MLDSA_Q) &&+           a <  (((int64_t)1 << 31) * MLDSA_Q))+  ensures(return_value > -MLDSA_Q && return_value < MLDSA_Q)+)+{+  /* check-magic: 58728449 == unsigned_mod(pow(MLDSA_Q, -1, 2^32), 2^32) */+  const uint64_t QINV = 58728449;++  /*  Compute a*q^{-1} mod 2^32 in unsigned representatives */+  const uint32_t a_reduced = mld_cast_int64_to_uint32(a);+  const uint32_t a_inverted = (a_reduced * QINV) & UINT32_MAX;++  /* Lift to signed canonical representative mod 2^32. */+  const int32_t t = mld_cast_uint32_to_int32(a_inverted);++  int64_t r;++  mld_assert(a < +(INT64_MAX - (((int64_t)1 << 31) * MLDSA_Q)) &&+             a > -(INT64_MAX - (((int64_t)1 << 31) * MLDSA_Q)));++  r = a - (int64_t)t * MLDSA_Q;++  /*+   * PORTABILITY: Right-shift on a signed integer is, strictly-speaking,+   * implementation-defined for negative left argument. Here,+   * we assume it's sign-preserving "arithmetic" shift right. (C99 6.5.7 (5))+   */+  r = r >> 32;++  /* Bounds:+   *+   * By construction of the Montgomery multiplication, by the time we+   * compute r >> 32, r is divisible by 2^32, and hence+   *+   *   |r >> 32|  = |r| / 2^32+   *             <= |a| / 2^32 + MLDSA_Q / 2+   *+   * (In general, we would only have |x >> n| <= ceil(|x| / 2^n)).+   *+   * In particular, if |a| < 2^31 * MLDSA_Q, then |return_value| < MLDSA_Q.+   */+  return (int32_t)r;+}++/**+ * For finite field element a with a <= 2^{31} - 2^{22} - 1, compute+ * r congruent to a (mod MLDSA_Q) such that+ * -MLD_REDUCE32_RANGE_MAX <= r < MLD_REDUCE32_RANGE_MAX.+ *+ * @param a Finite field element.+ *+ * @return r.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int32_t mld_reduce32(int32_t a)+__contract__(+  requires(a <= MLD_REDUCE32_DOMAIN_MAX)+  ensures(return_value >= -MLD_REDUCE32_RANGE_MAX)+  ensures(return_value <   MLD_REDUCE32_RANGE_MAX)+)+{+  int32_t t;++  t = (a + ((int32_t)1 << 22)) >> 23;+  t = a - t * MLDSA_Q;+  mld_assert((t - a) % MLDSA_Q == 0);+  return t;+}++/**+ * Add MLDSA_Q if input coefficient is negative.+ *+ * @param a Finite field element.+ *+ * @return r.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int32_t mld_caddq(int32_t a)+__contract__(+  requires(a > -MLDSA_Q)+  requires(a < MLDSA_Q)+  ensures(return_value >= 0)+  ensures(return_value < MLDSA_Q)+  ensures(return_value == ((a >= 0) ? a : (a + MLDSA_Q)))+)+{+  return mld_ct_sel_int32(a + MLDSA_Q, a, mld_ct_cmask_neg_i32(a));+}+++#endif /* !MLD_REDUCE_H */
+ cbits/mldsa/src/rounding.h view
@@ -0,0 +1,265 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ */++#ifndef MLD_ROUNDING_H+#define MLD_ROUNDING_H++#include "cbmc.h"+#include "common.h"+#include "ct.h"+#include "debug.h"++/* Parameter set namespacing+ * This is to facilitate building multiple instances+ * of mldsa-native (e.g. with varying parameter sets)+ * within a single compilation unit. */+#define mld_power2round MLD_ADD_PARAM_SET(mld_power2round)+#define mld_decompose MLD_ADD_PARAM_SET(mld_decompose)+#define mld_make_hint MLD_ADD_PARAM_SET(mld_make_hint)+#define mld_use_hint MLD_ADD_PARAM_SET(mld_use_hint)+/* End of parameter set namespacing */++#define MLD_2_POW_D (1 << MLDSA_D)++/**+ * For finite field element a, compute a0, a1 such that+ * a mod^+ MLDSA_Q = a1*2^MLDSA_D + a0 with+ * -2^{MLDSA_D-1} < a0 <= 2^{MLDSA_D-1}. Assumes a to be standard+ * representative.+ *+ * @spec{Implements @[FIPS204, Algorithm 35, Power2Round].}+ *+ * @reference{In the reference implementation, a1 is passed as a return value+ * instead.}+ *+ * @param[out] a0 Pointer to output element a0.+ * @param[out] a1 Pointer to output element a1.+ * @param      a  Input element.+ */+static MLD_INLINE void mld_power2round(int32_t *a0, int32_t *a1, int32_t a)+__contract__(+  requires(memory_no_alias(a0, sizeof(int32_t)))+  requires(memory_no_alias(a1, sizeof(int32_t)))+  requires(a >= 0 && a < MLDSA_Q)+  assigns(memory_slice(a0, sizeof(int32_t)))+  assigns(memory_slice(a1, sizeof(int32_t)))+  ensures(*a0 > -(MLD_2_POW_D/2) && *a0 <= (MLD_2_POW_D/2))+  ensures(*a1 >= 0 && *a1 <= (MLDSA_Q - 1) / MLD_2_POW_D)+  ensures((*a1 * MLD_2_POW_D + *a0 - a) % MLDSA_Q == 0)+)+{+  *a1 = (a + (1 << (MLDSA_D - 1)) - 1) >> MLDSA_D;+  *a0 = a - (*a1 << MLDSA_D);+}++/**+ * For finite field element a, compute high and low bits a0, a1 such that+ * a mod^+ MLDSA_Q = a1 * 2 * MLDSA_GAMMA2 + a0 with+ * -MLDSA_GAMMA2 < a0 <= MLDSA_GAMMA2 except if+ * a1 = (MLDSA_Q-1)/(MLDSA_GAMMA2*2) where we set a1 = 0 and+ * -MLDSA_GAMMA2 <= a0 = a mod^+ MLDSA_Q - MLDSA_Q < 0. Assumes a to be+ * standard representative.+ *+ * @spec{Implements @[FIPS204, Algorithm 36, Decompose].}+ *+ * @reference{In the reference implementation, a1 is passed as a return value+ * instead.}+ *+ * @param[out] a0 Pointer to output element a0.+ * @param[out] a1 Pointer to output element a1.+ * @param      a  Input element.+ */+static MLD_INLINE void mld_decompose(int32_t *a0, int32_t *a1, int32_t a)+__contract__(+  requires(memory_no_alias(a0, sizeof(int32_t)))+  requires(memory_no_alias(a1, sizeof(int32_t)))+  requires(a >= 0 && a < MLDSA_Q)+  assigns(memory_slice(a0, sizeof(int32_t)))+  assigns(memory_slice(a1, sizeof(int32_t)))+  /* a0 = -MLDSA_GAMMA2 occurs exactly when a = MLDSA_Q - MLDSA_GAMMA2: the+   * border case of Decompose where a1 = (MLDSA_Q-1)/(2*MLDSA_GAMMA2) is+   * wrapped to 0 and a0 = a - MLDSA_Q (@[FIPS204, Algorithm 36, Decompose]) */+  ensures(*a0 >= -MLDSA_GAMMA2  && *a0 <= MLDSA_GAMMA2)+  ensures(*a1 >= 0 && *a1 < (MLDSA_Q-1)/(2*MLDSA_GAMMA2))+  ensures((*a1 * 2 * MLDSA_GAMMA2 + *a0 - a) % MLDSA_Q == 0)+)+{+  /*+   * The goal is to compute f1 = round-(f / (2*GAMMA2)), which can be computed+   * alternatively as round-(f / (128B)) = round-(ceil(f / 128) / B) where+   * B = 2*GAMMA2 / 128. Here round-() denotes "round half down".+   *+   * The equality round-(f / (128B)) = round-(ceil(f / 128) / B) can deduced+   * as follows. Since changing f to align-up(f, 128) can move f onto but not+   * across a rounding boundary for division by 128*B (note that we need B to be+   * even for this to work), and round- rounds down on the boundary, we have+   *+   *   round-(f / (128B)) = round-(align-up(f, 128) / (128B))+   *                      = round-((align-up(f, 128) / 128) / B)+   *                      = round-(ceil(f / 128) / B).+   */+  *a1 = (a + 127) >> 7;+  /* We know a >= 0 and a < MLDSA_Q, so... */+  /* check-magic: 65472 == round((MLDSA_Q-1)/128) */+  mld_assert(*a1 >= 0 && *a1 <= 65472);++#if MLD_CONFIG_PARAMETER_SET == 44+  /* check-magic: 1488 == 2 * intdiv(intdiv(MLDSA_Q - 1, 88), 128) */+  /* check-magic: 11275 == floor(2**24 / 1488) */+  /* check-magic: 1560281088 == 1 / (1 / 1488 - 11275 / 2**24) */+  /*+   * Compute f1 = round-(f1' / B) ≈ round(f1' * 11275 / 2^24). This is exact for+   * 0 <= f1' < 2^16.+   *+   * To see this, consider the (signed) error f1' * (1 / B - 11275 / 2^24)+   * between f1' / B and the (under-)approximation f1' * 11275 / 2^24. Because+   * eps := 1 / B - 11275 / 2^24 is 1 / 1560281088 ≈ 2^(-30.54) < 2^(-30), we+   * have 0 <= f1' * eps < 2^16 * 2^(-30) = 1 / 2^14 < 1 / 2^11 < 1 / B (note+   * that f1' is non-negative).+   *+   * On the other hand, 1 / B is the spacing between the integral multiples+   * of 1 / B, which includes all rounding boundaries n + 0.5 (since B is even).+   * Hence, if f1' / B is not of the form n + 0.5, then it is at least 1 / B+   * away from the nearest rounding boundary, so moving from f1' / B to+   * f1' * 11275 / 2^24 does not affect the rounding result, no matter the type+   * of rounding used in either side. In particular, we have round-(f1' / B) =+   * round(f1' * 11275 / 2^24) as claimed.+   *+   * As for the remaining case where f1' / B _is_ of the form n + 0.5, because+   * f1' * 11275 / 2^24 is slightly but strictly below f1' / B = n + 0.5 (note+   * that f1' and thus the error f1' * eps cannot be 0 here), it is always+   * rounded down to n. More precisely, we have round-(f1' / B) =+   * round(f1' * 11275 / 2^24), where the round-down on the LHS is essential,+   * and on the RHS the type of rounding again does not matter. This concludes+   * the proof.+   *+   * See proofs/isabelle/compress for a formalization of the above argument.+   */+  *a1 = (*a1 * 11275 + ((int32_t)1 << 23)) >> 24;+  mld_assert(*a1 >= 0 && *a1 <= 44);++  *a1 = mld_ct_sel_int32(0, *a1, mld_ct_cmask_neg_i32(43 - *a1));+  mld_assert(*a1 >= 0 && *a1 <= 43);+#else /* MLD_CONFIG_PARAMETER_SET == 44 */+  /* check-magic: 4092 == 2 * intdiv(intdiv(MLDSA_Q - 1, 32), 128) */+  /* check-magic: 1025 == floor(2**22 / 4092) */+  /* check-magic: 4290772992 == 1 / (1 / 4092 - 1025 / 2**22) */+  /*+   * Compute f1 = round-(f1' / B) ≈ round(f1' * 1025 / 2^22). This is exact for+   * 0 <= f1' < 2^16. Following the same argument above, it suffices to show+   * that f1' * eps < 1 / B, where eps := 1 / B - 1025 / 2^22. Indeed, we have+   * eps = 1 / 4290772992 ≈ 2^(-31.99) < 2^(-31), therefore f1' * eps <+   * 2^16 * 2^(-31) = 1 / 2^15 < 1 / 2^12 < 1 / B.+   */+  *a1 = (*a1 * 1025 + ((int32_t)1 << 21)) >> 22;+  mld_assert(*a1 >= 0 && *a1 <= 16);++  *a1 &= 15;+  mld_assert(*a1 >= 0 && *a1 <= 15);++#endif /* MLD_CONFIG_PARAMETER_SET != 44 */++  *a0 = a - *a1 * 2 * MLDSA_GAMMA2;+  *a0 = mld_ct_sel_int32(*a0 - MLDSA_Q, *a0,+                         mld_ct_cmask_neg_i32((MLDSA_Q - 1) / 2 - *a0));+}++/**+ * Decide a single hint bit from the low part a0 and high part a1 of a+ * coefficient: return 1 unless a0 lies in the range (-GAMMA2, GAMMA2] that+ * LowBits would produce, with the boundary value -GAMMA2 also admitted when+ * a1 == 0 (the Decompose border case).+ *+ * @note This is not a line-for-line implementation of FIPS 204's MakeHint(z, r)+ * (@[FIPS204, Algorithm 39, MakeHint]), which takes two ring elements and+ * returns [[HighBits(r) != HighBits(r + z)]]. Instead, it takes the already+ * decomposed low/high parts (a0, a1) of a coefficient and decides the hint bit+ * from them directly. As explained in the block comment of+ * mld_attempt_signature_generation (sign.c), for the specific values that arise+ * during signing -- a0 = w0 - cs2 + ct0 and a1 = w1 = HighBits(w) -- this is+ * equivalent to the spec's MakeHint(-ct0, w - cs2 + ct0) coefficient-wise.+ * Because it consumes (a0, a1) rather than (z, r), it relies on the caller+ * having computed a compatible decomposition.+ *+ * @param a0 Low bits of input element.+ * @param a1 High bits of input element.+ *+ * @return 1 if overflow, 0 otherwise.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE unsigned int mld_make_hint(int32_t a0, int32_t a1)+__contract__(+  ensures(return_value >= 0 && return_value <= 1)+  ensures(return_value == (a0 > MLDSA_GAMMA2 || a0 < -MLDSA_GAMMA2 ||+                           (a0 == -MLDSA_GAMMA2 && a1 != 0)))+)+{+  if (a0 > MLDSA_GAMMA2 || a0 < -MLDSA_GAMMA2 ||+      (a0 == -MLDSA_GAMMA2 && a1 != 0))+  {+    return 1;+  }++  return 0;+}++/**+ * Correct high bits according to hint.+ *+ * @spec{Implements @[FIPS204, Algorithm 40, UseHint].}+ *+ * @param a    Input element.+ * @param hint Hint bit.+ *+ * @return Corrected high bits.+ */+MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int32_t mld_use_hint(int32_t a, int32_t hint)+__contract__(+  requires(hint >= 0 && hint <= 1)+  requires(a >= 0 && a < MLDSA_Q)+  ensures(return_value >= 0 && return_value < (MLDSA_Q-1)/(2*MLDSA_GAMMA2))+)+{+  int32_t a0, a1;++  mld_decompose(&a0, &a1, a);+  if (hint == 0)+  {+    return a1;+  }++#if MLD_CONFIG_PARAMETER_SET == 44+  if (a0 > 0)+  {+    return (a1 == 43) ? 0 : a1 + 1;+  }+  else+  {+    return (a1 == 0) ? 43 : a1 - 1;+  }+#else  /* MLD_CONFIG_PARAMETER_SET == 44 */+  if (a0 > 0)+  {+    return (a1 + 1) & 15;+  }+  else+  {+    return (a1 - 1) & 15;+  }+#endif /* MLD_CONFIG_PARAMETER_SET != 44 */+}+++#endif /* !MLD_ROUNDING_H */
+ cbits/mldsa/src/sign.c view
@@ -0,0 +1,1720 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS140_3_IG]+ *   Implementation Guidance for FIPS 140-3 and the Cryptographic Module+ *   Validation Program+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/projects/cryptographic-module-validation-program/fips-140-3-ig-announcements+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ *+ * - [FIPS204_UPDATES]+ *   FIPS 204 Potential Updates (Errata)+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/files/pubs/fips/204/final/docs/fips-204-potential-updates.xlsx+ *+ * - [Round3_Spec]+ *   CRYSTALS-Dilithium Algorithm Specifications and Supporting Documentation+ *   (Version 3.1)+ *   Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé+ *   https://pq-crystals.org/dilithium/data/dilithium-specification-round3-20210208.pdf+ */++#include "sign.h"++#include "cbmc.h"+#include "ct.h"+#include "debug.h"+#include "packing.h"+#include "poly.h"+#include "poly_kl.h"+#include "polyvec.h"+#include "randombytes.h"+#include "symmetric.h"++/* Parameter set namespacing+ * This is to facilitate building multiple instances+ * of mldsa-native (e.g. with varying parameter sets)+ * within a single compilation unit. */+#define mld_check_pct MLD_ADD_PARAM_SET(mld_check_pct) MLD_CONTEXT_PARAMETERS_2+#define mld_sample_s1_s2 MLD_ADD_PARAM_SET(mld_sample_s1_s2)+#define mld_validate_hash_length MLD_ADD_PARAM_SET(mld_validate_hash_length)+#define mld_get_hash_oid MLD_ADD_PARAM_SET(mld_get_hash_oid)+#define mld_H MLD_ADD_PARAM_SET(mld_H)+#define mld_compute_pack_z MLD_ADD_PARAM_SET(mld_compute_pack_z)+#define mld_attempt_signature_generation \+  MLD_ADD_PARAM_SET(mld_attempt_signature_generation) MLD_CONTEXT_PARAMETERS_8+#define mld_compute_pack_t0_t1 \+  MLD_ADD_PARAM_SET(mld_compute_pack_t0_t1) MLD_CONTEXT_PARAMETERS_5+#define mld_get_max_signing_attempts \+  MLD_ADD_PARAM_SET(mld_get_max_signing_attempts)++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+static int mld_check_pct(uint8_t const pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+                         uint8_t const sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                         MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES))+  requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES))+  ensures(return_value == 0 || return_value == MLD_ERR_OUT_OF_MEMORY ||+          return_value == MLD_ERR_RNG_FAIL ||+          return_value == MLD_ERR_SIGNING_PAUSED ||+          return_value == MLD_ERR_PCT_FAIL)+);++#if defined(MLD_CONFIG_KEYGEN_PCT)+/**+ * Pair-wise Consistency Test (PCT) for DSA keypairs.+ *+ * @[FIPS140_3_IG] TE10.35.02+ * (https://csrc.nist.gov/csrc/media/Projects/cryptographic-module-validation-program/documents/fips%20140-3/FIPS%20140-3%20IG.pdf).+ *+ * Validates that a generated public/private key pair can correctly sign and+ * verify data. Performs signature generation using the private key (sk),+ * followed by signature verification using the public key (pk).+ *+ * @note @[FIPS204] requires that public/private key pairs are to be used+ * only for the calculation and/or verification of digital signatures.+ *+ * @param[in] pk      Public key.+ * @param[in] sk      Secret key.+ * @param     context Application context. Only present when+ *                    MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                    MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                      Success.+ * @retval MLD_ERR_OUT_OF_MEMORY  MLD_CONFIG_CUSTOM_ALLOC_FREE was used and an+ *                                allocation via MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_RNG_FAIL       Random number generation failed.+ * @retval MLD_ERR_SIGNING_PAUSED The PCT's signing step was paused by a+ *                                MLD_CONFIG_SIGN_HOOK_ATTEMPT hook.+ * @retval MLD_ERR_PCT_FAIL       The consistency check failed.+ */+static int mld_check_pct(uint8_t const pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+                         uint8_t const sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                         MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  MLD_ALIGN uint8_t message[1] = {0};+  int ret;+  MLD_ALLOC(signature, uint8_t, MLDSA_CRYPTO_BYTES, context);+  MLD_ALLOC(pk_test, uint8_t, MLDSA_CRYPTO_PUBLICKEYBYTES, context);++  if (signature == NULL || pk_test == NULL)+  {+    ret = MLD_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  /* Copy public key for testing */+  mld_memcpy(pk_test, pk, MLDSA_CRYPTO_PUBLICKEYBYTES);++  /* Sign a test message using the original secret key */+  ret = mld_sign_signature(signature, message, sizeof(message), NULL, 0, sk,+                           context);+  if (ret != 0)+  {+    goto cleanup;+  }++#if defined(MLD_CONFIG_KEYGEN_PCT_BREAKAGE_TEST)+  /* Deliberately break public key for testing purposes */+  if (mld_break_pct())+  {+    pk_test[0] = ~pk_test[0];+  }+#endif /* MLD_CONFIG_KEYGEN_PCT_BREAKAGE_TEST */++  /* Verify the signature using the (potentially corrupted) public key. */+  ret = mld_sign_verify(signature, message, sizeof(message), NULL, 0, pk_test,+                        context);+  if (ret != 0)+  {+    goto cleanup;+  }++cleanup:+  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  MLD_FREE(pk_test, uint8_t, MLDSA_CRYPTO_PUBLICKEYBYTES, context);+  MLD_FREE(signature, uint8_t, MLDSA_CRYPTO_BYTES, context);++  /* A failed signing operation or an invalid signature hint at a faulty+   * implementation and map to a dedicated error code for PCT failure. */+  if (ret == MLD_ERR_INVALID_SIGNATURE ||+      ret == MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED)+  {+    ret = MLD_ERR_PCT_FAIL;+  }++  /* Other error codes, e.g. platform failures like out of memory or+   * randomness failure, are passed on unmodified. */++  return ret;+}+#else  /* MLD_CONFIG_KEYGEN_PCT */+static int mld_check_pct(uint8_t const pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+                         uint8_t const sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                         MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  /* Skip PCT */+  ((void)pk);+  ((void)sk);+  MLD_CONTEXT_UNUSED(context);+  return 0;+}+#endif /* !MLD_CONFIG_KEYGEN_PCT */++/**+ * Sample the short secret vectors s1 (length MLDSA_L) and s2 (length MLDSA_K)+ * with coefficients in [-MLDSA_ETA, MLDSA_ETA] from the seed.+ *+ * @spec{Implements @[FIPS204, Algorithm 33, ExpandS].}+ *+ * @param[out] s1   Output vector s1.+ * @param[out] s2   Output vector s2.+ * @param[in]  seed Byte array with seed of length MLDSA_CRHBYTES.+ */+static void mld_sample_s1_s2(mld_polyvecl *s1, mld_polyveck *s2,+                             const uint8_t seed[MLDSA_CRHBYTES])+__contract__(+  requires(memory_no_alias(s1, sizeof(mld_polyvecl)))+  requires(memory_no_alias(s2, sizeof(mld_polyveck)))+  requires(memory_no_alias(seed, MLDSA_CRHBYTES))+  assigns(object_whole(s1), object_whole(s2))+  ensures(forall(l0, 0, MLDSA_L, array_abs_bound(s1->vec[l0].coeffs, 0, MLDSA_N, MLDSA_ETA + 1)))+  ensures(forall(k0, 0, MLDSA_K, array_abs_bound(s2->vec[k0].coeffs, 0, MLDSA_N, MLDSA_ETA + 1)))+)+{+/* Sample short vectors s1 and s2 */+#if defined(MLD_CONFIG_SERIAL_FIPS202_ONLY)+  int i;+  uint16_t nonce = 0;+  /* Safety: The nonces are at most 14 (MLDSA_L + MLDSA_K - 1), and, hence, the+   * casts are safe. */+  for (i = 0; i < MLDSA_L; i++)+  {+    mld_poly_uniform_eta(&s1->vec[i], seed, (uint8_t)(nonce + i));+  }+  for (i = 0; i < MLDSA_K; i++)+  {+    mld_poly_uniform_eta(&s2->vec[i], seed, (uint8_t)(nonce + MLDSA_L + i));+  }+#else /* MLD_CONFIG_SERIAL_FIPS202_ONLY */+#if MLD_CONFIG_PARAMETER_SET == 44+  mld_poly_uniform_eta_4x(&s1->vec[0], &s1->vec[1], &s1->vec[2], &s1->vec[3],+                          seed, 0, 1, 2, 3);+  mld_poly_uniform_eta_4x(&s2->vec[0], &s2->vec[1], &s2->vec[2], &s2->vec[3],+                          seed, 4, 5, 6, 7);+#elif MLD_CONFIG_PARAMETER_SET == 65+  mld_poly_uniform_eta_4x(&s1->vec[0], &s1->vec[1], &s1->vec[2], &s1->vec[3],+                          seed, 0, 1, 2, 3);+  mld_poly_uniform_eta_4x(&s1->vec[4], &s2->vec[0], &s2->vec[1],+                          &s2->vec[2] /* irrelevant */, seed, 4, 5, 6,+                          0xFF /* irrelevant */);+  mld_poly_uniform_eta_4x(&s2->vec[2], &s2->vec[3], &s2->vec[4], &s2->vec[5],+                          seed, 7, 8, 9, 10);+#elif MLD_CONFIG_PARAMETER_SET == 87+  mld_poly_uniform_eta_4x(&s1->vec[0], &s1->vec[1], &s1->vec[2], &s1->vec[3],+                          seed, 0, 1, 2, 3);+  mld_poly_uniform_eta_4x(&s1->vec[4], &s1->vec[5], &s1->vec[6],+                          &s2->vec[0] /* irrelevant */, seed, 4, 5, 6,+                          0xFF /* irrelevant */);+  mld_poly_uniform_eta_4x(&s2->vec[0], &s2->vec[1], &s2->vec[2], &s2->vec[3],+                          seed, 7, 8, 9, 10);+  mld_poly_uniform_eta_4x(&s2->vec[4], &s2->vec[5], &s2->vec[6], &s2->vec[7],+                          seed, 11, 12, 13, 14);+#endif /* MLD_CONFIG_PARAMETER_SET == 87 */+#endif /* !MLD_CONFIG_SERIAL_FIPS202_ONLY */+}++/**+ * Compute t = A*s1hat + s2 row by row, decompose each row into t0[k] and+ * t1[k] via power2round, and bit-pack t1[k] into pk_t1 and t0[k] into the+ * t0_packed buffer. Used by both keygen and pk_from_sk.+ *+ * @spec{Partially implements @[FIPS204, Algorithm 22, pkEncode] (t1) and+ * @[FIPS204, Algorithm 24, skEncode] (t0).}+ *+ * @param[out] pk_t1     Output buffer for packed t1 (size+ *                       MLDSA_K * MLDSA_POLYT1_PACKEDBYTES; i.e. the t1+ *                       region of pk).+ * @param[out] t0_packed Output buffer for packed t0 (size+ *                       MLDSA_K * MLDSA_POLYT0_PACKEDBYTES).+ * @param[in]  s1hat     s1 in NTT domain.+ * @param[in]  s2        s2.+ * @param[in]  rho       Byte array containing seed rho.+ * @param      context   Application context. Only present when+ *                       MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                       MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @return - 0: Success.+ *         - MLD_ERR_OUT_OF_MEMORY: If MLD_CONFIG_CUSTOM_ALLOC_FREE is used and+ *           an allocation via MLD_CUSTOM_ALLOC returned NULL.+ */+MLD_MUST_CHECK_RETURN_VALUE+static int mld_compute_pack_t0_t1(+    uint8_t pk_t1[MLDSA_K * MLDSA_POLYT1_PACKEDBYTES],+    uint8_t t0_packed[MLDSA_K * MLDSA_POLYT0_PACKEDBYTES],+    const mld_polyvecl *s1hat, const mld_polyveck *s2,+    const uint8_t rho[MLDSA_SEEDBYTES],+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(pk_t1, MLDSA_K * MLDSA_POLYT1_PACKEDBYTES))+  requires(memory_no_alias(t0_packed, MLDSA_K * MLDSA_POLYT0_PACKEDBYTES))+  requires(memory_no_alias(s1hat, sizeof(mld_polyvecl)))+  requires(memory_no_alias(s2, sizeof(mld_polyveck)))+  requires(memory_no_alias(rho, MLDSA_SEEDBYTES))+  requires(forall(l1, 0, MLDSA_L,+    array_abs_bound(s1hat->vec[l1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+  requires(forall(k2, 0, MLDSA_K,+    array_bound(s2->vec[k2].coeffs, 0, MLDSA_N,+      MLD_POLYETA_UNPACK_LOWER_BOUND, MLDSA_ETA + 1)))+  assigns(memory_slice(pk_t1, MLDSA_K * MLDSA_POLYT1_PACKEDBYTES))+  assigns(memory_slice(t0_packed, MLDSA_K * MLDSA_POLYT0_PACKEDBYTES))+  ensures(return_value == 0 || return_value == MLD_ERR_OUT_OF_MEMORY))+{+  unsigned int k;+  int ret;+  MLD_ALLOC(mat, mld_polymat, 1, context);+  MLD_ALLOC(t0k, mld_poly, 1, context);+  MLD_ALLOC(t1k, mld_poly, 1, context);++  if (mat == NULL || t0k == NULL || t1k == NULL)+  {+    ret = MLD_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  /* Expand matrix */+  mld_polyvec_matrix_expand(mat, rho);++  for (k = 0; k < MLDSA_K; k++)+  __loop__(+    assigns(k, memory_slice(pk_t1, MLDSA_K * MLDSA_POLYT1_PACKEDBYTES),+            memory_slice(t0_packed, MLDSA_K * MLDSA_POLYT0_PACKEDBYTES),+            memory_slice(t0k, sizeof(mld_poly)),+            memory_slice(t1k, sizeof(mld_poly))+            MLD_IF_REDUCE_RAM(, memory_slice(mat, sizeof(mld_polymat))))+    invariant(k <= MLDSA_K)+    decreases(MLDSA_K - k)+  )+  {+    /* t0k = (A * s1hat)_k in NTT domain */+    mld_polyvec_matrix_pointwise_montgomery_row(t0k, mat, s1hat, k);++    /* t0k = invNTT(t0k) */+    mld_poly_invntt_tomont(t0k);++    /* t0k += s2[k] */+    mld_poly_add(t0k, &s2->vec[k]);++    /* Reference: The following reduction is not present in the reference+     *            implementation. Omitting this reduction requires the output+     *            of the invntt to be small enough such that the addition of+     *            s2 does not result in absolute values >= MLDSA_Q. While our+     *            C, x86_64, and AArch64 invntt implementations produce small+     *            enough values for this to work out, it complicates the+     *            bounds reasoning. We instead add an additional reduction,+     *            and can consequently, relax the bounds requirements for the+     *            invntt.+     */+    mld_poly_reduce(t0k);++    /* Decompose into t1[k] and t0[k] (in place into t0k). */+    mld_poly_caddq(t0k);+    mld_poly_power2round(t1k, t0k, t0k);++    /* Pack t1[k] into pk and t0[k] into the t0 output buffer. */+    mld_polyt1_pack(pk_t1 + k * MLDSA_POLYT1_PACKEDBYTES, t1k);+    mld_polyt0_pack(t0_packed + k * MLDSA_POLYT0_PACKEDBYTES, t0k);+  }++  ret = 0;+cleanup:+  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  MLD_FREE(t1k, mld_poly, 1, context);+  MLD_FREE(t0k, mld_poly, 1, context);+  MLD_FREE(mat, mld_polymat, 1, context);+  return ret;+}++MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_keypair_internal(uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+                              uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                              const uint8_t seed[MLDSA_SEEDBYTES],+                              MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  int ret;+  const uint8_t *rho, *rhoprime, *key;++  MLD_ALLOC(seedbuf, uint8_t, 2 * MLDSA_SEEDBYTES + MLDSA_CRHBYTES, context);+  MLD_ALLOC(inbuf, uint8_t, MLDSA_SEEDBYTES + 2, context);+  MLD_ALLOC(tr, uint8_t, MLDSA_TRBYTES, context);+  MLD_ALLOC(s1, mld_polyvecl, 1, context);+  MLD_ALLOC(s2, mld_polyveck, 1, context);++  if (seedbuf == NULL || inbuf == NULL || tr == NULL || s1 == NULL ||+      s2 == NULL)+  {+    ret = MLD_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  /* Get randomness for rho, rhoprime and key */+  mld_memcpy(inbuf, seed, MLDSA_SEEDBYTES);+  inbuf[MLDSA_SEEDBYTES + 0] = MLDSA_K;+  inbuf[MLDSA_SEEDBYTES + 1] = MLDSA_L;+  mld_shake256(seedbuf, 2 * MLDSA_SEEDBYTES + MLDSA_CRHBYTES, inbuf,+               MLDSA_SEEDBYTES + 2);+  rho = seedbuf;+  rhoprime = rho + MLDSA_SEEDBYTES;+  key = rhoprime + MLDSA_CRHBYTES;++  /* Constant time: rho is part of the public key and, hence, public. */+  MLD_CT_TESTING_DECLASSIFY(rho, MLDSA_SEEDBYTES);++  /* Sample s1 and s2 */+  mld_sample_s1_s2(s1, s2, rhoprime);++  /* Pack s1 into sk before NTT */+  mld_pack_sk_s1(sk, s1);++  /* NTT s1 in place to use as s1hat */+  mld_polyvecl_ntt(s1);++  /* Pack rho into pk */+  mld_memcpy(pk + MLDSA_PK_RHO_OFFSET, rho, MLDSA_SEEDBYTES);++  /* Compute t = A*s1hat + s2 row by row, decompose into t1/t0, and pack+   * t1 into pk and t0 directly into the t0 region of sk. */+  ret = mld_compute_pack_t0_t1(pk + MLDSA_PK_T1_OFFSET, sk + MLDSA_SK_T0_OFFSET,+                               s1, s2, rho, context);+  if (ret != 0)+  {+    goto cleanup;+  }++  /* Compute tr = H(pk) */+  mld_shake256(tr, MLDSA_TRBYTES, pk, MLDSA_CRYPTO_PUBLICKEYBYTES);++  /* Pack remaining secret key components (s1 and t0 already packed) */+  mld_pack_sk_rho_key_tr_s2(sk, rho, tr, key, s2);++  /* Constant time: pk is the public key, inherently public data */+  MLD_CT_TESTING_DECLASSIFY(pk, MLDSA_CRYPTO_PUBLICKEYBYTES);++cleanup:+  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  MLD_FREE(s2, mld_polyveck, 1, context);+  MLD_FREE(s1, mld_polyvecl, 1, context);+  MLD_FREE(tr, uint8_t, MLDSA_TRBYTES, context);+  MLD_FREE(inbuf, uint8_t, MLDSA_SEEDBYTES + 2, context);+  MLD_FREE(seedbuf, uint8_t, 2 * MLDSA_SEEDBYTES + MLDSA_CRHBYTES, context);++  /* Pairwise Consistency Test (PCT) @[FIPS140_3_IG, p.87] */+  /* Do this after freeing all temporaries. */+  if (ret == 0)+  {+    ret = mld_check_pct(pk, sk, context);+  }++  if (ret != 0)+  {+    /* Clear caller outputs on failure. */+    mld_zeroize(pk, MLDSA_CRYPTO_PUBLICKEYBYTES);+    mld_zeroize(sk, MLDSA_CRYPTO_SECRETKEYBYTES);+  }++  return ret;+}++#if !defined(MLD_CONFIG_CORE_API_ONLY)+#if !defined(MLD_CONFIG_NO_RANDOMIZED_API)+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_keypair(uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+                     uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                     MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  int ret;+  MLD_ALLOC(seed, uint8_t, MLDSA_SEEDBYTES, context);++  if (seed == NULL)+  {+    ret = MLD_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  if (mld_randombytes(seed, MLDSA_SEEDBYTES) != 0)+  {+    ret = MLD_ERR_RNG_FAIL;+    goto cleanup;+  }+  MLD_CT_TESTING_SECRET(seed, MLDSA_SEEDBYTES);+  ret = mld_sign_keypair_internal(pk, sk, seed, context);++cleanup:+  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  MLD_FREE(seed, uint8_t, MLDSA_SEEDBYTES, context);+  return ret;+}+#endif /* !MLD_CONFIG_NO_RANDOMIZED_API */+#endif /* !MLD_CONFIG_CORE_API_ONLY */+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+/**+ * Abstracts application of SHAKE256 to one, two or three blocks of data,+ * yielding a user-requested size of output.+ *+ * @param[out] out    Pointer to output.+ * @param      outlen Requested output length in bytes.+ * @param[in]  in1    Pointer to input block 1. Must NOT be NULL.+ * @param      in1len Length of input in1 in bytes.+ * @param[in]  in2    Pointer to input block 2. May be NULL if in2len == 0,+ *                    in which case this block is ignored.+ * @param      in2len Length of input in2 in bytes.+ * @param[in]  in3    Pointer to input block 3. May be NULL if in3len == 0,+ *                    in which case this block is ignored.+ * @param      in3len Length of input in3 in bytes.+ */+static void mld_H(uint8_t *out, size_t outlen, const uint8_t *in1,+                  size_t in1len, const uint8_t *in2, size_t in2len,+                  const uint8_t *in3, size_t in3len)+__contract__(+  requires(in1len <= MLD_MAX_BUFFER_SIZE)+  requires(in2len <= MLD_MAX_BUFFER_SIZE)+  requires(in3len <= MLD_MAX_BUFFER_SIZE)+  requires(outlen <= 8 * SHAKE256_RATE /* somewhat arbitrary bound */)+  requires(memory_no_alias(in1, in1len))+  requires(in2len == 0 || memory_no_alias(in2, in2len))+  requires(in3len == 0 || memory_no_alias(in3, in3len))+  requires(memory_no_alias(out, outlen))+  assigns(memory_slice(out, outlen))+)+{+  mld_shake256ctx state;+  mld_shake256_init(&state);+  mld_shake256_absorb(&state, in1, in1len);+  if (in2len != 0)+  {+    mld_shake256_absorb(&state, in2, in2len);+  }+  if (in3len != 0)+  {+    mld_shake256_absorb(&state, in3, in3len);+  }+  mld_shake256_finalize(&state);+  mld_shake256_squeeze(out, outlen, &state);+  mld_shake256_release(&state);++  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(&state, sizeof(state));+}+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+/* MLD_MAX_KAPPA (see params.h) bounds the rejection-sampling counter kappa;+ * MLD_MAX_SIGNING_ATTEMPTS below turns that into a bound on attempts. */++/**+ * Compute z = y + s1*c, check that z has coefficients smaller than+ * MLDSA_GAMMA1 - MLDSA_BETA, and pack z into the signature buffer.+ *+ * @reference{This function is inlined into mld_sign_signature in the+ * reference implementation.}+ *+ * @param[in,out] sig   Output signature.+ * @param[in]     cp    Challenge polynomial.+ * @param[in]     s1hat Secret vector s1 in NTT domain.+ * @param[in]     y     Masking vector y (or seed in REDUCE_RAM mode).+ * @param[out]    z     Scratch polynomial for z computation.+ * @param[out]    tmp   Scratch polynomial.+ *+ * @return - 0: Success (z has coefficients smaller than+ *           MLDSA_GAMMA1 - MLDSA_BETA).+ *         - MLD_ERR_FAIL: z rejected (norm check failed).+ *         - MLD_ERR_OUT_OF_MEMORY: If MLD_CONFIG_CUSTOM_ALLOC_FREE is used and+ *           an allocation via MLD_CUSTOM_ALLOC returned NULL.+ */+MLD_MUST_CHECK_RETURN_VALUE+static int mld_compute_pack_z(uint8_t sig[MLDSA_CRYPTO_BYTES],+                              const mld_poly *cp, const mld_sk_s1hat *s1hat,+                              const mld_yvec *y, mld_poly *z, mld_poly *tmp)+__contract__(+  requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES))+  requires(memory_no_alias(cp, sizeof(mld_poly)))+  requires(memory_no_alias(s1hat, sizeof(mld_sk_s1hat)))+  requires(memory_no_alias(y, sizeof(mld_yvec)))+  requires(memory_no_alias(z, sizeof(mld_poly)))+  requires(memory_no_alias(tmp, sizeof(mld_poly)))+  requires(array_abs_bound(cp->coeffs, 0, MLDSA_N, MLD_NTT_BOUND))+  MLD_IF_NOT_REDUCE_RAM(+    requires(forall(k0, 0, MLDSA_L,+      array_bound(y->vec.vec[k0].coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)))+    requires(forall(k1, 0, MLDSA_L, array_abs_bound(s1hat->vec.vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+  )+  MLD_IF_REDUCE_RAM(+    requires(memory_no_alias(s1hat->packed, MLDSA_L * MLDSA_POLYETA_PACKEDBYTES))+    requires(memory_no_alias(y->rhoprime, MLDSA_CRHBYTES))+    requires(y->kappa <= MLD_MAX_KAPPA)+  )+  assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES))+  assigns(memory_slice(z, sizeof(mld_poly)))+  assigns(memory_slice(tmp, sizeof(mld_poly)))+  ensures(return_value == 0 || return_value == MLD_ERR_FAIL ||+          return_value == MLD_ERR_OUT_OF_MEMORY)+)+{+  unsigned int i;+  uint32_t z_invalid;+  for (i = 0; i < MLDSA_L; i++)+  __loop__(+    assigns(i, memory_slice(z, sizeof(mld_poly)),+            memory_slice(tmp, sizeof(mld_poly)),+            memory_slice(sig, MLDSA_CRYPTO_BYTES))+    invariant(i <= MLDSA_L)+    decreases(MLDSA_L - i)+  )+  {+    mld_sk_s1hat_get_poly(z, s1hat, i);+    mld_poly_pointwise_montgomery(z, cp);+    mld_poly_invntt_tomont(z);+    mld_yvec_get_poly(tmp, y, i);+    mld_poly_add(z, tmp);+    mld_poly_reduce(z);++    z_invalid = mld_poly_chknorm(z, MLDSA_GAMMA1 - MLDSA_BETA);+    /* Constant time: It is fine (and prohibitively expensive to avoid)+     * to leak the result of the norm check and which polynomial in z caused a+     * rejection. It would even be okay to leak which coefficient led to+     * rejection as the candidate signature will be discarded anyway.+     * See Section 5.5 of @[Round3_Spec]. */+    MLD_CT_TESTING_DECLASSIFY(&z_invalid, sizeof(uint32_t));+    if (z_invalid)+    {+      return MLD_ERR_FAIL; /* reject */+    }+    /* If z is valid, then its coefficients are bounded by+     * MLDSA_GAMMA1 - MLDSA_BETA. This will be needed below+     * to prove the pre-condition of pack_sig_z() */+    mld_assert_abs_bound(z, MLDSA_N, (MLDSA_GAMMA1 - MLDSA_BETA));++    /* After the norm check, the distribution of each coefficient of z is+     * independent of the secret key and it can, hence, be considered+     * public. It is, hence, okay to immediately pack it into the user-provided+     * signature buffer. */+    mld_pack_sig_z(sig, z, i);+  }+  return 0;+}++/* Effective bound on signing attempts: the configured bound+ * MLD_CONFIG_MAX_SIGNING_ATTEMPTS (see mldsa_native_config.h) if set, otherwise+ * the hard type-safety bound MLD_MAX_KAPPA / MLDSA_L (see MLD_MAX_KAPPA in+ * params.h). */+#if defined(MLD_CONFIG_MAX_SIGNING_ATTEMPTS)++#if !defined(MLD_ALLOW_NONCOMPLIANT_SIGNING_BOUND) && \+    MLD_CONFIG_MAX_SIGNING_ATTEMPTS < 821+#error Bad configuration: MLD_CONFIG_MAX_SIGNING_ATTEMPTS must be >= 821 for FIPS 204 compliance @[FIPS204, Appendix C] @[FIPS204_UPDATES]+#endif++#if MLD_CONFIG_MAX_SIGNING_ATTEMPTS < 1+#error Bad configuration: MLD_CONFIG_MAX_SIGNING_ATTEMPTS must be >= 1+#endif++#if MLD_CONFIG_MAX_SIGNING_ATTEMPTS > MLD_MAX_KAPPA / MLDSA_L+#error Bad configuration: MLD_CONFIG_MAX_SIGNING_ATTEMPTS exceeds the maximum allowed value.+#endif++#define MLD_MAX_SIGNING_ATTEMPTS MLD_CONFIG_MAX_SIGNING_ATTEMPTS+#else /* MLD_CONFIG_MAX_SIGNING_ATTEMPTS */+#define MLD_MAX_SIGNING_ATTEMPTS (MLD_MAX_KAPPA / MLDSA_L)+#endif /* !MLD_CONFIG_MAX_SIGNING_ATTEMPTS */++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE uint16_t mld_get_max_signing_attempts(void)+__contract__(+  ensures(return_value >= 1)+  ensures(return_value <= MLD_MAX_KAPPA / MLDSA_L)+)+{+  /* cassert(0) ensures CBMC uses the contract rather than inlining the body,+   * keeping proofs agnostic of the configured value. */+  cassert(0);+  return MLD_MAX_SIGNING_ATTEMPTS;+}++/**+ * Attempt to generate a single signature: one iteration of the+ * ML-DSA.Sign_internal rejection-sampling loop.+ *+ * @spec{Implements one iteration of the rejection-sampling loop body of+ * @[FIPS204, Algorithm 7, ML-DSA.Sign_internal] (lines 11-30) plus, on success,+ * the sigEncode step (line 33). The per-signature setup (Algorithm 7 lines 1-7:+ * skDecode, NTT of s1/s2/t0, ExpandA, and computation of mu and rhoprime) and+ * the loop itself (lines 8-10, 31-32) live in the caller+ * mld_sign_signature_internal; kappa is this iteration's counter, used to+ * sample y.}+ *+ * @reference{This code differs from the reference implementation in that it+ * factors out the core signature generation step into a distinct function+ * here in order to improve efficiency of CBMC proof.}+ *+ * @param[out] sig      Pointer to output signature.+ * @param[in]  mu       Pointer to message or hash of exactly MLDSA_CRHBYTES+ *                      bytes.+ * @param[in]  rhoprime Pointer to randomness seed.+ * @param      kappa    Counter for this iteration (= attempt*MLDSA_L).+ * @param[in]  mat      Expanded matrix.+ * @param[in]  s1hat    Secret vector s1 in NTT domain.+ * @param[in]  s2hat    Secret vector s2 in NTT domain.+ * @param[in]  t0hat    Vector t0 in NTT domain.+ * @param      context  Application context. Only present when+ *                      MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                      MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @return - 0: Signature generation succeeded.+ *         - MLD_ERR_FAIL: Signature rejected (norm check failed).+ *         - MLD_ERR_OUT_OF_MEMORY: If MLD_CONFIG_CUSTOM_ALLOC_FREE is used and+ *           an allocation via MLD_CUSTOM_ALLOC returned NULL.+ */+MLD_MUST_CHECK_RETURN_VALUE+/* NOLINTNEXTLINE(readability-function-cognitive-complexity) */+static int mld_attempt_signature_generation(+    uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *mu,+    const uint8_t rhoprime[MLDSA_CRHBYTES], uint16_t kappa, mld_polymat *mat,+    const mld_sk_s1hat *s1hat, const mld_sk_s2hat *s2hat,+    const mld_sk_t0hat *t0hat, MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES))+  requires(memory_no_alias(mu, MLDSA_CRHBYTES))+  requires(memory_no_alias(rhoprime, MLDSA_CRHBYTES))+  requires(memory_no_alias(mat, sizeof(mld_polymat)))+  requires(memory_no_alias(s1hat, sizeof(mld_sk_s1hat)))+  requires(memory_no_alias(s2hat, sizeof(mld_sk_s2hat)))+  requires(memory_no_alias(t0hat, sizeof(mld_sk_t0hat)))+  requires(kappa <= MLD_MAX_KAPPA)+  MLD_IF_NOT_REDUCE_RAM(+    requires(forall(k1, 0, MLDSA_K, forall(l1, 0, MLDSA_L,+                                           array_bound(mat->vec[k1].vec[l1].coeffs, 0, MLDSA_N, 0, MLDSA_Q))))+    requires(forall(k2, 0, MLDSA_K, array_abs_bound(t0hat->vec.vec[k2].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+    requires(forall(k3, 0, MLDSA_L, array_abs_bound(s1hat->vec.vec[k3].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+    requires(forall(k4, 0, MLDSA_K, array_abs_bound(s2hat->vec.vec[k4].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+  )+  MLD_IF_REDUCE_RAM(+    requires(memory_no_alias(s1hat->packed, MLDSA_L * MLDSA_POLYETA_PACKEDBYTES))+    requires(memory_no_alias(s2hat->packed, MLDSA_K * MLDSA_POLYETA_PACKEDBYTES))+    requires(memory_no_alias(t0hat->packed, MLDSA_K * MLDSA_POLYT0_PACKEDBYTES))+  )+  assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES))+  MLD_IF_REDUCE_RAM(+    assigns(memory_slice(mat, sizeof(mld_polymat)))+  )+  ensures(return_value == 0 || return_value == MLD_ERR_FAIL ||+          return_value == MLD_ERR_OUT_OF_MEMORY)+)+{+  unsigned int k;+  uint32_t w0_invalid, h_invalid;+  int ret;++  typedef union+  {+    mld_polyveck w1;+    mld_polyvecl tmp;+  } w1tmp_u;+  mld_polyveck *w1;+  mld_polyvecl *tmp;++  MLD_ALLOC(challenge_bytes, uint8_t, MLDSA_CTILDEBYTES, context);+  MLD_ALLOC(y, mld_yvec, 1, context);+  MLD_ALLOC(z, mld_poly, 1, context);+  MLD_ALLOC(w1tmp, w1tmp_u, 1, context);+  MLD_ALLOC(w0, mld_polyveck, 1, context);+  MLD_ALLOC(cp, mld_poly, 1, context);+  MLD_ALLOC(t, mld_poly, 1, context);++  if (challenge_bytes == NULL || y == NULL || z == NULL || w1tmp == NULL ||+      w0 == NULL || cp == NULL || t == NULL)+  {+    ret = MLD_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }+  w1 = &w1tmp->w1;+  tmp = &w1tmp->tmp;++  /* @[FIPS204, Algorithm 7, line 11] y <- ExpandMask(rhoprime, kappa). */+  mld_yvec_init(y, rhoprime, kappa);++  /* @[FIPS204, Algorithm 7, line 12] w <- invNTT(A_hat o NTT(y)). This call+   * performs the whole line: it NTTs y, accumulates the pointwise product with+   * A_hat, and applies the inverse NTT. In REDUCE_RAM mode the y sampling is+   * fused into the same pass. */+  mld_polyvec_matrix_pointwise_montgomery_yvec(w0, mat, y, tmp);++  /* @[FIPS204, Algorithm 7, line 13] w1 <- HighBits(w), here together with the+   * low part: Decompose yields w = 2*GAMMA2*w1 + w0, keeping both w1 and w0+   * (w0 is reused below in the line-21/26 alternative, see further down). */+  mld_polyveck_caddq(w0);+  mld_polyveck_decompose(w1, w0);++  /* @[FIPS204, Algorithm 7, line 15] ctilde <- H(mu || w1Encode(w1), lambda/4).+   * w1Encode(w1) is packed into the w1 region of sig (mld_polyveck_pack_w1),+   * then absorbed by H together with mu. */+  mld_polyveck_pack_w1(sig, w1);++  mld_H(challenge_bytes, MLDSA_CTILDEBYTES, mu, MLDSA_CRHBYTES, sig,+        MLDSA_K * MLDSA_POLYW1_PACKEDBYTES, NULL, 0);+  /* Constant time: Leaking challenge_bytes does not reveal any information+   * about the secret key as H() is modelled as random oracle.+   * This also applies to challenges for rejected signatures.+   * See Section 5.5 of @[Round3_Spec]. */+  MLD_CT_TESTING_DECLASSIFY(challenge_bytes, MLDSA_CTILDEBYTES);+  /* @[FIPS204, Algorithm 7, line 16] c <- SampleInBall(ctilde) and+   * @[FIPS204, Algorithm 7, line 17] c_hat <- NTT(c). */+  mld_poly_challenge(cp, challenge_bytes);+  mld_poly_ntt(cp);++  /* @[FIPS204, Algorithm 7, lines 18+20] cs1 <- invNTT(c_hat o s1_hat) and+   * z <- y + cs1, followed by the line-23 norm check ||z||_inf >= GAMMA1 -+   * BETA. mld_compute_pack_z fuses all three per polynomial and, on success,+   * packs z into sig; it returns MLD_ERR_FAIL if the norm check rejects z. */+  ret = mld_compute_pack_z(sig, cp, s1hat, y, t, z);+  if (ret != 0)+  {+    goto cleanup;+  }++  /* The remaining steps realize @[FIPS204, Algorithm 7, lines 21-28] (the+   * low-bits norm check and the hint h) via the faster alternative formulation+   * of @[Round3_Spec, Section 5.1]. @[FIPS204] explicitly permits this: the+   * note accompanying Algorithm 7 states that the validity checks on z and the+   * computation of h may instead be implemented "as described in Section 5.1 of+   * [6]", and that reference is @[Round3_Spec, Section 5.1].+   *+   * The loop below builds w0 - cs2 + ct0 in place in w0; w1 is unmodified, and+   * is HighBits(w) from line 13. Those are the inputs to the streamlined+   * computation of MakeHint explained below.+   *+   * Low-bits norm check:+   *   @[FIPS204, Algorithm 7, line 21] computes r0 = LowBits(w - cs2) and line+   *   23 rejects when ||r0||_inf >= GAMMA2 - BETA. By @[Round3_Spec, Section+   *   5.1] (Lemma 3), this line-23 check on r0 = LowBits(w - cs2) is implied by+   *   ||w0 - cs2||_inf < GAMMA2 - BETA, where w0 is the low part of w. In our+   *   context, w0 already holds the low part of w from the line-13 Decompose;+   *   after subtracting cs2 from it in place, the mld_poly_chknorm(w0, GAMMA2 -+   *   BETA) call below is exactly that check.+   *+   * Hint:+   *   @[FIPS204, Algorithm 7, line 26] sets h = MakeHint(-ct0, w - cs2 + ct0),+   *   and line 28 rejects when ||ct0||_inf >= GAMMA2 or h has more than OMEGA+   *   nonzero coefficients. @[Round3_Spec, Section 5.1] provides the following+   *   alternative description for MakeHint(-ct0, w - cs2 + ct0): a hint bit is+   *   zero exactly when the coefficient of w0 - cs2 + ct0 lies in+   *   (-GAMMA2, GAMMA2], or equals -GAMMA2 while the matching w1 coefficient is+   *   zero (the Decompose border case), and is set otherwise. This equivalence+   *   is precisely what mld_pack_sig_h -> mld_make_hint compute from w0+   *   (= w0 - cs2 + ct0) and w1. The line-28 ||ct0||_inf >= GAMMA2 check is the+   *   mld_poly_chknorm(z, GAMMA2) call on ct0 below; the weight bound is+   *   enforced by mld_pack_sig_h.+   *+   * Building w0 per-component and checking norms incrementally also avoids+   * allocating a full polyveck for h. */+  for (k = 0; k < MLDSA_K; k++)+  __loop__(+    assigns(k,+            object_whole(z),+            object_whole(w0))+    invariant(k <= MLDSA_K)+    invariant(forall(k0, k, MLDSA_K,+      array_abs_bound(w0->vec[k0].coeffs, 0, MLDSA_N, MLDSA_GAMMA2 + 1)))+    decreases(MLDSA_K - k)+  )+  {+    /* @[FIPS204, Algorithm 7, line 19] cs2[k] <- invNTT(c_hat o s2_hat)[k],+     * then subtract from w0[k] to form (w0 - cs2)[k]. */+    mld_sk_s2hat_get_poly(z, s2hat, k);+    mld_poly_pointwise_montgomery(z, cp);+    mld_poly_invntt_tomont(z);++    mld_poly_sub(&w0->vec[k], z);+    mld_poly_reduce(&w0->vec[k]);++    /* Low-bits norm check (see block comment above): the line-23 check on+     * r0 = LowBits(w - cs2) holds via ||w0 - cs2||_inf < GAMMA2 - BETA. */+    w0_invalid = mld_poly_chknorm(&w0->vec[k], MLDSA_GAMMA2 - MLDSA_BETA);+    /* Constant time: w0_invalid may be leaked - see comment for z_invalid. */+    MLD_CT_TESTING_DECLASSIFY(&w0_invalid, sizeof(uint32_t));+    if (w0_invalid)+    {+      ret = MLD_ERR_FAIL; /* reject */+      goto cleanup;+    }++    /* @[FIPS204, Algorithm 7, line 25] ct0[k] <- invNTT(c_hat o t0_hat)[k]. */+    mld_sk_t0hat_get_poly(z, t0hat, k);+    mld_poly_pointwise_montgomery(z, cp);+    mld_poly_invntt_tomont(z);+    mld_poly_reduce(z);++    /* @[FIPS204, Algorithm 7, line 28] reject when ||ct0||_inf >= GAMMA2 (the+     * second part, the OMEGA weight bound, is enforced by mld_pack_sig_h). */+    h_invalid = mld_poly_chknorm(z, MLDSA_GAMMA2);+    /* Constant time: h_invalid may be leaked - see comment for z_invalid. */+    MLD_CT_TESTING_DECLASSIFY(&h_invalid, sizeof(uint32_t));+    if (h_invalid)+    {+      ret = MLD_ERR_FAIL; /* reject */+      goto cleanup;+    }++    /* Add ct0[k] to (w0 - cs2)[k], leaving (w0 - cs2 + ct0)[k] in w0[k] -- the+     * MakeHint input prepared for mld_pack_sig_h (see block comment above). */+    mld_poly_add(&w0->vec[k], z);+  }++  /* Constant time: At this point all norm checks have passed and we, hence,+   * know that the signature does not leak any secret information.+   * Consequently, any value that can be computed from the signature and public+   * key is considered public.+   * w0 and w1 are public as they can be computed from Az - ct = \alpha w1 + w0.+   * h=c*t0 is public as both c and t0 are considered public.+   * While t0 is not part of the public key, it can be reconstructed from+   * a small number of signatures and need not be regarded as secret+   * (see @[FIPS204, Section 6.1]).+   */+  MLD_CT_TESTING_DECLASSIFY(w0, sizeof(*w0));+  MLD_CT_TESTING_DECLASSIFY(w1, sizeof(*w1));++  /* @[FIPS204, Algorithm 7, line 33] sigEncode(ctilde, z mod+/- q, h) is split+   * across three calls: z was already packed by mld_compute_pack_z, this call+   * packs ctilde, and mld_pack_sig_h below packs the hint h. */+  mld_pack_sig_c(sig, challenge_bytes);++  /* @[FIPS204, Algorithm 7, line 26] h <- MakeHint(-ct0, w - cs2 + ct0),+   * computed from (w0 = w0 - cs2 + ct0, w1) as described in the block comment+   * above, and packed as the h component of the line-33 sigEncode. Returns+   * MLD_ERR_FAIL if h would exceed OMEGA nonzero coefficients (the remaining+   * part of the line-28 check), in which case we reject. */+  ret = mld_pack_sig_h(sig, w0, w1);+  if (ret != 0)+  {+    goto cleanup;+  }++  /* Constant time: At this point it is clear that the signature is valid - it+   * can, hence, be considered public. */+  MLD_CT_TESTING_DECLASSIFY(sig, MLDSA_CRYPTO_BYTES);+  ret = 0; /* success */++cleanup:+  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  MLD_FREE(t, mld_poly, 1, context);+  MLD_FREE(cp, mld_poly, 1, context);+  MLD_FREE(w0, mld_polyveck, 1, context);+  MLD_FREE(w1tmp, w1tmp_u, 1, context);+  MLD_FREE(z, mld_poly, 1, context);+  MLD_FREE(y, mld_yvec, 1, context);+  MLD_FREE(challenge_bytes, uint8_t, MLDSA_CTILDEBYTES, context);++  return ret;+}+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_signature_internal(uint8_t sig[MLDSA_CRYPTO_BYTES],+                                const uint8_t *m, size_t mlen,+                                const uint8_t *pre, size_t prelen,+                                const uint8_t rnd[MLDSA_RNDBYTES],+                                const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                                int externalmu,+                                MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  int ret;+  uint8_t *rho, *tr, *key, *mu, *rhoprime;+  uint16_t attempt;+  const uint16_t max_signing_attempts = mld_get_max_signing_attempts();+  MLD_ALLOC(seedbuf, uint8_t,+            2 * MLDSA_SEEDBYTES + MLDSA_TRBYTES + 2 * MLDSA_CRHBYTES, context);+  MLD_ALLOC(mat, mld_polymat, 1, context);+  MLD_ALLOC(s1hat, mld_sk_s1hat, 1, context);+  MLD_ALLOC(t0hat, mld_sk_t0hat, 1, context);+  MLD_ALLOC(s2hat, mld_sk_s2hat, 1, context);++  if (seedbuf == NULL || mat == NULL || s1hat == NULL || t0hat == NULL ||+      s2hat == NULL)+  {+    ret = MLD_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  /* If a resume hook is configured via MLD_CONFIG_SIGN_HOOK_RESUME, it provides+   * the attempt to resume from after an earlier pause. Otherwise, we start at+   * 0. Clamp to max_signing_attempts. */+  attempt = mld_sign_resume(context);+  if (attempt > max_signing_attempts)+  {+    attempt = max_signing_attempts;+  }++  rho = seedbuf;+  tr = rho + MLDSA_SEEDBYTES;+  key = tr + MLDSA_TRBYTES;+  mu = key + MLDSA_SEEDBYTES;+  rhoprime = mu + MLDSA_CRHBYTES;+  /* @[FIPS204, Algorithm 7, line 1] (rho, K, tr, s1, s2, t0) <- skDecode(sk)+   * and @[FIPS204, Algorithm 7, lines 2-4] s1_hat/s2_hat/t0_hat <- NTT(...):+   * mld_unpack_sk returns s1hat, s2hat, t0hat already in NTT domain. The spec's+   * private random seed K is held in the local variable key. */+  mld_unpack_sk(rho, tr, key, t0hat, s1hat, s2hat, sk);++  if (!externalmu)+  {+    /* @[FIPS204, Algorithm 7, line 6] mu <- H(BytesToBits(tr) || M', 64). */+    mld_H(mu, MLDSA_CRHBYTES, tr, MLDSA_TRBYTES, pre, prelen, m, mlen);+  }+  else+  {+    /* mu has been provided directly (external-mu variant; line 6 done by the+     * caller in a separate cryptographic module). */+    mld_memcpy(mu, m, MLDSA_CRHBYTES);+  }++  /* @[FIPS204, Algorithm 7, line 7] rhoprime <- H(K || rnd || mu, 64). */+  mld_H(rhoprime, MLDSA_CRHBYTES, key, MLDSA_SEEDBYTES, rnd, MLDSA_RNDBYTES, mu,+        MLDSA_CRHBYTES);++  /* Constant time: rho is part of the public key and, hence, public. */+  MLD_CT_TESTING_DECLASSIFY(rho, MLDSA_SEEDBYTES);+  /* @[FIPS204, Algorithm 7, line 5] A_hat <- ExpandA(rho). */+  mld_polyvec_matrix_expand(mat, rho);++  /* @[FIPS204, Algorithm 7, lines 8-10 and 31-32] the rejection-sampling loop,+   * tracked by attempt (kappa = attempt*MLDSA_L). Each iteration's body (lines+   * 11-30) plus, on success, the line-33 sigEncode are performed by+   * mld_attempt_signature_generation. */++  /* Reference: the reference loops unboundedly; we instead iterate over the+   * bounded range [0, max_signing_attempts) for predictable termination.+   * A success or fatal error exits via goto cleanup; running to completion+   * means every attempt was rejected; with a FIPS compliant choice of+   * MLD_CONFIG_MAX_SIGNING_ATTEMPTS, this should never happen. */+  for (; attempt < max_signing_attempts; attempt++)+  __loop__(+    MLD_IF_NOT_REDUCE_RAM(+      assigns(attempt, ret, memory_slice(sig, MLDSA_CRYPTO_BYTES))+    )+    MLD_IF_REDUCE_RAM(+      assigns(attempt, ret, memory_slice(sig, MLDSA_CRYPTO_BYTES),+              memory_slice(mat, sizeof(mld_polymat)))+    )+    invariant(attempt <= max_signing_attempts)++    /* t0, s1, s2, and mat are initialized above and are NOT changed by this */+    /* loop. We can therefore re-assert their bounds here as part of the     */+    /* loop invariant. This makes proof noticeably faster with CBMC          */+    MLD_IF_NOT_REDUCE_RAM(+      invariant(forall(k1, 0, MLDSA_K, forall(l1, 0, MLDSA_L,+                array_bound(mat->vec[k1].vec[l1].coeffs, 0, MLDSA_N, 0, MLDSA_Q))))+      invariant(forall(k2, 0, MLDSA_K, array_abs_bound(t0hat->vec.vec[k2].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+      invariant(forall(k3, 0, MLDSA_L, array_abs_bound(s1hat->vec.vec[k3].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+      invariant(forall(k4, 0, MLDSA_K, array_abs_bound(s2hat->vec.vec[k4].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)))+    )+    decreases(max_signing_attempts - attempt)+  )+  {+    /* Safety: attempt < max_signing_attempts <= MLD_MAX_KAPPA / MLDSA_L, so+     * kappa <= MLD_MAX_KAPPA and the cast is safe. */+    const uint16_t kappa = (uint16_t)(attempt * MLDSA_L);++    /* Query configurable signing hook whether signing should be paused.+     * This is skipped by default and only used if the user sets the+     * configuration option MLD_CONFIG_SIGN_HOOK_ATTEMPT. */+    if (mld_sign_attempt(attempt, context) != 0)+    {+      ret = MLD_ERR_SIGNING_PAUSED;+      goto cleanup;+    }++    ret = mld_attempt_signature_generation(sig, mu, rhoprime, kappa, mat, s1hat,+                                           s2hat, t0hat, context);++    /* Decide whether to keep trying based on the return value:+     *  - ret == 0: a valid signature was produced; we are done.+     *  - ret == MLD_ERR_FAIL: this candidate was rejected by one of the norm+     *    or hint checks. We continue the loop and try again with the next+     *    nonce.+     *  - any other value (e.g. MLD_ERR_OUT_OF_MEMORY): an unrecoverable error+     *    occurred, so we propagate it to the caller. */+    if (ret == 0)+    {+      /* Signing succeeded: record the attempt that succeeded. No-op in the+       * default build. */+      mld_sign_finish(attempt, context);+      goto cleanup;+    }+    if (ret != MLD_ERR_FAIL)+    {+      goto cleanup;+    }+  }++  /* Loop ran to completion: all attempts rejected, budget exhausted.+   * This should never happen with a FIPS compliant choice of+   * MLD_CONFIG_MAX_SIGNING_ATTEMPTS. */+  ret = MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED;++cleanup:++  if (ret != 0)+  {+    /* To be on the safe-side, we zeroize the signature buffer. */+    mld_zeroize(sig, MLDSA_CRYPTO_BYTES);+  }++  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  MLD_FREE(s2hat, mld_sk_s2hat, 1, context);+  MLD_FREE(t0hat, mld_sk_t0hat, 1, context);+  MLD_FREE(s1hat, mld_sk_s1hat, 1, context);+  MLD_FREE(mat, mld_polymat, 1, context);+  MLD_FREE(seedbuf, uint8_t,+           2 * MLDSA_SEEDBYTES + MLDSA_TRBYTES + 2 * MLDSA_CRHBYTES, context);+  return ret;+}++#if !defined(MLD_CONFIG_CORE_API_ONLY)+#if !defined(MLD_CONFIG_NO_RANDOMIZED_API)+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_signature(uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *m,+                       size_t mlen, const uint8_t *ctx, size_t ctxlen,+                       const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                       MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  size_t pre_len;+  int ret;+  MLD_ALLOC(pre, uint8_t, MLD_DOMAIN_SEPARATION_MAX_BYTES, context);+  MLD_ALLOC(rnd, uint8_t, MLDSA_RNDBYTES, context);++  if (pre == NULL || rnd == NULL)+  {+    ret = MLD_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  /* Prepare domain separation prefix for pure ML-DSA */+  pre_len = mld_prepare_domain_separation_prefix(pre, NULL, 0, ctx, ctxlen,+                                                 MLD_PREHASH_NONE);+  if (pre_len == 0)+  {+    ret = MLD_ERR_INVALID_ARG;+    goto cleanup;+  }++  /* Randomized variant of ML-DSA. If you need the deterministic variant,+   * call mld_sign_signature_internal directly with all-zero rnd. */+  if (mld_randombytes(rnd, MLDSA_RNDBYTES) != 0)+  {+    ret = MLD_ERR_RNG_FAIL;+    goto cleanup;+  }+  MLD_CT_TESTING_SECRET(rnd, MLDSA_RNDBYTES);++  ret = mld_sign_signature_internal(sig, m, mlen, pre, pre_len, rnd, sk, 0,+                                    context);++cleanup:+  if (ret != 0)+  {+    /* To be on the safe-side, make sure sig has a well-defined value, even in+     * the case of error.+     *+     * If we come from mld_sign_signature_internal, this is redundant, but the+     * error case should not be the norm, and the added cost of the zeroization+     * insignificant. */+    mld_zeroize(sig, MLDSA_CRYPTO_BYTES);+  }++  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  MLD_FREE(rnd, uint8_t, MLDSA_RNDBYTES, context);+  MLD_FREE(pre, uint8_t, MLD_DOMAIN_SEPARATION_MAX_BYTES, context);++  return ret;+}+#endif /* !MLD_CONFIG_NO_RANDOMIZED_API */++#if !defined(MLD_CONFIG_NO_RANDOMIZED_API)+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_signature_extmu(uint8_t sig[MLDSA_CRYPTO_BYTES],+                             const uint8_t mu[MLDSA_CRHBYTES],+                             const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                             MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  int ret;+  MLD_ALLOC(rnd, uint8_t, MLDSA_RNDBYTES, context);++  if (rnd == NULL)+  {+    ret = MLD_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  /* Randomized variant of ML-DSA. If you need the deterministic variant,+   * call mld_sign_signature_internal directly with all-zero rnd. */+  if (mld_randombytes(rnd, MLDSA_RNDBYTES) != 0)+  {+    ret = MLD_ERR_RNG_FAIL;+    goto cleanup;+  }+  MLD_CT_TESTING_SECRET(rnd, MLDSA_RNDBYTES);++  ret = mld_sign_signature_internal(sig, mu, MLDSA_CRHBYTES, NULL, 0, rnd, sk,+                                    1, context);++cleanup:+  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  MLD_FREE(rnd, uint8_t, MLDSA_RNDBYTES, context);++  return ret;+}+#endif /* !MLD_CONFIG_NO_RANDOMIZED_API */+#endif /* !MLD_CONFIG_CORE_API_ONLY */+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_verify_internal(const uint8_t sig[MLDSA_CRYPTO_BYTES],+                             const uint8_t *m, size_t mlen, const uint8_t *pre,+                             size_t prelen,+                             const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+                             int externalmu,+                             MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  int ret, cmp;+  unsigned int i;++  MLD_ALLOC(buf, uint8_t, (MLDSA_K * MLDSA_POLYW1_PACKEDBYTES), context);+  MLD_ALLOC(mu, uint8_t, MLDSA_CRHBYTES, context);+  MLD_ALLOC(c, uint8_t, MLDSA_CTILDEBYTES, context);+  MLD_ALLOC(c2, uint8_t, MLDSA_CTILDEBYTES, context);+  MLD_ALLOC(z, mld_polyvecl, 1, context);+  MLD_ALLOC(cp, mld_poly, 1, context);+  MLD_ALLOC(mat, mld_polymat, 1, context);+  MLD_ALLOC(w1, mld_poly, 1, context);+  MLD_ALLOC(tmp, mld_poly, 1, context);++  if (buf == NULL || mu == NULL || c == NULL || c2 == NULL || z == NULL ||+      cp == NULL || mat == NULL || w1 == NULL || tmp == NULL)+  {+    ret = MLD_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  mld_memcpy(c, sig, MLDSA_CTILDEBYTES);+  mld_polyvecl_unpack_z(z, sig + MLDSA_SIG_Z_OFFSET);++  /* mld_polyvecl_chknorm signals failure through a single non-zero error code+   * that's not yet aligned with MLD_ERR_XXX. A norm-check failure here means+   * the signature is invalid, so map it to MLD_ERR_INVALID_SIGNATURE. */+  if (mld_polyvecl_chknorm(z, MLDSA_GAMMA1 - MLDSA_BETA))+  {+    ret = MLD_ERR_INVALID_SIGNATURE;+    goto cleanup;+  }++  if (!externalmu)+  {+    /* Compute CRH(H(rho, t1), pre, msg) */+    MLD_ALIGN uint8_t hpk[MLDSA_CRHBYTES];+    mld_H(hpk, MLDSA_TRBYTES, pk, MLDSA_CRYPTO_PUBLICKEYBYTES, NULL, 0, NULL,+          0);+    mld_H(mu, MLDSA_CRHBYTES, hpk, MLDSA_TRBYTES, pre, prelen, m, mlen);++    /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+    mld_zeroize(hpk, sizeof(hpk));+  }+  else+  {+    /* mu has been provided directly */+    mld_memcpy(mu, m, MLDSA_CRHBYTES);+  }++  /* Matrix-vector multiplication and per-row reconstruction of w1. */+  mld_polyvecl_ntt(z);+  mld_polyvec_matrix_expand(mat, pk);+  mld_poly_challenge(cp, c);+  mld_poly_ntt(cp);++  for (i = 0; i < MLDSA_K; ++i)+  __loop__(+    assigns(MLD_IF_REDUCE_RAM(memory_slice(mat, sizeof(mld_polymat)),)+            i, ret,+            memory_slice(w1, sizeof(mld_poly)),+            memory_slice(tmp, sizeof(mld_poly)),+            memory_slice(buf, MLDSA_K * MLDSA_POLYW1_PACKEDBYTES)+    )+    invariant(i <= MLDSA_K)+    decreases(MLDSA_K - i)+  )+  {+    /* w1 = (A * z)_i in NTT domain */+    mld_polyvec_matrix_pointwise_montgomery_row(w1, mat, z, i);++    /* tmp = c * t1_i * 2^d in NTT domain */+    mld_unpack_pk_t1(tmp, pk, i);+    mld_poly_shiftl(tmp);+    mld_poly_ntt(tmp);+    mld_poly_pointwise_montgomery(tmp, cp);++    /* w1 = invNTT(w1 - c * t1_i * 2^d) */+    mld_poly_sub(w1, tmp);+    mld_poly_reduce(w1);+    mld_poly_invntt_tomont(w1);+    mld_poly_caddq(w1);++    /* tmp = h_i (decoded and validated from signature). A non-zero return+     * means the hint encoding is malformed, i.e. the signature is invalid. */+    if (mld_sig_unpack_hints(tmp, sig, i) != 0)+    {+      ret = MLD_ERR_INVALID_SIGNATURE;+      goto cleanup;+    }++    /* w1 = use_hint(w1, tmp), then pack into buf[i] */+    mld_poly_use_hint(w1, tmp);+    mld_polyw1_pack(buf + i * MLDSA_POLYW1_PACKEDBYTES, w1);+  }++  /* Call random oracle and verify challenge */+  mld_H(c2, MLDSA_CTILDEBYTES, mu, MLDSA_CRHBYTES, buf,+        MLDSA_K * MLDSA_POLYW1_PACKEDBYTES, NULL, 0);++  cmp = mld_ct_memcmp(c, c2, MLDSA_CTILDEBYTES);++  /* Declassify the result of the verification. */+  MLD_CT_TESTING_DECLASSIFY(&cmp, sizeof(cmp));++  ret = cmp == 0 ? 0 : MLD_ERR_INVALID_SIGNATURE;++cleanup:+  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  MLD_FREE(tmp, mld_poly, 1, context);+  MLD_FREE(w1, mld_poly, 1, context);+  MLD_FREE(mat, mld_polymat, 1, context);+  MLD_FREE(cp, mld_poly, 1, context);+  MLD_FREE(z, mld_polyvecl, 1, context);+  MLD_FREE(c2, uint8_t, MLDSA_CTILDEBYTES, context);+  MLD_FREE(c, uint8_t, MLDSA_CTILDEBYTES, context);+  MLD_FREE(mu, uint8_t, MLDSA_CRHBYTES, context);+  MLD_FREE(buf, uint8_t, (MLDSA_K * MLDSA_POLYW1_PACKEDBYTES), context);+  return ret;+}++#if !defined(MLD_CONFIG_CORE_API_ONLY)+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_verify(const uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *m,+                    size_t mlen, const uint8_t *ctx, size_t ctxlen,+                    const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+                    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  MLD_ALIGN uint8_t pre[MLD_DOMAIN_SEPARATION_MAX_BYTES];+  size_t pre_len;+  int ret;++  pre_len = mld_prepare_domain_separation_prefix(pre, NULL, 0, ctx, ctxlen,+                                                 MLD_PREHASH_NONE);+  if (pre_len == 0)+  {+    ret = MLD_ERR_INVALID_ARG;+    goto cleanup;+  }++  ret = mld_sign_verify_internal(sig, m, mlen, pre, pre_len, pk, 0, context);++cleanup:+  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(pre, sizeof(pre));++  return ret;+}++MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_verify_extmu(const uint8_t sig[MLDSA_CRYPTO_BYTES],+                          const uint8_t mu[MLDSA_CRHBYTES],+                          const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+                          MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  return mld_sign_verify_internal(sig, mu, MLDSA_CRHBYTES, NULL, 0, pk, 1,+                                  context);+}+#endif /* !MLD_CONFIG_CORE_API_ONLY */+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_CORE_API_ONLY)+#if !defined(MLD_CONFIG_NO_SIGN_API)+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_signature_pre_hash_internal(+    uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *ph, size_t phlen,+    const uint8_t *ctx, size_t ctxlen, const uint8_t rnd[MLDSA_RNDBYTES],+    const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], int hashalg,+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  MLD_ALIGN uint8_t pre[MLD_DOMAIN_SEPARATION_MAX_BYTES];+  size_t pre_len;+  int ret;++  if (hashalg == MLD_PREHASH_NONE)+  {+    ret = MLD_ERR_INVALID_ARG;+    goto cleanup;+  }++  pre_len = mld_prepare_domain_separation_prefix(pre, ph, phlen, ctx, ctxlen,+                                                 hashalg);+  if (pre_len == 0)+  {+    ret = MLD_ERR_INVALID_ARG;+    goto cleanup;+  }++  ret = mld_sign_signature_internal(sig, pre, pre_len, NULL, 0, rnd, sk, 0,+                                    context);+cleanup:+  if (ret != 0)+  {+    /* To be on the safe-side, make sure sig has a well-defined value, even in+     * the case of error.+     *+     * If we come from mld_sign_signature_internal, this is redundant, but the+     * error case should not be the norm, and the added cost of the zeroization+     * insignificant. */+    mld_zeroize(sig, MLDSA_CRYPTO_BYTES);+  }++  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(pre, sizeof(pre));+  return ret;+}+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_verify_pre_hash_internal(+    const uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *ph, size_t phlen,+    const uint8_t *ctx, size_t ctxlen,+    const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], int hashalg,+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  MLD_ALIGN uint8_t pre[MLD_DOMAIN_SEPARATION_MAX_BYTES];+  size_t pre_len;+  int ret;++  if (hashalg == MLD_PREHASH_NONE)+  {+    ret = MLD_ERR_INVALID_ARG;+    goto cleanup;+  }++  pre_len = mld_prepare_domain_separation_prefix(pre, ph, phlen, ctx, ctxlen,+                                                 hashalg);+  if (pre_len == 0)+  {+    ret = MLD_ERR_INVALID_ARG;+    goto cleanup;+  }++  ret = mld_sign_verify_internal(sig, pre, pre_len, NULL, 0, pk, 0, context);++cleanup:+  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(pre, sizeof(pre));+  return ret;+}+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_signature_pre_hash_shake256(+    uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *m, size_t mlen,+    const uint8_t *ctx, size_t ctxlen, const uint8_t rnd[MLDSA_RNDBYTES],+    const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  MLD_ALIGN uint8_t ph[64];+  int ret;+  mld_shake256(ph, sizeof(ph), m, mlen);+  ret = mld_sign_signature_pre_hash_internal(sig, ph, sizeof(ph), ctx, ctxlen,+                                             rnd, sk, MLD_PREHASH_SHAKE_256,+                                             context);+  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(ph, sizeof(ph));+  return ret;+}+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_verify_pre_hash_shake256(+    const uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *m, size_t mlen,+    const uint8_t *ctx, size_t ctxlen,+    const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  MLD_ALIGN uint8_t ph[64];+  int ret;+  mld_shake256(ph, sizeof(ph), m, mlen);+  ret = mld_sign_verify_pre_hash_internal(sig, ph, sizeof(ph), ctx, ctxlen, pk,+                                          MLD_PREHASH_SHAKE_256, context);+  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  mld_zeroize(ph, sizeof(ph));+  return ret;+}+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+#define MLD_PRE_HASH_OID_LEN 11++/**+ * Return the OID of a given SHA-2/SHA-3 hash function.+ *+ * @param[out] oid     Pointer to output OID.+ * @param      hashalg Hash algorithm constant (MLD_PREHASH_*).+ */+static void mld_get_hash_oid(uint8_t oid[MLD_PRE_HASH_OID_LEN], int hashalg)+{+  unsigned int i;+  static const struct+  {+    int alg;+    uint8_t oid[MLD_PRE_HASH_OID_LEN];+  } oid_map[] = {+      {MLD_PREHASH_SHA2_224,+       {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x04}},+      {MLD_PREHASH_SHA2_256,+       {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x01}},+      {MLD_PREHASH_SHA2_384,+       {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x02}},+      {MLD_PREHASH_SHA2_512,+       {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x03}},+      {MLD_PREHASH_SHA2_512_224,+       {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x05}},+      {MLD_PREHASH_SHA2_512_256,+       {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x06}},+      {MLD_PREHASH_SHA3_224,+       {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x07}},+      {MLD_PREHASH_SHA3_256,+       {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x08}},+      {MLD_PREHASH_SHA3_384,+       {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x09}},+      {MLD_PREHASH_SHA3_512,+       {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x0A}},+      {MLD_PREHASH_SHAKE_128,+       {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x0B}},+      {MLD_PREHASH_SHAKE_256,+       {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x0C}}};++  for (i = 0; i < sizeof(oid_map) / sizeof(oid_map[0]); i++)+  __loop__(+    invariant(i <= sizeof(oid_map) / sizeof(oid_map[0]))+    decreases(sizeof(oid_map) / sizeof(oid_map[0]) - i)+  )+  {+    if (oid_map[i].alg == hashalg)+    {+      mld_memcpy(oid, oid_map[i].oid, MLD_PRE_HASH_OID_LEN);+      return;+    }+  }+}++static int mld_validate_hash_length(int hashalg, size_t len)+{+  switch (hashalg)+  {+    case MLD_PREHASH_SHA2_224:+      return (len == 224 / 8) ? 0 : -1;+    case MLD_PREHASH_SHA2_256:+      return (len == 256 / 8) ? 0 : -1;+    case MLD_PREHASH_SHA2_384:+      return (len == 384 / 8) ? 0 : -1;+    case MLD_PREHASH_SHA2_512:+      return (len == 512 / 8) ? 0 : -1;+    case MLD_PREHASH_SHA2_512_224:+      return (len == 224 / 8) ? 0 : -1;+    case MLD_PREHASH_SHA2_512_256:+      return (len == 256 / 8) ? 0 : -1;+    case MLD_PREHASH_SHA3_224:+      return (len == 224 / 8) ? 0 : -1;+    case MLD_PREHASH_SHA3_256:+      return (len == 256 / 8) ? 0 : -1;+    case MLD_PREHASH_SHA3_384:+      return (len == 384 / 8) ? 0 : -1;+    case MLD_PREHASH_SHA3_512:+      return (len == 512 / 8) ? 0 : -1;+    case MLD_PREHASH_SHAKE_128:+      return (len == 256 / 8) ? 0 : -1;+    case MLD_PREHASH_SHAKE_256:+      return (len == 512 / 8) ? 0 : -1;+    default:+      return -1;+  }+}++MLD_EXTERNAL_API+size_t mld_prepare_domain_separation_prefix(+    uint8_t prefix[MLD_DOMAIN_SEPARATION_MAX_BYTES], const uint8_t *ph,+    size_t phlen, const uint8_t *ctx, size_t ctxlen, int hashalg)+{+  if (ctxlen > 255)+  {+    return 0;+  }++  if (hashalg != MLD_PREHASH_NONE)+  {+    if (ph == NULL || mld_validate_hash_length(hashalg, phlen) != 0)+    {+      return 0;+    }+  }++  /* Common prefix: 0x00/0x01 || ctxlen || ctx */+  prefix[0] = (hashalg == MLD_PREHASH_NONE) ? 0 : 1;+  prefix[1] = (uint8_t)ctxlen;+  if (ctxlen > 0)+  {+    mld_memcpy(prefix + 2, ctx, ctxlen);+  }++  if (hashalg == MLD_PREHASH_NONE)+  {+    return 2 + ctxlen;+  }++  /* HashML-DSA: append oid || ph */+  mld_get_hash_oid(prefix + 2 + ctxlen, hashalg);+  mld_memcpy(prefix + 2 + ctxlen + MLD_PRE_HASH_OID_LEN, ph, phlen);+  return 2 + ctxlen + MLD_PRE_HASH_OID_LEN + phlen;+}+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+MLD_EXTERNAL_API+int mld_sign_pk_from_sk(uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+                        const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                        MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  uint8_t check, cmp0, cmp1, chk1, chk2;+  int ret;+  MLD_ALLOC(rho, uint8_t, MLDSA_SEEDBYTES, context);+  MLD_ALLOC(tr, uint8_t, MLDSA_TRBYTES, context);+  MLD_ALLOC(tr_computed, uint8_t, MLDSA_TRBYTES, context);+  MLD_ALLOC(key, uint8_t, MLDSA_SEEDBYTES, context);+  MLD_ALLOC(s1, mld_polyvecl, 1, context);+  MLD_ALLOC(s2, mld_polyveck, 1, context);+  MLD_ALLOC(t0_packed, uint8_t, MLDSA_K *MLDSA_POLYT0_PACKEDBYTES, context);++  if (rho == NULL || tr == NULL || tr_computed == NULL || key == NULL ||+      s1 == NULL || s2 == NULL || t0_packed == NULL)+  {+    ret = MLD_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  /* Inline unpack_sk: mld_unpack_sk uses lazy types for s1/s2/t0 which+   * we cannot use here. t0 stays in packed form -- we compare it against+   * the recomputed value below. */+  mld_memcpy(rho, sk + MLDSA_SK_RHO_OFFSET, MLDSA_SEEDBYTES);+  mld_memcpy(key, sk + MLDSA_SK_KEY_OFFSET, MLDSA_SEEDBYTES);+  mld_memcpy(tr, sk + MLDSA_SK_TR_OFFSET, MLDSA_TRBYTES);+  mld_polyvecl_unpack_eta(s1, sk + MLDSA_SK_S1_OFFSET);+  mld_polyveck_unpack_eta(s2, sk + MLDSA_SK_S2_OFFSET);++  /* Validate s1 and s2 coefficients are within [-MLDSA_ETA, MLDSA_ETA] */+  chk1 = mld_polyvecl_chknorm(s1, MLDSA_ETA + 1) & 0xFF;+  chk2 = mld_polyveck_chknorm(s2, MLDSA_ETA + 1) & 0xFF;++  /* NTT s1 in place to use as s1hat */+  mld_polyvecl_ntt(s1);++  /* Pack rho into pk */+  mld_memcpy(pk + MLDSA_PK_RHO_OFFSET, rho, MLDSA_SEEDBYTES);++  /* Recompute t row by row, decompose, and pack t1 into pk and t0 into+   * t0_packed. */+  ret = mld_compute_pack_t0_t1(pk + MLDSA_PK_T1_OFFSET, t0_packed, s1, s2, rho,+                               context);+  if (ret != 0)+  {+    goto cleanup;+  }++  /* Compare recomputed packed t0 against the t0 region of sk. */+  cmp0 = mld_ct_memcmp(t0_packed, sk + MLDSA_SK_T0_OFFSET,+                       MLDSA_K * MLDSA_POLYT0_PACKEDBYTES);++  /* Compute tr_computed = H(pk) and compare to the stored tr */+  mld_shake256(tr_computed, MLDSA_TRBYTES, pk, MLDSA_CRYPTO_PUBLICKEYBYTES);+  cmp1 = mld_ct_memcmp((const uint8_t *)tr, (const uint8_t *)tr_computed,+                       MLDSA_TRBYTES);+  check = mld_value_barrier_u8(cmp0 | cmp1 | chk1 | chk2);++  /* Declassify the final result of the validity check. */+  MLD_CT_TESTING_DECLASSIFY(&check, sizeof(check));+  ret = (check != 0) ? MLD_ERR_INVALID_KEY : 0;++cleanup:++  if (ret != 0)+  {+    mld_zeroize(pk, MLDSA_CRYPTO_PUBLICKEYBYTES);+  }++  /* Constant time: pk is either the valid public key or zeroed on error */+  MLD_CT_TESTING_DECLASSIFY(pk, MLDSA_CRYPTO_PUBLICKEYBYTES);++  /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */+  MLD_FREE(t0_packed, uint8_t, MLDSA_K *MLDSA_POLYT0_PACKEDBYTES, context);+  MLD_FREE(s2, mld_polyveck, 1, context);+  MLD_FREE(s1, mld_polyvecl, 1, context);+  MLD_FREE(key, uint8_t, MLDSA_SEEDBYTES, context);+  MLD_FREE(tr_computed, uint8_t, MLDSA_TRBYTES, context);+  MLD_FREE(tr, uint8_t, MLDSA_TRBYTES, context);+  MLD_FREE(rho, uint8_t, MLDSA_SEEDBYTES, context);++  return ret;+}+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */+#endif /* !MLD_CONFIG_CORE_API_ONLY */++/* To facilitate single-compilation-unit (SCU) builds, undefine all macros.+ * Don't modify by hand -- this is auto-generated by scripts/autogen. */+#undef mld_check_pct+#undef mld_sample_s1_s2+#undef mld_validate_hash_length+#undef mld_get_hash_oid+#undef mld_H+#undef mld_compute_pack_z+#undef mld_attempt_signature_generation+#undef mld_compute_pack_t0_t1+#undef mld_get_max_signing_attempts+#undef MLD_MAX_SIGNING_ATTEMPTS+#undef MLD_PRE_HASH_OID_LEN
+ cbits/mldsa/src/sign.h view
@@ -0,0 +1,850 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS204]+ *   FIPS 204 Module-Lattice-Based Digital Signature Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/204/final+ */++#ifndef MLD_SIGN_H+#define MLD_SIGN_H++#include <stddef.h>+#include "cbmc.h"+#include "common.h"+#include "poly.h"+#include "polyvec.h"+#include "sys.h"++#if defined(MLD_CHECK_APIS)+/* Include to ensure consistency between internal sign.h+ * and external mldsa_native.h. */+#include "mldsa_native.h"++#if MLDSA_CRYPTO_SECRETKEYBYTES != \+    MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET)+#error Mismatch for SECRETKEYBYTES between sign.h and mldsa_native.h+#endif++#if MLDSA_CRYPTO_PUBLICKEYBYTES != \+    MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET)+#error Mismatch for PUBLICKEYBYTES between sign.h and mldsa_native.h+#endif++#if MLDSA_CRYPTO_BYTES != MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)+#error Mismatch for BYTES between sign.h and mldsa_native.h+#endif++#endif /* MLD_CHECK_APIS */++#define mld_sign_keypair_internal \+  MLD_NAMESPACE_KL(keypair_internal) MLD_CONTEXT_PARAMETERS_3+#define mld_sign_keypair MLD_NAMESPACE_KL(keypair) MLD_CONTEXT_PARAMETERS_2+#define mld_sign_signature_internal \+  MLD_NAMESPACE_KL(signature_internal) MLD_CONTEXT_PARAMETERS_8+#define mld_sign_signature MLD_NAMESPACE_KL(signature) MLD_CONTEXT_PARAMETERS_6+#define mld_sign_signature_extmu \+  MLD_NAMESPACE_KL(signature_extmu) MLD_CONTEXT_PARAMETERS_3+#define mld_sign_verify_internal \+  MLD_NAMESPACE_KL(verify_internal) MLD_CONTEXT_PARAMETERS_7+#define mld_sign_verify MLD_NAMESPACE_KL(verify) MLD_CONTEXT_PARAMETERS_6+#define mld_sign_verify_extmu \+  MLD_NAMESPACE_KL(verify_extmu) MLD_CONTEXT_PARAMETERS_3+#define mld_sign_signature_pre_hash_internal \+  MLD_NAMESPACE_KL(signature_pre_hash_internal) MLD_CONTEXT_PARAMETERS_8+#define mld_sign_verify_pre_hash_internal \+  MLD_NAMESPACE_KL(verify_pre_hash_internal) MLD_CONTEXT_PARAMETERS_7+#define mld_sign_signature_pre_hash_shake256 \+  MLD_NAMESPACE_KL(signature_pre_hash_shake256) MLD_CONTEXT_PARAMETERS_7+#define mld_sign_verify_pre_hash_shake256 \+  MLD_NAMESPACE_KL(verify_pre_hash_shake256) MLD_CONTEXT_PARAMETERS_6+#define mld_prepare_domain_separation_prefix \+  MLD_NAMESPACE_KL(prepare_domain_separation_prefix)+#define mld_sign_pk_from_sk \+  MLD_NAMESPACE_KL(pk_from_sk) MLD_CONTEXT_PARAMETERS_2++/* Hash algorithm constants for domain separation */+#define MLD_PREHASH_NONE 0+#define MLD_PREHASH_SHA2_224 1+#define MLD_PREHASH_SHA2_256 2+#define MLD_PREHASH_SHA2_384 3+#define MLD_PREHASH_SHA2_512 4+#define MLD_PREHASH_SHA2_512_224 5+#define MLD_PREHASH_SHA2_512_256 6+#define MLD_PREHASH_SHA3_224 7+#define MLD_PREHASH_SHA3_256 8+#define MLD_PREHASH_SHA3_384 9+#define MLD_PREHASH_SHA3_512 10+#define MLD_PREHASH_SHAKE_128 11+#define MLD_PREHASH_SHAKE_256 12++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+/**+ * Generate a public-private key pair from a seed.+ *+ * When MLD_CONFIG_KEYGEN_PCT is set, performs a Pairwise Consistency Test+ * (PCT) as required by FIPS 140-3 IG.+ *+ * @spec{Implements @[FIPS204, Algorithm 6, ML-DSA.KeyGen_internal].}+ *+ * @param[out] pk      Output public key.+ * @param[out] sk      Output private key.+ * @param[in]  seed    Input random seed.+ * @param      context Application context. Only present when+ *                     MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                               Success.+ * @retval MLD_ERR_OUT_OF_MEMORY           MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                         used and an allocation via+ *                                         MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_RNG_FAIL                Random number generation failed+ *                                         during the PCT. Only possible when+ *                                         MLD_CONFIG_KEYGEN_PCT is enabled.+ * @retval MLD_ERR_SIGNING_PAUSED          The PCT's signing step was paused by+ *                                         a MLD_CONFIG_SIGN_HOOK_ATTEMPT hook.+ *                                         This should currently never happen:+ *                                         signing hooks require+ *                                         MLD_CONFIG_NO_RANDOMIZED_API, which+ *                                         is incompatible with+ *                                         MLD_CONFIG_KEYGEN_PCT, so the two+ *                                         cannot be enabled simultaneously.+ * @retval MLD_ERR_PCT_FAIL                MLD_CONFIG_KEYGEN_PCT is enabled and+ *                                         the PCT check failed.+ */+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_keypair_internal(uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+                              uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                              const uint8_t seed[MLDSA_SEEDBYTES],+                              MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES))+  requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES))+  requires(memory_no_alias(seed, MLDSA_SEEDBYTES))+  assigns(object_whole(pk))+  assigns(object_whole(sk))+  ensures(return_value == 0 || return_value == MLD_ERR_OUT_OF_MEMORY ||+          return_value == MLD_ERR_RNG_FAIL ||+          return_value == MLD_ERR_SIGNING_PAUSED ||+          return_value == MLD_ERR_PCT_FAIL)+  /* Output buffers on error, per API-CONVENTIONS.md */+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(pk, MLDSA_CRYPTO_PUBLICKEYBYTES))+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(sk, MLDSA_CRYPTO_SECRETKEYBYTES))+);++#if !defined(MLD_CONFIG_CORE_API_ONLY)+#if !defined(MLD_CONFIG_NO_RANDOMIZED_API)+/**+ * Generate a public-private key pair.+ *+ * When MLD_CONFIG_KEYGEN_PCT is set, performs a Pairwise Consistency Test+ * (PCT) as required by FIPS 140-3 IG.+ *+ * @spec{Implements @[FIPS204, Algorithm 1, ML-DSA.KeyGen].}+ *+ * @param[out] pk      Output public key.+ * @param[out] sk      Output private key.+ * @param      context Application context. Only present when+ *                     MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                               Success.+ * @retval MLD_ERR_OUT_OF_MEMORY           MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                         used and an allocation via+ *                                         MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_RNG_FAIL                Random number generation failed.+ * @retval MLD_ERR_SIGNING_PAUSED          The PCT's signing step was paused by+ *                                         a MLD_CONFIG_SIGN_HOOK_ATTEMPT hook.+ *                                         This should currently never happen:+ *                                         signing hooks require+ *                                         MLD_CONFIG_NO_RANDOMIZED_API, which+ *                                         is incompatible with+ *                                         MLD_CONFIG_KEYGEN_PCT, so the two+ *                                         cannot be enabled simultaneously.+ * @retval MLD_ERR_PCT_FAIL                MLD_CONFIG_KEYGEN_PCT is enabled and+ *                                         the PCT check failed.+ */+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_keypair(uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+                     uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                     MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES))+  requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES))+  assigns(object_whole(pk))+  assigns(object_whole(sk))+  ensures(return_value == 0 || return_value == MLD_ERR_OUT_OF_MEMORY ||+          return_value == MLD_ERR_RNG_FAIL ||+          return_value == MLD_ERR_SIGNING_PAUSED ||+          return_value == MLD_ERR_PCT_FAIL)+  /* Output buffers on error, per API-CONVENTIONS.md */+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(pk, MLDSA_CRYPTO_PUBLICKEYBYTES))+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(sk, MLDSA_CRYPTO_SECRETKEYBYTES))+);+#endif /* !MLD_CONFIG_NO_RANDOMIZED_API */+#endif /* !MLD_CONFIG_CORE_API_ONLY */+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+/**+ * Compute signature using a caller-supplied random seed and prefix.+ *+ * On error (non-zero return value), the signature buffer sig is zeroized.+ *+ * @spec{Implements @[FIPS204, Algorithm 7, ML-DSA.Sign_internal].}+ *+ * @warning This function does not perform secret key validation.+ *          Callers importing serialized keys can use mld_sign_pk_from_sk+ *          to validate them before signing.+ *+ * @param[out] sig        Pointer to buffer to hold the generated signature of+ *                        MLDSA_CRYPTO_BYTES bytes.+ * @param[in]  m          Pointer to message to be signed (when+ *                        externalmu == 0), or to a precomputed+ *                        message representative mu (when externalmu != 0).+ * @param      mlen       Length of m. Must equal MLDSA_CRHBYTES when+ *                        externalmu != 0.+ * @param[in]  pre        Pointer to prefix string. Ignored when+ *                        externalmu != 0.+ * @param      prelen     Length of prefix string. Ignored when+ *                        externalmu != 0.+ * @param[in]  rnd        Random seed.+ * @param[in]  sk         Bit-packed secret key; assumed to be valid.+ * @param      externalmu 0: m/mlen is the raw message; mu = H(tr, pre, m) is+ *                        computed internally.+ *                        non-zero: m points to a precomputed mu of+ *                        MLDSA_CRHBYTES bytes; pre/prelen unused.+ * @param      context    Application context. Only present when+ *                        MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                        MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                               Success.+ * @retval MLD_ERR_OUT_OF_MEMORY           MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                         used and an allocation via+ *                                         MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded+ *                                         MLD_CONFIG_MAX_SIGNING_ATTEMPTS+ *                                         iterations.+ * @retval MLD_ERR_SIGNING_PAUSED          A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook+ *                                         paused signing; re-invoke to resume.+ */+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_signature_internal(uint8_t sig[MLDSA_CRYPTO_BYTES],+                                const uint8_t *m, size_t mlen,+                                const uint8_t *pre, size_t prelen,+                                const uint8_t rnd[MLDSA_RNDBYTES],+                                const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                                int externalmu,+                                MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(mlen <= MLD_MAX_BUFFER_SIZE)+  requires(prelen <= MLD_MAX_BUFFER_SIZE)+  requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES))+  requires(memory_no_alias(m, mlen))+  requires(memory_no_alias(rnd, MLDSA_RNDBYTES))+  requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES))+  requires((externalmu == 0) ==> ((prelen == 0) || memory_no_alias(pre, prelen)))+  requires((externalmu != 0) ==> (mlen == MLDSA_CRHBYTES))+  assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES))+  ensures(return_value == 0 ||+          return_value == MLD_ERR_OUT_OF_MEMORY ||+          return_value == MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED ||+          return_value == MLD_ERR_SIGNING_PAUSED)+  /* Output buffers on error, per API-CONVENTIONS.md */+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(sig, MLDSA_CRYPTO_BYTES)));++#if !defined(MLD_CONFIG_CORE_API_ONLY)+#if !defined(MLD_CONFIG_NO_RANDOMIZED_API)+/**+ * Compute signature. This function implements the randomized variant of+ * ML-DSA. If you require the deterministic variant, use+ * mld_sign_signature_internal directly.+ *+ * @spec{Implements @[FIPS204, Algorithm 2, ML-DSA.Sign].}+ *+ * @warning This function does not perform secret key validation.+ *          Callers importing serialized keys can use mld_sign_pk_from_sk+ *          to validate them before signing.+ *+ * @param[out] sig     Output signature.+ * @param[in]  m       Pointer to message to be signed.+ * @param      mlen    Length of message.+ * @param[in]  ctx     Pointer to context string. May be NULL if ctxlen == 0.+ * @param      ctxlen  Length of context string. Should be <= 255.+ * @param[in]  sk      Bit-packed secret key; assumed to be valid.+ * @param      context Application context. Only present when+ *                     MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                               Success.+ * @retval MLD_ERR_OUT_OF_MEMORY           MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                         used and an allocation via+ *                                         MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_RNG_FAIL                Random number generation failed.+ * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded+ *                                         MLD_CONFIG_MAX_SIGNING_ATTEMPTS+ *                                         iterations.+ * @retval MLD_ERR_SIGNING_PAUSED          A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook+ *                                         paused signing; re-invoke to resume.+ * @retval MLD_ERR_INVALID_ARG             The context string exceeded 255+ *                                         bytes.+ */+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_signature(uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *m,+                       size_t mlen, const uint8_t *ctx, size_t ctxlen,+                       const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                       MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(mlen <= MLD_MAX_BUFFER_SIZE)+  requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES))+  requires(memory_no_alias(m, mlen))+  requires(ctxlen <= MLD_MAX_BUFFER_SIZE)+  requires(ctxlen == 0 || memory_no_alias(ctx, ctxlen))+  requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES))+  assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES))+  ensures(return_value == 0 || return_value == MLD_ERR_OUT_OF_MEMORY ||+          return_value == MLD_ERR_RNG_FAIL ||+          return_value == MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED ||+          return_value == MLD_ERR_SIGNING_PAUSED ||+          return_value == MLD_ERR_INVALID_ARG)+  ensures((return_value == MLD_ERR_INVALID_ARG) ==> (ctxlen > 255))+  /* Output buffers on error, per API-CONVENTIONS.md */+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(sig, MLDSA_CRYPTO_BYTES))+);++/**+ * Compute signature in "external mu" mode: the caller has already computed+ * the message representative mu = SHAKE256(tr || M', 64), where+ * tr = SHAKE256(pk, 64) and M' is the FIPS 204 formatted message (e.g.+ * 0x00 || ctxlen || ctx || msg for pure ML-DSA). This is the randomized+ * variant; for the deterministic variant, use mld_sign_signature_internal+ * directly with externalmu set to non-zero and an all-zero rnd.+ *+ * @spec{Implements @[FIPS204, Algorithm 2, ML-DSA.Sign external mu variant].}+ *+ * @warning This function does not perform secret key validation.+ *          Callers importing serialized keys can use mld_sign_pk_from_sk+ *          to validate them before signing.+ *+ * @param[out] sig     Output signature.+ * @param[in]  mu      Precomputed message representative.+ * @param[in]  sk      Bit-packed secret key; assumed to be valid.+ * @param      context Application context. Only present when+ *                     MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                               Success.+ * @retval MLD_ERR_OUT_OF_MEMORY           MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                         used and an allocation via+ *                                         MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_RNG_FAIL                Random number generation failed.+ * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded+ *                                         MLD_CONFIG_MAX_SIGNING_ATTEMPTS+ *                                         iterations.+ * @retval MLD_ERR_SIGNING_PAUSED          A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook+ *                                         paused signing; re-invoke to resume.+ */+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_signature_extmu(uint8_t sig[MLDSA_CRYPTO_BYTES],+                             const uint8_t mu[MLDSA_CRHBYTES],+                             const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                             MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES))+  requires(memory_no_alias(mu, MLDSA_CRHBYTES))+  requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES))+  assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES))+  ensures(return_value == 0 || return_value == MLD_ERR_OUT_OF_MEMORY ||+          return_value == MLD_ERR_RNG_FAIL ||+          return_value == MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED ||+          return_value == MLD_ERR_SIGNING_PAUSED)+  /* Output buffers on error, per API-CONVENTIONS.md */+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(sig, MLDSA_CRYPTO_BYTES))+);++#endif /* !MLD_CONFIG_NO_RANDOMIZED_API */+#endif /* !MLD_CONFIG_CORE_API_ONLY */+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+/**+ * Verify signature.+ *+ * @spec{Implements @[FIPS204, Algorithm 8, ML-DSA.Verify_internal].}+ *+ * @param[in] sig        Pointer to input signature of+ *                       MLDSA_CRYPTO_BYTES bytes.+ * @param[in] m          Pointer to message (when externalmu == 0), or to a+ *                       precomputed message representative mu (when+ *                       externalmu != 0).+ * @param     mlen       Length of m. Must equal MLDSA_CRHBYTES when+ *                       externalmu != 0.+ * @param[in] pre        Pointer to prefix string. Ignored when externalmu != 0.+ * @param     prelen     Length of prefix string. Ignored when externalmu != 0.+ * @param[in] pk         Bit-packed public key.+ * @param     externalmu 0: m/mlen is the raw message; mu = H(H(pk), pre, m) is+ *                       computed internally.+ *                       non-zero: m points to a precomputed mu of+ *                       MLDSA_CRHBYTES bytes; pre/prelen unused.+ * @param     context    Application context. Only present when+ *                       MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                       MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                         Success.+ * @retval MLD_ERR_OUT_OF_MEMORY     MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                   used and an allocation via+ *                                   MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed.+ */+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_verify_internal(const uint8_t sig[MLDSA_CRYPTO_BYTES],+                             const uint8_t *m, size_t mlen, const uint8_t *pre,+                             size_t prelen,+                             const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+                             int externalmu,+                             MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(prelen <= MLD_MAX_BUFFER_SIZE)+  requires(mlen <= MLD_MAX_BUFFER_SIZE)+  requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES))+  requires(memory_no_alias(m, mlen))+  requires((externalmu == 0) ==> ((prelen == 0) || memory_no_alias(pre, prelen)))+  requires((externalmu != 0) ==> (mlen == MLDSA_CRHBYTES))+  requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES))+  ensures(return_value == 0 || return_value == MLD_ERR_INVALID_SIGNATURE || return_value == MLD_ERR_OUT_OF_MEMORY)+);++#if !defined(MLD_CONFIG_CORE_API_ONLY)+/**+ * Verify signature.+ *+ * @spec{Implements @[FIPS204, Algorithm 3, ML-DSA.Verify].}+ *+ * @param[in] sig     Pointer to input signature.+ * @param[in] m       Pointer to message.+ * @param     mlen    Length of message.+ * @param[in] ctx     Pointer to context string. May be NULL if ctxlen == 0.+ * @param     ctxlen  Length of context string.+ * @param[in] pk      Bit-packed public key.+ * @param     context Application context. Only present when+ *                    MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                    MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                         Success.+ * @retval MLD_ERR_OUT_OF_MEMORY     MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                   used and an allocation via+ *                                   MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed.+ * @retval MLD_ERR_INVALID_ARG       The context string exceeded 255 bytes.+ */+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_verify(const uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *m,+                    size_t mlen, const uint8_t *ctx, size_t ctxlen,+                    const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+                    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(mlen <= MLD_MAX_BUFFER_SIZE)+  requires(ctxlen <= MLD_MAX_BUFFER_SIZE)+  requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES))+  requires(memory_no_alias(m, mlen))+  requires(ctxlen == 0 || memory_no_alias(ctx, ctxlen))+  requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES))+  ensures(return_value == 0 || return_value == MLD_ERR_INVALID_SIGNATURE || return_value == MLD_ERR_INVALID_ARG || return_value == MLD_ERR_OUT_OF_MEMORY)+  ensures((return_value == MLD_ERR_INVALID_ARG) ==> (ctxlen > 255))+);++/**+ * Verify signature in "external mu" mode: the caller has already computed+ * the message representative mu = SHAKE256(tr || M', 64), where+ * tr = SHAKE256(pk, 64) and M' is the FIPS 204 formatted message (e.g.+ * 0x00 || ctxlen || ctx || msg for pure ML-DSA). The same mu must have been+ * used at signing time.+ *+ * @spec{Implements @[FIPS204, Algorithm 3, ML-DSA.Verify external mu variant].}+ *+ * @param[in] sig     Pointer to input signature.+ * @param[in] mu      Precomputed message representative.+ * @param[in] pk      Bit-packed public key.+ * @param     context Application context. Only present when+ *                    MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                    MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                         Success.+ * @retval MLD_ERR_OUT_OF_MEMORY     MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                   used and an allocation via+ *                                   MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed.+ */+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_verify_extmu(const uint8_t sig[MLDSA_CRYPTO_BYTES],+                          const uint8_t mu[MLDSA_CRHBYTES],+                          const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+                          MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(  requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES))+  requires(memory_no_alias(mu, MLDSA_CRHBYTES))+  requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES))+  ensures(return_value == 0 || return_value == MLD_ERR_INVALID_SIGNATURE || return_value == MLD_ERR_OUT_OF_MEMORY)+);++#endif /* !MLD_CONFIG_CORE_API_ONLY */+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_CORE_API_ONLY)+#if !defined(MLD_CONFIG_NO_SIGN_API)+/**+ * Compute signature with pre-hashed message.+ *+ * @spec{Implements @[FIPS204, Algorithm 4, HashML-DSA.Sign].}+ *+ * Supported hash algorithm constants:+ *   MLD_PREHASH_SHA2_224, MLD_PREHASH_SHA2_256, MLD_PREHASH_SHA2_384,+ *   MLD_PREHASH_SHA2_512, MLD_PREHASH_SHA2_512_224, MLD_PREHASH_SHA2_512_256,+ *   MLD_PREHASH_SHA3_224, MLD_PREHASH_SHA3_256, MLD_PREHASH_SHA3_384,+ *   MLD_PREHASH_SHA3_512, MLD_PREHASH_SHAKE_128, MLD_PREHASH_SHAKE_256.+ *+ * MLD_PREHASH_NONE is rejected by this API.+ *+ * @warning This is an unstable API that may change in the future. If you need+ * a stable API use mld_sign_signature_pre_hash_shake256.+ *+ * @warning This function does not perform secret key validation.+ *          Callers importing serialized keys can use mld_sign_pk_from_sk+ *          to validate them before signing.+ *+ * @param[out] sig     Output signature.+ * @param[in]  ph      Pointer to pre-hashed message.+ * @param      phlen   Length of pre-hashed message. Must match the output+ *                     length of hashalg (the digest size for SHA-2/SHA-3,+ *                     32 bytes for MLD_PREHASH_SHAKE_128, 64 bytes for+ *                     MLD_PREHASH_SHAKE_256).+ * @param[in]  ctx     Pointer to context string. May be NULL if ctxlen == 0.+ * @param      ctxlen  Length of context string.+ * @param[in]  rnd     Random seed.+ * @param[in]  sk      Bit-packed secret key; assumed to be valid.+ * @param      hashalg Hash algorithm constant (one of MLD_PREHASH_*).+ * @param      context Application context. Only present when+ *                     MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                               Success.+ * @retval MLD_ERR_OUT_OF_MEMORY           MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                         used and an allocation via+ *                                         MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded+ *                                         MLD_CONFIG_MAX_SIGNING_ATTEMPTS+ *                                         iterations.+ * @retval MLD_ERR_SIGNING_PAUSED          A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook+ *                                         paused signing; re-invoke to resume.+ * @retval MLD_ERR_INVALID_ARG             The pre-hash algorithm was+ *                                         MLD_PREHASH_NONE or unsupported,+ *                                         phlen did not match the output+ *                                         length of hashalg, or the context+ *                                         string exceeded 255 bytes.+ */+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_signature_pre_hash_internal(+    uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *ph, size_t phlen,+    const uint8_t *ctx, size_t ctxlen, const uint8_t rnd[MLDSA_RNDBYTES],+    const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], int hashalg,+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(ctxlen <= MLD_MAX_BUFFER_SIZE)+  requires(phlen <= MLD_MAX_BUFFER_SIZE)+  requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES))+  requires(memory_no_alias(ph, phlen))+  requires(ctxlen == 0 || memory_no_alias(ctx, ctxlen))+  requires(memory_no_alias(rnd, MLDSA_RNDBYTES))+  requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES))+  assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES))+  ensures(return_value == 0 || return_value == MLD_ERR_OUT_OF_MEMORY || return_value == MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED || return_value == MLD_ERR_SIGNING_PAUSED || return_value == MLD_ERR_INVALID_ARG)+  /* Output buffers on error, per API-CONVENTIONS.md */+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(sig, MLDSA_CRYPTO_BYTES))+);+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+/**+ * Verify signature with pre-hashed message.+ *+ * @spec{Implements @[FIPS204, Algorithm 5, HashML-DSA.Verify].}+ *+ * Supported hash algorithm constants:+ *   MLD_PREHASH_SHA2_224, MLD_PREHASH_SHA2_256, MLD_PREHASH_SHA2_384,+ *   MLD_PREHASH_SHA2_512, MLD_PREHASH_SHA2_512_224, MLD_PREHASH_SHA2_512_256,+ *   MLD_PREHASH_SHA3_224, MLD_PREHASH_SHA3_256, MLD_PREHASH_SHA3_384,+ *   MLD_PREHASH_SHA3_512, MLD_PREHASH_SHAKE_128, MLD_PREHASH_SHAKE_256.+ *+ * MLD_PREHASH_NONE is rejected by this API.+ *+ * @warning This is an unstable API that may change in the future. If you need+ * a stable API use mld_sign_verify_pre_hash_shake256.+ *+ * @param[in] sig     Pointer to input signature.+ * @param[in] ph      Pointer to pre-hashed message.+ * @param     phlen   Length of pre-hashed message. Must match the output+ *                    length of hashalg (the digest size for SHA-2/SHA-3,+ *                    32 bytes for MLD_PREHASH_SHAKE_128, 64 bytes for+ *                    MLD_PREHASH_SHAKE_256).+ * @param[in] ctx     Pointer to context string. May be NULL if ctxlen == 0.+ * @param     ctxlen  Length of context string.+ * @param[in] pk      Bit-packed public key.+ * @param     hashalg Hash algorithm constant (one of MLD_PREHASH_*).+ * @param     context Application context. Only present when+ *                    MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                    MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                         Success.+ * @retval MLD_ERR_OUT_OF_MEMORY     MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                   used and an allocation via+ *                                   MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed.+ * @retval MLD_ERR_INVALID_ARG       The pre-hash algorithm was+ *                                   MLD_PREHASH_NONE or unsupported, phlen+ *                                   did not match the output length of+ *                                   hashalg, or the context string exceeded+ *                                   255 bytes.+ */+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_verify_pre_hash_internal(+    const uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *ph, size_t phlen,+    const uint8_t *ctx, size_t ctxlen,+    const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], int hashalg,+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(phlen <= MLD_MAX_BUFFER_SIZE)+  requires(ctxlen <= MLD_MAX_BUFFER_SIZE - 77)+  requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES))+  requires(memory_no_alias(ph, phlen))+  requires(ctxlen == 0 || memory_no_alias(ctx, ctxlen))+  requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES))+  ensures(return_value == 0 || return_value == MLD_ERR_INVALID_SIGNATURE || return_value == MLD_ERR_INVALID_ARG || return_value == MLD_ERR_OUT_OF_MEMORY)+);+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_SIGN_API)+/**+ * Compute signature with pre-hashed message using SHAKE256. This function+ * computes the SHAKE256 hash of the message internally.+ *+ * @spec{Implements @[FIPS204, Algorithm 4, HashML-DSA.Sign] with SHAKE256 as+ * the pre-hash.}+ *+ * @warning This function does not perform secret key validation.+ *          Callers importing serialized keys can use mld_sign_pk_from_sk+ *          to validate them before signing.+ *+ * @param[out] sig     Output signature.+ * @param[in]  m       Pointer to message to be hashed and signed.+ * @param      mlen    Length of message.+ * @param[in]  ctx     Pointer to context string. May be NULL if ctxlen == 0.+ * @param      ctxlen  Length of context string.+ * @param[in]  rnd     Random seed.+ * @param[in]  sk      Bit-packed secret key; assumed to be valid.+ * @param      context Application context. Only present when+ *                     MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                               Success.+ * @retval MLD_ERR_OUT_OF_MEMORY           MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                         used and an allocation via+ *                                         MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded+ *                                         MLD_CONFIG_MAX_SIGNING_ATTEMPTS+ *                                         iterations.+ * @retval MLD_ERR_SIGNING_PAUSED          A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook+ *                                         paused signing; re-invoke to resume.+ * @retval MLD_ERR_INVALID_ARG             The context string exceeded 255+ *                                         bytes.+ */+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_signature_pre_hash_shake256(+    uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *m, size_t mlen,+    const uint8_t *ctx, size_t ctxlen, const uint8_t rnd[MLDSA_RNDBYTES],+    const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(mlen <= MLD_MAX_BUFFER_SIZE)+  requires(ctxlen <= MLD_MAX_BUFFER_SIZE)+  requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES))+  requires(memory_no_alias(m, mlen))+  requires(ctxlen == 0 || memory_no_alias(ctx, ctxlen))+  requires(memory_no_alias(rnd, MLDSA_RNDBYTES))+  requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES))+  assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES))+  ensures(return_value == 0 || return_value == MLD_ERR_OUT_OF_MEMORY || return_value == MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED || return_value == MLD_ERR_SIGNING_PAUSED || return_value == MLD_ERR_INVALID_ARG)+  /* Output buffers on error, per API-CONVENTIONS.md */+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(sig, MLDSA_CRYPTO_BYTES))+);+#endif /* !MLD_CONFIG_NO_SIGN_API */++#if !defined(MLD_CONFIG_NO_VERIFY_API)+/**+ * Verify signature with pre-hashed message using SHAKE256. This function+ * computes the SHAKE256 hash of the message internally.+ *+ * @spec{Implements @[FIPS204, Algorithm 5, HashML-DSA.Verify] with SHAKE256 as+ * the pre-hash.}+ *+ * @param[in] sig     Pointer to input signature.+ * @param[in] m       Pointer to message to be hashed and verified.+ * @param     mlen    Length of message.+ * @param[in] ctx     Pointer to context string. May be NULL if ctxlen == 0.+ * @param     ctxlen  Length of context string.+ * @param[in] pk      Bit-packed public key.+ * @param     context Application context. Only present when+ *                    MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                    MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                         Success.+ * @retval MLD_ERR_OUT_OF_MEMORY     MLD_CONFIG_CUSTOM_ALLOC_FREE was+ *                                   used and an allocation via+ *                                   MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed.+ * @retval MLD_ERR_INVALID_ARG       The context string exceeded 255 bytes.+ */+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_verify_pre_hash_shake256(+    const uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *m, size_t mlen,+    const uint8_t *ctx, size_t ctxlen,+    const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+    MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(mlen <= MLD_MAX_BUFFER_SIZE)+  requires(ctxlen <= MLD_MAX_BUFFER_SIZE - 77)+  requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES))+  requires(memory_no_alias(m, mlen))+  requires(ctxlen == 0 || memory_no_alias(ctx, ctxlen))+  requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES))+  ensures(return_value == 0 || return_value == MLD_ERR_INVALID_SIGNATURE || return_value == MLD_ERR_INVALID_ARG || return_value == MLD_ERR_OUT_OF_MEMORY)+);+#endif /* !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)+/* Maximum formatted domain separation message length:+ * - Pure ML-DSA: 0x00 || ctxlen || ctx (max 255)+ * - HashML-DSA: 0x01 || ctxlen || ctx (max 255) || oid (11) || ph (max 64) */+#define MLD_DOMAIN_SEPARATION_MAX_BYTES (2 + 255 + 11 + 64)++/**+ * Prepare domain separation prefix for ML-DSA signing.+ *+ * For pure ML-DSA (hashalg == MLD_PREHASH_NONE):+ *   Format: 0x00 || ctxlen (1 byte) || ctx.+ *+ * For HashML-DSA (hashalg != MLD_PREHASH_NONE):+ *   Format: 0x01 || ctxlen (1 byte) || ctx || oid (11 bytes) || ph.+ *+ * This function is useful for building incremental signing APIs.+ *+ * @spec{For HashML-DSA (hashalg != MLD_PREHASH_NONE), implements+ * @[FIPS204, Algorithm 4, line 23]. For Pure ML-DSA+ * (hashalg == MLD_PREHASH_NONE), implements+ * ```+ *    M' <- BytesToBits(IntegerToBytes(0, 1)+ *           || IntegerToBytes(|ctx|, 1)+ *           || ctx+ * ```+ * which is part of @[FIPS204, Algorithm 2, ML-DSA.Sign, line 10] and+ * @[FIPS204, Algorithm 3, ML-DSA.Verify, line 5].}+ *+ * @param[out] prefix  Output domain separation prefix buffer.+ * @param[in]  ph      Pointer to pre-hashed message (ignored for pure+ *                     ML-DSA).+ * @param      phlen   Length of pre-hashed message; must match the output+ *                     length of hashalg (ignored for pure ML-DSA).+ * @param[in]  ctx     Pointer to context string. May be NULL if ctxlen == 0.+ * @param      ctxlen  Length of context string.+ * @param      hashalg Hash algorithm constant (MLD_PREHASH_NONE for pure+ *                     ML-DSA, or MLD_PREHASH_* for HashML-DSA).+ *+ * @return The total length of the formatted prefix, or 0 on error.+ *         Errors are:+ *         - The context string exceeded 255 bytes.+ *         - For HashML-DSA: hashalg was unsupported, ph was NULL, or phlen+ *           did not match the output length of hashalg.+ */+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+size_t mld_prepare_domain_separation_prefix(+    uint8_t prefix[MLD_DOMAIN_SEPARATION_MAX_BYTES], const uint8_t *ph,+    size_t phlen, const uint8_t *ctx, size_t ctxlen, int hashalg)+__contract__(+  requires(ctxlen <= 255)+  requires(phlen <= MLD_MAX_BUFFER_SIZE)+  requires(ctxlen == 0 || memory_no_alias(ctx, ctxlen))+  requires(hashalg == MLD_PREHASH_NONE || memory_no_alias(ph, phlen))+  requires(memory_no_alias(prefix, MLD_DOMAIN_SEPARATION_MAX_BYTES))+  assigns(memory_slice(prefix, MLD_DOMAIN_SEPARATION_MAX_BYTES))+  ensures(return_value <= MLD_DOMAIN_SEPARATION_MAX_BYTES)+);+#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */++#if !defined(MLD_CONFIG_NO_KEYPAIR_API)+/**+ * Perform basic validity checks on secret key, and derive public key.+ *+ * Referring to the decoding of the secret key `sk=(rho, K, tr, s1, s2, t0)`+ * (cf. @[FIPS204, Algorithm 25, skDecode]), the following checks are+ * performed:+ *   - Check that s1 and s2 have coefficients in [-MLDSA_ETA, MLDSA_ETA].+ *   - Check that t0 and tr stored in sk match recomputed values.+ *+ * @note This function leaks whether the secret key is valid or invalid+ * through its return value and timing.+ *+ * @param[out] pk      Output public key.+ * @param[in]  sk      Input secret key.+ * @param      context Application context. Only present when+ *                     MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLD_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                    Success.+ * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was used and an+ *                               allocation via MLD_CUSTOM_ALLOC returned NULL.+ * @retval MLD_ERR_INVALID_KEY   Secret key validation failed.+ */+MLD_MUST_CHECK_RETURN_VALUE+MLD_EXTERNAL_API+int mld_sign_pk_from_sk(uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES],+                        const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES],+                        MLD_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES))+  requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES))+  assigns(memory_slice(pk, MLDSA_CRYPTO_PUBLICKEYBYTES))+  ensures(return_value == 0 || return_value == MLD_ERR_INVALID_KEY || return_value == MLD_ERR_OUT_OF_MEMORY)+  /* Output buffers on error, per API-CONVENTIONS.md */+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(pk, MLDSA_CRYPTO_PUBLICKEYBYTES))+);+#endif /* !MLD_CONFIG_NO_KEYPAIR_API */+#endif /* !MLD_CONFIG_CORE_API_ONLY */++#endif /* !MLD_SIGN_H */
+ cbits/mldsa/src/symmetric.h view
@@ -0,0 +1,68 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLD_SYMMETRIC_H+#define MLD_SYMMETRIC_H++#include "cbmc.h"+#include "common.h"++#include MLD_FIPS202_HEADER_FILE+#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY)+#include MLD_FIPS202X4_HEADER_FILE+#endif++#define MLD_STREAM128_BLOCKBYTES SHAKE128_RATE+#define MLD_STREAM256_BLOCKBYTES SHAKE256_RATE++#define mld_xof256_ctx mld_shake256ctx+#define mld_xof256_init(CTX) mld_shake256_init(CTX)++#define mld_xof256_absorb_once(CTX, IN, INBYTES) \+  do                                             \+  {                                              \+    mld_shake256_absorb(CTX, IN, INBYTES);       \+    mld_shake256_finalize(CTX);                  \+  } while (0)+++#define mld_xof256_release(CTX) mld_shake256_release(CTX)+#define mld_xof256_squeezeblocks(OUT, OUTBLOCKS, STATE) \+  mld_shake256_squeeze(OUT, (OUTBLOCKS) * SHAKE256_RATE, STATE)++#define mld_xof128_ctx mld_shake128ctx+#define mld_xof128_init(CTX) mld_shake128_init(CTX)++#define mld_xof128_absorb_once(CTX, IN, INBYTES) \+  do                                             \+  {                                              \+    mld_shake128_absorb(CTX, IN, INBYTES);       \+    mld_shake128_finalize(CTX);                  \+  } while (0)++#define mld_xof128_release(CTX) mld_shake128_release(CTX)+#define mld_xof128_squeezeblocks(OUT, OUTBLOCKS, STATE) \+  mld_shake128_squeeze(OUT, (OUTBLOCKS) * SHAKE128_RATE, STATE)++#define mld_xof256_x4_ctx mld_shake256x4ctx+#define mld_xof256_x4_init(CTX) mld_shake256x4_init((CTX))+#define mld_xof256_x4_absorb(CTX, IN, INBYTES)                          \+  mld_shake256x4_absorb_once((CTX), (IN)[0], (IN)[1], (IN)[2], (IN)[3], \+                             (INBYTES))+#define mld_xof256_x4_squeezeblocks(BUF, NBLOCKS, CTX)                 \+  mld_shake256x4_squeezeblocks((BUF)[0], (BUF)[1], (BUF)[2], (BUF)[3], \+                               (NBLOCKS), (CTX))+#define mld_xof256_x4_release(CTX) mld_shake256x4_release((CTX))++#define mld_xof128_x4_ctx mld_shake128x4ctx+#define mld_xof128_x4_init(CTX) mld_shake128x4_init((CTX))+#define mld_xof128_x4_absorb(CTX, IN, INBYTES)                          \+  mld_shake128x4_absorb_once((CTX), (IN)[0], (IN)[1], (IN)[2], (IN)[3], \+                             (INBYTES))+#define mld_xof128_x4_squeezeblocks(BUF, NBLOCKS, CTX)                 \+  mld_shake128x4_squeezeblocks((BUF)[0], (BUF)[1], (BUF)[2], (BUF)[3], \+                               (NBLOCKS), (CTX))+#define mld_xof128_x4_release(CTX) mld_shake128x4_release((CTX))++#endif /* !MLD_SYMMETRIC_H */
+ cbits/mldsa/src/sys.h view
@@ -0,0 +1,327 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLD_SYS_H+#define MLD_SYS_H++#if !defined(MLD_CONFIG_NO_ASM) && (defined(__GNUC__) || defined(__clang__))+#define MLD_HAVE_INLINE_ASM+#endif++/* Try to find endianness, if not forced through CFLAGS already */+#if !defined(MLD_SYS_LITTLE_ENDIAN) && !defined(MLD_SYS_BIG_ENDIAN)+#if defined(__BYTE_ORDER__)+#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__+#define MLD_SYS_LITTLE_ENDIAN+#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__+#define MLD_SYS_BIG_ENDIAN+#else+#error "__BYTE_ORDER__ defined, but don't recognize value."+#endif+#endif /* __BYTE_ORDER__ */++/* MSVC does not define __BYTE_ORDER__. However, MSVC only supports+ * little endian x86, x86_64, and AArch64. It is, hence, safe to assume+ * little endian. */+#if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_AMD64) || \+                          defined(_M_IX86) || defined(_M_ARM64))+#define MLD_SYS_LITTLE_ENDIAN+#endif++#endif /* !MLD_SYS_LITTLE_ENDIAN && !MLD_SYS_BIG_ENDIAN */++/* Check if we're running on an AArch64 little endian system. _M_ARM64 is set by+ * MSVC. */+#if defined(__AARCH64EL__) || defined(_M_ARM64)+#define MLD_SYS_AARCH64+#endif++/* Check if the AArch64 compilation target supports NEON (Advanced SIMD).+ *+ * Some compilers also define __ARM_NEON__, but __ARM_NEON is the most reliable+ * signal. Specifically, clang on Apple appears to keep __ARM_NEON__ set even if+ * -march=armv8-a+nosimd is set.+ *+ * gcc 4.8 -- the first gcc version introducing Neon support -- sets neither+ * __ARM_NEON nor __ARM_NEON__; in fact, there is no preprocessor signal that+ * Neon is enabled. If you use gcc 4.8 and need Neon, you should set+ * MLD_SYS_AARCH64_NEON manually. gcc 4.9 onwards do set __ARM_NEON.+ */+#if defined(MLD_SYS_AARCH64) && defined(__ARM_NEON)+#define MLD_SYS_AARCH64_NEON+#endif++/* Check if we're running on an AArch64 big endian system. */+#if defined(__AARCH64EB__)+#define MLD_SYS_AARCH64_EB+#endif++/* Check if we're running on an Armv8.1-M system with MVE */+#if defined(__ARM_ARCH_8_1M_MAIN__) || defined(__ARM_FEATURE_MVE)+#define MLD_SYS_ARMV81M_MVE+#endif++/* Check if we're running on an x86_64 system. */+#if defined(__x86_64__) || defined(_M_X64) || defined(_M_AMD64)+#define MLD_SYS_X86_64+#if defined(__AVX2__)+#define MLD_SYS_X86_64_AVX2+#endif+#endif /* __x86_64__ || _M_X64 || _M_AMD64 */++#if defined(MLD_SYS_LITTLE_ENDIAN) && defined(__powerpc64__)+#define MLD_SYS_PPC64LE+#endif++#if defined(__riscv) && defined(__riscv_xlen) && __riscv_xlen == 64+#define MLD_SYS_RISCV64+#endif++#if defined(MLD_SYS_RISCV64) && defined(__riscv_vector) && \+    defined(__riscv_v_intrinsic)+#define MLD_SYS_RISCV64_RVV+#endif++#if defined(__riscv) && defined(__riscv_xlen) && __riscv_xlen == 32+#define MLD_SYS_RISCV32+#endif++#if defined(_WIN64) || defined(_WIN32)+#define MLD_SYS_WINDOWS+#endif++#if defined(__linux__)+#define MLD_SYS_LINUX+#endif++#if defined(__APPLE__)+#define MLD_SYS_APPLE+#endif++/* If MLD_FORCE_AARCH64 is set, assert that we're indeed on an AArch64 system.+ */+#if defined(MLD_FORCE_AARCH64) && !defined(MLD_SYS_AARCH64)+#error "MLD_FORCE_AARCH64 is set, but we don't seem to be on an AArch64 system."+#endif++/* If MLD_FORCE_AARCH64_EB is set, assert that we're indeed on a big endian+ * AArch64 system. */+#if defined(MLD_FORCE_AARCH64_EB) && !defined(MLD_SYS_AARCH64_EB)+#error \+    "MLD_FORCE_AARCH64_EB is set, but we don't seem to be on an AArch64 system."+#endif++/* If MLD_FORCE_X86_64 is set, assert that we're indeed on an X86_64 system. */+#if defined(MLD_FORCE_X86_64) && !defined(MLD_SYS_X86_64)+#error "MLD_FORCE_X86_64 is set, but we don't seem to be on an X86_64 system."+#endif++#if defined(MLD_FORCE_PPC64LE) && !defined(MLD_SYS_PPC64LE)+#error "MLD_FORCE_PPC64LE is set, but we don't seem to be on a PPC64LE system."+#endif++#if defined(MLD_FORCE_RISCV64) && !defined(MLD_SYS_RISCV64)+#error "MLD_FORCE_RISCV64 is set, but we don't seem to be on a RISCV64 system."+#endif++#if defined(MLD_FORCE_RISCV32) && !defined(MLD_SYS_RISCV32)+#error "MLD_FORCE_RISCV32 is set, but we don't seem to be on a RISCV32 system."+#endif++/*+ * MLD_INLINE: Hint for inlining.+ * - MSVC: __inline+ * - C99+: inline+ * - GCC/Clang C90: __attribute__((unused)) to silence warnings+ * - Other C90: empty+ */+#if !defined(MLD_INLINE)+#if defined(_MSC_VER)+#define MLD_INLINE __inline+#elif defined(inline) || \+    (defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L)+#define MLD_INLINE inline+#elif defined(__GNUC__) || defined(__clang__)+#define MLD_INLINE __attribute__((unused))+#else+#define MLD_INLINE+#endif+#endif /* !MLD_INLINE */++/*+ * MLD_ALWAYS_INLINE: Force inlining.+ * - MSVC: __forceinline+ * - GCC/Clang C99+: MLD_INLINE __attribute__((always_inline))+ * - Other: MLD_INLINE (no forced inlining)+ */+#if !defined(MLD_ALWAYS_INLINE)+#if defined(_MSC_VER)+#define MLD_ALWAYS_INLINE __forceinline+#elif (defined(__GNUC__) || defined(__clang__)) && \+    (defined(inline) ||                            \+     (defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L))+#define MLD_ALWAYS_INLINE MLD_INLINE __attribute__((always_inline))+#else+#define MLD_ALWAYS_INLINE MLD_INLINE+#endif+#endif /* !MLD_ALWAYS_INLINE */++/*+ * MLD_NOINLINE: Prevent inlining.+ * - MSVC: __declspec(noinline)+ * - GCC/Clang: __attribute__((noinline))+ * - Other: empty+ */+#if !defined(MLD_NOINLINE)+#if defined(_MSC_VER)+#define MLD_NOINLINE __declspec(noinline)+#elif defined(__GNUC__) || defined(__clang__)+#define MLD_NOINLINE __attribute__((noinline))+#else+#define MLD_NOINLINE+#endif+#endif /* !MLD_NOINLINE */++#ifndef MLD_STATIC_TESTABLE+#define MLD_STATIC_TESTABLE static+#endif++/*+ * C90 does not have the restrict compiler directive yet.+ * We don't use it in C90 builds.+ */+#if !defined(restrict)+#if defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L+#define MLD_RESTRICT restrict+#else+#define MLD_RESTRICT+#endif++#else /* !restrict */++#define MLD_RESTRICT restrict+#endif /* restrict */++#define MLD_DEFAULT_ALIGN 32+#define MLD_ALIGN_UP(N) \+  ((((N) + (MLD_DEFAULT_ALIGN - 1)) / MLD_DEFAULT_ALIGN) * MLD_DEFAULT_ALIGN)+#if defined(__GNUC__)+#define MLD_ALIGN __attribute__((aligned(MLD_DEFAULT_ALIGN)))+#elif defined(_MSC_VER)+#define MLD_ALIGN __declspec(align(MLD_DEFAULT_ALIGN))+#else+#define MLD_ALIGN /* No known support for alignment constraints */+#endif+++/* New X86_64 CPUs support control-flow protection using the CET instructions.+ * When enabled (through -fcf-protection=), all compilation units (including+ * empty ones) need to support CET for this to work.+ * For assembly, this means that source files need to signal support for+ * CET by setting the appropriate note.gnu.property section.+ * This can be achieved by including the <cet.h> header in all assembly file.+ * This file also provides the _CET_ENDBR macro which needs to be placed at+ * every potential target of an indirect branch.+ * If CET is enabled _CET_ENDBR maps to the endbr64 instruction, otherwise+ * it is empty.+ * In case the compiler does not support CET (e.g., <gcc8, <clang11),+ * the __CET__ macro is not set and we default to nothing.+ * Note that we only issue _CET_ENDBR instructions through the MLD_ASM_FN_SYMBOL+ * macro as the global symbols are the only possible targets of indirect+ * branches in our code.+ */+#if defined(MLD_SYS_X86_64)+#if defined(__CET__)+#include <cet.h>+#define MLD_CET_ENDBR _CET_ENDBR+#else+#define MLD_CET_ENDBR+#endif+#endif /* MLD_SYS_X86_64 */++#if defined(MLD_CONFIG_CT_TESTING_ENABLED) && !defined(__ASSEMBLER__)+#include <valgrind/memcheck.h>+#define MLD_CT_TESTING_SECRET(ptr, len) \+  VALGRIND_MAKE_MEM_UNDEFINED((ptr), (len))+#define MLD_CT_TESTING_DECLASSIFY(ptr, len) \+  VALGRIND_MAKE_MEM_DEFINED((ptr), (len))+#else /* MLD_CONFIG_CT_TESTING_ENABLED && !__ASSEMBLER__ */+#define MLD_CT_TESTING_SECRET(ptr, len) \+  do                                    \+  {                                     \+  } while (0)+#define MLD_CT_TESTING_DECLASSIFY(ptr, len) \+  do                                        \+  {                                         \+  } while (0)+#endif /* !(MLD_CONFIG_CT_TESTING_ENABLED && !__ASSEMBLER__) */++#if defined(__GNUC__) || defined(__clang__)+#define MLD_MUST_CHECK_RETURN_VALUE __attribute__((warn_unused_result))+#else+#define MLD_MUST_CHECK_RETURN_VALUE+#endif++/* The x86_64 assembly backend uses the SysV calling convention. On Windows,+ * where the Microsoft x64 calling convention is the default, it can still be+ * used with compilers that allow choosing the calling convention per+ * function: GCC and Clang support __attribute__((sysv_abi)), which makes+ * calls to the annotated function follow the SysV calling convention.+ *+ * MLD_SYSV_ABI_SUPPORTED signals that the toolchain can call SysV assembly+ * routines; the x86_64 assembly backend is only enabled if it is defined.+ * MLD_SYSV_ABI is the attribute carried by declarations of x86_64 assembly+ * routines. Both macros can be set externally for toolchains offering an+ * equivalent mechanism that is not recognized here. */+#if defined(MLD_SYS_X86_64) && !defined(MLD_SYSV_ABI_SUPPORTED)+#if !defined(MLD_SYS_WINDOWS) || defined(__GNUC__) || defined(__clang__)+#define MLD_SYSV_ABI_SUPPORTED+#endif+#endif++#if !defined(MLD_SYSV_ABI)+#if defined(MLD_SYS_WINDOWS) && defined(MLD_SYSV_ABI_SUPPORTED)+#define MLD_SYSV_ABI __attribute__((sysv_abi))+#else+#define MLD_SYSV_ABI+#endif+#endif /* !MLD_SYSV_ABI */++#if !defined(__ASSEMBLER__)+/* System capability enumeration */+typedef enum+{+  /* x86_64 */+  MLD_SYS_CAP_X86_64_AVX2,+  /* AArch64 */+  MLD_SYS_CAP_AARCH64_NEON,+  MLD_SYS_CAP_AARCH64_SHA3,+  /* Armv8.1-M */+  MLD_SYS_CAP_ARMV81M_MVE+} mld_sys_cap;++#if !defined(MLD_CONFIG_CUSTOM_CAPABILITY_FUNC)+#include "cbmc.h"++MLD_MUST_CHECK_RETURN_VALUE+static MLD_INLINE int mld_sys_check_capability(mld_sys_cap cap)+__contract__(+  ensures(return_value == 0 || return_value == 1)+)+{+  /* By default, we rely on compile-time feature detection/specification:+   * If a feature is enabled at compile-time, we assume it is supported by+   * the host that the resulting library/binary will be built on.+   * If this assumption is not true, you MUST overwrite this function.+   * See the documentation of MLD_CONFIG_CUSTOM_CAPABILITY_FUNC in+   * mldsa_native_config.h for more information. */+  (void)cap;+  return 1;+}+#endif /* !MLD_CONFIG_CUSTOM_CAPABILITY_FUNC */+#endif /* !__ASSEMBLER__ */++#endif /* !MLD_SYS_H */
+ cbits/mldsa/src/zetas.inc view
@@ -0,0 +1,55 @@+/*+ * Copyright (c) The mldsa-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mldsa-native repository.+ *          Do not modify it directly.+ */+++/*+ * Table of zeta values used in the reference NTT and inverse NTT.+ * See autogen for details.+ */+static const int32_t mld_zetas[MLDSA_N] = {+    0,        25847,    -2608894, -518909,  237124,   -777960,  -876248,+    466468,   1826347,  2353451,  -359251,  -2091905, 3119733,  -2884855,+    3111497,  2680103,  2725464,  1024112,  -1079900, 3585928,  -549488,+    -1119584, 2619752,  -2108549, -2118186, -3859737, -1399561, -3277672,+    1757237,  -19422,   4010497,  280005,   2706023,  95776,    3077325,+    3530437,  -1661693, -3592148, -2537516, 3915439,  -3861115, -3043716,+    3574422,  -2867647, 3539968,  -300467,  2348700,  -539299,  -1699267,+    -1643818, 3505694,  -3821735, 3507263,  -2140649, -1600420, 3699596,+    811944,   531354,   954230,   3881043,  3900724,  -2556880, 2071892,+    -2797779, -3930395, -1528703, -3677745, -3041255, -1452451, 3475950,+    2176455,  -1585221, -1257611, 1939314,  -4083598, -1000202, -3190144,+    -3157330, -3632928, 126922,   3412210,  -983419,  2147896,  2715295,+    -2967645, -3693493, -411027,  -2477047, -671102,  -1228525, -22981,+    -1308169, -381987,  1349076,  1852771,  -1430430, -3343383, 264944,+    508951,   3097992,  44288,    -1100098, 904516,   3958618,  -3724342,+    -8578,    1653064,  -3249728, 2389356,  -210977,  759969,   -1316856,+    189548,   -3553272, 3159746,  -1851402, -2409325, -177440,  1315589,+    1341330,  1285669,  -1584928, -812732,  -1439742, -3019102, -3881060,+    -3628969, 3839961,  2091667,  3407706,  2316500,  3817976,  -3342478,+    2244091,  -2446433, -3562462, 266997,   2434439,  -1235728, 3513181,+    -3520352, -3759364, -1197226, -3193378, 900702,   1859098,  909542,+    819034,   495491,   -1613174, -43260,   -522500,  -655327,  -3122442,+    2031748,  3207046,  -3556995, -525098,  -768622,  -3595838, 342297,+    286988,   -2437823, 4108315,  3437287,  -3342277, 1735879,  203044,+    2842341,  2691481,  -2590150, 1265009,  4055324,  1247620,  2486353,+    1595974,  -3767016, 1250494,  2635921,  -3548272, -2994039, 1869119,+    1903435,  -1050970, -1333058, 1237275,  -3318210, -1430225, -451100,+    1312455,  3306115,  -1962642, -1279661, 1917081,  -2546312, -1374803,+    1500165,  777191,   2235880,  3406031,  -542412,  -2831860, -1671176,+    -1846953, -2584293, -3724270, 594136,   -3776993, -2013608, 2432395,+    2454455,  -164721,  1957272,  3369112,  185531,   -1207385, -3183426,+    162844,   1616392,  3014001,  810149,   1652634,  -3694233, -1799107,+    -3038916, 3523897,  3866901,  269760,   2213111,  -975884,  1717735,+    472078,   -426683,  1723600,  -1803090, 1910376,  -1667432, -1104333,+    -260646,  -3833893, -2939036, -2235985, -420899,  -2286327, 183443,+    -976891,  1612842,  -3545687, -554416,  3919660,  -48306,   -1362209,+    3937738,  1400424,  -846154,  1976782,+};
+ cbits/mlkem/COMMIT view
@@ -0,0 +1,2 @@+d1b2fe782888bdb761a50336012923180be7f502+v2.0.0
+ cbits/mlkem/LICENSE view
@@ -0,0 +1,312 @@+mlkem-native is a fork of the public domain Kyber reference+implementation, available on https://github.com/pq-crystals/kyber.++All source code in mlkem/* and dev/* is licensed under your choice+of the Apache-2.0 license OR the ISC license OR the MIT license.+These licenses are reproduced at the bottom of this file.+The copyright holders are indicated at the top of each file.++Files outside the library itself may carry different terms. Every file+states its own SPDX-License-Identifier, which determines the terms that+apply to it. In particular:++The code in test/notrandombytes/*, and its copies in+examples/*/test_only_rng/* and scripts/notrandombytes, is derived from+https://cr.yp.to/papers.html#surf and licensed under+LicenseRef-PD-hp OR CC0-1.0 OR 0BSD OR MIT-0 OR MIT.+It is only used for testing purposes.++The benchmarking code in test/hal/* carries the+MIT license. It is only used for testing purposes.++The tiny_sha3 code in+examples/bring_your_own_fips202/custom_fips202/tiny_sha3/* and+examples/custom_backend/mlkem_native/src/fips202/native/custom/src/*+carries the MIT license. It is only used to demonstrate custom+FIPS-202 implementations.++The proofs in proofs/* are in part derived from Amazon Web Services+verification infrastructure. Individual files carry, among others,+Apache-2.0 OR ISC OR MIT-0, MIT-0, and MIT-0 AND Apache-2.0. None of the+proofs are part of the library.++Documentation is licensed under CC-BY-4.0.++```+Copyright (c) The mlkem-native project authors+Copyright (c) 2020 Dougall Johnson+Copyright (c) 2022 Arm Limited+SPDX-License-Identifier: MIT++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in+all copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.+```++The tiny_sha3 implementation in examples/custom_backend/mlkem_native/src/fips202/native/custom/src/+carries the MIT license. It is only used for testing purposes.++```+SPDX-License-Identifier: MIT++19-Nov-11  Markku-Juhani O. Saarinen <mjos@iki.fi>+```++ISC license+-----------++Copyright <YEAR> <OWNER>++Permission to use, copy, modify, and/or distribute this software for any purpose+with or without fee is hereby granted, provided that the above copyright notice+and this permission notice appear in all copies.++THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH+REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND+FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,+INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS+OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER+TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF+THIS SOFTWARE.+++MIT license+-----------++Copyright <YEAR> <COPYRIGHT HOLDER>++Permission is hereby granted, free of charge, to any person obtaining a copy of+this software and associated documentation files (the “Software”), to deal in+the Software without restriction, including without limitation the rights to+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of+the Software, and to permit persons to whom the Software is furnished to do so,+subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.++Apache-2.0 license+------------------++                                 Apache License+                           Version 2.0, January 2004+                        http://www.apache.org/licenses/++   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION++   1. Definitions.++      "License" shall mean the terms and conditions for use, reproduction,+      and distribution as defined by Sections 1 through 9 of this document.++      "Licensor" shall mean the copyright owner or entity authorized by+      the copyright owner that is granting the License.++      "Legal Entity" shall mean the union of the acting entity and all+      other entities that control, are controlled by, or are under common+      control with that entity. For the purposes of this definition,+      "control" means (i) the power, direct or indirect, to cause the+      direction or management of such entity, whether by contract or+      otherwise, or (ii) ownership of fifty percent (50%) or more of the+      outstanding shares, or (iii) beneficial ownership of such entity.++      "You" (or "Your") shall mean an individual or Legal Entity+      exercising permissions granted by this License.++      "Source" form shall mean the preferred form for making modifications,+      including but not limited to software source code, documentation+      source, and configuration files.++      "Object" form shall mean any form resulting from mechanical+      transformation or translation of a Source form, including but+      not limited to compiled object code, generated documentation,+      and conversions to other media types.++      "Work" shall mean the work of authorship, whether in Source or+      Object form, made available under the License, as indicated by a+      copyright notice that is included in or attached to the work+      (an example is provided in the Appendix below).++      "Derivative Works" shall mean any work, whether in Source or Object+      form, that is based on (or derived from) the Work and for which the+      editorial revisions, annotations, elaborations, or other modifications+      represent, as a whole, an original work of authorship. For the purposes+      of this License, Derivative Works shall not include works that remain+      separable from, or merely link (or bind by name) to the interfaces of,+      the Work and Derivative Works thereof.++      "Contribution" shall mean any work of authorship, including+      the original version of the Work and any modifications or additions+      to that Work or Derivative Works thereof, that is intentionally+      submitted to Licensor for inclusion in the Work by the copyright owner+      or by an individual or Legal Entity authorized to submit on behalf of+      the copyright owner. For the purposes of this definition, "submitted"+      means any form of electronic, verbal, or written communication sent+      to the Licensor or its representatives, including but not limited to+      communication on electronic mailing lists, source code control systems,+      and issue tracking systems that are managed by, or on behalf of, the+      Licensor for the purpose of discussing and improving the Work, but+      excluding communication that is conspicuously marked or otherwise+      designated in writing by the copyright owner as "Not a Contribution."++      "Contributor" shall mean Licensor and any individual or Legal Entity+      on behalf of whom a Contribution has been received by Licensor and+      subsequently incorporated within the Work.++   2. Grant of Copyright License. Subject to the terms and conditions of+      this License, each Contributor hereby grants to You a perpetual,+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable+      copyright license to reproduce, prepare Derivative Works of,+      publicly display, publicly perform, sublicense, and distribute the+      Work and such Derivative Works in Source or Object form.++   3. Grant of Patent License. Subject to the terms and conditions of+      this License, each Contributor hereby grants to You a perpetual,+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable+      (except as stated in this section) patent license to make, have made,+      use, offer to sell, sell, import, and otherwise transfer the Work,+      where such license applies only to those patent claims licensable+      by such Contributor that are necessarily infringed by their+      Contribution(s) alone or by combination of their Contribution(s)+      with the Work to which such Contribution(s) was submitted. If You+      institute patent litigation against any entity (including a+      cross-claim or counterclaim in a lawsuit) alleging that the Work+      or a Contribution incorporated within the Work constitutes direct+      or contributory patent infringement, then any patent licenses+      granted to You under this License for that Work shall terminate+      as of the date such litigation is filed.++   4. Redistribution. You may reproduce and distribute copies of the+      Work or Derivative Works thereof in any medium, with or without+      modifications, and in Source or Object form, provided that You+      meet the following conditions:++      (a) You must give any other recipients of the Work or+          Derivative Works a copy of this License; and++      (b) You must cause any modified files to carry prominent notices+          stating that You changed the files; and++      (c) You must retain, in the Source form of any Derivative Works+          that You distribute, all copyright, patent, trademark, and+          attribution notices from the Source form of the Work,+          excluding those notices that do not pertain to any part of+          the Derivative Works; and++      (d) If the Work includes a "NOTICE" text file as part of its+          distribution, then any Derivative Works that You distribute must+          include a readable copy of the attribution notices contained+          within such NOTICE file, excluding those notices that do not+          pertain to any part of the Derivative Works, in at least one+          of the following places: within a NOTICE text file distributed+          as part of the Derivative Works; within the Source form or+          documentation, if provided along with the Derivative Works; or,+          within a display generated by the Derivative Works, if and+          wherever such third-party notices normally appear. The contents+          of the NOTICE file are for informational purposes only and+          do not modify the License. You may add Your own attribution+          notices within Derivative Works that You distribute, alongside+          or as an addendum to the NOTICE text from the Work, provided+          that such additional attribution notices cannot be construed+          as modifying the License.++      You may add Your own copyright statement to Your modifications and+      may provide additional or different license terms and conditions+      for use, reproduction, or distribution of Your modifications, or+      for any such Derivative Works as a whole, provided Your use,+      reproduction, and distribution of the Work otherwise complies with+      the conditions stated in this License.++   5. Submission of Contributions. Unless You explicitly state otherwise,+      any Contribution intentionally submitted for inclusion in the Work+      by You to the Licensor shall be under the terms and conditions of+      this License, without any additional terms or conditions.+      Notwithstanding the above, nothing herein shall supersede or modify+      the terms of any separate license agreement you may have executed+      with Licensor regarding such Contributions.++   6. Trademarks. This License does not grant permission to use the trade+      names, trademarks, service marks, or product names of the Licensor,+      except as required for reasonable and customary use in describing the+      origin of the Work and reproducing the content of the NOTICE file.++   7. Disclaimer of Warranty. Unless required by applicable law or+      agreed to in writing, Licensor provides the Work (and each+      Contributor provides its Contributions) on an "AS IS" BASIS,+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or+      implied, including, without limitation, any warranties or conditions+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A+      PARTICULAR PURPOSE. You are solely responsible for determining the+      appropriateness of using or redistributing the Work and assume any+      risks associated with Your exercise of permissions under this License.++   8. Limitation of Liability. In no event and under no legal theory,+      whether in tort (including negligence), contract, or otherwise,+      unless required by applicable law (such as deliberate and grossly+      negligent acts) or agreed to in writing, shall any Contributor be+      liable to You for damages, including any direct, indirect, special,+      incidental, or consequential damages of any character arising as a+      result of this License or out of the use or inability to use the+      Work (including but not limited to damages for loss of goodwill,+      work stoppage, computer failure or malfunction, or any and all+      other commercial damages or losses), even if such Contributor+      has been advised of the possibility of such damages.++   9. Accepting Warranty or Additional Liability. While redistributing+      the Work or Derivative Works thereof, You may choose to offer,+      and charge a fee for, acceptance of support, warranty, indemnity,+      or other liability obligations and/or rights consistent with this+      License. However, in accepting such obligations, You may act only+      on Your own behalf and on Your sole responsibility, not on behalf+      of any other Contributor, and only if You agree to indemnify,+      defend, and hold each Contributor harmless for any liability+      incurred by, or claims asserted against, such Contributor by reason+      of your accepting any such warranty or additional liability.++   END OF TERMS AND CONDITIONS++   APPENDIX: How to apply the Apache License to your work.++      To apply the Apache License to your work, attach the following+      boilerplate notice, with the fields enclosed by brackets "[]"+      replaced with your own identifying information. (Don't include+      the brackets!)  The text should be enclosed in the appropriate+      comment syntax for the file format. We also recommend that a+      file or class name and description of purpose be included on the+      same "printed page" as the copyright notice for easier+      identification within third-party archives.++   Copyright [yyyy] [name of copyright owner]++   Licensed under the Apache License, Version 2.0 (the "License");+   you may not use this file except in compliance with the License.+   You may obtain a copy of the License at++       http://www.apache.org/licenses/LICENSE-2.0++   Unless required by applicable law or agreed to in writing, software+   distributed under the License is distributed on an "AS IS" BASIS,+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+   See the License for the specific language governing permissions and+   limitations under the License.
+ cbits/mlkem/README.md view
@@ -0,0 +1,59 @@+# mlkem-native++ML-KEM (FIPS 203) from the PQ Code Package, vendored here and built for all+three parameter sets.++## Where it comes from++<https://github.com/pq-code-package/mlkem-native>.  `COMMIT` holds the+revision this tree is at; `import.sh` puts it there and is how the tree is+refreshed.  Nothing here is edited by hand -- every choice crypton makes is+made in `crypton_mlkem.h`, `crypton_mlkem.c` and `crypton.cabal`, so that a+re-import is a straight overwrite.++## Licence++`Apache-2.0 OR ISC OR MIT`, the same three-way form as `cbits/s2n` and by+some of the same authors.  crypton takes it under ISC, which is already in+the package's `license:` field.  `LICENSE` is the upstream file and is+listed in `license-files:`.++## What is taken and what is not++The whole of `mlkem/`, less the backends for architectures crypton does not+build for: `src/native/riscv64`, `src/native/ppc64le` and the 32-bit+`src/fips202/native/armv81m`.  Every reference to those is behind an+`MLK_SYS_` guard that cannot be true on the architectures crypton does+build for, so dropping them changes no build and keeps a few dozen files of+unreachable assembly out of the release tarball.  To take one back, delete+its line from `import.sh` and add its directory to `extra-source-files:`.++## How it is built++Upstream builds for one parameter set at a time.  `crypton_mlkem.c`+includes the amalgamation once per set -- the level-independent half kept by+exactly one of them -- which is how a single crypton offers ML-KEM-512, 768+and 1024.  `crypton_mlkem_asm.S` does the same for the assembly, which is+level-independent and so is included once.++This is the shape `cbits/aes/armv8.c` already uses for the three AES key+sizes, and it has the same hazard: **cabal does not know that the wrapper+depends on the tree it includes.**  After changing anything under+`cbits/mlkem`, touch `crypton_mlkem.c`, or the build keeps the object it+already has and the change is not tested.++The hand-written backends are selected by `CRYPTON_MLKEM_NATIVE_BACKEND`,+which `crypton.cabal` defines on x86-64 and AArch64 other than Windows --+the same exclusion, and for the same reasons, as `cbits/s2n`.  Everywhere+else the portable C is built, which is the same code and passes the same+tests.++There is no randomised API: `MLK_CONFIG_NO_RANDOMIZED_API` is set, no+`randombytes()` is needed, and randomness is drawn in Haskell through+`MonadRandom` as it is for every other key crypton generates.++The symbols are `crypton_mlkem512_*`, `crypton_mlkem768_*` and+`crypton_mlkem1024_*` rather than upstream's defaults, so that an+application linking another copy of mlkem-native -- through some other+library, or its own -- does not present the linker with two sets of+functions answering to one set of names.
+ cbits/mlkem/crypton_mlkem.c view
@@ -0,0 +1,31 @@+/*+ * All three ML-KEM parameter sets in one translation unit.+ *+ * mlkem-native is built for one parameter set at a time; a build wanting+ * several includes the amalgamation once per set, with the level-independent+ * half kept by exactly one of them.  This is the same shape as+ * cbits/aes/armv8.c, which includes cbits/aes/armv8_impl.c three times for+ * the three AES key sizes.+ *+ * NOTE, as there: cabal does not know that this file depends on the tree it+ * includes.  After changing anything under cbits/mlkem, touch this file, or+ * the build will quietly keep the object it already has.+ */+#include "crypton_mlkem.h"++#define MLK_CONFIG_MULTILEVEL_WITH_SHARED 1+#define MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS+#define MLK_CONFIG_PARAMETER_SET 512+#include "mlkem_native.c"+#undef MLK_CONFIG_MULTILEVEL_WITH_SHARED+#undef MLK_CONFIG_PARAMETER_SET++#define MLK_CONFIG_MULTILEVEL_NO_SHARED+#define MLK_CONFIG_PARAMETER_SET 768+#include "mlkem_native.c"+#undef MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS+#undef MLK_CONFIG_PARAMETER_SET++#define MLK_CONFIG_PARAMETER_SET 1024+#include "mlkem_native.c"+#undef MLK_CONFIG_PARAMETER_SET
+ cbits/mlkem/crypton_mlkem.h view
@@ -0,0 +1,79 @@+/*+ * What crypton asks of the vendored mlkem-native, in one place.  The tree+ * under cbits/mlkem is upstream's and is overwritten by import.sh, so every+ * choice crypton makes is made here instead of by editing it.+ *+ * Included first by both crypton_mlkem.c and crypton_mlkem_asm.S, so it must+ * hold nothing the assembler cannot read: directives, and C behind+ * __ASSEMBLER__.+ */+#ifndef CRYPTON_MLKEM_H+#define CRYPTON_MLKEM_H++/*+ * The symbols are crypton's own, not the default PQCP_MLKEM_NATIVE_*.  An+ * application is free to link another copy of mlkem-native -- through some+ * other library, or its own -- and two copies answering to one set of names+ * is a problem the linker resolves silently and in nobody's favour.  With+ * MLK_CONFIG_MULTILEVEL_BUILD the level is appended, so the entry points+ * are crypton_mlkem512_*, crypton_mlkem768_* and crypton_mlkem1024_*.+ */+#define MLK_CONFIG_NAMESPACE_PREFIX crypton_mlkem+#define MLK_CONFIG_MULTILEVEL_BUILD++/*+ * No randomised API, so no randombytes() to provide.  Randomness is drawn+ * in Haskell through MonadRandom, the way every other key in crypton is+ * generated, and the deterministic entry points are what the FFI calls.+ * That also keeps the C free of any opinion about where entropy comes from.+ */+#define MLK_CONFIG_NO_RANDOMIZED_API++/*+ * The hand-written backends, where crypton.cabal says the architecture has+ * them.  Without this the portable C is built, which is correct everywhere+ * and is what every other architecture gets.+ */+#ifdef CRYPTON_MLKEM_NATIVE_BACKEND+#define MLK_CONFIG_USE_NATIVE_BACKEND_ARITH+#define MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202++/*+ * Which of the backend's instructions this processor actually has.+ *+ * Without this both packages decide at compile time: the backend is built+ * because the architecture has the instructions in principle, and is then+ * called with no check.  Upstream says so plainly -- if the machine that+ * runs the binary may differ from the one that built it, this function+ * MUST be provided -- and crypton is a library whose builds are copied+ * between machines and shipped by distributions.  Without it a binary+ * built on a host with AVX2 dies with SIGILL on one without (#316).+ *+ * The dispatch sites are upstream's; each falls back to the portable C+ * when this answers no.  The answers come from crypton's own detection,+ * the same that AES-NI, PCLMUL and the s2n-bignum variants use.+ */+#define MLK_CONFIG_CUSTOM_CAPABILITY_FUNC+#ifndef __ASSEMBLER__+#include "src/sys.h"+#include "crypton_cpu.h"++static MLK_INLINE int mlk_sys_check_capability(mlk_sys_cap cap)+{+#if defined(__x86_64__)+    if (cap == MLK_SYS_CAP_X86_64_AVX2)+        return (crypton_x86_simd_features() & CRYPTON_X86_AVX2) != 0;+#elif defined(__aarch64__)+    /* NEON is not optional on AArch64; the Keccak instructions are. */+    if (cap == MLK_SYS_CAP_AARCH64_NEON)+        return 1;+    if (cap == MLK_SYS_CAP_AARCH64_SHA3)+        return (crypton_arm_features() & CRYPTON_ARM_SHA3) != 0;+#endif+    (void)cap;+    return 0;+}+#endif /* !__ASSEMBLER__ */+#endif++#endif /* CRYPTON_MLKEM_H */
+ cbits/mlkem/crypton_mlkem_asm.S view
@@ -0,0 +1,14 @@+/*+ * The assembly half, which is level-independent: it is included once, with+ * the shared directives kept, and covers all three parameter sets.  See the+ * comment at the top of mlkem_native_asm.S.+ *+ * Built only where crypton.cabal defines CRYPTON_MLKEM_NATIVE_BACKEND; on+ * every other architecture this file is not in asm-sources at all.+ */+#include "crypton_mlkem.h"++#define MLK_CONFIG_MULTILEVEL_WITH_SHARED 1+#define MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS+#define MLK_CONFIG_PARAMETER_SET 768+#include "mlkem_native_asm.S"
+ cbits/mlkem/import.sh view
@@ -0,0 +1,42 @@+#!/bin/sh+# Re-import the vendored parts of the PQ Code Package's mlkem-native.+#+# The files are kept unmodified.  Everything crypton decides -- which+# parameter sets exist, what the symbols are called, that there is no+# randomised API -- is decided in cbits/mlkem/crypton_mlkem.c and in+# crypton.cabal, not by editing anything here.  Run this from cbits/mlkem:+#+#     ./import.sh [tag-or-commit]+#+# and commit the result together with the COMMIT line it writes, so that the+# tree always says which upstream revision it holds.+set -eu++REPO=https://github.com/pq-code-package/mlkem-native+REV=${1:-v2.0.0}+HERE=$(cd "$(dirname "$0")" && pwd)+TMP=$(mktemp -d)+trap 'rm -rf "$TMP"' EXIT++git clone -q "$REPO" "$TMP/u"+git -C "$TMP/u" checkout -q "$REV"++rm -rf "$HERE/src"+cp "$TMP/u/mlkem/mlkem_native.c" "$HERE/"+cp "$TMP/u/mlkem/mlkem_native.h" "$HERE/"+cp "$TMP/u/mlkem/mlkem_native_asm.S" "$HERE/"+cp "$TMP/u/mlkem/mlkem_native_config.h" "$HERE/"+cp -R "$TMP/u/mlkem/src" "$HERE/src"+cp "$TMP/u/LICENSE" "$HERE/LICENSE"++# The backends for architectures crypton does not build for.  Every+# reference to them is behind an MLK_SYS_ guard that cannot be true on the+# two it does, so dropping them changes no build and keeps a few dozen files+# of unreachable assembly out of the release tarball.  If crypton ever+# wants one of them, delete its line here rather than patching anything.+rm -rf "$HERE/src/native/riscv64" "$HERE/src/native/ppc64le"+rm -rf "$HERE/src/fips202/native/armv81m"++git -C "$TMP/u" rev-parse HEAD > "$HERE/COMMIT"+git -C "$TMP/u" describe --tags --exact-match 2>/dev/null >> "$HERE/COMMIT" || true+echo "imported $(head -1 "$HERE/COMMIT")"
+ cbits/mlkem/mlkem_native.c view
@@ -0,0 +1,692 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mlkem-native repository.+ *          Do not modify it directly.+ */++/******************************************************************************+ *+ * Single compilation unit (SCU) for fixed-level build of mlkem-native+ *+ * This compilation unit bundles together all source files for a build+ * of mlkem-native for a fixed security level (MLKEM-512/768/1024).+ *+ * # API+ *+ * The API exposed by this file is described in mlkem_native.h.+ *+ * # Multi-level build+ *+ * If you want an SCU build of mlkem-native with support for multiple security+ * levels, you need to include this file multiple times, and set+ * MLK_CONFIG_MULTILEVEL_WITH_SHARED and MLK_CONFIG_MULTILEVEL_NO_SHARED+ * appropriately. This is exemplified in examples/monolithic_build_multilevel+ * and examples/monolithic_build_multilevel_native.+ *+ * # Configuration+ *+ * The following options from the mlkem-native configuration are relevant:+ *+ * - MLK_CONFIG_FIPS202_CUSTOM_HEADER+ *   Set this option if you use a custom FIPS202 implementation.+ *+ * - MLK_CONFIG_USE_NATIVE_BACKEND_ARITH+ *   Set this option if you want to include the native arithmetic backends+ *   in your build.+ *+ * - MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202+ *   Set this option if you want to include the native FIPS202 backends+ *   in your build.+ *+ * - MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS+ *   Set this option if you want to keep the directives defined in+ *   level-independent headers. This is needed for a multi-level build.+ */++/* If parts of the mlkem-native source tree are not used,+ * consider reducing this header via `unifdef`.+ *+ * Example:+ * ```bash+ * unifdef -UMLK_CONFIG_USE_NATIVE_BACKEND_ARITH mlkem_native.c+ * ```+ */++#include "src/common.h"++#include "src/compress.c"+#include "src/debug.c"+#include "src/indcpa.c"+#include "src/kem.c"+#include "src/poly.c"+#include "src/poly_k.c"+#include "src/sampling.c"+#include "src/verify.c"++#if !defined(MLK_CONFIG_FIPS202_CUSTOM_HEADER)+#include "src/fips202/fips202.c"+#include "src/fips202/fips202x4.c"+#include "src/fips202/keccakf1600.c"+#endif++#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_ARITH)+#if defined(MLK_SYS_AARCH64)+#include "src/native/aarch64/src/aarch64_zetas.c"+#include "src/native/aarch64/src/rej_uniform_table.c"+#endif+#if defined(MLK_SYS_X86_64)+#include "src/native/x86_64/src/compress_consts.c"+#include "src/native/x86_64/src/consts.c"+#include "src/native/x86_64/src/rej_uniform_table.c"+#endif+#if defined(MLK_SYS_RISCV64)+#include "src/native/riscv64/src/rv64v_debug.c"+#include "src/native/riscv64/src/rv64v_poly.c"+#endif+#if defined(MLK_SYS_PPC64LE)+#include "src/native/ppc64le/src/consts.c"+#endif+#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_ARITH */++#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202)+#if defined(MLK_SYS_AARCH64)+#include "src/fips202/native/aarch64/src/keccakf1600_round_constants.c"+#endif+#if defined(MLK_SYS_X86_64)+#include "src/fips202/native/x86_64/src/keccakf1600_constants.c"+#endif+#if defined(MLK_SYS_ARMV81M_MVE)+#include "src/fips202/native/armv81m/src/keccak_f1600_x4_mve.c"+#include "src/fips202/native/armv81m/src/keccakf1600_round_constants.c"+#endif+#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 */++/* Macro #undef's+ *+ * The following undefines macros from headers+ * included by the source files imported above.+ *+ * This is to allow building and linking multiple builds+ * of mlkem-native for varying parameter sets through concatenation+ * of this file, as if the files had been compiled separately.+ * If this is not relevant to you, you may remove the following.+ */++/*+ * Undefine macros from MLK_CONFIG_PARAMETER_SET-specific files+ */+/* mlkem/mlkem_native.h */+#undef MLKEM1024_BYTES+#undef MLKEM1024_CIPHERTEXTBYTES+#undef MLKEM1024_PUBLICKEYBYTES+#undef MLKEM1024_SECRETKEYBYTES+#undef MLKEM1024_SYMBYTES+#undef MLKEM512_BYTES+#undef MLKEM512_CIPHERTEXTBYTES+#undef MLKEM512_PUBLICKEYBYTES+#undef MLKEM512_SECRETKEYBYTES+#undef MLKEM512_SYMBYTES+#undef MLKEM768_BYTES+#undef MLKEM768_CIPHERTEXTBYTES+#undef MLKEM768_PUBLICKEYBYTES+#undef MLKEM768_SECRETKEYBYTES+#undef MLKEM768_SYMBYTES+#undef MLKEM_BYTES+#undef MLKEM_CIPHERTEXTBYTES+#undef MLKEM_CIPHERTEXTBYTES_+#undef MLKEM_PUBLICKEYBYTES+#undef MLKEM_PUBLICKEYBYTES_+#undef MLKEM_SECRETKEYBYTES+#undef MLKEM_SECRETKEYBYTES_+#undef MLKEM_SYMBYTES+#undef MLK_API_CONCAT+#undef MLK_API_CONCAT_+#undef MLK_API_CONCAT_UNDERSCORE+#undef MLK_API_MUST_CHECK_RETURN_VALUE+#undef MLK_API_NAMESPACE+#undef MLK_API_NAMESPACE_PREFIX+#undef MLK_API_QUALIFIER+#undef MLK_ERR_FAIL+#undef MLK_ERR_INVALID_PK+#undef MLK_ERR_INVALID_SK+#undef MLK_ERR_OUT_OF_MEMORY+#undef MLK_ERR_PCT_FAIL+#undef MLK_ERR_RNG_FAIL+#undef MLK_H+#undef MLK_MAX3_+#undef MLK_TOTAL_ALLOC_1024+#undef MLK_TOTAL_ALLOC_1024_DECAPS+#undef MLK_TOTAL_ALLOC_1024_ENCAPS+#undef MLK_TOTAL_ALLOC_1024_KEYPAIR+#undef MLK_TOTAL_ALLOC_1024_KEYPAIR_NO_PCT+#undef MLK_TOTAL_ALLOC_1024_KEYPAIR_PCT+#undef MLK_TOTAL_ALLOC_512+#undef MLK_TOTAL_ALLOC_512_DECAPS+#undef MLK_TOTAL_ALLOC_512_ENCAPS+#undef MLK_TOTAL_ALLOC_512_KEYPAIR+#undef MLK_TOTAL_ALLOC_512_KEYPAIR_NO_PCT+#undef MLK_TOTAL_ALLOC_512_KEYPAIR_PCT+#undef MLK_TOTAL_ALLOC_768+#undef MLK_TOTAL_ALLOC_768_DECAPS+#undef MLK_TOTAL_ALLOC_768_ENCAPS+#undef MLK_TOTAL_ALLOC_768_KEYPAIR+#undef MLK_TOTAL_ALLOC_768_KEYPAIR_NO_PCT+#undef MLK_TOTAL_ALLOC_768_KEYPAIR_PCT+/* mlkem/src/common.h */+#undef MLK_ADD_PARAM_SET+#undef MLK_ALLOC+#undef MLK_APPLY+#undef MLK_ASM_FN_SIZE+#undef MLK_ASM_FN_SYMBOL+#undef MLK_ASM_NAMESPACE+#undef MLK_BUILD_INTERNAL+#undef MLK_COMMON_H+#undef MLK_CONCAT+#undef MLK_CONCAT_+#undef MLK_EMPTY_CU+#undef MLK_ERR_FAIL+#undef MLK_ERR_INVALID_PK+#undef MLK_ERR_INVALID_SK+#undef MLK_ERR_OUT_OF_MEMORY+#undef MLK_ERR_PCT_FAIL+#undef MLK_ERR_RNG_FAIL+#undef MLK_EXTERNAL_API+#undef MLK_FIPS202X4_HEADER_FILE+#undef MLK_FIPS202_HEADER_FILE+#undef MLK_FREE+#undef MLK_INTERNAL_API+#undef MLK_INTERNAL_DATA_DECLARATION+#undef MLK_INTERNAL_DATA_DEFINITION+#undef MLK_NAMESPACE+#undef MLK_NAMESPACE_K+#undef MLK_NAMESPACE_PREFIX+#undef MLK_NAMESPACE_PREFIX_K+#undef mlk_memcpy+#undef mlk_memset+/* mlkem/src/indcpa.h */+#undef MLK_INDCPA_H+#undef mlk_gen_matrix+#undef mlk_indcpa_dec+#undef mlk_indcpa_enc+#undef mlk_indcpa_keypair_derand+/* mlkem/src/kem.h */+#undef MLK_KEM_H+#undef mlk_kem_check_pk+#undef mlk_kem_check_sk+#undef mlk_kem_dec+#undef mlk_kem_enc+#undef mlk_kem_enc_derand+#undef mlk_kem_keypair+#undef mlk_kem_keypair_derand+/* mlkem/src/params.h */+#undef MLKEM_DU+#undef MLKEM_DV+#undef MLKEM_ETA1+#undef MLKEM_ETA2+#undef MLKEM_INDCCA_CIPHERTEXTBYTES+#undef MLKEM_INDCCA_PUBLICKEYBYTES+#undef MLKEM_INDCCA_SECRETKEYBYTES+#undef MLKEM_INDCPA_BYTES+#undef MLKEM_INDCPA_MSGBYTES+#undef MLKEM_INDCPA_PUBLICKEYBYTES+#undef MLKEM_INDCPA_SECRETKEYBYTES+#undef MLKEM_K+#undef MLKEM_N+#undef MLKEM_POLYBYTES+#undef MLKEM_POLYCOMPRESSEDBYTES_D10+#undef MLKEM_POLYCOMPRESSEDBYTES_D11+#undef MLKEM_POLYCOMPRESSEDBYTES_D4+#undef MLKEM_POLYCOMPRESSEDBYTES_D5+#undef MLKEM_POLYCOMPRESSEDBYTES_DU+#undef MLKEM_POLYCOMPRESSEDBYTES_DV+#undef MLKEM_POLYVECBYTES+#undef MLKEM_POLYVECCOMPRESSEDBYTES_DU+#undef MLKEM_Q+#undef MLKEM_Q_HALF+#undef MLKEM_SSBYTES+#undef MLKEM_SYMBYTES+#undef MLKEM_UINT12_LIMIT+#undef MLK_PARAMS_H+/* mlkem/src/poly_k.h */+#undef MLK_POLY_K_H+#undef mlk_poly_compress_du+#undef mlk_poly_compress_dv+#undef mlk_poly_decompress_du+#undef mlk_poly_decompress_dv+#undef mlk_poly_getnoise_eta1122_4x+#undef mlk_poly_getnoise_eta1_4x+#undef mlk_poly_getnoise_eta2+#undef mlk_poly_getnoise_eta2_4x+#undef mlk_polymat+#undef mlk_polyvec+#undef mlk_polyvec_add+#undef mlk_polyvec_basemul_acc_montgomery_cached+#undef mlk_polyvec_compress_du+#undef mlk_polyvec_decompress_du+#undef mlk_polyvec_frombytes+#undef mlk_polyvec_invntt_tomont+#undef mlk_polyvec_mulcache+#undef mlk_polyvec_mulcache_compute+#undef mlk_polyvec_ntt+#undef mlk_polyvec_reduce+#undef mlk_polyvec_tobytes+#undef mlk_polyvec_tomont++#if !defined(MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS)+/*+ * Undefine macros from MLK_CONFIG_PARAMETER_SET-generic files+ */+/* mlkem/src/compress.h */+#undef MLK_COMPRESS_H+#undef mlk_poly_compress_d10+#undef mlk_poly_compress_d11+#undef mlk_poly_compress_d4+#undef mlk_poly_compress_d5+#undef mlk_poly_decompress_d10+#undef mlk_poly_decompress_d11+#undef mlk_poly_decompress_d4+#undef mlk_poly_decompress_d5+#undef mlk_poly_frombytes+#undef mlk_poly_frommsg+#undef mlk_poly_tobytes+#undef mlk_poly_tomsg+/* mlkem/src/context.h */+#undef MLK_CONTEXT_H+#undef MLK_CONTEXT_PARAMETERS_0+#undef MLK_CONTEXT_PARAMETERS_1+#undef MLK_CONTEXT_PARAMETERS_2+#undef MLK_CONTEXT_PARAMETERS_3+#undef MLK_CONTEXT_PARAMETERS_4+#undef MLK_CONTEXT_UNUSED+/* mlkem/src/debug.h */+#undef MLK_DEBUG_H+#undef mlk_assert+#undef mlk_assert_abs_bound+#undef mlk_assert_abs_bound_2d+#undef mlk_assert_bound+#undef mlk_assert_bound_2d+#undef mlk_debug_check_assert+#undef mlk_debug_check_bounds+/* mlkem/src/poly.h */+#undef MLK_INVNTT_BOUND+#undef MLK_NTT_BOUND+#undef MLK_POLY_H+#undef mlk_poly_add+#undef mlk_poly_invntt_tomont+#undef mlk_poly_mulcache_compute+#undef mlk_poly_ntt+#undef mlk_poly_reduce+#undef mlk_poly_sub+#undef mlk_poly_tomont+/* mlkem/src/randombytes.h */+#undef MLK_RANDOMBYTES_H+/* mlkem/src/sampling.h */+#undef MLK_SAMPLING_H+#undef mlk_poly_cbd2+#undef mlk_poly_cbd3+#undef mlk_poly_rej_uniform+#undef mlk_poly_rej_uniform_x4+/* mlkem/src/symmetric.h */+#undef MLK_SYMMETRIC_H+#undef MLK_XOF_RATE+#undef mlk_hash_g+#undef mlk_hash_h+#undef mlk_hash_j+#undef mlk_prf_eta+#undef mlk_prf_eta1+#undef mlk_prf_eta1_x4+#undef mlk_prf_eta2+#undef mlk_xof_absorb+#undef mlk_xof_ctx+#undef mlk_xof_init+#undef mlk_xof_release+#undef mlk_xof_squeezeblocks+#undef mlk_xof_x4_absorb+#undef mlk_xof_x4_ctx+#undef mlk_xof_x4_init+#undef mlk_xof_x4_release+#undef mlk_xof_x4_squeezeblocks+/* mlkem/src/sys.h */+#undef MLK_ALIGN+#undef MLK_ALIGN_UP+#undef MLK_ALWAYS_INLINE+#undef MLK_CET_ENDBR+#undef MLK_CT_TESTING_DECLASSIFY+#undef MLK_CT_TESTING_SECRET+#undef MLK_DEFAULT_ALIGN+#undef MLK_HAVE_INLINE_ASM+#undef MLK_INLINE+#undef MLK_MUST_CHECK_RETURN_VALUE+#undef MLK_NOINLINE+#undef MLK_RESTRICT+#undef MLK_STATIC_TESTABLE+#undef MLK_SYSV_ABI+#undef MLK_SYSV_ABI_SUPPORTED+#undef MLK_SYS_AARCH64+#undef MLK_SYS_AARCH64_EB+#undef MLK_SYS_AARCH64_NEON+#undef MLK_SYS_APPLE+#undef MLK_SYS_ARMV81M_MVE+#undef MLK_SYS_BIG_ENDIAN+#undef MLK_SYS_H+#undef MLK_SYS_LINUX+#undef MLK_SYS_LITTLE_ENDIAN+#undef MLK_SYS_PPC64LE+#undef MLK_SYS_RISCV32+#undef MLK_SYS_RISCV64+#undef MLK_SYS_RISCV64_RVV+#undef MLK_SYS_WINDOWS+#undef MLK_SYS_X86_64+#undef MLK_SYS_X86_64_AVX2+/* mlkem/src/verify.h */+#undef MLK_USE_ASM_VALUE_BARRIER+#undef MLK_VERIFY_H+#undef mlk_ct_opt_blocker_u64+/* mlkem/src/cbmc.h */+#undef MLK_CBMC_H+#undef __contract__+#undef __loop__++#if !defined(MLK_CONFIG_FIPS202_CUSTOM_HEADER)+/*+ * Undefine macros from FIPS-202 files+ */+/* mlkem/src/fips202/fips202.h */+#undef FIPS202_X4_DEFAULT_IMPLEMENTATION+#undef MLK_FIPS202_FIPS202_H+#undef SHA3_256_HASHBYTES+#undef SHA3_256_RATE+#undef SHA3_384_RATE+#undef SHA3_512_HASHBYTES+#undef SHA3_512_RATE+#undef SHAKE128_RATE+#undef SHAKE256_RATE+#undef mlk_sha3_256+#undef mlk_sha3_512+#undef mlk_shake128_absorb_once+#undef mlk_shake128_init+#undef mlk_shake128_release+#undef mlk_shake128_squeezeblocks+#undef mlk_shake256+/* mlkem/src/fips202/fips202x4.h */+#undef MLK_FIPS202_FIPS202X4_H+#undef mlk_shake128x4_absorb_once+#undef mlk_shake128x4_init+#undef mlk_shake128x4_release+#undef mlk_shake128x4_squeezeblocks+#undef mlk_shake256x4+/* mlkem/src/fips202/keccakf1600.h */+#undef MLK_FIPS202_KECCAKF1600_H+#undef MLK_KECCAK_LANES+#undef MLK_KECCAK_WAY+#undef mlk_keccakf1600_extract_bytes+#undef mlk_keccakf1600_permute+#undef mlk_keccakf1600_xor_bytes+#undef mlk_keccakf1600x4_extract_bytes+#undef mlk_keccakf1600x4_permute+#undef mlk_keccakf1600x4_xor_bytes+#endif /* !MLK_CONFIG_FIPS202_CUSTOM_HEADER */++#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202)+/* mlkem/src/fips202/native/api.h */+#undef MLK_FIPS202_NATIVE_API_H+#undef MLK_NATIVE_FUNC_FALLBACK+#undef MLK_NATIVE_FUNC_SUCCESS+/* mlkem/src/fips202/native/auto.h */+#undef MLK_FIPS202_NATIVE_AUTO_H+#if defined(MLK_SYS_AARCH64)+/*+ * Undefine macros from native code (FIPS202, AArch64)+ */+/* mlkem/src/fips202/native/aarch64/auto.h */+#undef MLK_FIPS202_NATIVE_AARCH64_AUTO_H+/* mlkem/src/fips202/native/aarch64/src/fips202_native_aarch64.h */+#undef MLK_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H+#undef mlk_keccak_f1600_x1_scalar_aarch64_asm+#undef mlk_keccak_f1600_x1_v84a_aarch64_asm+#undef mlk_keccak_f1600_x2_v84a_aarch64_asm+#undef mlk_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm+#undef mlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm+#undef mlk_keccakf1600_round_constants+/* mlkem/src/fips202/native/aarch64/x1_scalar.h */+#undef MLK_FIPS202_AARCH64_NEED_X1_SCALAR+#undef MLK_FIPS202_NATIVE_AARCH64_X1_SCALAR_H+#undef MLK_USE_NATIVE_FIPS202_X1+/* mlkem/src/fips202/native/aarch64/x1_v84a.h */+#undef MLK_FIPS202_AARCH64_NEED_X1_V84A+#undef MLK_FIPS202_NATIVE_AARCH64_X1_V84A_H+#undef MLK_USE_NATIVE_FIPS202_X1+/* mlkem/src/fips202/native/aarch64/x2_v84a.h */+#undef MLK_FIPS202_AARCH64_NEED_X2_V84A+#undef MLK_FIPS202_NATIVE_AARCH64_X2_V84A_H+#undef MLK_USE_NATIVE_FIPS202_X4+/* mlkem/src/fips202/native/aarch64/x4_v8a_scalar.h */+#undef MLK_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID+#undef MLK_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H+#undef MLK_USE_NATIVE_FIPS202_X4+/* mlkem/src/fips202/native/aarch64/x4_v8a_v84a_scalar.h */+#undef MLK_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID+#undef MLK_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H+#undef MLK_USE_NATIVE_FIPS202_X4+#endif /* MLK_SYS_AARCH64 */+#if defined(MLK_SYS_X86_64)+/*+ * Undefine macros from native code (FIPS202, x86_64)+ */+/* mlkem/src/fips202/native/x86_64/keccak_f1600_x4_avx2.h */+#undef MLK_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H+#undef MLK_FIPS202_X86_64_NEED_X4_AVX2+#undef MLK_USE_NATIVE_FIPS202_X4+/* mlkem/src/fips202/native/x86_64/src/fips202_native_x86_64.h */+#undef MLK_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H+#undef mlk_keccak_f1600_x4_avx2_asm+#undef mlk_keccak_rho56+#undef mlk_keccak_rho8+#undef mlk_keccakf1600_round_constants+#endif /* MLK_SYS_X86_64 */+#if defined(MLK_SYS_ARMV81M_MVE)+/*+ * Undefine macros from native code (FIPS202, Armv8.1-M)+ */+/* mlkem/src/fips202/native/armv81m/mve.h */+#undef MLK_FIPS202_ARMV81M_NEED_X4+#undef MLK_FIPS202_NATIVE_ARMV81M+#undef MLK_FIPS202_NATIVE_ARMV81M_MVE_H+#undef MLK_USE_NATIVE_FIPS202_X4+#undef MLK_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES+#undef MLK_USE_NATIVE_FIPS202_X4_XOR_BYTES+#undef mlk_keccak_f1600_x4_native_impl+/* mlkem/src/fips202/native/armv81m/src/fips202_native_armv81m.h */+#undef MLK_FIPS202_NATIVE_ARMV81M_SRC_FIPS202_NATIVE_ARMV81M_H+#undef mlk_keccak_f1600_x4_mve_asm+#undef mlk_keccak_f1600_x4_state_extract_bytes_asm+#undef mlk_keccak_f1600_x4_state_xor_bytes_asm+#undef mlk_keccakf1600_round_constants+#endif /* MLK_SYS_ARMV81M_MVE */+#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 */+#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_ARITH)+/* mlkem/src/native/api.h */+#undef MLK_INVNTT_BOUND+#undef MLK_NATIVE_API_H+#undef MLK_NATIVE_FUNC_FALLBACK+#undef MLK_NATIVE_FUNC_SUCCESS+#undef MLK_NTT_BOUND+/* mlkem/src/native/meta.h */+#undef MLK_NATIVE_META_H+#if defined(MLK_SYS_AARCH64)+/*+ * Undefine macros from native code (Arith, AArch64)+ */+/* mlkem/src/native/aarch64/meta.h */+#undef MLK_ARITH_BACKEND_AARCH64+#undef MLK_NATIVE_AARCH64_META_H+#undef MLK_USE_NATIVE_INTT+#undef MLK_USE_NATIVE_NTT+#undef MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED+#undef MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE+#undef MLK_USE_NATIVE_POLY_REDUCE+#undef MLK_USE_NATIVE_POLY_TOBYTES+#undef MLK_USE_NATIVE_POLY_TOMONT+#undef MLK_USE_NATIVE_REJ_UNIFORM+/* mlkem/src/native/aarch64/src/arith_native_aarch64.h */+#undef MLK_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H+#undef mlk_aarch64_invntt_zetas_layer12345+#undef mlk_aarch64_invntt_zetas_layer67+#undef mlk_aarch64_ntt_zetas_layer12345+#undef mlk_aarch64_ntt_zetas_layer67+#undef mlk_aarch64_zetas_mulcache_native+#undef mlk_aarch64_zetas_mulcache_twisted_native+#undef mlk_intt_aarch64_asm+#undef mlk_ntt_aarch64_asm+#undef mlk_poly_mulcache_compute_aarch64_asm+#undef mlk_poly_reduce_aarch64_asm+#undef mlk_poly_tobytes_aarch64_asm+#undef mlk_poly_tomont_aarch64_asm+#undef mlk_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm+#undef mlk_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm+#undef mlk_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm+#undef mlk_rej_uniform_aarch64_asm+#undef mlk_rej_uniform_table+#endif /* MLK_SYS_AARCH64 */+#if defined(MLK_SYS_X86_64)+/*+ * Undefine macros from native code (Arith, X86_64)+ */+/* mlkem/src/native/x86_64/meta.h */+#undef MLK_ARITH_BACKEND_X86_64_DEFAULT+#undef MLK_NATIVE_X86_64_META_H+#undef MLK_USE_NATIVE_INTT+#undef MLK_USE_NATIVE_NTT+#undef MLK_USE_NATIVE_NTT_CUSTOM_ORDER+#undef MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED+#undef MLK_USE_NATIVE_POLY_COMPRESS_D10+#undef MLK_USE_NATIVE_POLY_COMPRESS_D11+#undef MLK_USE_NATIVE_POLY_COMPRESS_D4+#undef MLK_USE_NATIVE_POLY_COMPRESS_D5+#undef MLK_USE_NATIVE_POLY_DECOMPRESS_D10+#undef MLK_USE_NATIVE_POLY_DECOMPRESS_D11+#undef MLK_USE_NATIVE_POLY_DECOMPRESS_D4+#undef MLK_USE_NATIVE_POLY_DECOMPRESS_D5+#undef MLK_USE_NATIVE_POLY_FROMBYTES+#undef MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE+#undef MLK_USE_NATIVE_POLY_REDUCE+#undef MLK_USE_NATIVE_POLY_TOBYTES+#undef MLK_USE_NATIVE_POLY_TOMONT+#undef MLK_USE_NATIVE_REJ_UNIFORM+/* mlkem/src/native/x86_64/src/arith_native_x86_64.h */+#undef MLK_AVX2_REJ_UNIFORM_BUFLEN+#undef MLK_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H+#undef mlk_invntt_avx2_asm+#undef mlk_ntt_avx2_asm+#undef mlk_nttfrombytes_avx2_asm+#undef mlk_ntttobytes_avx2_asm+#undef mlk_nttunpack_avx2_asm+#undef mlk_poly_compress_d10_avx2_asm+#undef mlk_poly_compress_d11_avx2_asm+#undef mlk_poly_compress_d4_avx2_asm+#undef mlk_poly_compress_d5_avx2_asm+#undef mlk_poly_decompress_d10_avx2_asm+#undef mlk_poly_decompress_d11_avx2_asm+#undef mlk_poly_decompress_d4_avx2_asm+#undef mlk_poly_decompress_d5_avx2_asm+#undef mlk_poly_mulcache_compute_avx2_asm+#undef mlk_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm+#undef mlk_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm+#undef mlk_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm+#undef mlk_reduce_avx2_asm+#undef mlk_rej_uniform_avx2_asm+#undef mlk_rej_uniform_table+#undef mlk_tomont_avx2_asm+/* mlkem/src/native/x86_64/src/compress_consts.h */+#undef MLK_NATIVE_X86_64_SRC_COMPRESS_CONSTS_H+#undef mlk_compress_d10_data+#undef mlk_compress_d11_data+#undef mlk_compress_d4_data+#undef mlk_compress_d5_data+#undef mlk_decompress_d10_data+#undef mlk_decompress_d11_data+#undef mlk_decompress_d4_data+#undef mlk_decompress_d5_data+/* mlkem/src/native/x86_64/src/consts.h */+#undef MLK_AVX2_BACKEND_DATA_OFFSET_MULCACHE_TWIDDLES+#undef MLK_AVX2_BACKEND_DATA_OFFSET_REVIDXB+#undef MLK_AVX2_BACKEND_DATA_OFFSET_REVIDXD+#undef MLK_AVX2_BACKEND_DATA_OFFSET_ZETAS_EXP+#undef MLK_NATIVE_X86_64_SRC_CONSTS_H+#undef mlk_qdata+#endif /* MLK_SYS_X86_64 */+#if defined(MLK_SYS_RISCV64)+/*+ * Undefine macros from native code (Arith, RISC-V 64)+ */+/* mlkem/src/native/riscv64/meta.h */+#undef MLK_ARITH_BACKEND_RISCV64+#undef MLK_NATIVE_RISCV64_META_H+#undef MLK_USE_NATIVE_INTT+#undef MLK_USE_NATIVE_NTT+#undef MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED+#undef MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE+#undef MLK_USE_NATIVE_POLY_REDUCE+#undef MLK_USE_NATIVE_POLY_TOMONT+#undef MLK_USE_NATIVE_REJ_UNIFORM+/* mlkem/src/native/riscv64/src/arith_native_riscv64.h */+#undef MLK_NATIVE_RISCV64_SRC_ARITH_NATIVE_RISCV64_H+#undef mlk_rv64v_poly_add+#undef mlk_rv64v_poly_basemul_mont_add_k2+#undef mlk_rv64v_poly_basemul_mont_add_k3+#undef mlk_rv64v_poly_basemul_mont_add_k4+#undef mlk_rv64v_poly_invntt_tomont+#undef mlk_rv64v_poly_ntt+#undef mlk_rv64v_poly_reduce+#undef mlk_rv64v_poly_sub+#undef mlk_rv64v_poly_tomont+#undef mlk_rv64v_rej_uniform+/* mlkem/src/native/riscv64/src/rv64v_debug.h */+#undef MLK_NATIVE_RISCV64_SRC_RV64V_DEBUG_H+#undef mlk_assert_abs_bound_int16m1+#undef mlk_assert_abs_bound_int16m2+#undef mlk_assert_bound_int16m1+#undef mlk_assert_bound_int16m2+#undef mlk_debug_check_bounds_int16m1+#undef mlk_debug_check_bounds_int16m2+#endif /* MLK_SYS_RISCV64 */+#if defined(MLK_SYS_PPC64LE)+/*+ * Undefine macros from native code (Arith, PPC64LE)+ */+/* mlkem/src/native/ppc64le/meta.h */+#undef MLK_ARITH_BACKEND_NAME+#undef MLK_ARITH_BACKEND_PPC64LE_DEFAULT+#undef MLK_NATIVE_PPC64LE_META_H+#undef MLK_USE_NATIVE_INTT+#undef MLK_USE_NATIVE_NTT+#undef MLK_USE_NATIVE_POLY_REDUCE+#undef MLK_USE_NATIVE_POLY_TOMONT+/* mlkem/src/native/ppc64le/src/arith_native_ppc64le.h */+#undef MLK_NATIVE_PPC64LE_SRC_ARITH_NATIVE_PPC64LE_H+#undef mlk_intt_ppc_asm+#undef mlk_ntt_ppc_asm+#undef mlk_poly_tomont_ppc_asm+#undef mlk_reduce_ppc_asm+/* mlkem/src/native/ppc64le/src/consts.h */+#undef MLK_NATIVE_PPC64LE_SRC_CONSTS_H+#undef MLK_PPC_C20159_OFFSET+#undef MLK_PPC_NQ_OFFSET+#undef MLK_PPC_N_INV_OFFSET+#undef MLK_PPC_N_INV_TW_OFFSET+#undef MLK_PPC_Q_OFFSET+#undef MLK_PPC_TOMONT_OFFSET+#undef MLK_PPC_TOMONT_TW_OFFSET+#undef MLK_PPC_ZETA_INTT_OFFSET+#undef MLK_PPC_ZETA_INTT_TW_OFFSET+#undef MLK_PPC_ZETA_NTT_OFFSET+#undef MLK_PPC_ZETA_NTT_TW_OFFSET+#undef mlk_ppc_qdata+#endif /* MLK_SYS_PPC64LE */+#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_ARITH */+#endif /* !MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS */
+ cbits/mlkem/mlkem_native.h view
@@ -0,0 +1,464 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS140_3_IG]+ *   Implementation Guidance for FIPS 140-3 and the Cryptographic Module+ *   Validation Program+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/projects/cryptographic-module-validation-program/fips-140-3-ig-announcements+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ */++#ifndef MLK_H+#define MLK_H++/*+ * Public API for mlkem-native.+ *+ * This header defines the public API of a single build of mlkem-native.+ *+ * Make sure the configuration file is in the include path+ * (this is "mlkem_native_config.h" by default, or MLK_CONFIG_FILE if defined).+ *+ * # API conventions+ *+ * Conventions shared by all functions below (return values, pointer validity,+ * output buffers on error) are documented in API-CONVENTIONS.md.+ *+ * # Multi-level builds+ *+ * This header specifies a build of mlkem-native for a fixed security level.+ * If you need multiple security levels, leave the security level unspecified+ * in the configuration file and include this header multiple times, setting+ * MLK_CONFIG_PARAMETER_SET accordingly for each, and #undef'ing the MLK_H+ * guard to allow multiple inclusions.+ *+ * In this case, the configuration file must also set+ * MLK_CONFIG_MULTILEVEL_BUILD. Without it, the parameter set is not appended+ * to the namespace prefix and all inclusions declare the same symbol names.+ */++/******************************* Key sizes ************************************/++/* Sizes of cryptographic material, per parameter set */+/* See mlkem/src/params.h for the arithmetic expressions giving rise to these */+/* check-magic: off */+#define MLKEM512_SECRETKEYBYTES 1632+#define MLKEM512_PUBLICKEYBYTES 800+#define MLKEM512_CIPHERTEXTBYTES 768++#define MLKEM768_SECRETKEYBYTES 2400+#define MLKEM768_PUBLICKEYBYTES 1184+#define MLKEM768_CIPHERTEXTBYTES 1088++#define MLKEM1024_SECRETKEYBYTES 3168+#define MLKEM1024_PUBLICKEYBYTES 1568+#define MLKEM1024_CIPHERTEXTBYTES 1568+/* check-magic: on */++/* Size of randomness coins in bytes (level-independent) */+#define MLKEM_SYMBYTES 32+#define MLKEM512_SYMBYTES MLKEM_SYMBYTES+#define MLKEM768_SYMBYTES MLKEM_SYMBYTES+#define MLKEM1024_SYMBYTES MLKEM_SYMBYTES+/* Size of shared secret in bytes (level-independent) */+#define MLKEM_BYTES 32+#define MLKEM512_BYTES MLKEM_BYTES+#define MLKEM768_BYTES MLKEM_BYTES+#define MLKEM1024_BYTES MLKEM_BYTES++/* Sizes of cryptographic material, as a function of LVL=512,768,1024 */+#define MLKEM_SECRETKEYBYTES_(LVL) MLKEM##LVL##_SECRETKEYBYTES+#define MLKEM_PUBLICKEYBYTES_(LVL) MLKEM##LVL##_PUBLICKEYBYTES+#define MLKEM_CIPHERTEXTBYTES_(LVL) MLKEM##LVL##_CIPHERTEXTBYTES+#define MLKEM_SECRETKEYBYTES(LVL) MLKEM_SECRETKEYBYTES_(LVL)+#define MLKEM_PUBLICKEYBYTES(LVL) MLKEM_PUBLICKEYBYTES_(LVL)+#define MLKEM_CIPHERTEXTBYTES(LVL) MLKEM_CIPHERTEXTBYTES_(LVL)++/****************************** Error codes ***********************************/++/* Generic failure condition. Currently not returned by any function;+ * reserved for failures that no more specific code covers. */+#define MLK_ERR_FAIL (-1)+/* An allocation failed. This can only happen if MLK_CONFIG_CUSTOM_ALLOC_FREE+ * is defined and the provided MLK_CUSTOM_ALLOC can fail. */+#define MLK_ERR_OUT_OF_MEMORY (-2)+/* An RNG failure occurred. Might be due to insufficient entropy or+ * system misconfiguration. */+#define MLK_ERR_RNG_FAIL (-3)+/* Public key validation failed: the @[FIPS203, Section 7.2, 'modulus check']+ * found a coefficient outside [0,q-1]. Returned by check_pk and by the+ * encapsulation API. */+#define MLK_ERR_INVALID_PK (-4)+/* Secret key validation failed: the @[FIPS203, Section 7.3, 'hash check']+ * found the embedded public key hash inconsistent. Returned by check_sk and+ * by the decapsulation API. */+#define MLK_ERR_INVALID_SK (-5)+/* The 'Pairwise Consistency Test' @[FIPS140_3_IG, p.87] and+ * @[FIPS203, Section 7.1, Pairwise Consistency] failed. Only possible when+ * MLK_CONFIG_KEYGEN_PCT is enabled; signals that the freshly generated key+ * pair failed its encaps/decaps self-test. */+#define MLK_ERR_PCT_FAIL (-6)++/********************* Namespacing and Qualifiers *****************************/++#define MLK_API_CONCAT_(x, y) x##y+#define MLK_API_CONCAT(x, y) MLK_API_CONCAT_(x, y)+#define MLK_API_CONCAT_UNDERSCORE(x, y) MLK_API_CONCAT(MLK_API_CONCAT(x, _), y)++/* You need to make sure the config file is in the include path. */+#if defined(MLK_CONFIG_FILE)+#include MLK_CONFIG_FILE+#else+#include "mlkem_native_config.h"+#endif++/* Namespace prefix for the public API symbols. For multi-level builds, the+ * parameter set is appended to disambiguate the security levels. */+#if defined(MLK_CONFIG_MULTILEVEL_BUILD)+#define MLK_API_NAMESPACE_PREFIX \+  MLK_API_CONCAT(MLK_CONFIG_NAMESPACE_PREFIX, MLK_CONFIG_PARAMETER_SET)+#else+#define MLK_API_NAMESPACE_PREFIX MLK_CONFIG_NAMESPACE_PREFIX+#endif++#define MLK_API_NAMESPACE(sym) \+  MLK_API_CONCAT_UNDERSCORE(MLK_API_NAMESPACE_PREFIX, sym)++#if defined(__GNUC__) || defined(__clang__)+#define MLK_API_MUST_CHECK_RETURN_VALUE __attribute__((warn_unused_result))+#else+#define MLK_API_MUST_CHECK_RETURN_VALUE+#endif++#if defined(MLK_CONFIG_EXTERNAL_API_QUALIFIER)+#define MLK_API_QUALIFIER MLK_CONFIG_EXTERNAL_API_QUALIFIER+#else+#define MLK_API_QUALIFIER+#endif++/****************************** Function API **********************************/++#if !defined(MLK_CONFIG_CONSTANTS_ONLY)++#include <stdint.h>++#ifdef __cplusplus+extern "C"+{+#endif++#if !defined(MLK_CONFIG_NO_KEYPAIR_API)+/**+ * Generate a public/private keypair for the ML-KEM key encapsulation mechanism.+ *+ * @spec{Implements @[FIPS203, Algorithm 16, ML-KEM.KeyGen_Internal].}+ *+ * @param[out] pk      Output public key, an array of+ *                     MLKEM{512,768,1024}_PUBLICKEYBYTES bytes.+ * @param[out] sk      Output private key, an array of+ *                     MLKEM{512,768,1024}_SECRETKEYBYTES bytes.+ * @param[in]  coins   Input randomness, an array of 2*MLKEM_SYMBYTES uniformly+ *                     random bytes.+ * @param      context Application context. Only present when+ *                     MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_PCT_FAIL      MLK_CONFIG_KEYGEN_PCT enabled and PCT failed.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ * @retval MLK_ERR_RNG_FAIL      MLK_CONFIG_KEYGEN_PCT enabled and random+ *                               number generation failed within the PCT.+ */+MLK_API_QUALIFIER+MLK_API_MUST_CHECK_RETURN_VALUE+int MLK_API_NAMESPACE(keypair_derand)(+    uint8_t pk[MLKEM_PUBLICKEYBYTES(MLK_CONFIG_PARAMETER_SET)],+    uint8_t sk[MLKEM_SECRETKEYBYTES(MLK_CONFIG_PARAMETER_SET)],+    const uint8_t coins[2 * MLKEM_SYMBYTES]+#ifdef MLK_CONFIG_CONTEXT_PARAMETER+    ,+    MLK_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);+++#if !defined(MLK_CONFIG_NO_RANDOMIZED_API)+/**+ * Generate a public/private keypair for the ML-KEM key encapsulation mechanism.+ *+ * @spec{Implements @[FIPS203, Algorithm 19, ML-KEM.KeyGen].}+ *+ * @param[out] pk      Output public key, an array of+ *                     MLKEM{512,768,1024}_PUBLICKEYBYTES bytes.+ * @param[out] sk      Output private key, an array of+ *                     MLKEM{512,768,1024}_SECRETKEYBYTES bytes.+ * @param      context Application context. Only present when+ *                     MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_PCT_FAIL      MLK_CONFIG_KEYGEN_PCT enabled and PCT failed.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ * @retval MLK_ERR_RNG_FAIL      Random number generation failed.+ */+MLK_API_QUALIFIER+MLK_API_MUST_CHECK_RETURN_VALUE+int MLK_API_NAMESPACE(keypair)(+    uint8_t pk[MLKEM_PUBLICKEYBYTES(MLK_CONFIG_PARAMETER_SET)],+    uint8_t sk[MLKEM_SECRETKEYBYTES(MLK_CONFIG_PARAMETER_SET)]+#ifdef MLK_CONFIG_CONTEXT_PARAMETER+    ,+    MLK_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);+#endif /* !MLK_CONFIG_NO_RANDOMIZED_API */+#endif /* !MLK_CONFIG_NO_KEYPAIR_API */++#if !defined(MLK_CONFIG_NO_ENCAPS_API)+/**+ * Generate ciphertext and shared secret for a given public key.+ *+ * @spec{Implements @[FIPS203, Algorithm 17, ML-KEM.Encaps_Internal].}+ *+ * @param[out] ct      Output ciphertext, an array of+ *                     MLKEM{512,768,1024}_CIPHERTEXTBYTES bytes.+ * @param[out] ss      Output shared secret, an array of MLKEM_BYTES bytes.+ * @param[in]  pk      Input public key, an array of+ *                     MLKEM{512,768,1024}_PUBLICKEYBYTES bytes.+ * @param[in]  coins   Input randomness, an array of MLKEM_SYMBYTES bytes.+ * @param      context Application context. Only present when+ *                     MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_INVALID_PK    The 'modulus check' @[FIPS203, Section 7.2]+ *                               for the public key failed.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ */+MLK_API_QUALIFIER+MLK_API_MUST_CHECK_RETURN_VALUE+int MLK_API_NAMESPACE(enc_derand)(+    uint8_t ct[MLKEM_CIPHERTEXTBYTES(MLK_CONFIG_PARAMETER_SET)],+    uint8_t ss[MLKEM_BYTES],+    const uint8_t pk[MLKEM_PUBLICKEYBYTES(MLK_CONFIG_PARAMETER_SET)],+    const uint8_t coins[MLKEM_SYMBYTES]+#ifdef MLK_CONFIG_CONTEXT_PARAMETER+    ,+    MLK_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);++#if !defined(MLK_CONFIG_NO_RANDOMIZED_API)+/**+ * Generate ciphertext and shared secret for a given public key.+ *+ * @spec{Implements @[FIPS203, Algorithm 20, ML-KEM.Encaps].}+ *+ * @param[out] ct      Output ciphertext, an array of+ *                     MLKEM{512,768,1024}_CIPHERTEXTBYTES bytes.+ * @param[out] ss      Output shared secret, an array of MLKEM_BYTES bytes.+ * @param[in]  pk      Input public key, an array of+ *                     MLKEM{512,768,1024}_PUBLICKEYBYTES bytes.+ * @param      context Application context. Only present when+ *                     MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_INVALID_PK    The 'modulus check' @[FIPS203, Section 7.2]+ *                               for the public key failed.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ * @retval MLK_ERR_RNG_FAIL      Random number generation failed.+ */+MLK_API_QUALIFIER+MLK_API_MUST_CHECK_RETURN_VALUE+int MLK_API_NAMESPACE(enc)(+    uint8_t ct[MLKEM_CIPHERTEXTBYTES(MLK_CONFIG_PARAMETER_SET)],+    uint8_t ss[MLKEM_BYTES],+    const uint8_t pk[MLKEM_PUBLICKEYBYTES(MLK_CONFIG_PARAMETER_SET)]+#ifdef MLK_CONFIG_CONTEXT_PARAMETER+    ,+    MLK_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);+#endif /* !MLK_CONFIG_NO_RANDOMIZED_API */+#endif /* !MLK_CONFIG_NO_ENCAPS_API */++#if !defined(MLK_CONFIG_NO_DECAPS_API)+/**+ * Generate shared secret for a given ciphertext and private key.+ *+ * @spec{Implements @[FIPS203, Algorithm 21, ML-KEM.Decaps].}+ *+ * @param[out] ss      Output shared secret, an array of MLKEM_BYTES bytes.+ * @param[in]  ct      Input ciphertext, an array of+ *                     MLKEM{512,768,1024}_CIPHERTEXTBYTES bytes.+ * @param[in]  sk      Input private key, an array of+ *                     MLKEM{512,768,1024}_SECRETKEYBYTES bytes.+ * @param      context Application context. Only present when+ *                     MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_INVALID_SK    The 'hash check' @[FIPS203, Section 7.3]+ *                               for the secret key failed.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ */+MLK_API_QUALIFIER+MLK_API_MUST_CHECK_RETURN_VALUE+int MLK_API_NAMESPACE(dec)(+    uint8_t ss[MLKEM_BYTES],+    const uint8_t ct[MLKEM_CIPHERTEXTBYTES(MLK_CONFIG_PARAMETER_SET)],+    const uint8_t sk[MLKEM_SECRETKEYBYTES(MLK_CONFIG_PARAMETER_SET)]+#ifdef MLK_CONFIG_CONTEXT_PARAMETER+    ,+    MLK_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);+#endif /* !MLK_CONFIG_NO_DECAPS_API */+++/**+ * Implements modulus check mandated by FIPS 203, i.e., ensures that+ * coefficients are in [0,q-1].+ *+ * @spec{Implements @[FIPS203, Section 7.2, 'modulus check'].}+ *+ * @param[in] pk      Input public key, an array of+ *                    MLKEM{512,768,1024}_PUBLICKEYBYTES bytes.+ * @param     context Application context. Only present when+ *                    MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                    MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_INVALID_PK    Modulus check failed.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ */+#if !defined(MLK_CONFIG_NO_ENCAPS_API)+MLK_API_QUALIFIER+MLK_API_MUST_CHECK_RETURN_VALUE+int MLK_API_NAMESPACE(check_pk)(+    const uint8_t pk[MLKEM_PUBLICKEYBYTES(MLK_CONFIG_PARAMETER_SET)]+#ifdef MLK_CONFIG_CONTEXT_PARAMETER+    ,+    MLK_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);+#endif /* !MLK_CONFIG_NO_ENCAPS_API */++/**+ * Implements public key hash check mandated by FIPS 203, i.e., ensures that+ * sk[768𝑘+32 ∶ 768𝑘+64] = H(pk) = H(sk[384𝑘 : 768𝑘+32]).+ *+ * @spec{Implements @[FIPS203, Section 7.3, 'hash check'].}+ *+ * @param[in] sk      Input private key, an array of+ *                    MLKEM{512,768,1024}_SECRETKEYBYTES bytes.+ * @param     context Application context. Only present when+ *                    MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                    MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_INVALID_SK    Public key hash check failed.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ */+#if !defined(MLK_CONFIG_NO_DECAPS_API)+MLK_API_QUALIFIER+MLK_API_MUST_CHECK_RETURN_VALUE+int MLK_API_NAMESPACE(check_sk)(+    const uint8_t sk[MLKEM_SECRETKEYBYTES(MLK_CONFIG_PARAMETER_SET)]+#ifdef MLK_CONFIG_CONTEXT_PARAMETER+    ,+    MLK_CONFIG_CONTEXT_PARAMETER_TYPE context+#endif+);+#endif /* !MLK_CONFIG_NO_DECAPS_API */++#ifdef __cplusplus+}+#endif++#undef MLK_API_NAMESPACE_PREFIX++#endif /* !MLK_CONFIG_CONSTANTS_ONLY */+++/***************************** Memory Usage **********************************/++/*+ * By default mlkem-native performs all memory allocations on the stack.+ * Alternatively, mlkem-native supports custom allocation of large structures+ * through the `MLK_CONFIG_CUSTOM_ALLOC_FREE` configuration option.+ * See mlkem_native_config.h for details.+ *+ * `MLK_TOTAL_ALLOC_{512,768,1024}_{KEYPAIR,ENCAPS,DECAPS}` indicates the+ * maximum (accumulative) allocation via MLK_ALLOC for each parameter set and+ * operation. Note that some stack allocation remains even when using custom+ * allocators, so these values are lower than total stack usage with the default+ * stack-only allocation.+ *+ * These constants may be used to implement custom allocations using a+ * fixed-sized buffer and a simple allocator (e.g., bump allocator).+ */+/* check-magic: off */+#define MLK_TOTAL_ALLOC_512_KEYPAIR_NO_PCT 5824+#define MLK_TOTAL_ALLOC_512_KEYPAIR_PCT 10048+#define MLK_TOTAL_ALLOC_512_ENCAPS 8384+#define MLK_TOTAL_ALLOC_512_DECAPS 9152+#define MLK_TOTAL_ALLOC_768_KEYPAIR_NO_PCT 10176+#define MLK_TOTAL_ALLOC_768_KEYPAIR_PCT 15552+#define MLK_TOTAL_ALLOC_768_ENCAPS 13248+#define MLK_TOTAL_ALLOC_768_DECAPS 14336+#define MLK_TOTAL_ALLOC_1024_KEYPAIR_NO_PCT 15552+#define MLK_TOTAL_ALLOC_1024_KEYPAIR_PCT 22400+#define MLK_TOTAL_ALLOC_1024_ENCAPS 19136+#define MLK_TOTAL_ALLOC_1024_DECAPS 20704+/* check-magic: on */++/*+ * MLK_TOTAL_ALLOC_*_KEYPAIR adapts based on MLK_CONFIG_KEYGEN_PCT.+ */+#if defined(MLK_CONFIG_KEYGEN_PCT)+#define MLK_TOTAL_ALLOC_512_KEYPAIR MLK_TOTAL_ALLOC_512_KEYPAIR_PCT+#define MLK_TOTAL_ALLOC_768_KEYPAIR MLK_TOTAL_ALLOC_768_KEYPAIR_PCT+#define MLK_TOTAL_ALLOC_1024_KEYPAIR MLK_TOTAL_ALLOC_1024_KEYPAIR_PCT+#else+#define MLK_TOTAL_ALLOC_512_KEYPAIR MLK_TOTAL_ALLOC_512_KEYPAIR_NO_PCT+#define MLK_TOTAL_ALLOC_768_KEYPAIR MLK_TOTAL_ALLOC_768_KEYPAIR_NO_PCT+#define MLK_TOTAL_ALLOC_1024_KEYPAIR MLK_TOTAL_ALLOC_1024_KEYPAIR_NO_PCT+#endif++#define MLK_MAX3_(a, b, c) \+  ((a) > (b) ? ((a) > (c) ? (a) : (c)) : ((b) > (c) ? (b) : (c)))++/*+ * `MLK_TOTAL_ALLOC_{512,768,1024}` is the maximum across all operations for+ * each parameter set.+ */+#define MLK_TOTAL_ALLOC_512                                          \+  MLK_MAX3_(MLK_TOTAL_ALLOC_512_KEYPAIR, MLK_TOTAL_ALLOC_512_ENCAPS, \+            MLK_TOTAL_ALLOC_512_DECAPS)+#define MLK_TOTAL_ALLOC_768                                          \+  MLK_MAX3_(MLK_TOTAL_ALLOC_768_KEYPAIR, MLK_TOTAL_ALLOC_768_ENCAPS, \+            MLK_TOTAL_ALLOC_768_DECAPS)+#define MLK_TOTAL_ALLOC_1024                                           \+  MLK_MAX3_(MLK_TOTAL_ALLOC_1024_KEYPAIR, MLK_TOTAL_ALLOC_1024_ENCAPS, \+            MLK_TOTAL_ALLOC_1024_DECAPS)++#endif /* !MLK_H */
+ cbits/mlkem/mlkem_native_asm.S view
@@ -0,0 +1,716 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mlkem-native repository.+ *          Do not modify it directly.+ */++/******************************************************************************+ *+ * Single assembly unit for fixed-level build of mlkem-native+ *+ * This assembly unit bundles together all assembly files for a build+ * of mlkem-native for a fixed security level (MLKEM-512/768/1024).+ *+ * # Multi-level build+ *+ * If you want an SCU build of mlkem-native with support for multiple security+ * levels, you should include this file once with+ * MLK_CONFIG_MULTILEVEL_WITH_SHARED set.+ *+ * (You could also follow the same pattern as for mlkem_native.c+ *  and include it for every level, setting MLK_CONFIG_MULTILEVEL_NO_SHARED+ *  for all but one. For builds with MLK_CONFIG_MULTILEVEL_NO_SHARED, this+ *  file will then be ignored.)+ *+ * # Configuration+ *+ * The following options from the mlkem-native configuration are relevant:+ *+ * - MLK_CONFIG_FIPS202_CUSTOM_HEADER+ *   Set this option if you use a custom FIPS202 implementation.+ *+ * - MLK_CONFIG_USE_NATIVE_BACKEND_ARITH+ *   Set this option if you want to include the native arithmetic backends+ *   in your build.+ *+ * - MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202+ *   Set this option if you want to include the native FIPS202 backends+ *   in your build.+ *+ * - MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS+ *   Set this option if you want to keep the directives defined in+ *   level-independent headers. This is needed for a multi-level build.+ */++/* If parts of the mlkem-native source tree are not used,+ * consider reducing this header via `unifdef`.+ *+ * Example:+ * ```bash+ * unifdef -UMLK_CONFIG_USE_NATIVE_BACKEND_ARITH mlkem_native_asm.S+ * ```+ */++#include "src/common.h"++#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_ARITH)+#if defined(MLK_SYS_AARCH64)+#include "src/native/aarch64/src/mlkem_intt_aarch64_asm.S"+#include "src/native/aarch64/src/mlkem_ntt_aarch64_asm.S"+#include "src/native/aarch64/src/mlkem_poly_mulcache_compute_aarch64_asm.S"+#include "src/native/aarch64/src/mlkem_poly_reduce_aarch64_asm.S"+#include "src/native/aarch64/src/mlkem_poly_tobytes_aarch64_asm.S"+#include "src/native/aarch64/src/mlkem_poly_tomont_aarch64_asm.S"+#include "src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm.S"+#include "src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm.S"+#include "src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm.S"+#include "src/native/aarch64/src/mlkem_rej_uniform_aarch64_asm.S"+#endif /* MLK_SYS_AARCH64 */+#if defined(MLK_SYS_X86_64)+#include "src/native/x86_64/src/mlkem_intt_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_ntt_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_nttfrombytes_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_ntttobytes_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_nttunpack_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_poly_compress_d10_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_poly_compress_d11_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_poly_compress_d4_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_poly_compress_d5_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_poly_decompress_d10_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_poly_decompress_d11_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_poly_decompress_d4_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_poly_decompress_d5_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_poly_mulcache_compute_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_reduce_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_rej_uniform_avx2_asm.S"+#include "src/native/x86_64/src/mlkem_tomont_avx2_asm.S"+#endif /* MLK_SYS_X86_64 */+#if defined(MLK_SYS_RISCV64)+#endif+#if defined(MLK_SYS_PPC64LE)+#include "src/native/ppc64le/src/mlkem_intt_ppc_asm.S"+#include "src/native/ppc64le/src/mlkem_ntt_ppc_asm.S"+#include "src/native/ppc64le/src/mlkem_poly_tomont_ppc_asm.S"+#include "src/native/ppc64le/src/mlkem_reduce_ppc_asm.S"+#endif /* MLK_SYS_PPC64LE */+#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_ARITH */++#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202)+#if defined(MLK_SYS_AARCH64)+#include "src/fips202/native/aarch64/src/keccak_f1600_x1_scalar_aarch64_asm.S"+#include "src/fips202/native/aarch64/src/keccak_f1600_x1_v84a_aarch64_asm.S"+#include "src/fips202/native/aarch64/src/keccak_f1600_x2_v84a_aarch64_asm.S"+#include "src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.S"+#include "src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.S"+#endif /* MLK_SYS_AARCH64 */+#if defined(MLK_SYS_X86_64)+#include "src/fips202/native/x86_64/src/keccak_f1600_x4_avx2_asm.S"+#endif+#if defined(MLK_SYS_ARMV81M_MVE)+#include "src/fips202/native/armv81m/src/keccak_f1600_x4_mve.S"+#include "src/fips202/native/armv81m/src/keccak_f1600_x4_state_extract_bytes_mve.S"+#include "src/fips202/native/armv81m/src/keccak_f1600_x4_state_xor_bytes_mve.S"+#endif+#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 */+++/* Macro #undef's+ *+ * The following undefines macros from headers+ * included by the source files imported above.+ *+ * This is to allow building and linking multiple builds+ * of mlkem-native for varying parameter sets through concatenation+ * of this file, as if the files had been compiled separately.+ * If this is not relevant to you, you may remove the following.+ *+ * NOTE: This is not needed for the assembly SCU since, at present,+ * there is no need to include it multiple times.+ * We keep it for uniformity with mlkem_native.c only.+ *+ * NOTE: To avoid having to distinguish between which headers are included+ * from the assembly files, we #undef the same set of directives+ * as in mlkem_native.c+ */++/*+ * Undefine macros from MLK_CONFIG_PARAMETER_SET-specific files+ */+/* mlkem/mlkem_native.h */+#undef MLKEM1024_BYTES+#undef MLKEM1024_CIPHERTEXTBYTES+#undef MLKEM1024_PUBLICKEYBYTES+#undef MLKEM1024_SECRETKEYBYTES+#undef MLKEM1024_SYMBYTES+#undef MLKEM512_BYTES+#undef MLKEM512_CIPHERTEXTBYTES+#undef MLKEM512_PUBLICKEYBYTES+#undef MLKEM512_SECRETKEYBYTES+#undef MLKEM512_SYMBYTES+#undef MLKEM768_BYTES+#undef MLKEM768_CIPHERTEXTBYTES+#undef MLKEM768_PUBLICKEYBYTES+#undef MLKEM768_SECRETKEYBYTES+#undef MLKEM768_SYMBYTES+#undef MLKEM_BYTES+#undef MLKEM_CIPHERTEXTBYTES+#undef MLKEM_CIPHERTEXTBYTES_+#undef MLKEM_PUBLICKEYBYTES+#undef MLKEM_PUBLICKEYBYTES_+#undef MLKEM_SECRETKEYBYTES+#undef MLKEM_SECRETKEYBYTES_+#undef MLKEM_SYMBYTES+#undef MLK_API_CONCAT+#undef MLK_API_CONCAT_+#undef MLK_API_CONCAT_UNDERSCORE+#undef MLK_API_MUST_CHECK_RETURN_VALUE+#undef MLK_API_NAMESPACE+#undef MLK_API_NAMESPACE_PREFIX+#undef MLK_API_QUALIFIER+#undef MLK_ERR_FAIL+#undef MLK_ERR_INVALID_PK+#undef MLK_ERR_INVALID_SK+#undef MLK_ERR_OUT_OF_MEMORY+#undef MLK_ERR_PCT_FAIL+#undef MLK_ERR_RNG_FAIL+#undef MLK_H+#undef MLK_MAX3_+#undef MLK_TOTAL_ALLOC_1024+#undef MLK_TOTAL_ALLOC_1024_DECAPS+#undef MLK_TOTAL_ALLOC_1024_ENCAPS+#undef MLK_TOTAL_ALLOC_1024_KEYPAIR+#undef MLK_TOTAL_ALLOC_1024_KEYPAIR_NO_PCT+#undef MLK_TOTAL_ALLOC_1024_KEYPAIR_PCT+#undef MLK_TOTAL_ALLOC_512+#undef MLK_TOTAL_ALLOC_512_DECAPS+#undef MLK_TOTAL_ALLOC_512_ENCAPS+#undef MLK_TOTAL_ALLOC_512_KEYPAIR+#undef MLK_TOTAL_ALLOC_512_KEYPAIR_NO_PCT+#undef MLK_TOTAL_ALLOC_512_KEYPAIR_PCT+#undef MLK_TOTAL_ALLOC_768+#undef MLK_TOTAL_ALLOC_768_DECAPS+#undef MLK_TOTAL_ALLOC_768_ENCAPS+#undef MLK_TOTAL_ALLOC_768_KEYPAIR+#undef MLK_TOTAL_ALLOC_768_KEYPAIR_NO_PCT+#undef MLK_TOTAL_ALLOC_768_KEYPAIR_PCT+/* mlkem/src/common.h */+#undef MLK_ADD_PARAM_SET+#undef MLK_ALLOC+#undef MLK_APPLY+#undef MLK_ASM_FN_SIZE+#undef MLK_ASM_FN_SYMBOL+#undef MLK_ASM_NAMESPACE+#undef MLK_BUILD_INTERNAL+#undef MLK_COMMON_H+#undef MLK_CONCAT+#undef MLK_CONCAT_+#undef MLK_EMPTY_CU+#undef MLK_ERR_FAIL+#undef MLK_ERR_INVALID_PK+#undef MLK_ERR_INVALID_SK+#undef MLK_ERR_OUT_OF_MEMORY+#undef MLK_ERR_PCT_FAIL+#undef MLK_ERR_RNG_FAIL+#undef MLK_EXTERNAL_API+#undef MLK_FIPS202X4_HEADER_FILE+#undef MLK_FIPS202_HEADER_FILE+#undef MLK_FREE+#undef MLK_INTERNAL_API+#undef MLK_INTERNAL_DATA_DECLARATION+#undef MLK_INTERNAL_DATA_DEFINITION+#undef MLK_NAMESPACE+#undef MLK_NAMESPACE_K+#undef MLK_NAMESPACE_PREFIX+#undef MLK_NAMESPACE_PREFIX_K+#undef mlk_memcpy+#undef mlk_memset+/* mlkem/src/indcpa.h */+#undef MLK_INDCPA_H+#undef mlk_gen_matrix+#undef mlk_indcpa_dec+#undef mlk_indcpa_enc+#undef mlk_indcpa_keypair_derand+/* mlkem/src/kem.h */+#undef MLK_KEM_H+#undef mlk_kem_check_pk+#undef mlk_kem_check_sk+#undef mlk_kem_dec+#undef mlk_kem_enc+#undef mlk_kem_enc_derand+#undef mlk_kem_keypair+#undef mlk_kem_keypair_derand+/* mlkem/src/params.h */+#undef MLKEM_DU+#undef MLKEM_DV+#undef MLKEM_ETA1+#undef MLKEM_ETA2+#undef MLKEM_INDCCA_CIPHERTEXTBYTES+#undef MLKEM_INDCCA_PUBLICKEYBYTES+#undef MLKEM_INDCCA_SECRETKEYBYTES+#undef MLKEM_INDCPA_BYTES+#undef MLKEM_INDCPA_MSGBYTES+#undef MLKEM_INDCPA_PUBLICKEYBYTES+#undef MLKEM_INDCPA_SECRETKEYBYTES+#undef MLKEM_K+#undef MLKEM_N+#undef MLKEM_POLYBYTES+#undef MLKEM_POLYCOMPRESSEDBYTES_D10+#undef MLKEM_POLYCOMPRESSEDBYTES_D11+#undef MLKEM_POLYCOMPRESSEDBYTES_D4+#undef MLKEM_POLYCOMPRESSEDBYTES_D5+#undef MLKEM_POLYCOMPRESSEDBYTES_DU+#undef MLKEM_POLYCOMPRESSEDBYTES_DV+#undef MLKEM_POLYVECBYTES+#undef MLKEM_POLYVECCOMPRESSEDBYTES_DU+#undef MLKEM_Q+#undef MLKEM_Q_HALF+#undef MLKEM_SSBYTES+#undef MLKEM_SYMBYTES+#undef MLKEM_UINT12_LIMIT+#undef MLK_PARAMS_H+/* mlkem/src/poly_k.h */+#undef MLK_POLY_K_H+#undef mlk_poly_compress_du+#undef mlk_poly_compress_dv+#undef mlk_poly_decompress_du+#undef mlk_poly_decompress_dv+#undef mlk_poly_getnoise_eta1122_4x+#undef mlk_poly_getnoise_eta1_4x+#undef mlk_poly_getnoise_eta2+#undef mlk_poly_getnoise_eta2_4x+#undef mlk_polymat+#undef mlk_polyvec+#undef mlk_polyvec_add+#undef mlk_polyvec_basemul_acc_montgomery_cached+#undef mlk_polyvec_compress_du+#undef mlk_polyvec_decompress_du+#undef mlk_polyvec_frombytes+#undef mlk_polyvec_invntt_tomont+#undef mlk_polyvec_mulcache+#undef mlk_polyvec_mulcache_compute+#undef mlk_polyvec_ntt+#undef mlk_polyvec_reduce+#undef mlk_polyvec_tobytes+#undef mlk_polyvec_tomont++#if !defined(MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS)+/*+ * Undefine macros from MLK_CONFIG_PARAMETER_SET-generic files+ */+/* mlkem/src/compress.h */+#undef MLK_COMPRESS_H+#undef mlk_poly_compress_d10+#undef mlk_poly_compress_d11+#undef mlk_poly_compress_d4+#undef mlk_poly_compress_d5+#undef mlk_poly_decompress_d10+#undef mlk_poly_decompress_d11+#undef mlk_poly_decompress_d4+#undef mlk_poly_decompress_d5+#undef mlk_poly_frombytes+#undef mlk_poly_frommsg+#undef mlk_poly_tobytes+#undef mlk_poly_tomsg+/* mlkem/src/context.h */+#undef MLK_CONTEXT_H+#undef MLK_CONTEXT_PARAMETERS_0+#undef MLK_CONTEXT_PARAMETERS_1+#undef MLK_CONTEXT_PARAMETERS_2+#undef MLK_CONTEXT_PARAMETERS_3+#undef MLK_CONTEXT_PARAMETERS_4+#undef MLK_CONTEXT_UNUSED+/* mlkem/src/debug.h */+#undef MLK_DEBUG_H+#undef mlk_assert+#undef mlk_assert_abs_bound+#undef mlk_assert_abs_bound_2d+#undef mlk_assert_bound+#undef mlk_assert_bound_2d+#undef mlk_debug_check_assert+#undef mlk_debug_check_bounds+/* mlkem/src/poly.h */+#undef MLK_INVNTT_BOUND+#undef MLK_NTT_BOUND+#undef MLK_POLY_H+#undef mlk_poly_add+#undef mlk_poly_invntt_tomont+#undef mlk_poly_mulcache_compute+#undef mlk_poly_ntt+#undef mlk_poly_reduce+#undef mlk_poly_sub+#undef mlk_poly_tomont+/* mlkem/src/randombytes.h */+#undef MLK_RANDOMBYTES_H+/* mlkem/src/sampling.h */+#undef MLK_SAMPLING_H+#undef mlk_poly_cbd2+#undef mlk_poly_cbd3+#undef mlk_poly_rej_uniform+#undef mlk_poly_rej_uniform_x4+/* mlkem/src/symmetric.h */+#undef MLK_SYMMETRIC_H+#undef MLK_XOF_RATE+#undef mlk_hash_g+#undef mlk_hash_h+#undef mlk_hash_j+#undef mlk_prf_eta+#undef mlk_prf_eta1+#undef mlk_prf_eta1_x4+#undef mlk_prf_eta2+#undef mlk_xof_absorb+#undef mlk_xof_ctx+#undef mlk_xof_init+#undef mlk_xof_release+#undef mlk_xof_squeezeblocks+#undef mlk_xof_x4_absorb+#undef mlk_xof_x4_ctx+#undef mlk_xof_x4_init+#undef mlk_xof_x4_release+#undef mlk_xof_x4_squeezeblocks+/* mlkem/src/sys.h */+#undef MLK_ALIGN+#undef MLK_ALIGN_UP+#undef MLK_ALWAYS_INLINE+#undef MLK_CET_ENDBR+#undef MLK_CT_TESTING_DECLASSIFY+#undef MLK_CT_TESTING_SECRET+#undef MLK_DEFAULT_ALIGN+#undef MLK_HAVE_INLINE_ASM+#undef MLK_INLINE+#undef MLK_MUST_CHECK_RETURN_VALUE+#undef MLK_NOINLINE+#undef MLK_RESTRICT+#undef MLK_STATIC_TESTABLE+#undef MLK_SYSV_ABI+#undef MLK_SYSV_ABI_SUPPORTED+#undef MLK_SYS_AARCH64+#undef MLK_SYS_AARCH64_EB+#undef MLK_SYS_AARCH64_NEON+#undef MLK_SYS_APPLE+#undef MLK_SYS_ARMV81M_MVE+#undef MLK_SYS_BIG_ENDIAN+#undef MLK_SYS_H+#undef MLK_SYS_LINUX+#undef MLK_SYS_LITTLE_ENDIAN+#undef MLK_SYS_PPC64LE+#undef MLK_SYS_RISCV32+#undef MLK_SYS_RISCV64+#undef MLK_SYS_RISCV64_RVV+#undef MLK_SYS_WINDOWS+#undef MLK_SYS_X86_64+#undef MLK_SYS_X86_64_AVX2+/* mlkem/src/verify.h */+#undef MLK_USE_ASM_VALUE_BARRIER+#undef MLK_VERIFY_H+#undef mlk_ct_opt_blocker_u64+/* mlkem/src/cbmc.h */+#undef MLK_CBMC_H+#undef __contract__+#undef __loop__++#if !defined(MLK_CONFIG_FIPS202_CUSTOM_HEADER)+/*+ * Undefine macros from FIPS-202 files+ */+/* mlkem/src/fips202/fips202.h */+#undef FIPS202_X4_DEFAULT_IMPLEMENTATION+#undef MLK_FIPS202_FIPS202_H+#undef SHA3_256_HASHBYTES+#undef SHA3_256_RATE+#undef SHA3_384_RATE+#undef SHA3_512_HASHBYTES+#undef SHA3_512_RATE+#undef SHAKE128_RATE+#undef SHAKE256_RATE+#undef mlk_sha3_256+#undef mlk_sha3_512+#undef mlk_shake128_absorb_once+#undef mlk_shake128_init+#undef mlk_shake128_release+#undef mlk_shake128_squeezeblocks+#undef mlk_shake256+/* mlkem/src/fips202/fips202x4.h */+#undef MLK_FIPS202_FIPS202X4_H+#undef mlk_shake128x4_absorb_once+#undef mlk_shake128x4_init+#undef mlk_shake128x4_release+#undef mlk_shake128x4_squeezeblocks+#undef mlk_shake256x4+/* mlkem/src/fips202/keccakf1600.h */+#undef MLK_FIPS202_KECCAKF1600_H+#undef MLK_KECCAK_LANES+#undef MLK_KECCAK_WAY+#undef mlk_keccakf1600_extract_bytes+#undef mlk_keccakf1600_permute+#undef mlk_keccakf1600_xor_bytes+#undef mlk_keccakf1600x4_extract_bytes+#undef mlk_keccakf1600x4_permute+#undef mlk_keccakf1600x4_xor_bytes+#endif /* !MLK_CONFIG_FIPS202_CUSTOM_HEADER */++#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202)+/* mlkem/src/fips202/native/api.h */+#undef MLK_FIPS202_NATIVE_API_H+#undef MLK_NATIVE_FUNC_FALLBACK+#undef MLK_NATIVE_FUNC_SUCCESS+/* mlkem/src/fips202/native/auto.h */+#undef MLK_FIPS202_NATIVE_AUTO_H+#if defined(MLK_SYS_AARCH64)+/*+ * Undefine macros from native code (FIPS202, AArch64)+ */+/* mlkem/src/fips202/native/aarch64/auto.h */+#undef MLK_FIPS202_NATIVE_AARCH64_AUTO_H+/* mlkem/src/fips202/native/aarch64/src/fips202_native_aarch64.h */+#undef MLK_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H+#undef mlk_keccak_f1600_x1_scalar_aarch64_asm+#undef mlk_keccak_f1600_x1_v84a_aarch64_asm+#undef mlk_keccak_f1600_x2_v84a_aarch64_asm+#undef mlk_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm+#undef mlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm+#undef mlk_keccakf1600_round_constants+/* mlkem/src/fips202/native/aarch64/x1_scalar.h */+#undef MLK_FIPS202_AARCH64_NEED_X1_SCALAR+#undef MLK_FIPS202_NATIVE_AARCH64_X1_SCALAR_H+#undef MLK_USE_NATIVE_FIPS202_X1+/* mlkem/src/fips202/native/aarch64/x1_v84a.h */+#undef MLK_FIPS202_AARCH64_NEED_X1_V84A+#undef MLK_FIPS202_NATIVE_AARCH64_X1_V84A_H+#undef MLK_USE_NATIVE_FIPS202_X1+/* mlkem/src/fips202/native/aarch64/x2_v84a.h */+#undef MLK_FIPS202_AARCH64_NEED_X2_V84A+#undef MLK_FIPS202_NATIVE_AARCH64_X2_V84A_H+#undef MLK_USE_NATIVE_FIPS202_X4+/* mlkem/src/fips202/native/aarch64/x4_v8a_scalar.h */+#undef MLK_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID+#undef MLK_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H+#undef MLK_USE_NATIVE_FIPS202_X4+/* mlkem/src/fips202/native/aarch64/x4_v8a_v84a_scalar.h */+#undef MLK_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID+#undef MLK_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H+#undef MLK_USE_NATIVE_FIPS202_X4+#endif /* MLK_SYS_AARCH64 */+#if defined(MLK_SYS_X86_64)+/*+ * Undefine macros from native code (FIPS202, x86_64)+ */+/* mlkem/src/fips202/native/x86_64/keccak_f1600_x4_avx2.h */+#undef MLK_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H+#undef MLK_FIPS202_X86_64_NEED_X4_AVX2+#undef MLK_USE_NATIVE_FIPS202_X4+/* mlkem/src/fips202/native/x86_64/src/fips202_native_x86_64.h */+#undef MLK_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H+#undef mlk_keccak_f1600_x4_avx2_asm+#undef mlk_keccak_rho56+#undef mlk_keccak_rho8+#undef mlk_keccakf1600_round_constants+#endif /* MLK_SYS_X86_64 */+#if defined(MLK_SYS_ARMV81M_MVE)+/*+ * Undefine macros from native code (FIPS202, Armv8.1-M)+ */+/* mlkem/src/fips202/native/armv81m/mve.h */+#undef MLK_FIPS202_ARMV81M_NEED_X4+#undef MLK_FIPS202_NATIVE_ARMV81M+#undef MLK_FIPS202_NATIVE_ARMV81M_MVE_H+#undef MLK_USE_NATIVE_FIPS202_X4+#undef MLK_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES+#undef MLK_USE_NATIVE_FIPS202_X4_XOR_BYTES+#undef mlk_keccak_f1600_x4_native_impl+/* mlkem/src/fips202/native/armv81m/src/fips202_native_armv81m.h */+#undef MLK_FIPS202_NATIVE_ARMV81M_SRC_FIPS202_NATIVE_ARMV81M_H+#undef mlk_keccak_f1600_x4_mve_asm+#undef mlk_keccak_f1600_x4_state_extract_bytes_asm+#undef mlk_keccak_f1600_x4_state_xor_bytes_asm+#undef mlk_keccakf1600_round_constants+#endif /* MLK_SYS_ARMV81M_MVE */+#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 */+#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_ARITH)+/* mlkem/src/native/api.h */+#undef MLK_INVNTT_BOUND+#undef MLK_NATIVE_API_H+#undef MLK_NATIVE_FUNC_FALLBACK+#undef MLK_NATIVE_FUNC_SUCCESS+#undef MLK_NTT_BOUND+/* mlkem/src/native/meta.h */+#undef MLK_NATIVE_META_H+#if defined(MLK_SYS_AARCH64)+/*+ * Undefine macros from native code (Arith, AArch64)+ */+/* mlkem/src/native/aarch64/meta.h */+#undef MLK_ARITH_BACKEND_AARCH64+#undef MLK_NATIVE_AARCH64_META_H+#undef MLK_USE_NATIVE_INTT+#undef MLK_USE_NATIVE_NTT+#undef MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED+#undef MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE+#undef MLK_USE_NATIVE_POLY_REDUCE+#undef MLK_USE_NATIVE_POLY_TOBYTES+#undef MLK_USE_NATIVE_POLY_TOMONT+#undef MLK_USE_NATIVE_REJ_UNIFORM+/* mlkem/src/native/aarch64/src/arith_native_aarch64.h */+#undef MLK_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H+#undef mlk_aarch64_invntt_zetas_layer12345+#undef mlk_aarch64_invntt_zetas_layer67+#undef mlk_aarch64_ntt_zetas_layer12345+#undef mlk_aarch64_ntt_zetas_layer67+#undef mlk_aarch64_zetas_mulcache_native+#undef mlk_aarch64_zetas_mulcache_twisted_native+#undef mlk_intt_aarch64_asm+#undef mlk_ntt_aarch64_asm+#undef mlk_poly_mulcache_compute_aarch64_asm+#undef mlk_poly_reduce_aarch64_asm+#undef mlk_poly_tobytes_aarch64_asm+#undef mlk_poly_tomont_aarch64_asm+#undef mlk_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm+#undef mlk_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm+#undef mlk_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm+#undef mlk_rej_uniform_aarch64_asm+#undef mlk_rej_uniform_table+#endif /* MLK_SYS_AARCH64 */+#if defined(MLK_SYS_X86_64)+/*+ * Undefine macros from native code (Arith, X86_64)+ */+/* mlkem/src/native/x86_64/meta.h */+#undef MLK_ARITH_BACKEND_X86_64_DEFAULT+#undef MLK_NATIVE_X86_64_META_H+#undef MLK_USE_NATIVE_INTT+#undef MLK_USE_NATIVE_NTT+#undef MLK_USE_NATIVE_NTT_CUSTOM_ORDER+#undef MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED+#undef MLK_USE_NATIVE_POLY_COMPRESS_D10+#undef MLK_USE_NATIVE_POLY_COMPRESS_D11+#undef MLK_USE_NATIVE_POLY_COMPRESS_D4+#undef MLK_USE_NATIVE_POLY_COMPRESS_D5+#undef MLK_USE_NATIVE_POLY_DECOMPRESS_D10+#undef MLK_USE_NATIVE_POLY_DECOMPRESS_D11+#undef MLK_USE_NATIVE_POLY_DECOMPRESS_D4+#undef MLK_USE_NATIVE_POLY_DECOMPRESS_D5+#undef MLK_USE_NATIVE_POLY_FROMBYTES+#undef MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE+#undef MLK_USE_NATIVE_POLY_REDUCE+#undef MLK_USE_NATIVE_POLY_TOBYTES+#undef MLK_USE_NATIVE_POLY_TOMONT+#undef MLK_USE_NATIVE_REJ_UNIFORM+/* mlkem/src/native/x86_64/src/arith_native_x86_64.h */+#undef MLK_AVX2_REJ_UNIFORM_BUFLEN+#undef MLK_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H+#undef mlk_invntt_avx2_asm+#undef mlk_ntt_avx2_asm+#undef mlk_nttfrombytes_avx2_asm+#undef mlk_ntttobytes_avx2_asm+#undef mlk_nttunpack_avx2_asm+#undef mlk_poly_compress_d10_avx2_asm+#undef mlk_poly_compress_d11_avx2_asm+#undef mlk_poly_compress_d4_avx2_asm+#undef mlk_poly_compress_d5_avx2_asm+#undef mlk_poly_decompress_d10_avx2_asm+#undef mlk_poly_decompress_d11_avx2_asm+#undef mlk_poly_decompress_d4_avx2_asm+#undef mlk_poly_decompress_d5_avx2_asm+#undef mlk_poly_mulcache_compute_avx2_asm+#undef mlk_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm+#undef mlk_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm+#undef mlk_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm+#undef mlk_reduce_avx2_asm+#undef mlk_rej_uniform_avx2_asm+#undef mlk_rej_uniform_table+#undef mlk_tomont_avx2_asm+/* mlkem/src/native/x86_64/src/compress_consts.h */+#undef MLK_NATIVE_X86_64_SRC_COMPRESS_CONSTS_H+#undef mlk_compress_d10_data+#undef mlk_compress_d11_data+#undef mlk_compress_d4_data+#undef mlk_compress_d5_data+#undef mlk_decompress_d10_data+#undef mlk_decompress_d11_data+#undef mlk_decompress_d4_data+#undef mlk_decompress_d5_data+/* mlkem/src/native/x86_64/src/consts.h */+#undef MLK_AVX2_BACKEND_DATA_OFFSET_MULCACHE_TWIDDLES+#undef MLK_AVX2_BACKEND_DATA_OFFSET_REVIDXB+#undef MLK_AVX2_BACKEND_DATA_OFFSET_REVIDXD+#undef MLK_AVX2_BACKEND_DATA_OFFSET_ZETAS_EXP+#undef MLK_NATIVE_X86_64_SRC_CONSTS_H+#undef mlk_qdata+#endif /* MLK_SYS_X86_64 */+#if defined(MLK_SYS_RISCV64)+/*+ * Undefine macros from native code (Arith, RISC-V 64)+ */+/* mlkem/src/native/riscv64/meta.h */+#undef MLK_ARITH_BACKEND_RISCV64+#undef MLK_NATIVE_RISCV64_META_H+#undef MLK_USE_NATIVE_INTT+#undef MLK_USE_NATIVE_NTT+#undef MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED+#undef MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE+#undef MLK_USE_NATIVE_POLY_REDUCE+#undef MLK_USE_NATIVE_POLY_TOMONT+#undef MLK_USE_NATIVE_REJ_UNIFORM+/* mlkem/src/native/riscv64/src/arith_native_riscv64.h */+#undef MLK_NATIVE_RISCV64_SRC_ARITH_NATIVE_RISCV64_H+#undef mlk_rv64v_poly_add+#undef mlk_rv64v_poly_basemul_mont_add_k2+#undef mlk_rv64v_poly_basemul_mont_add_k3+#undef mlk_rv64v_poly_basemul_mont_add_k4+#undef mlk_rv64v_poly_invntt_tomont+#undef mlk_rv64v_poly_ntt+#undef mlk_rv64v_poly_reduce+#undef mlk_rv64v_poly_sub+#undef mlk_rv64v_poly_tomont+#undef mlk_rv64v_rej_uniform+/* mlkem/src/native/riscv64/src/rv64v_debug.h */+#undef MLK_NATIVE_RISCV64_SRC_RV64V_DEBUG_H+#undef mlk_assert_abs_bound_int16m1+#undef mlk_assert_abs_bound_int16m2+#undef mlk_assert_bound_int16m1+#undef mlk_assert_bound_int16m2+#undef mlk_debug_check_bounds_int16m1+#undef mlk_debug_check_bounds_int16m2+#endif /* MLK_SYS_RISCV64 */+#if defined(MLK_SYS_PPC64LE)+/*+ * Undefine macros from native code (Arith, PPC64LE)+ */+/* mlkem/src/native/ppc64le/meta.h */+#undef MLK_ARITH_BACKEND_NAME+#undef MLK_ARITH_BACKEND_PPC64LE_DEFAULT+#undef MLK_NATIVE_PPC64LE_META_H+#undef MLK_USE_NATIVE_INTT+#undef MLK_USE_NATIVE_NTT+#undef MLK_USE_NATIVE_POLY_REDUCE+#undef MLK_USE_NATIVE_POLY_TOMONT+/* mlkem/src/native/ppc64le/src/arith_native_ppc64le.h */+#undef MLK_NATIVE_PPC64LE_SRC_ARITH_NATIVE_PPC64LE_H+#undef mlk_intt_ppc_asm+#undef mlk_ntt_ppc_asm+#undef mlk_poly_tomont_ppc_asm+#undef mlk_reduce_ppc_asm+/* mlkem/src/native/ppc64le/src/consts.h */+#undef MLK_NATIVE_PPC64LE_SRC_CONSTS_H+#undef MLK_PPC_C20159_OFFSET+#undef MLK_PPC_NQ_OFFSET+#undef MLK_PPC_N_INV_OFFSET+#undef MLK_PPC_N_INV_TW_OFFSET+#undef MLK_PPC_Q_OFFSET+#undef MLK_PPC_TOMONT_OFFSET+#undef MLK_PPC_TOMONT_TW_OFFSET+#undef MLK_PPC_ZETA_INTT_OFFSET+#undef MLK_PPC_ZETA_INTT_TW_OFFSET+#undef MLK_PPC_ZETA_NTT_OFFSET+#undef MLK_PPC_ZETA_NTT_TW_OFFSET+#undef mlk_ppc_qdata+#endif /* MLK_SYS_PPC64LE */+#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_ARITH */+#endif /* !MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS */
+ cbits/mlkem/mlkem_native_config.h view
@@ -0,0 +1,683 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS140_3_IG]+ *   Implementation Guidance for FIPS 140-3 and the Cryptographic Module+ *   Validation Program+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/projects/cryptographic-module-validation-program/fips-140-3-ig-announcements+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ */++#ifndef MLK_CONFIG_H+#define MLK_CONFIG_H++/**+ * Specifies the parameter set for ML-KEM:+ *  - MLK_CONFIG_PARAMETER_SET=512 corresponds to ML-KEM-512+ *  - MLK_CONFIG_PARAMETER_SET=768 corresponds to ML-KEM-768+ *  - MLK_CONFIG_PARAMETER_SET=1024 corresponds to ML-KEM-1024+ *+ * If you want to support multiple parameter sets, build the library multiple+ * times and set MLK_CONFIG_MULTILEVEL_BUILD. See MLK_CONFIG_MULTILEVEL_BUILD+ * for how to do this while minimizing code duplication.+ *+ * This can also be set using CFLAGS.+ */+#ifndef MLK_CONFIG_PARAMETER_SET+#define MLK_CONFIG_PARAMETER_SET \+  768 /* Change this for different security strengths */+#endif++/**+ * MLK_CONFIG_FILE+ *+ * If defined, this is a header that will be included instead of the default+ * configuration file mlkem/mlkem_native_config.h.+ *+ * When you need to build mlkem-native in multiple configurations, using+ * varying MLK_CONFIG_FILE can be more convenient than configuring everything+ * through CFLAGS.+ *+ * To use, MLK_CONFIG_FILE _must_ be defined prior to the inclusion of any+ * mlkem-native headers. For example, it can be set by passing+ * `-DMLK_CONFIG_FILE="..."` on the command line.+ */+/* #define MLK_CONFIG_FILE "mlkem_native_config.h" */++/**+ * The prefix to use to namespace global symbols from mlkem/.+ *+ * In a multi-level build, level-dependent symbols will additionally be+ * prefixed with the parameter set (512/768/1024).+ *+ * This can also be set using CFLAGS.+ */+#if !defined(MLK_CONFIG_NAMESPACE_PREFIX)+#define MLK_CONFIG_NAMESPACE_PREFIX MLK_DEFAULT_NAMESPACE_PREFIX+#endif++/**+ * MLK_CONFIG_MULTILEVEL_BUILD+ *+ * Set this if the build is part of a multi-level build supporting multiple+ * parameter sets.+ *+ * If you need only a single parameter set, keep this unset.+ *+ * To build mlkem-native with support for all parameter sets, build it three+ * times -- once per parameter set -- and set the option+ * MLK_CONFIG_MULTILEVEL_WITH_SHARED for exactly one of them, and+ * MLK_CONFIG_MULTILEVEL_NO_SHARED for the others.+ * MLK_CONFIG_MULTILEVEL_BUILD should be set for all of them.+ *+ * See examples/multilevel_build for an example.+ *+ * This can also be set using CFLAGS.+ */+/* #define MLK_CONFIG_MULTILEVEL_BUILD */++/**+ * MLK_CONFIG_EXTERNAL_API_QUALIFIER+ *+ * If set, this option provides an additional function qualifier to be added+ * to declarations of mlkem-native's public API.+ *+ * The primary use case for this option are single-CU builds where the public+ * API exposed by mlkem-native is wrapped by another API in the consuming+ * application. In this case, even mlkem-native's public API can be marked+ * `static`.+ */+/* #define MLK_CONFIG_EXTERNAL_API_QUALIFIER */++/**+ * MLK_CONFIG_NO_KEYPAIR_API+ *+ * By default, mlkem-native includes support for generating key pairs.+ * If you don't need this, set MLK_CONFIG_NO_KEYPAIR_API to exclude+ * keypair and keypair_derand, and all internal+ * APIs only needed by those functions.+ */+/* #define MLK_CONFIG_NO_KEYPAIR_API */++/**+ * MLK_CONFIG_NO_ENCAPS_API+ *+ * By default, mlkem-native includes support for encapsulation. If you+ * don't need this, set MLK_CONFIG_NO_ENCAPS_API to exclude+ * enc, enc_derand, check_pk, and+ * all internal APIs only needed by those functions.+ *+ * @note Setting this option is incompatible with MLK_CONFIG_KEYGEN_PCT+ *       as the current PCT implementation requires enc().+ */+/* #define MLK_CONFIG_NO_ENCAPS_API */++/**+ * MLK_CONFIG_NO_DECAPS_API+ *+ * By default, mlkem-native includes support for decapsulation. If you+ * don't need this, set MLK_CONFIG_NO_DECAPS_API to exclude+ * dec, check_sk, and all internal APIs only+ * needed by those functions.+ *+ * @note Setting this option is incompatible with MLK_CONFIG_KEYGEN_PCT+ *       as the current PCT implementation requires dec().+ */+/* #define MLK_CONFIG_NO_DECAPS_API */++/**+ * MLK_CONFIG_NO_RANDOMIZED_API+ *+ * If this option is set, mlkem-native will be built without the randomized+ * API functions (keypair and enc). This allows users+ * to build mlkem-native without providing a randombytes() implementation+ * if they only need the deterministic API (keypair_derand,+ * enc_derand, dec).+ *+ * @note This option is incompatible with MLK_CONFIG_KEYGEN_PCT as the+ *       current PCT implementation requires enc().+ */+/* #define MLK_CONFIG_NO_RANDOMIZED_API */++/**+ * MLK_CONFIG_CONSTANTS_ONLY+ *+ * If you only need the size constants (MLKEM_PUBLICKEYBYTES, etc.) but no+ * function declarations, set MLK_CONFIG_CONSTANTS_ONLY.+ *+ * This only affects the public header mlkem_native.h, not the+ * implementation.+ */+/* #define MLK_CONFIG_CONSTANTS_ONLY */++/******************************************************************************+ *+ * Build-only configuration options+ *+ * The remaining configurations are build-options only.+ * They do not affect the API described in mlkem_native.h.+ *+ *****************************************************************************/++#if defined(MLK_BUILD_INTERNAL)+/**+ * MLK_CONFIG_MULTILEVEL_WITH_SHARED+ *+ * This is for multi-level builds of mlkem-native only. If you need only a+ * single parameter set, keep this unset.+ *+ * If this is set, all MLK_CONFIG_PARAMETER_SET-independent code will be+ * included in the build, including code needed only for other parameter+ * sets.+ *+ * Example: mlk_poly_cbd3 is only needed for MLK_CONFIG_PARAMETER_SET == 512.+ * Yet, if this option is set for a build with+ * MLK_CONFIG_PARAMETER_SET == 768/1024, it would be included.+ *+ * To build mlkem-native with support for all parameter sets, build it three+ * times -- once per parameter set -- and set the option+ * MLK_CONFIG_MULTILEVEL_WITH_SHARED for exactly one of them, and+ * MLK_CONFIG_MULTILEVEL_NO_SHARED for the others.+ * MLK_CONFIG_MULTILEVEL_BUILD should be set for all of them.+ *+ * See examples/multilevel_build for an example.+ *+ * This can also be set using CFLAGS.+ */+/* #define MLK_CONFIG_MULTILEVEL_WITH_SHARED */++/**+ * MLK_CONFIG_MULTILEVEL_NO_SHARED+ *+ * This is for multi-level builds of mlkem-native only. If you need only a+ * single parameter set, keep this unset.+ *+ * If this is set, no MLK_CONFIG_PARAMETER_SET-independent code will be+ * included in the build.+ *+ * To build mlkem-native with support for all parameter sets, build it three+ * times -- once per parameter set -- and set the option+ * MLK_CONFIG_MULTILEVEL_WITH_SHARED for exactly one of them, and+ * MLK_CONFIG_MULTILEVEL_NO_SHARED for the others.+ * MLK_CONFIG_MULTILEVEL_BUILD should be set for all of them.+ *+ * See examples/multilevel_build for an example.+ *+ * This can also be set using CFLAGS.+ */+/* #define MLK_CONFIG_MULTILEVEL_NO_SHARED */++/**+ * MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS+ *+ * This is only relevant for single compilation unit (SCU) builds of+ * mlkem-native. In this case, it determines whether directives defined in+ * parameter-set-independent headers should be #undef'ined or not at the+ * end of the SCU file. This is needed in multilevel builds.+ *+ * See examples/multilevel_build_native for an example.+ *+ * This can also be set using CFLAGS.+ */+/* #define MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS */++/**+ * MLK_CONFIG_USE_NATIVE_BACKEND_ARITH+ *+ * Determines whether a native arithmetic backend should be used.+ *+ * The arithmetic backend covers performance-critical functions such as the+ * number-theoretic transform (NTT).+ *+ * If this option is unset, the C backend will be used.+ *+ * If this option is set, the arithmetic backend to be used is determined+ * by MLK_CONFIG_ARITH_BACKEND_FILE: if the latter is unset, the default+ * backend for the target architecture will be used. If set, it must be the+ * name of a backend metadata file.+ *+ * This can also be set using CFLAGS.+ */+#if !defined(MLK_CONFIG_USE_NATIVE_BACKEND_ARITH)+/* #define MLK_CONFIG_USE_NATIVE_BACKEND_ARITH */+#endif++/**+ * MLK_CONFIG_ARITH_BACKEND_FILE+ *+ * The arithmetic backend to use.+ *+ * If MLK_CONFIG_USE_NATIVE_BACKEND_ARITH is unset, this option is ignored.+ *+ * If MLK_CONFIG_USE_NATIVE_BACKEND_ARITH is set, this option must either+ * be undefined or the filename of an arithmetic backend. If unset, the+ * default backend will be used.+ *+ * This can be set using CFLAGS.+ */+#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_ARITH) && \+    !defined(MLK_CONFIG_ARITH_BACKEND_FILE)+#define MLK_CONFIG_ARITH_BACKEND_FILE "native/meta.h"+#endif++/**+ * MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202+ *+ * Determines whether a native FIPS202 backend should be used.+ *+ * The FIPS202 backend covers 1x/2x/4x-fold Keccak-f1600, which is the+ * performance bottleneck of SHA3 and SHAKE.+ *+ * If this option is unset, the C backend will be used.+ *+ * If this option is set, the FIPS202 backend to be used is determined by+ * MLK_CONFIG_FIPS202_BACKEND_FILE: if the latter is unset, the default+ * backend for the target architecture will be used. If set, it must be+ * the name of a backend metadata file.+ *+ * This can also be set using CFLAGS.+ */+#if !defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202)+/* #define MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 */+#endif++/**+ * MLK_CONFIG_FIPS202_BACKEND_FILE+ *+ * The FIPS-202 backend to use.+ *+ * If MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 is set, this option must either+ * be undefined or the filename of a FIPS202 backend. If unset, the default+ * backend will be used.+ *+ * This can be set using CFLAGS.+ */+#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202) && \+    !defined(MLK_CONFIG_FIPS202_BACKEND_FILE)+#define MLK_CONFIG_FIPS202_BACKEND_FILE "fips202/native/auto.h"+#endif++/**+ * MLK_CONFIG_FIPS202_CUSTOM_HEADER+ *+ * Custom header to use for FIPS-202.+ *+ * This should only be set if you intend to use a custom FIPS-202+ * implementation, different from the one shipped with mlkem-native.+ *+ * If set, it must be the name of a file serving as the replacement for+ * mlkem/src/fips202/fips202.h, and exposing the same API (see FIPS202.md).+ */+/* #define MLK_CONFIG_FIPS202_CUSTOM_HEADER "SOME_FILE.h" */++/**+ * MLK_CONFIG_FIPS202X4_CUSTOM_HEADER+ *+ * Custom header to use for FIPS-202-X4.+ *+ * This should only be set if you intend to use a custom FIPS-202+ * implementation, different from the one shipped with mlkem-native.+ *+ * If set, it must be the name of a file serving as the replacement for+ * mlkem/src/fips202/fips202x4.h, and exposing the same API (see FIPS202.md).+ */+/* #define MLK_CONFIG_FIPS202X4_CUSTOM_HEADER "SOME_FILE.h" */++/**+ * MLK_CONFIG_CUSTOM_ZEROIZE+ *+ * In compliance with @[FIPS203, Section 3.3], mlkem-native zeroizes+ * intermediate buffers before returning from function calls. By default,+ * those buffers are allocated from the stack; if MLK_CONFIG_CUSTOM_ALLOC_FREE+ * is set, they are (mostly -- few exceptions remain at present) allocated from+ * the configured custom allocator.+ *+ * mlkem-native also zeroizes caller-owned output buffers as needed to uphold+ * the API convention that outputs be either unmodified or zeroized upon+ * failure.+ *+ * Set this option and define `mlk_zeroize` if you want to use a custom+ * method to zeroize intermediate and output buffers.+ *+ * The default implementation uses SecureZeroMemory on Windows and a+ * memset + compiler barrier otherwise. If neither of those is available on+ * the target platform, compilation will fail, and you will need to use+ * MLK_CONFIG_CUSTOM_ZEROIZE to provide a custom implementation of+ * `mlk_zeroize()`.+ *+ * @warning+ *   The zeroization conducted by mlkem-native reduces the likelihood of data+ *   leaking on the stack or custom allocators, but it does not eliminate it.+ *   For example, the C standard makes no guarantee about where a compiler+ *   allocates local structures and whether/where it makes copies of them.+ *   Also, in addition to entire structures, there may also be potentially+ *   exploitable leakage of individual values on the stack. If you need+ *   bullet-proof zeroization of the stack, you need to consider additional+ *   measures instead of what this feature provides. In this case, you can+ *   set mlk_zeroize to a no-op. Note that in this case you are also responsible+ *   for zeroizing output buffers upon failure.+ */+/* #define MLK_CONFIG_CUSTOM_ZEROIZE+   #if !defined(__ASSEMBLER__)+   #include <stdint.h>+   #include "src/sys.h"+   static MLK_INLINE void mlk_zeroize(void *ptr, size_t len)+   {+       ... your implementation ...+   }+   #endif+*/++/**+ * MLK_CONFIG_CUSTOM_RANDOMBYTES+ *+ * mlkem-native does not provide a secure randombytes implementation. Such+ * an implementation has to be provided by the consumer.+ *+ * If this option is not set, mlkem-native expects a function+ * int randombytes(uint8_t *out, size_t outlen). It is expected to return+ * zero on success, and non-zero on failure. In case of failure, the+ * top-level APIs will return an MLK_ERR_RNG_FAIL error code.+ *+ * Set this option and define `mlk_randombytes` (with the same signature+ * and behaviour) if you want to use a custom method to sample randombytes+ * with a different name or signature.+ */+/* #define MLK_CONFIG_CUSTOM_RANDOMBYTES+   #if !defined(__ASSEMBLER__)+   #include <stdint.h>+   #include "src/sys.h"+   static MLK_INLINE int mlk_randombytes(uint8_t *ptr, size_t len)+   {+       ... your implementation ...+       return 0;+   }+   #endif+*/++/**+ * MLK_CONFIG_CUSTOM_CAPABILITY_FUNC+ *+ * mlkem-native backends may rely on specific hardware features. Those+ * backends will only be included in an mlkem-native build if support for+ * the respective features is enabled at compile-time. However, when+ * building for a heterogeneous set of CPUs to run the resulting+ * binary/library on, feature detection at _runtime_ is needed to decide+ * whether a backend can be used or not.+ *+ * Set this option and define `mlk_sys_check_capability` if you want to+ * use a custom method to dispatch between implementations.+ *+ * If this option is not set, mlkem-native uses compile-time feature+ * detection only to decide which backend to use.+ *+ * If you compile mlkem-native on a system with different capabilities+ * than the system that the resulting binary/library will be run on, you+ * must use this option.+ */+/* #define MLK_CONFIG_CUSTOM_CAPABILITY_FUNC+   static MLK_INLINE int mlk_sys_check_capability(mlk_sys_cap cap)+   __contract__(+     ensures(return_value == 0 || return_value == 1)+   )+   {+       ... your implementation ...+   }+*/++/**+ * MLK_CONFIG_CUSTOM_ALLOC_FREE [EXPERIMENTAL]+ *+ * Set this option and define `MLK_CUSTOM_ALLOC` and `MLK_CUSTOM_FREE` if+ * you want to use custom allocation for large local structures or buffers.+ *+ * By default, all buffers/structures are allocated on the stack. If this+ * option is set, most of them will be allocated via MLK_CUSTOM_ALLOC.+ *+ * Parameters to MLK_CUSTOM_ALLOC:+ *  - T* v: Target pointer to declare.+ *  - T: Type of structure to be allocated.+ *  - N: Number of elements to be allocated.+ *+ * Parameters to MLK_CUSTOM_FREE:+ *  - T* v: Target pointer to free. May be NULL.+ *  - T: Type of structure to be freed.+ *  - N: Number of elements to be freed.+ *+ * @warning This option is experimental. Its scope, configuration and+ *          function/macro signatures may change at any time. We expect a+ *          stable API in a future version.+ *+ * @note Even if this option is set, some allocations further down the call+ *       stack will still be made from the stack, consuming up to 3KB of+ *       stack space. Those will likely be added to the scope of this+ *       option in the future.+ *+ * @note MLK_CUSTOM_ALLOC need not guarantee a successful allocation nor+ *       include error handling. Upon failure, the target pointer should+ *       simply be set to NULL. The calling code will handle this case and+ *       invoke MLK_CUSTOM_FREE.+ */+/* #define MLK_CONFIG_CUSTOM_ALLOC_FREE+   #if !defined(__ASSEMBLER__)+   #include <stdlib.h>+   #define MLK_CUSTOM_ALLOC(v, T, N)                              \+     T* (v) = (T *)aligned_alloc(MLK_DEFAULT_ALIGN,               \+                                 MLK_ALIGN_UP(sizeof(T) * (N)))+   #define MLK_CUSTOM_FREE(v, T, N) free(v)+   #endif+*/++/**+ * MLK_CONFIG_CUSTOM_MEMCPY+ *+ * Set this option and define `mlk_memcpy` if you want to use a custom+ * method to copy memory instead of the standard library memcpy function.+ *+ * The custom implementation must have the same signature and behavior as+ * the standard memcpy function:+ * void *mlk_memcpy(void *dest, const void *src, size_t n)+ */+/* #define MLK_CONFIG_CUSTOM_MEMCPY+   #if !defined(__ASSEMBLER__)+   #include <stdint.h>+   #include "src/sys.h"+   static MLK_INLINE void *mlk_memcpy(void *dest, const void *src, size_t n)+   {+       ... your implementation ...+   }+   #endif+*/++/**+ * MLK_CONFIG_CUSTOM_MEMSET+ *+ * Set this option and define `mlk_memset` if you want to use a custom+ * method to set memory instead of the standard library memset function.+ *+ * The custom implementation must have the same signature and behavior as+ * the standard memset function:+ * void *mlk_memset(void *s, int c, size_t n)+ */+/* #define MLK_CONFIG_CUSTOM_MEMSET+   #if !defined(__ASSEMBLER__)+   #include <stdint.h>+   #include "src/sys.h"+   static MLK_INLINE void *mlk_memset(void *s, int c, size_t n)+   {+       ... your implementation ...+   }+   #endif+*/++/**+ * MLK_CONFIG_INTERNAL_API_QUALIFIER+ *+ * If set, this option provides an additional qualifier to be added to+ * declarations of internal API functions and data.+ *+ * The primary use case for this option are single-CU builds, in which case+ * this option can be set to `static`.+ */+/* #define MLK_CONFIG_INTERNAL_API_QUALIFIER */++/**+ * MLK_CONFIG_CT_TESTING_ENABLED+ *+ * If set, mlkem-native annotates data as secret/public using valgrind's+ * annotations VALGRIND_MAKE_MEM_UNDEFINED and VALGRIND_MAKE_MEM_DEFINED,+ * enabling various checks for secret-dependent control flow or+ * variable-time execution (depending on the exact version of valgrind+ * installed).+ */+/* #define MLK_CONFIG_CT_TESTING_ENABLED */++/**+ * MLK_CONFIG_NO_ASM+ *+ * If this option is set, mlkem-native will be built without use of native+ * code or inline assembly.+ *+ * By default, inline assembly is used to implement value barriers. Without+ * inline assembly, mlkem-native will use a global volatile 'opt blocker'+ * instead; see verify.h.+ *+ * Inline assembly is also used to implement a secure zeroization function+ * on non-Windows platforms. If this option is set and the target platform+ * is not Windows, you MUST set MLK_CONFIG_CUSTOM_ZEROIZE and provide a+ * custom zeroization function.+ *+ * If this option is set, MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 and+ * MLK_CONFIG_USE_NATIVE_BACKEND_ARITH will be ignored, and no native+ * backends will be used.+ */+/* #define MLK_CONFIG_NO_ASM */++/**+ * MLK_CONFIG_NO_ASM_VALUE_BARRIER+ *+ * If this option is set, mlkem-native will be built without use of native+ * code or inline assembly for value barriers.+ *+ * By default, inline assembly (if available) is used to implement value+ * barriers. Without inline assembly, mlkem-native will use a global+ * volatile 'opt blocker' instead; see verify.h.+ */+/* #define MLK_CONFIG_NO_ASM_VALUE_BARRIER */++/**+ * MLK_CONFIG_KEYGEN_PCT+ *+ * Compliance with @[FIPS140_3_IG, p.87] requires a Pairwise Consistency+ * Test (PCT) to be carried out on a freshly generated keypair before it+ * can be exported.+ *+ * Set this option if such a check should be implemented. In this case,+ * keypair_derand and keypair will return+ * MLK_ERR_PCT_FAIL if the PCT failed.+ *+ * @note This feature will drastically lower the performance of key+ *       generation.+ */+/* #define MLK_CONFIG_KEYGEN_PCT */++/**+ * MLK_CONFIG_KEYGEN_PCT_BREAKAGE_TEST+ *+ * If this option is set, the user must provide a runtime function+ * `static inline int mlk_break_pct() { ... }` to indicate whether the PCT+ * should be made to fail.+ *+ * This option only has an effect if MLK_CONFIG_KEYGEN_PCT is set.+ */+/* #define MLK_CONFIG_KEYGEN_PCT_BREAKAGE_TEST+   #if !defined(__ASSEMBLER__)+   #include "src/sys.h"+   static MLK_INLINE int mlk_break_pct(void)+   {+       ... return 0/1 depending on whether PCT should be broken ...+   }+   #endif+*/++/**+ * MLK_CONFIG_SERIAL_FIPS202_ONLY+ *+ * Set this to use a FIPS202 implementation with global state that supports+ * only one active Keccak computation at a time (e.g. some hardware+ * accelerators).+ *+ * If this option is set, batched Keccak operations are disabled for+ * rejection sampling during matrix generation. Instead, matrix entries+ * will be generated one at a time.+ *+ * This allows offloading Keccak computations to a hardware accelerator+ * that holds only a single Keccak state locally, rather than requiring+ * support for batched (4x) Keccak states.+ *+ * @note Depending on the target CPU, disabling batched Keccak may reduce+ *       performance when using software FIPS202 implementations. Only+ *       enable this when you have to.+ */+/* #define MLK_CONFIG_SERIAL_FIPS202_ONLY */++/**+ * MLK_CONFIG_CONTEXT_PARAMETER+ *+ * Set this to add a context parameter that is provided to public API+ * functions and is then available in custom callbacks.+ *+ * The type of the context parameter is configured via+ * MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ */+/* #define MLK_CONFIG_CONTEXT_PARAMETER */++/**+ * MLK_CONFIG_CONTEXT_PARAMETER_TYPE+ *+ * Set this to define the type for the context parameter used by+ * MLK_CONFIG_CONTEXT_PARAMETER.+ *+ * This is only relevant if MLK_CONFIG_CONTEXT_PARAMETER is set.+ */+/* #define MLK_CONFIG_CONTEXT_PARAMETER_TYPE void* */++/*************************  Config internals  ********************************/++#endif /* MLK_BUILD_INTERNAL */++/* Default namespace+ *+ * Don't change this. If you need a different namespace, re-define+ * MLK_CONFIG_NAMESPACE_PREFIX above instead, and remove the following.+ *+ * The default MLKEM namespace is+ *+ *   PQCP_MLKEM_NATIVE_MLKEM<LEVEL>_+ *+ * e.g., PQCP_MLKEM_NATIVE_MLKEM512_+ */++#if defined(MLK_CONFIG_MULTILEVEL_BUILD)+/* In a multi-level build the parameter set is appended by the namespacing+ * machinery, so the default prefix must not embed it. */+#define MLK_DEFAULT_NAMESPACE_PREFIX PQCP_MLKEM_NATIVE_MLKEM+#elif MLK_CONFIG_PARAMETER_SET == 512+#define MLK_DEFAULT_NAMESPACE_PREFIX PQCP_MLKEM_NATIVE_MLKEM512+#elif MLK_CONFIG_PARAMETER_SET == 768+#define MLK_DEFAULT_NAMESPACE_PREFIX PQCP_MLKEM_NATIVE_MLKEM768+#elif MLK_CONFIG_PARAMETER_SET == 1024+#define MLK_DEFAULT_NAMESPACE_PREFIX PQCP_MLKEM_NATIVE_MLKEM1024+#endif++#endif /* !MLK_CONFIG_H */
+ cbits/mlkem/src/cbmc.h view
@@ -0,0 +1,222 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLK_CBMC_H+#define MLK_CBMC_H+/***************************************************+ * Basic replacements for __CPROVER_XXX contracts+ ***************************************************/+/*+ * The `__contract__` / `__loop__` annotation macros use a+ * leading-double-underscore spelling in line with other CBMC macros.+ * clang-tidy flags these as reserved identifiers; we suppress the diagnostic+ * at each definition site (NOLINT) rather than disabling the check globally,+ * so it stays active for the rest of the tree.+ */+#ifndef CBMC++/* clang-format off */+#define __contract__(x) /* NOLINT(bugprone-reserved-identifier,cert-dcl37-c,cert-dcl51-cpp) */+#define __loop__(x) /* NOLINT(bugprone-reserved-identifier,cert-dcl37-c,cert-dcl51-cpp) */+/* clang-format on */++#else /* !CBMC */+++/* clang-format off */+#define __contract__(x) x /* NOLINT(bugprone-reserved-identifier,cert-dcl37-c,cert-dcl51-cpp) */+#define __loop__(x) x /* NOLINT(bugprone-reserved-identifier,cert-dcl37-c,cert-dcl51-cpp) */+/* clang-format on */++/* https://diffblue.github.io/cbmc/contracts-assigns.html */+#define assigns(...) __CPROVER_assigns(__VA_ARGS__)++/* https://diffblue.github.io/cbmc/contracts-requires-ensures.html */+#define requires(...) __CPROVER_requires(__VA_ARGS__)+#define ensures(...) __CPROVER_ensures(__VA_ARGS__)+/* https://diffblue.github.io/cbmc/contracts-loops.html */+#define invariant(...) __CPROVER_loop_invariant(__VA_ARGS__)+#define decreases(...) __CPROVER_decreases(__VA_ARGS__)+/* cassert to avoid confusion with in-built assert */+#define cassert(x) __CPROVER_assert(x, "cbmc assertion failed")+#define assume(...) __CPROVER_assume(__VA_ARGS__)++/***************************************************+ * Macros for "expression" forms that may appear+ * _inside_ top-level contracts.+ ***************************************************/++/*+ * function return value - useful inside ensures+ * https://diffblue.github.io/cbmc/contracts-functions.html+ */+#define return_value (__CPROVER_return_value)++/*+ * assigns l-value targets+ * https://diffblue.github.io/cbmc/contracts-assigns.html+ */+#define object_whole(...) __CPROVER_object_whole(__VA_ARGS__)+#define memory_slice(...) __CPROVER_object_upto(__VA_ARGS__)++/*+ * Pointer-related predicates+ * https://diffblue.github.io/cbmc/contracts-memory-predicates.html+ */+#define memory_no_alias(...) __CPROVER_is_fresh(__VA_ARGS__)+#define readable(...) __CPROVER_r_ok(__VA_ARGS__)+#define writeable(...) __CPROVER_w_ok(__VA_ARGS__)++/* Maximum supported buffer size+ *+ * Larger buffers may be supported, but due to internal modeling constraints+ * in CBMC, the proofs of memory- and type-safety won't be able to run.+ *+ * If you find yourself in need for a buffer size larger than this,+ * please contact the maintainers, so we can prioritize work to relax+ * this somewhat artificial bound.+ */+#define MLK_MAX_BUFFER_SIZE (SIZE_MAX >> 12)++/*+ * History variables+ * https://diffblue.github.io/cbmc/contracts-history-variables.html+ */+#define old(...) __CPROVER_old(__VA_ARGS__)+#define loop_entry(...) __CPROVER_loop_entry(__VA_ARGS__)++/*+ * Quantifiers+ * Note that the range on qvar is _exclusive_ between qvar_lb .. qvar_ub+ * https://diffblue.github.io/cbmc/contracts-quantifiers.html+ *+ * The quantified variable is declared as uint32_t, so these macros+ * quantify only over indices in [0, UINT32_MAX). Bounds larger than+ * UINT32_MAX (4 GiB) are NOT supported: the explicit (uint32_t) casts+ * on the bounds will trigger CBMC's conversion check if a wider bound+ * (e.g. a size_t > UINT32_MAX) is passed.+ *+ * Quantifying over size_t (64-bit) was found to blow up SMT proof+ * times, so we deliberately keep the index width at 32 bits. Callers+ * dealing with size_t-typed buffers must add an explicit+ *   requires(len <= UINT32_MAX)+ * precondition.+ */++/*+ * Prevent clang-format from corrupting CBMC's special ==> operator+ */+/* clang-format off */+#define forall(qvar, qvar_lb, qvar_ub, predicate)                              \+  __CPROVER_forall                                                             \+  {                                                                            \+    uint32_t qvar;                                                             \+    ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==> (predicate) \+  }++#define exists(qvar, qvar_lb, qvar_ub, predicate)                              \+  __CPROVER_exists                                                             \+  {                                                                            \+    uint32_t qvar;                                                             \+    ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) && (predicate) \+  }+/* clang-format on */++/***************************************************+ * Convenience macros for common contract patterns+ ***************************************************/++/*+ * Boolean-value predidate that asserts that "all values of array_var are in+ * range value_lb (inclusive) .. value_ub (exclusive)"+ * Example:+ *  array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q)+ * expands to+ *  __CPROVER_forall { int k; (0 <= k && k <= MLKEM_N-1) ==> (+ *  0 <= a->coeffs[k]) && a->coeffs[k] < MLKEM_Q)) }+ */++/*+ * Prevent clang-format from corrupting CBMC's special ==> operator+ */+/* clang-format off */+#define CBMC_CONCAT_(left, right) left##right+#define CBMC_CONCAT(left, right) CBMC_CONCAT_(left, right)++#define array_bound_core(qvar, qvar_lb, qvar_ub, array_var,            \+                         value_lb, value_ub)                           \+  __CPROVER_forall                                                     \+  {                                                                    \+    uint32_t qvar;                                                     \+    ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==> \+        (((int)(value_lb) <= ((array_var)[(qvar)])) &&                 \+         (((array_var)[(qvar)]) < (int)(value_ub)))                    \+  }++#define array_bound(array_var, qvar_lb, qvar_ub, value_lb, value_ub)    \+  array_bound_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), (qvar_lb),      \+      (qvar_ub), (array_var), (value_lb), (value_ub))++#define array_unchanged_core(qvar, qvar_lb, qvar_ub, array_var)        \+  __CPROVER_forall                                                     \+  {                                                                    \+    uint32_t qvar;                                                     \+    ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==> \+    ((array_var)[(qvar)]) == (old(* (int16_t (*)[(qvar_ub)])(array_var)))[(qvar)] \+  }++#define array_unchanged(array_var, N) \+    array_unchanged_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), 0, (N), (array_var))++#define array_unchanged_u64_core(qvar, qvar_lb, qvar_ub, array_var)    \+  __CPROVER_forall                                                     \+  {                                                                    \+    uint32_t qvar;                                                     \+    ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==> \+    ((array_var)[(qvar)]) == (old(* (uint64_t (*)[(qvar_ub)])(array_var)))[(qvar)] \+  }++#define array_unchanged_u64(array_var, N) \+    array_unchanged_u64_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), 0, (N), (array_var))++#define array_unchanged_u8_core(qvar, array_var, N)                     \+  forall(qvar, 0, (N),                                                  \+    ((array_var)[(qvar)]) == (old(* (uint8_t (*)[(N)])(array_var)))[(qvar)])++#define array_unchanged_u8(array_var, N) \+    array_unchanged_u8_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), (array_var), (N))++#define array_zeroized_u8_core(qvar, array_var, N) \+    forall(qvar, 0, (N), ((array_var)[(qvar)]) == 0)++#define array_zeroized_u8(array_var, N) \+    array_zeroized_u8_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), (array_var), (N))+/* clang-format on */++/*+ * Output-buffer discipline on failure, as documented in API-CONVENTIONS.md:+ * when a function fails, each caller-owned output buffer is left either+ * fully unchanged or fully zeroized -- never holding partially computed or+ * stale data that could be mistaken for a valid result.+ *+ * Note the disjunction is over the buffer as a whole: it is not enough for+ * each byte to be individually either unchanged or zero.+ */+#define array_unchanged_or_zeroized_u8(array_var, N) \+  (array_unchanged_u8((array_var), (N)) || array_zeroized_u8((array_var), (N)))++/* Wrapper around array_bound operating on absolute values.+ *+ * The absolute value bound `k` is exclusive.+ *+ * Note that since the lower bound in array_bound is inclusive, we have to+ * raise it by 1 here.+ */+#define array_abs_bound(arr, lb, ub, k) \+  array_bound((arr), (lb), (ub), -((int)(k)) + 1, (k))++#endif /* CBMC */++#endif /* !MLK_CBMC_H */
+ cbits/mlkem/src/common.h view
@@ -0,0 +1,296 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS140_3_IG]+ *   Implementation Guidance for FIPS 140-3 and the Cryptographic Module+ *   Validation Program+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/projects/cryptographic-module-validation-program/fips-140-3-ig-announcements+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ */++#ifndef MLK_COMMON_H+#define MLK_COMMON_H++#ifndef __ASSEMBLER__+#include <stdint.h>+#endif++#define MLK_BUILD_INTERNAL++#if defined(MLK_CONFIG_FILE)+#include MLK_CONFIG_FILE+#else+#include "mlkem_native_config.h"+#endif++#include "params.h"+#include "sys.h"++/* Internal and public API have external linkage by default, but+ * this can be overwritten by the user, e.g. for single-CU builds. */+#if !defined(MLK_CONFIG_INTERNAL_API_QUALIFIER)+#define MLK_INTERNAL_API+#define MLK_INTERNAL_DATA_DECLARATION extern+#define MLK_INTERNAL_DATA_DEFINITION+#else+#define MLK_INTERNAL_API MLK_CONFIG_INTERNAL_API_QUALIFIER+#define MLK_INTERNAL_DATA_DECLARATION MLK_CONFIG_INTERNAL_API_QUALIFIER+#define MLK_INTERNAL_DATA_DEFINITION MLK_CONFIG_INTERNAL_API_QUALIFIER+#endif++#if !defined(MLK_CONFIG_EXTERNAL_API_QUALIFIER)+#define MLK_EXTERNAL_API+#else+#define MLK_EXTERNAL_API MLK_CONFIG_EXTERNAL_API_QUALIFIER+#endif++#define MLK_CONCAT_(x1, x2) x1##x2+#define MLK_CONCAT(x1, x2) MLK_CONCAT_(x1, x2)++#if (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || \+     defined(MLK_CONFIG_MULTILEVEL_NO_SHARED))+#define MLK_ADD_PARAM_SET(s) MLK_CONCAT(s, MLK_CONFIG_PARAMETER_SET)+#else+#define MLK_ADD_PARAM_SET(s) s+#endif++#define MLK_NAMESPACE_PREFIX MLK_CONCAT(MLK_CONFIG_NAMESPACE_PREFIX, _)+#define MLK_NAMESPACE_PREFIX_K \+  MLK_CONCAT(MLK_ADD_PARAM_SET(MLK_CONFIG_NAMESPACE_PREFIX), _)++/* Functions are prefixed by MLK_CONFIG_NAMESPACE_PREFIX.+ *+ * If multiple parameter sets are used, functions depending on the parameter+ * set are additionally prefixed with 512/768/1024. See mlkem_native_config.h.+ *+ * Example: If MLK_CONFIG_NAMESPACE_PREFIX is mlkem, then+ * MLK_NAMESPACE_K(enc) becomes mlkem512_enc/mlkem768_enc/mlkem1024_enc.+ */+#define MLK_NAMESPACE(s) MLK_CONCAT(MLK_NAMESPACE_PREFIX, s)+#define MLK_NAMESPACE_K(s) MLK_CONCAT(MLK_NAMESPACE_PREFIX_K, s)++/* On Apple platforms, we need to emit leading underscore+ * in front of assembly symbols. We thus introduce a separate+ * namespace wrapper for ASM symbols. */+#if !defined(__APPLE__)+#define MLK_ASM_NAMESPACE(sym) MLK_NAMESPACE(sym)+#else+#define MLK_ASM_NAMESPACE(sym) MLK_CONCAT(_, MLK_NAMESPACE(sym))+#endif++/*+ * On X86_64 if control-flow protections (CET) are enabled (through+ * -fcf-protection=), we add an endbr64 instruction at every global function+ * label.  See sys.h for more details+ */+#if defined(MLK_SYS_X86_64)+#define MLK_ASM_FN_SYMBOL(sym) MLK_ASM_NAMESPACE(sym) : MLK_CET_ENDBR+#elif defined(MLK_SYS_ARMV81M_MVE)+/* clang-format off */+#define MLK_ASM_FN_SYMBOL(sym) \+  .type MLK_ASM_NAMESPACE(sym), %function; \+  MLK_ASM_NAMESPACE(sym) :+/* clang-format on */+#else /* !MLK_SYS_X86_64 && MLK_SYS_ARMV81M_MVE */+#define MLK_ASM_FN_SYMBOL(sym) MLK_ASM_NAMESPACE(sym) :+#endif /* !MLK_SYS_X86_64 && !MLK_SYS_ARMV81M_MVE */++/*+ * Output the size of an assembly function.+ */+#if defined(__ELF__)+#define MLK_ASM_FN_SIZE(sym) \+  .size MLK_ASM_NAMESPACE(sym), .- MLK_ASM_NAMESPACE(sym)+#else+#define MLK_ASM_FN_SIZE(sym)+#endif++/* We aim to simplify the user's life by supporting builds where+ * all source files are included, even those that are not needed.+ * Those files are appropriately guarded and will be empty when unneeded.+ * The following is to avoid compilers complaining about this. */+#define MLK_EMPTY_CU(s) extern int MLK_NAMESPACE_K(empty_cu_##s);++/* MLK_CONFIG_NO_ASM takes precedence over MLK_USE_NATIVE_XXX */+#if defined(MLK_CONFIG_NO_ASM)+#undef MLK_CONFIG_USE_NATIVE_BACKEND_ARITH+#undef MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202+#endif++#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_ARITH) && \+    !defined(MLK_CONFIG_ARITH_BACKEND_FILE)+#error Bad configuration: MLK_CONFIG_USE_NATIVE_BACKEND_ARITH is set, but MLK_CONFIG_ARITH_BACKEND_FILE is not.+#endif++#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202) && \+    !defined(MLK_CONFIG_FIPS202_BACKEND_FILE)+#error Bad configuration: MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 is set, but MLK_CONFIG_FIPS202_BACKEND_FILE is not.+#endif++#if defined(MLK_CONFIG_NO_RANDOMIZED_API) && defined(MLK_CONFIG_KEYGEN_PCT)+#error Bad configuration: MLK_CONFIG_NO_RANDOMIZED_API is incompatible with MLK_CONFIG_KEYGEN_PCT as the current PCT implementation requires enc()+#endif++#if defined(MLK_CONFIG_NO_ENCAPS_API) && defined(MLK_CONFIG_KEYGEN_PCT)+#error Bad configuration: MLK_CONFIG_NO_ENCAPS_API is incompatible with MLK_CONFIG_KEYGEN_PCT as the current PCT implementation requires enc()+#endif++#if defined(MLK_CONFIG_NO_DECAPS_API) && defined(MLK_CONFIG_KEYGEN_PCT)+#error Bad configuration: MLK_CONFIG_NO_DECAPS_API is incompatible with MLK_CONFIG_KEYGEN_PCT as the current PCT implementation requires dec()+#endif++#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_ARITH)+#include MLK_CONFIG_ARITH_BACKEND_FILE+/* Include to enforce consistency of API and implementation,+ * and conduct sanity checks on the backend.+ *+ * Keep this _after_ the inclusion of the backend; otherwise,+ * the sanity checks won't have an effect. */+#if defined(MLK_CHECK_APIS) && !defined(__ASSEMBLER__)+#include "native/api.h"+#endif+#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_ARITH */++#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202)+#include MLK_CONFIG_FIPS202_BACKEND_FILE+/* Include to enforce consistency of API and implementation,+ * and conduct sanity checks on the backend.+ *+ * Keep this _after_ the inclusion of the backend; otherwise,+ * the sanity checks won't have an effect. */+#if defined(MLK_CHECK_APIS) && !defined(__ASSEMBLER__)+#include "fips202/native/api.h"+#endif+#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 */++#if !defined(MLK_CONFIG_FIPS202_CUSTOM_HEADER)+#define MLK_FIPS202_HEADER_FILE "fips202/fips202.h"+#else+#define MLK_FIPS202_HEADER_FILE MLK_CONFIG_FIPS202_CUSTOM_HEADER+#endif++#if !defined(MLK_CONFIG_FIPS202X4_CUSTOM_HEADER)+#define MLK_FIPS202X4_HEADER_FILE "fips202/fips202x4.h"+#else+#define MLK_FIPS202X4_HEADER_FILE MLK_CONFIG_FIPS202X4_CUSTOM_HEADER+#endif++/* Standard library function replacements */+#if !defined(__ASSEMBLER__)+#if !defined(MLK_CONFIG_CUSTOM_MEMCPY)+#include <string.h>+#define mlk_memcpy memcpy+#endif++#if !defined(MLK_CONFIG_CUSTOM_MEMSET)+#include <string.h>+#define mlk_memset memset+#endif+++/* Allocation macros for large local structures+ *+ * MLK_ALLOC(v, T, N) declares T *v and attempts to point it to an T[N]+ * MLK_FREE(v, T, N) zeroizes and frees the allocation+ *+ * Default implementation uses stack allocation.+ * Can be overridden by setting the config option MLK_CONFIG_CUSTOM_ALLOC_FREE+ * and defining MLK_CUSTOM_ALLOC and MLK_CUSTOM_FREE.+ */+#if defined(MLK_CONFIG_CUSTOM_ALLOC_FREE) != \+    (defined(MLK_CUSTOM_ALLOC) && defined(MLK_CUSTOM_FREE))+#error Bad configuration: MLK_CONFIG_CUSTOM_ALLOC_FREE must be set together with MLK_CUSTOM_ALLOC and MLK_CUSTOM_FREE+#endif++/* Context-parameter machinery (MLK_CONTEXT_PARAMETERS_n and related config+ * checks). Kept in a separate, level-generic header for readability; included+ * here so it is available to the allocation macros below and to all consumers+ * of common.h. */+#include "context.h"++#if !defined(MLK_CONFIG_CUSTOM_ALLOC_FREE)+/* Default: stack allocation */++/* This is a declaration macro, not an expression macro: T is a type and v is+ * a declarator, neither of which can be wrapped in parentheses. The+ * bugprone-macro-parentheses diagnostic is therefore a false positive here. */+#define MLK_ALLOC(v, T, N, context) \+  MLK_ALIGN T mlk_alloc_##v[N];     \+  T *v = mlk_alloc_##v /* NOLINT(bugprone-macro-parentheses) */++/* The MLK_FREE macro body references mlk_zeroize(), which is declared in+ * verify.h. We deliberately do NOT include verify.h here: doing so would+ * create a circular dependency (verify.h includes common.h), and common.h+ * itself never calls mlk_zeroize() -- only the macro expansion does. Each+ * translation unit that uses MLK_FREE therefore includes verify.h directly. */+#define MLK_FREE(v, T, N, context)                     \+  do                                                   \+  {                                                    \+    MLK_CONTEXT_UNUSED(context);                       \+    mlk_zeroize(mlk_alloc_##v, sizeof(mlk_alloc_##v)); \+    (v) = NULL;                                        \+  } while (0)++#else /* !MLK_CONFIG_CUSTOM_ALLOC_FREE */++/* Custom allocation */++/*+ * The indirection here is necessary to use MLK_CONTEXT_PARAMETERS_3 here.+ */+#define MLK_APPLY(f, args) f args++#define MLK_ALLOC(v, T, N, context) \+  MLK_APPLY(MLK_CUSTOM_ALLOC, MLK_CONTEXT_PARAMETERS_3(v, T, N, context))++#define MLK_FREE(v, T, N, context)                                            \+  do                                                                          \+  {                                                                           \+    if (v != NULL)                                                            \+    {                                                                         \+      mlk_zeroize(v, sizeof(T) * (N));                                        \+      MLK_APPLY(MLK_CUSTOM_FREE, MLK_CONTEXT_PARAMETERS_3(v, T, N, context)); \+      v = NULL;                                                               \+    }                                                                         \+  } while (0)++#endif /* MLK_CONFIG_CUSTOM_ALLOC_FREE */++/****************************** Error codes ***********************************/++/* Generic failure condition. Currently not returned by any function;+ * reserved for failures that no more specific code covers. */+#define MLK_ERR_FAIL (-1)+/* An allocation failed. This can only happen if MLK_CONFIG_CUSTOM_ALLOC_FREE+ * is defined and the provided MLK_CUSTOM_ALLOC can fail. */+#define MLK_ERR_OUT_OF_MEMORY (-2)+/* An RNG failure occurred. Might be due to insufficient entropy or+ * system misconfiguration. */+#define MLK_ERR_RNG_FAIL (-3)+/* Public key validation failed: the @[FIPS203, Section 7.2, 'modulus check']+ * found a coefficient outside [0,q-1]. Returned by check_pk and by the+ * encapsulation API. */+#define MLK_ERR_INVALID_PK (-4)+/* Secret key validation failed: the @[FIPS203, Section 7.3, 'hash check']+ * found the embedded public key hash inconsistent. Returned by check_sk and+ * by the decapsulation API. */+#define MLK_ERR_INVALID_SK (-5)+/* The 'Pairwise Consistency Test' @[FIPS140_3_IG, p.87] and+ * @[FIPS203, Section 7.1, Pairwise Consistency] failed. Only possible when+ * MLK_CONFIG_KEYGEN_PCT is enabled; signals that the freshly generated key+ * pair failed its encaps/decaps self-test. */+#define MLK_ERR_PCT_FAIL (-6)++#endif /* !__ASSEMBLER__ */++#endif /* !MLK_COMMON_H */
+ cbits/mlkem/src/compress.c view
@@ -0,0 +1,763 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ *+ * - [REF]+ *   CRYSTALS-Kyber C reference implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/ref+ */++#include "common.h"+#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)+++#include "cbmc.h"+#include "compress.h"+#include "debug.h"+#include "verify.h"++#if (!defined(MLK_CONFIG_NO_ENCAPS_API) ||                         \+     !defined(MLK_CONFIG_NO_DECAPS_API)) &&                        \+    (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2 || \+     MLKEM_K == 3)+/* Reference: `poly_compress()` in the reference implementation @[REF],+ *            for ML-KEM-{512,768}.+ *            - In contrast to the reference implementation, we assume+ *              unsigned canonical coefficients here.+ *              The reference implementation works with coefficients+ *              in the range [-(MLKEM_Q-1), MLKEM_Q-1]. */+MLK_STATIC_TESTABLE void mlk_poly_compress_d4_c(+    uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D4], const mlk_poly *a)+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D4))+  requires(memory_no_alias(a, sizeof(mlk_poly)))+  requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D4))+)+{+  unsigned i;+  mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q);++  for (i = 0; i < MLKEM_N / 8; i++)+  __loop__(invariant(i <= MLKEM_N / 8)+           decreases(MLKEM_N / 8 - i))+  {+    unsigned j;+    uint8_t t[8] = {0};+    for (j = 0; j < 8; j++)+    __loop__(+      invariant(i <= MLKEM_N / 8 && j <= 8)+      invariant(array_bound(t, 0, j, 0, 16))+      decreases(8 - j))+    {+      t[j] = mlk_scalar_compress_d4(a->coeffs[8 * i + j]);+    }++    /* All t[i] are 4-bit wide, so the truncations don't alter the value. */+    r[i * 4] = (uint8_t)(t[0] | (t[1] << 4));+    r[i * 4 + 1] = (uint8_t)(t[2] | (t[3] << 4));+    r[i * 4 + 2] = (uint8_t)(t[4] | (t[5] << 4));+    r[i * 4 + 3] = (uint8_t)(t[6] | (t[7] << 4));+  }+}++MLK_INTERNAL_API+void mlk_poly_compress_d4(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D4],+                          const mlk_poly *a)+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D4))+  requires(memory_no_alias(a, sizeof(mlk_poly)))+  requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D4))+)+{+#if defined(MLK_USE_NATIVE_POLY_COMPRESS_D4)+  int ret;+  mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q);+  ret = mlk_poly_compress_d4_native(r, a->coeffs);+  if (ret == MLK_NATIVE_FUNC_SUCCESS)+  {+    return;+  }+#endif /* MLK_USE_NATIVE_POLY_COMPRESS_D4 */++  mlk_poly_compress_d4_c(r, a);+}++/* Reference: Embedded into `polyvec_compress()` in the+ *            reference implementation, for ML-KEM-{512,768}.+ *            - In contrast to the reference implementation, we assume+ *              unsigned canonical coefficients here.+ *              The reference implementation works with coefficients+ *              in the range [-(MLKEM_Q-1), MLKEM_Q-1]. */+MLK_STATIC_TESTABLE void mlk_poly_compress_d10_c(+    uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D10], const mlk_poly *a)+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D10))+  requires(memory_no_alias(a, sizeof(mlk_poly)))+  requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D10))+)+{+  unsigned j;+  mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q);+  for (j = 0; j < MLKEM_N / 4; j++)+  __loop__(invariant(j <= MLKEM_N / 4)+           decreases(MLKEM_N / 4 - j))+  {+    unsigned k;+    uint16_t t[4];+    for (k = 0; k < 4; k++)+    __loop__(+      invariant(k <= 4)+      invariant(forall(r, 0, k, t[r] < (1u << 10)))+      decreases(4 - k))+    {+      t[k] = mlk_scalar_compress_d10(a->coeffs[4 * j + k]);+    }++    /*+     * Make all implicit truncation explicit. No data is being+     * truncated for the LHS's since each t[i] is 10-bit in size.+     */+    r[5 * j + 0] = (uint8_t)((t[0] >> 0) & 0xFF);+    r[5 * j + 1] = (uint8_t)((t[0] >> 8) | ((t[1] << 2) & 0xFF));+    r[5 * j + 2] = (uint8_t)((t[1] >> 6) | ((t[2] << 4) & 0xFF));+    r[5 * j + 3] = (uint8_t)((t[2] >> 4) | ((t[3] << 6) & 0xFF));+    r[5 * j + 4] = (uint8_t)(t[3] >> 2);+  }+}++MLK_INTERNAL_API+void mlk_poly_compress_d10(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D10],+                           const mlk_poly *a)+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D10))+  requires(memory_no_alias(a, sizeof(mlk_poly)))+  requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D10))+)+{+#if defined(MLK_USE_NATIVE_POLY_COMPRESS_D10)+  int ret;+  mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q);+  ret = mlk_poly_compress_d10_native(r, a->coeffs);+  if (ret == MLK_NATIVE_FUNC_SUCCESS)+  {+    return;+  }+#endif /* MLK_USE_NATIVE_POLY_COMPRESS_D10 */++  mlk_poly_compress_d10_c(r, a);+}++#if !defined(MLK_CONFIG_NO_DECAPS_API)+/* Reference: `poly_decompress()` in the reference implementation @[REF],+ *            for ML-KEM-{512,768}. */+MLK_STATIC_TESTABLE void mlk_poly_decompress_d4_c(+    mlk_poly *r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D4])+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D4))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+)+{+  unsigned i;+  for (i = 0; i < MLKEM_N / 2; i++)+  __loop__(+    invariant(i <= MLKEM_N / 2)+    invariant(array_bound(r->coeffs, 0, 2 * i, 0, MLKEM_Q))+    decreases(MLKEM_N / 2 - i))+  {+    r->coeffs[2 * i + 0] = mlk_scalar_decompress_d4((a[i] >> 0) & 0xF);+    r->coeffs[2 * i + 1] = mlk_scalar_decompress_d4((a[i] >> 4) & 0xF);+  }++  mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q);+}++MLK_INTERNAL_API+void mlk_poly_decompress_d4(mlk_poly *r,+                            const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D4])+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D4))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+)+{+#if defined(MLK_USE_NATIVE_POLY_DECOMPRESS_D4)+  int ret;+  ret = mlk_poly_decompress_d4_native(r->coeffs, a);+  if (ret == MLK_NATIVE_FUNC_SUCCESS)+  {+    mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q);+    return;+  }+#endif /* MLK_USE_NATIVE_POLY_DECOMPRESS_D4 */++  mlk_poly_decompress_d4_c(r, a);+}++/* Reference: Embedded into `polyvec_decompress()` in the+ *            reference implementation, for ML-KEM-{512,768}. */+MLK_STATIC_TESTABLE void mlk_poly_decompress_d10_c(+    mlk_poly *r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D10])+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D10))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+)+{+  unsigned j;+  for (j = 0; j < MLKEM_N / 4; j++)+  __loop__(+    invariant(j <= MLKEM_N / 4)+    invariant(array_bound(r->coeffs, 0, 4 * j, 0, MLKEM_Q))+    decreases(MLKEM_N / 4 - j))+  {+    unsigned k;+    uint16_t t[4];+    uint8_t const *base = &a[5 * j];++    t[0] = 0x3FF & ((base[0] >> 0) | ((uint16_t)base[1] << 8));+    t[1] = 0x3FF & ((base[1] >> 2) | ((uint16_t)base[2] << 6));+    t[2] = 0x3FF & ((base[2] >> 4) | ((uint16_t)base[3] << 4));+    t[3] = 0x3FF & ((base[3] >> 6) | ((uint16_t)base[4] << 2));++    for (k = 0; k < 4; k++)+    __loop__(+      invariant(k <= 4)+      invariant(array_bound(r->coeffs, 0, 4 * j + k, 0, MLKEM_Q))+      decreases(4 - k))+    {+      r->coeffs[4 * j + k] = mlk_scalar_decompress_d10(t[k]);+    }+  }++  mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q);+}++MLK_INTERNAL_API+void mlk_poly_decompress_d10(mlk_poly *r,+                             const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D10])+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D10))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+)+{+#if defined(MLK_USE_NATIVE_POLY_DECOMPRESS_D10)+  int ret;+  ret = mlk_poly_decompress_d10_native(r->coeffs, a);+  if (ret == MLK_NATIVE_FUNC_SUCCESS)+  {+    mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q);+    return;+  }+#endif /* MLK_USE_NATIVE_POLY_DECOMPRESS_D10 */++  mlk_poly_decompress_d10_c(r, a);+}+#endif /* !MLK_CONFIG_NO_DECAPS_API */+#endif /* (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) &&         \+          (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 || MLKEM_K == 3) \+        */++#if (!defined(MLK_CONFIG_NO_ENCAPS_API) ||  \+     !defined(MLK_CONFIG_NO_DECAPS_API)) && \+    (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4)+/* Reference: `poly_compress()` in the reference implementation @[REF],+ *            for ML-KEM-1024.+ *            - In contrast to the reference implementation, we assume+ *              unsigned canonical coefficients here.+ *              The reference implementation works with coefficients+ *              in the range [-(MLKEM_Q-1), MLKEM_Q-1]. */+MLK_STATIC_TESTABLE void mlk_poly_compress_d5_c(+    uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D5], const mlk_poly *a)+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D5))+  requires(memory_no_alias(a, sizeof(mlk_poly)))+  requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D5))+)+{+  unsigned i;+  mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q);++  for (i = 0; i < MLKEM_N / 8; i++)+  __loop__(invariant(i <= MLKEM_N / 8)+           decreases(MLKEM_N / 8 - i))+  {+    unsigned j;+    uint8_t t[8] = {0};+    for (j = 0; j < 8; j++)+    __loop__(+      invariant(i <= MLKEM_N / 8 && j <= 8)+      invariant(array_bound(t, 0, j, 0, 32))+      decreases(8 - j))+    {+      t[j] = mlk_scalar_compress_d5(a->coeffs[8 * i + j]);+    }++    r[i * 5] = (uint8_t)(0xFF & ((t[0] >> 0) | (t[1] << 5)));+    r[i * 5 + 1] = (uint8_t)(0xFF & ((t[1] >> 3) | (t[2] << 2) | (t[3] << 7)));+    r[i * 5 + 2] = (uint8_t)(0xFF & ((t[3] >> 1) | (t[4] << 4)));+    r[i * 5 + 3] = (uint8_t)(0xFF & ((t[4] >> 4) | (t[5] << 1) | (t[6] << 6)));+    r[i * 5 + 4] = (uint8_t)(0xFF & ((t[6] >> 2) | (t[7] << 3)));+  }+}++MLK_INTERNAL_API+void mlk_poly_compress_d5(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D5],+                          const mlk_poly *a)+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D5))+  requires(memory_no_alias(a, sizeof(mlk_poly)))+  requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D5))+)+{+#if defined(MLK_USE_NATIVE_POLY_COMPRESS_D5)+  int ret;+  mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q);+  ret = mlk_poly_compress_d5_native(r, a->coeffs);+  if (ret == MLK_NATIVE_FUNC_SUCCESS)+  {+    return;+  }+#endif /* MLK_USE_NATIVE_POLY_COMPRESS_D5 */++  mlk_poly_compress_d5_c(r, a);+}++/* Reference: Embedded into `polyvec_compress()` in the+ *            reference implementation, for ML-KEM-1024.+ *            - In contrast to the reference implementation, we assume+ *              unsigned canonical coefficients here.+ *              The reference implementation works with coefficients+ *              in the range [-(MLKEM_Q-1), MLKEM_Q-1]. */+MLK_STATIC_TESTABLE void mlk_poly_compress_d11_c(+    uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D11], const mlk_poly *a)+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D11))+  requires(memory_no_alias(a, sizeof(mlk_poly)))+  requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D11))+)+{+  unsigned j;+  mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q);++  for (j = 0; j < MLKEM_N / 8; j++)+  __loop__(invariant(j <= MLKEM_N / 8)+           decreases(MLKEM_N / 8 - j))+  {+    unsigned k;+    uint16_t t[8];+    for (k = 0; k < 8; k++)+    __loop__(+      invariant(k <= 8)+      invariant(forall(r, 0, k, t[r] < (1u << 11)))+      decreases(8 - k))+    {+      t[k] = mlk_scalar_compress_d11(a->coeffs[8 * j + k]);+    }++    /*+     * Make all implicit truncation explicit. No data is being+     * truncated for the LHS's since each t[i] is 11-bit in size.+     */+    r[11 * j + 0] = (uint8_t)((t[0] >> 0) & 0xFF);+    r[11 * j + 1] = (uint8_t)((t[0] >> 8) | ((t[1] << 3) & 0xFF));+    r[11 * j + 2] = (uint8_t)((t[1] >> 5) | ((t[2] << 6) & 0xFF));+    r[11 * j + 3] = (uint8_t)((t[2] >> 2) & 0xFF);+    r[11 * j + 4] = (uint8_t)((t[2] >> 10) | ((t[3] << 1) & 0xFF));+    r[11 * j + 5] = (uint8_t)((t[3] >> 7) | ((t[4] << 4) & 0xFF));+    r[11 * j + 6] = (uint8_t)((t[4] >> 4) | ((t[5] << 7) & 0xFF));+    r[11 * j + 7] = (uint8_t)((t[5] >> 1) & 0xFF);+    r[11 * j + 8] = (uint8_t)((t[5] >> 9) | ((t[6] << 2) & 0xFF));+    r[11 * j + 9] = (uint8_t)((t[6] >> 6) | ((t[7] << 5) & 0xFF));+    r[11 * j + 10] = (uint8_t)(t[7] >> 3);+  }+}++MLK_INTERNAL_API+void mlk_poly_compress_d11(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D11],+                           const mlk_poly *a)+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D11))+  requires(memory_no_alias(a, sizeof(mlk_poly)))+  requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D11))+)+{+#if defined(MLK_USE_NATIVE_POLY_COMPRESS_D11)+  int ret;+  mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q);+  ret = mlk_poly_compress_d11_native(r, a->coeffs);+  if (ret == MLK_NATIVE_FUNC_SUCCESS)+  {+    return;+  }+#endif /* MLK_USE_NATIVE_POLY_COMPRESS_D11 */++  mlk_poly_compress_d11_c(r, a);+}++#if !defined(MLK_CONFIG_NO_DECAPS_API)+/* Reference: `poly_decompress()` in the reference implementation @[REF],+ *            for ML-KEM-1024. */+MLK_STATIC_TESTABLE void mlk_poly_decompress_d5_c(+    mlk_poly *r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D5])+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D5))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+)+{+  unsigned i;+  for (i = 0; i < MLKEM_N / 8; i++)+  __loop__(+    invariant(i <= MLKEM_N / 8)+    invariant(array_bound(r->coeffs, 0, 8 * i, 0, MLKEM_Q))+    decreases(MLKEM_N / 8 - i))+  {+    unsigned j;+    uint8_t t[8];+    const unsigned offset = i * 5;+    /*+     * Explicitly truncate to avoid warning about+     * implicit truncation in CBMC and unwind loop for ease+     * of proof.+     */++    /*+     * Decompress 5 8-bit bytes (so 40 bits) into+     * 8 5-bit values stored in t[]+     */+    t[0] = 0x1F & (a[offset + 0] >> 0);+    t[1] = 0x1F & ((a[offset + 0] >> 5) | (a[offset + 1] << 3));+    t[2] = 0x1F & (a[offset + 1] >> 2);+    t[3] = 0x1F & ((a[offset + 1] >> 7) | (a[offset + 2] << 1));+    t[4] = 0x1F & ((a[offset + 2] >> 4) | (a[offset + 3] << 4));+    t[5] = 0x1F & (a[offset + 3] >> 1);+    t[6] = 0x1F & ((a[offset + 3] >> 6) | (a[offset + 4] << 2));+    t[7] = 0x1F & (a[offset + 4] >> 3);++    /* and copy to the correct slice in r[] */+    for (j = 0; j < 8; j++)+    __loop__(+      invariant(j <= 8 && i <= MLKEM_N / 8)+      invariant(array_bound(r->coeffs, 0, 8 * i + j, 0, MLKEM_Q))+      decreases(8 - j))+    {+      r->coeffs[8 * i + j] = mlk_scalar_decompress_d5(t[j]);+    }+  }++  mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q);+}++MLK_INTERNAL_API+void mlk_poly_decompress_d5(mlk_poly *r,+                            const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D5])+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D5))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+)+{+#if defined(MLK_USE_NATIVE_POLY_DECOMPRESS_D5)+  int ret;+  ret = mlk_poly_decompress_d5_native(r->coeffs, a);+  if (ret == MLK_NATIVE_FUNC_SUCCESS)+  {+    mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q);+    return;+  }+#endif /* MLK_USE_NATIVE_POLY_DECOMPRESS_D5 */++  mlk_poly_decompress_d5_c(r, a);+}++/* Reference: Embedded into `polyvec_decompress()` in the+ *            reference implementation, for ML-KEM-1024. */+MLK_STATIC_TESTABLE void mlk_poly_decompress_d11_c(+    mlk_poly *r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D11])+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D11))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+)+{+  unsigned j;+  for (j = 0; j < MLKEM_N / 8; j++)+  __loop__(+    invariant(j <= MLKEM_N / 8)+    invariant(array_bound(r->coeffs, 0, 8 * j, 0, MLKEM_Q))+    decreases(MLKEM_N / 8 - j))+  {+    unsigned k;+    uint16_t t[8];+    uint8_t const *base = &a[11 * j];+    t[0] = 0x7FF & ((base[0] >> 0) | ((uint16_t)base[1] << 8));+    t[1] = 0x7FF & ((base[1] >> 3) | ((uint16_t)base[2] << 5));+    t[2] = 0x7FF & ((base[2] >> 6) | ((uint16_t)base[3] << 2) |+                    ((uint16_t)base[4] << 10));+    t[3] = 0x7FF & ((base[4] >> 1) | ((uint16_t)base[5] << 7));+    t[4] = 0x7FF & ((base[5] >> 4) | ((uint16_t)base[6] << 4));+    t[5] = 0x7FF & ((base[6] >> 7) | ((uint16_t)base[7] << 1) |+                    ((uint16_t)base[8] << 9));+    t[6] = 0x7FF & ((base[8] >> 2) | ((uint16_t)base[9] << 6));+    t[7] = 0x7FF & ((base[9] >> 5) | ((uint16_t)base[10] << 3));++    for (k = 0; k < 8; k++)+    __loop__(+      invariant(k <= 8)+      invariant(array_bound(r->coeffs, 0, 8 * j + k, 0, MLKEM_Q))+      decreases(8 - k))+    {+      r->coeffs[8 * j + k] = mlk_scalar_decompress_d11(t[k]);+    }+  }++  mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q);+}++MLK_INTERNAL_API+void mlk_poly_decompress_d11(mlk_poly *r,+                             const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D11])+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D11))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+)+{+#if defined(MLK_USE_NATIVE_POLY_DECOMPRESS_D11)+  int ret;+  ret = mlk_poly_decompress_d11_native(r->coeffs, a);+  if (ret == MLK_NATIVE_FUNC_SUCCESS)+  {+    mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q);+    return;+  }+#endif /* MLK_USE_NATIVE_POLY_DECOMPRESS_D11 */++  mlk_poly_decompress_d11_c(r, a);+}++#endif /* !MLK_CONFIG_NO_DECAPS_API */+#endif /* (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) && \+          (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4) */++#if !defined(MLK_CONFIG_NO_KEYPAIR_API) || !defined(MLK_CONFIG_NO_ENCAPS_API)+/* Reference: `poly_tobytes()` in the reference implementation @[REF].+ *            - In contrast to the reference implementation, we assume+ *              unsigned canonical coefficients here.+ *              The reference implementation works with coefficients+ *              in the range [-(MLKEM_Q-1), MLKEM_Q-1]. */+MLK_STATIC_TESTABLE void mlk_poly_tobytes_c(uint8_t r[MLKEM_POLYBYTES],+                                            const mlk_poly *a)+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYBYTES))+  requires(memory_no_alias(a, sizeof(mlk_poly)))+  requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYBYTES))+)+{+  unsigned i;+  mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q);++  for (i = 0; i < MLKEM_N / 2; i++)+  __loop__(invariant(i <= MLKEM_N / 2)+           decreases(MLKEM_N / 2 - i))+  {+    /* The conversion to uint16_t is safe since we assume that+     * the coefficients of `a` are non-negative. */+    const uint16_t t0 = (uint16_t)a->coeffs[2 * i];+    const uint16_t t1 = (uint16_t)a->coeffs[2 * i + 1];+    /*+     * t0 and t1 are both < MLKEM_Q, so contain at most 12 bits each of+     * significant data, so these can be packed into 24 bits or exactly+     * 3 bytes, as follows.+     */++    /* Least significant bits 0 - 7 of t0. */+    r[3 * i + 0] = (uint8_t)(t0 & 0xFF);++    /*+     * Most significant bits 8 - 11 of t0 become the least significant+     * nibble of the second byte. The least significant 4 bits+     * of t1 become the upper nibble of the second byte.+     *+     * The conversion to uint8_t does not alter the value.+     */+    r[3 * i + 1] = (uint8_t)((t0 >> 8) | ((t1 << 4) & 0xF0));++    /* Bits 4 - 11 of t1 become the third byte. The conversion to uint8_t+     * does not alter the value because t1 is 12-bit wide. */+    r[3 * i + 2] = (uint8_t)(t1 >> 4);+  }+}++MLK_INTERNAL_API+void mlk_poly_tobytes(uint8_t r[MLKEM_POLYBYTES], const mlk_poly *a)+{+#if defined(MLK_USE_NATIVE_POLY_TOBYTES)+  int ret;+  mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q);+  ret = mlk_poly_tobytes_native(r, a->coeffs);+  if (ret == MLK_NATIVE_FUNC_SUCCESS)+  {+    return;+  }+#endif /* MLK_USE_NATIVE_POLY_TOBYTES */++  mlk_poly_tobytes_c(r, a);+}+#endif /* !MLK_CONFIG_NO_KEYPAIR_API || !MLK_CONFIG_NO_ENCAPS_API */++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+/* Reference: `poly_frombytes()` in the reference implementation @[REF]. */+MLK_STATIC_TESTABLE void mlk_poly_frombytes_c(mlk_poly *r,+                                              const uint8_t a[MLKEM_POLYBYTES])+__contract__(+  requires(memory_no_alias(a, MLKEM_POLYBYTES))+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT))+)+{+  unsigned i;+  for (i = 0; i < MLKEM_N / 2; i++)+  __loop__(+    invariant(i <= MLKEM_N / 2)+    invariant(array_bound(r->coeffs, 0, 2 * i, 0, MLKEM_UINT12_LIMIT))+    decreases(MLKEM_N / 2 - i))+  {+    const uint8_t t0 = a[3 * i + 0];+    const uint8_t t1 = a[3 * i + 1];+    const uint8_t t2 = a[3 * i + 2];+    /* Safety:+     * - The explicit cast to uint16_t ensures that << 8 does+     *   not signed-overflow even on a 16-bit system.+     * - The cast to int16_t is safe due to the explicit 0xFFF truncation.+     */+    r->coeffs[2 * i + 0] = (int16_t)(t0 | (((uint16_t)t1 << 8) & 0xFFF));+    r->coeffs[2 * i + 1] = (int16_t)((t1 >> 4) | (t2 << 4));+  }++  /* Note that the coefficients are not canonical */+  mlk_assert_bound(r, MLKEM_N, 0, MLKEM_UINT12_LIMIT);+}++MLK_INTERNAL_API+void mlk_poly_frombytes(mlk_poly *r, const uint8_t a[MLKEM_POLYBYTES])+{+#if defined(MLK_USE_NATIVE_POLY_FROMBYTES)+  int ret;+  ret = mlk_poly_frombytes_native(r->coeffs, a);+  if (ret == MLK_NATIVE_FUNC_SUCCESS)+  {+    return;+  }+#endif /* MLK_USE_NATIVE_POLY_FROMBYTES */++  mlk_poly_frombytes_c(r, a);+}+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+/* Reference: `poly_frommsg()` in the reference implementation @[REF].+ *            - We use a value barrier around the bit-selection mask to+ *              reduce the risk of compiler-introduced branches.+ *              The reference implementation contains the expression+ *              `(msg[i] >> j) & 1` which the compiler can reason must+ *              be either 0 or 1. */+MLK_INTERNAL_API+void mlk_poly_frommsg(mlk_poly *r, const uint8_t msg[MLKEM_INDCPA_MSGBYTES])+{+  unsigned i;+#if (MLKEM_INDCPA_MSGBYTES != MLKEM_N / 8)+#error "MLKEM_INDCPA_MSGBYTES must be equal to MLKEM_N/8 bytes!"+#endif++  for (i = 0; i < MLKEM_N / 8; i++)+  __loop__(+    invariant(i <= MLKEM_N / 8)+    invariant(array_bound(r->coeffs, 0, 8 * i, 0, MLKEM_Q))+    decreases(MLKEM_N / 8 - i))+  {+    unsigned j;+    for (j = 0; j < 8; j++)+    __loop__(+      invariant(i <  MLKEM_N / 8 && j <= 8)+      invariant(array_bound(r->coeffs, 0, 8 * i + j, 0, MLKEM_Q))+      decreases(8 - j))+    {+      /* mlk_ct_sel_int16(MLKEM_Q_HALF, 0, b) is `Decompress_1(b != 0)`+       * as per @[FIPS203, Eq (4.8)]. */++      /* Prevent the compiler from recognizing this as a bit selection */+      uint8_t mask = mlk_value_barrier_u8((uint8_t)(1u << j));+      r->coeffs[8 * i + j] = mlk_ct_sel_int16(MLKEM_Q_HALF, 0, msg[i] & mask);+    }+  }+  mlk_assert_abs_bound(r, MLKEM_N, MLKEM_Q);+}+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++#if !defined(MLK_CONFIG_NO_DECAPS_API)+/* Reference: `poly_tomsg()` in the reference implementation @[REF].+ *            - In contrast to the reference implementation, we assume+ *              unsigned canonical coefficients here.+ *              The reference implementation works with coefficients+ *              in the range [-(MLKEM_Q-1), MLKEM_Q-1].+ */+MLK_INTERNAL_API+void mlk_poly_tomsg(uint8_t msg[MLKEM_INDCPA_MSGBYTES], const mlk_poly *r)+{+  unsigned i;+  mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q);++  for (i = 0; i < MLKEM_N / 8; i++)+  __loop__(invariant(i <= MLKEM_N / 8)+           decreases(MLKEM_N / 8 - i))+  {+    unsigned j;+    msg[i] = 0;+    for (j = 0; j < 8; j++)+    __loop__(+      invariant(i <= MLKEM_N / 8 && j <= 8)+      decreases(8 - j))+    {+      uint32_t t = mlk_scalar_compress_d1(r->coeffs[8 * i + j]);+      msg[i] |= (uint8_t)(t << j);+    }+  }+}+#endif /* !MLK_CONFIG_NO_DECAPS_API */++#else /* !MLK_CONFIG_MULTILEVEL_NO_SHARED */++MLK_EMPTY_CU(compress)++#endif /* MLK_CONFIG_MULTILEVEL_NO_SHARED */
+ cbits/mlkem/src/compress.h view
@@ -0,0 +1,613 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ *+ * - [REF]+ *   CRYSTALS-Kyber C reference implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/ref+ */++#ifndef MLK_COMPRESS_H+#define MLK_COMPRESS_H+++#include "cbmc.h"+#include "common.h"+#include "debug.h"+#include "poly.h"+#include "verify.h"++/*+ * The multiplication in this routine will exceed UINT32_MAX+ * and wrap around for large values of u. This is expected and required.+ */+#ifdef CBMC+#pragma CPROVER check push+#pragma CPROVER check disable "unsigned-overflow"+#endif++/**+ * Compute round(u * 2 / MLKEM_Q).+ *+ * @spec{Compress_1 from @[FIPS203, Eq (4.7)].}+ *+ * @reference{Part of poly_tomsg() in the reference implementation @[REF].}+ *+ * @param u Unsigned canonical modulus modulo MLKEM_Q to be compressed.+ *+ * @return Compressed value.+ */+static MLK_INLINE uint8_t mlk_scalar_compress_d1(int16_t u)+__contract__(+  requires(0 <= u && u <= MLKEM_Q - 1)+  ensures(return_value < 2)+  ensures(return_value == (((uint32_t)u * 2 + MLKEM_Q / 2) / MLKEM_Q) % 2)  )+{+  /* Compute as follows:+   * ```+   * round(u * 2 / MLKEM_Q)+   *   = round(u * 2 * (2^31 / MLKEM_Q) / 2^31)+   *  ~= round(u * 2 * round(2^31 / MLKEM_Q) / 2^31)+   * ```+   */+  /* check-magic: 1290168 == 2*round(2^31 / MLKEM_Q) */+  uint32_t d0 = (uint32_t)u * 1290168;+  /* Unsigned shifting by 31 positions leaves only the top bit. */+  return (uint8_t)((d0 + ((uint32_t)1u << 30)) >> 31);+}+#ifdef CBMC+#pragma CPROVER check pop+#endif++/*+ * The multiplication in this routine will exceed UINT32_MAX+ * and wrap around for large values of u. This is expected and required.+ */+#ifdef CBMC+#pragma CPROVER check push+#pragma CPROVER check disable "unsigned-overflow"+#endif++/**+ * Compute round(u * 16 / MLKEM_Q) % 16.+ *+ * @spec{Compress_4 from @[FIPS203, Eq (4.7)].}+ *+ * @reference{Embedded into `poly_compress()` in the reference+ * implementation @[REF].}+ *+ * @param u Unsigned canonical modulus modulo MLKEM_Q to be compressed.+ *+ * @return Compressed value.+ */+static MLK_INLINE uint8_t mlk_scalar_compress_d4(int16_t u)+__contract__(+  requires(0 <= u && u <= MLKEM_Q - 1)+  ensures(return_value < 16)+  ensures(return_value == (((uint32_t)u * 16 + MLKEM_Q / 2) / MLKEM_Q) % 16))+{+  /* Compute as follows:+   * ```+   * round(u * 16 / MLKEM_Q)+   *   = round(u * 16 * (2^28 / MLKEM_Q) / 2^28)+   *  ~= round(u * 16 * round(2^28 / MLKEM_Q) / 2^28)+   * ```+   */+  /* check-magic: 1290160 == 16 * round(2^28 / MLKEM_Q) */+  uint32_t d0 = (uint32_t)u * 1290160;+  /* The return value is < 16, so not altered by the conversion to uint8_t. */+  return (uint8_t)((d0 + ((uint32_t)1u << 27)) >> 28); /* round(d0/2^28) */+}+#ifdef CBMC+#pragma CPROVER check pop+#endif++/**+ * Compute round(u * MLKEM_Q / 16).+ *+ * @spec{Decompress_4 from @[FIPS203, Eq (4.8)].}+ *+ * @reference{Embedded into `poly_decompress()` in the reference+ * implementation @[REF].}+ *+ * @param u Unsigned canonical modulus modulo 16 to be decompressed.+ *+ * @return Decompressed value.+ */+static MLK_INLINE int16_t mlk_scalar_decompress_d4(uint8_t u)+__contract__(+  requires(0 <= u && u < 16)+  ensures(return_value <= (MLKEM_Q - 1))+)+{+  /* The return value is in 0..MLKEM_Q-1, hence not altered by the+   * conversion to int16_t. */+  return (int16_t)((((uint32_t)u * MLKEM_Q) + 8) >> 4);+}++/*+ * The multiplication in this routine will exceed UINT32_MAX+ * and wrap around for large values of u. This is expected and required.+ */+#ifdef CBMC+#pragma CPROVER check push+#pragma CPROVER check disable "unsigned-overflow"+#endif++/**+ * Compute round(u * 32 / MLKEM_Q) % 32.+ *+ * @spec{Compress_5 from @[FIPS203, Eq (4.7)].}+ *+ * @reference{Embedded into `poly_compress()` in the reference+ * implementation @[REF].}+ *+ * @param u Unsigned canonical modulus modulo MLKEM_Q to be compressed.+ *+ * @return Compressed value.+ */+static MLK_INLINE uint8_t mlk_scalar_compress_d5(int16_t u)+__contract__(+  requires(0 <= u && u <= MLKEM_Q - 1)+  ensures(return_value < 32)+  ensures(return_value == (((uint32_t)u * 32 + MLKEM_Q / 2) / MLKEM_Q) % 32)  )+{+  /* Compute as follows:+   * ```+   * round(u * 32 / MLKEM_Q)+   *   = round(u * 32 * (2^27 / MLKEM_Q) / 2^27)+   *  ~= round(u * 32 * round(2^27 / MLKEM_Q) / 2^27)+   * ```+   */+  /* check-magic: 1290176 == 2^5 * round(2^27 / MLKEM_Q) */+  uint32_t d0 = (uint32_t)u * 1290176;+  /* The return value is < 32, so not altered by the conversion to uint8_t. */+  return (uint8_t)((d0 + ((uint32_t)1u << 26)) >> 27); /* round(d0/2^27) */+}+#ifdef CBMC+#pragma CPROVER check pop+#endif++/**+ * Compute round(u * MLKEM_Q / 32).+ *+ * @spec{Decompress_5 from @[FIPS203, Eq (4.8)].}+ *+ * @reference{Embedded into `poly_decompress()` in the reference+ * implementation @[REF].}+ *+ * @param u Unsigned canonical modulus modulo 32 to be decompressed.+ *+ * @return Decompressed value.+ */+static MLK_INLINE int16_t mlk_scalar_decompress_d5(uint8_t u)+__contract__(+  requires(0 <= u && u < 32)+  ensures(0 <= return_value && return_value <= MLKEM_Q - 1)+)+{+  /* The return value is in 0..MLKEM_Q-1, hence not altered by the+   * conversion to int16_t. */+  return (int16_t)((((uint32_t)u * MLKEM_Q) + 16) >> 5);+}++/*+ * The multiplication in this routine will exceed UINT32_MAX+ * and wrap around for large values of u. This is expected and required.+ */+#ifdef CBMC+#pragma CPROVER check push+#pragma CPROVER check disable "unsigned-overflow"+#endif++/**+ * Compute round(u * 2**10 / MLKEM_Q) % 2**10.+ *+ * @spec{Compress_10 from @[FIPS203, Eq (4.7)].}+ *+ * @reference{Embedded into `polyvec_compress()` in the reference+ * implementation @[REF].}+ *+ * @param u Unsigned canonical modulus modulo MLKEM_Q to be compressed.+ *+ * @return Compressed value.+ */+static MLK_INLINE uint16_t mlk_scalar_compress_d10(int16_t u)+__contract__(+  requires(0 <= u && u <= MLKEM_Q - 1)+  ensures(return_value < (1u << 10))+  ensures(return_value == (((uint32_t)u * (1u << 10) + MLKEM_Q / 2) / MLKEM_Q) % (1 << 10)))+{+  /* Compute as follows:+   * ```+   * round(u * 1024 / MLKEM_Q)+   *   = round(u * 1024 * (2^33 / MLKEM_Q) / 2^33)+   *  ~= round(u * 1024 * round(2^33 / MLKEM_Q) / 2^33)+   * ```+   */+  /* check-magic: 2642263040 == 2^10 * round(2^33 / MLKEM_Q) */+  uint64_t d0 = (uint64_t)u * 2642263040;+  d0 = (d0 + ((uint64_t)1u << 32)) >> 33; /* round(d0/2^33) */+  return (d0 & 0x3FF);+}+#ifdef CBMC+#pragma CPROVER check pop+#endif++/**+ * Compute round(u * MLKEM_Q / 1024).+ *+ * @spec{Decompress_10 from @[FIPS203, Eq (4.8)].}+ *+ * @reference{Embedded into `polyvec_decompress()` in the reference+ * implementation @[REF].}+ *+ * @param u Unsigned canonical modulus modulo 1024 to be decompressed.+ *+ * @return Decompressed value.+ */+static MLK_INLINE int16_t mlk_scalar_decompress_d10(uint16_t u)+__contract__(+  requires(0 <= u && u < 1024)+  ensures(0 <= return_value && return_value <= (MLKEM_Q - 1))+)+{+  /* The return value is in 0..MLKEM_Q-1, hence not altered by the+   * conversion to int16_t. */+  return (int16_t)((((uint32_t)u * MLKEM_Q) + 512) >> 10);+}++/*+ * The multiplication in this routine will exceed UINT32_MAX+ * and wrap around for large values of u. This is expected and required.+ */+#ifdef CBMC+#pragma CPROVER check push+#pragma CPROVER check disable "unsigned-overflow"+#endif++/**+ * Compute round(u * 2**11 / MLKEM_Q) % 2**11.+ *+ * @spec{Compress_11 from @[FIPS203, Eq (4.7)].}+ *+ * @reference{Embedded into `polyvec_compress()` in the reference+ * implementation @[REF].}+ *+ * @param u Unsigned canonical modulus modulo MLKEM_Q to be compressed.+ *+ * @return Compressed value.+ */+static MLK_INLINE uint16_t mlk_scalar_compress_d11(int16_t u)+__contract__(+  requires(0 <= u && u <= MLKEM_Q - 1)+  ensures(return_value < (1u << 11))+  ensures(return_value == (((uint32_t)u * (1u << 11) + MLKEM_Q / 2) / MLKEM_Q) % (1 << 11)))+{+  /* Compute as follows:+   * ```+   * round(u * 2048 / MLKEM_Q)+   *   = round(u * 2048 * (2^33 / MLKEM_Q) / 2^33)+   *  ~= round(u * 2048 * round(2^33 / MLKEM_Q) / 2^33)+   * ```+   */+  /* check-magic: 5284526080 == 2^11 * round(2^33 / MLKEM_Q) */+  uint64_t d0 = (uint64_t)u * 5284526080;+  d0 = (d0 + ((uint64_t)1u << 32)) >> 33; /* round(d0/2^33) */+  return (d0 & 0x7FF);+}+#ifdef CBMC+#pragma CPROVER check pop+#endif++/**+ * Compute round(u * MLKEM_Q / 2048).+ *+ * @spec{Decompress_11 from @[FIPS203, Eq (4.8)].}+ *+ * @reference{Embedded into `polyvec_decompress()` in the reference+ * implementation @[REF].}+ *+ * @param u Unsigned canonical modulus modulo 2048 to be decompressed.+ *+ * @return Decompressed value.+ */+static MLK_INLINE int16_t mlk_scalar_decompress_d11(uint16_t u)+__contract__(+  requires(0 <= u && u < 2048)+  ensures(0 <= return_value && return_value <= (MLKEM_Q - 1))+)+{+  /* The return value is in 0..MLKEM_Q-1, hence not altered by the+   * conversion to int16_t. */+  return (int16_t)((((uint32_t)u * MLKEM_Q) + 1024) >> 11);+}++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || (MLKEM_K == 2 || MLKEM_K == 3)+#define mlk_poly_compress_d4 MLK_NAMESPACE(poly_compress_d4)+/**+ * Compression (4 bits) and subsequent serialization of a polynomial.+ *+ * @spec{Implements `ByteEncode_4 (Compress_4 (a))`: ByteEncode_d+ * @[FIPS203, Algorithm 5], Compress_d @[FIPS203, Eq (4.7)], extended to+ * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays].+ * `ByteEncode_{d_v} (Compress_{d_v} (v))` appears in @[FIPS203, Algorithm+ * 14 (K-PKE.Encrypt), L23], where `d_v=4` for ML-KEM-{512,768} @[FIPS203,+ * Table 2].}+ *+ * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D4+ *               bytes).+ * @param[in]  a Input polynomial. Coefficients must be unsigned canonical,+ *               i.e. in [0,1,..,MLKEM_Q-1].+ */+MLK_INTERNAL_API+void mlk_poly_compress_d4(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D4],+                          const mlk_poly *a);++#define mlk_poly_compress_d10 MLK_NAMESPACE(poly_compress_d10)+/**+ * Compression (10 bits) and subsequent serialization of a polynomial.+ *+ * @spec{Implements `ByteEncode_10 (Compress_10 (a))`: ByteEncode_d+ * @[FIPS203, Algorithm 5], Compress_d @[FIPS203, Eq (4.7)], extended to+ * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays].+ * `ByteEncode_{d_u} (Compress_{d_u} (u))` appears in @[FIPS203, Algorithm+ * 14 (K-PKE.Encrypt), L22], where `d_u=10` for ML-KEM-{512,768} @[FIPS203,+ * Table 2].}+ *+ * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D10+ *               bytes).+ * @param[in]  a Input polynomial. Coefficients must be unsigned canonical,+ *               i.e. in [0,1,..,MLKEM_Q-1].+ */+MLK_INTERNAL_API+void mlk_poly_compress_d10(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D10],+                           const mlk_poly *a);++#if !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_poly_decompress_d4 MLK_NAMESPACE(poly_decompress_d4)+/**+ * De-serialization and subsequent decompression (4 bits) of a polynomial;+ * approximate inverse of mlk_poly_compress_d4.+ *+ * Upon return, the coefficients of the output polynomial are+ * unsigned-canonical (non-negative and smaller than MLKEM_Q).+ *+ * @spec{Implements `Decompress_4 (ByteDecode_4 (a))`: ByteDecode_d+ * @[FIPS203, Algorithm 6], Decompress_d @[FIPS203, Eq (4.8)], extended to+ * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays].+ * `Decompress_{d_v} (ByteDecode_{d_v} (v))` appears in @[FIPS203, Algorithm+ * 15 (K-PKE.Decrypt), L4], where `d_v=4` for ML-KEM-{512,768} @[FIPS203,+ * Table 2].}+ *+ * @param[out] r Output polynomial.+ * @param[in]  a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D4+ *               bytes).+ */+MLK_INTERNAL_API+void mlk_poly_decompress_d4(mlk_poly *r,+                            const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D4]);++#define mlk_poly_decompress_d10 MLK_NAMESPACE(poly_decompress_d10)+/**+ * De-serialization and subsequent decompression (10 bits) of a polynomial;+ * approximate inverse of mlk_poly_compress_d10.+ *+ * Upon return, the coefficients of the output polynomial are+ * unsigned-canonical (non-negative and smaller than MLKEM_Q).+ *+ * @spec{Implements `Decompress_10 (ByteDecode_10 (a))`: ByteDecode_d+ * @[FIPS203, Algorithm 6], Decompress_d @[FIPS203, Eq (4.8)], extended to+ * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays].+ * `Decompress_{d_u} (ByteDecode_{d_u} (u))` appears in @[FIPS203, Algorithm+ * 15 (K-PKE.Decrypt), L3], where `d_u=10` for ML-KEM-{512,768} @[FIPS203,+ * Table 2].}+ *+ * @param[out] r Output polynomial.+ * @param[in]  a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D10+ *               bytes).+ */+MLK_INTERNAL_API+void mlk_poly_decompress_d10(mlk_poly *r,+                             const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D10]);+#endif /* !MLK_CONFIG_NO_DECAPS_API */+#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 || MLKEM_K == 3 */++#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4+#define mlk_poly_compress_d5 MLK_NAMESPACE(poly_compress_d5)+/**+ * Compression (5 bits) and subsequent serialization of a polynomial.+ *+ * @spec{Implements `ByteEncode_5 (Compress_5 (a))`: ByteEncode_d+ * @[FIPS203, Algorithm 5], Compress_d @[FIPS203, Eq (4.7)], extended to+ * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays].+ * `ByteEncode_{d_v} (Compress_{d_v} (v))` appears in @[FIPS203, Algorithm+ * 14 (K-PKE.Encrypt), L23], where `d_v=5` for ML-KEM-1024 @[FIPS203,+ * Table 2].}+ *+ * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D5+ *               bytes).+ * @param[in]  a Input polynomial. Coefficients must be unsigned canonical,+ *               i.e. in [0,1,..,MLKEM_Q-1].+ */+MLK_INTERNAL_API+void mlk_poly_compress_d5(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D5],+                          const mlk_poly *a);++#define mlk_poly_compress_d11 MLK_NAMESPACE(poly_compress_d11)+/**+ * Compression (11 bits) and subsequent serialization of a polynomial.+ *+ * @spec{`ByteEncode_11 (Compress_11 (a))`: ByteEncode_d @[FIPS203,+ * Algorithm 5], Compress_d @[FIPS203, Eq (4.7)], extended to vectors as+ * per @[FIPS203, 2.4.8 Applying Algorithms to Arrays].+ * `ByteEncode_{d_u} (Compress_{d_u} (u))` appears in @[FIPS203, Algorithm+ * 14 (K-PKE.Encrypt), L22], where `d_u=11` for ML-KEM-1024 @[FIPS203,+ * Table 2].}+ *+ * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D11+ *               bytes).+ * @param[in]  a Input polynomial. Coefficients must be unsigned canonical,+ *               i.e. in [0,1,..,MLKEM_Q-1].+ */+MLK_INTERNAL_API+void mlk_poly_compress_d11(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D11],+                           const mlk_poly *a);++#if !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_poly_decompress_d5 MLK_NAMESPACE(poly_decompress_d5)+/**+ * De-serialization and subsequent decompression (5 bits) of a polynomial;+ * approximate inverse of mlk_poly_compress_d5.+ *+ * Upon return, the coefficients of the output polynomial are+ * unsigned-canonical (non-negative and smaller than MLKEM_Q).+ *+ * @spec{Implements `Decompress_5 (ByteDecode_5 (a))`: ByteDecode_d+ * @[FIPS203, Algorithm 6], Decompress_d @[FIPS203, Eq (4.8)], extended to+ * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays].+ * `Decompress_{d_v} (ByteDecode_{d_v} (v))` appears in @[FIPS203, Algorithm+ * 15 (K-PKE.Decrypt), L4], where `d_v=5` for ML-KEM-1024 @[FIPS203,+ * Table 2].}+ *+ * @param[out] r Output polynomial.+ * @param[in]  a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D5+ *               bytes).+ */+MLK_INTERNAL_API+void mlk_poly_decompress_d5(mlk_poly *r,+                            const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D5]);++#define mlk_poly_decompress_d11 MLK_NAMESPACE(poly_decompress_d11)+/**+ * De-serialization and subsequent decompression (11 bits) of a polynomial;+ * approximate inverse of mlk_poly_compress_d11.+ *+ * Upon return, the coefficients of the output polynomial are+ * unsigned-canonical (non-negative and smaller than MLKEM_Q).+ *+ * @spec{Implements `Decompress_11 (ByteDecode_11 (a))`: ByteDecode_d+ * @[FIPS203, Algorithm 6], Decompress_d @[FIPS203, Eq (4.8)], extended to+ * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays].+ * `Decompress_{d_u} (ByteDecode_{d_u} (u))` appears in @[FIPS203, Algorithm+ * 15 (K-PKE.Decrypt), L3], where `d_u=11` for ML-KEM-1024 @[FIPS203,+ * Table 2].}+ *+ * @param[out] r Output polynomial.+ * @param[in]  a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D11+ *               bytes).+ */+MLK_INTERNAL_API+void mlk_poly_decompress_d11(mlk_poly *r,+                             const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D11]);+#endif /* !MLK_CONFIG_NO_DECAPS_API */+#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4 */+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++#if !defined(MLK_CONFIG_NO_KEYPAIR_API) || !defined(MLK_CONFIG_NO_ENCAPS_API)+#define mlk_poly_tobytes MLK_NAMESPACE(poly_tobytes)+/**+ * Serialization of a polynomial. Signed coefficients are converted to+ * unsigned form before serialization.+ *+ * @spec{Implements ByteEncode_12 @[FIPS203, Algorithm 5]. Extended to+ * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays].}+ *+ * @param[out] r Output byte array (of MLKEM_POLYBYTES bytes).+ * @param[in]  a Input polynomial, with each coefficient in the range+ *               [0,1,..,MLKEM_Q-1].+ */+MLK_INTERNAL_API+void mlk_poly_tobytes(uint8_t r[MLKEM_POLYBYTES], const mlk_poly *a)+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYBYTES))+  requires(memory_no_alias(a, sizeof(mlk_poly)))+  requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYBYTES))+);+#endif /* !MLK_CONFIG_NO_KEYPAIR_API || !MLK_CONFIG_NO_ENCAPS_API */+++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_poly_frombytes MLK_NAMESPACE(poly_frombytes)+/**+ * De-serialization of a polynomial.+ *+ * @spec{Implements ByteDecode_12 @[FIPS203, Algorithm 6]. Extended to+ * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays].}+ *+ * @param[out] r Output polynomial, with each coefficient unsigned and in+ *               the range 0..4095.+ * @param[in]  a Input byte array (of MLKEM_POLYBYTES bytes).+ */+MLK_INTERNAL_API+void mlk_poly_frombytes(mlk_poly *r, const uint8_t a[MLKEM_POLYBYTES])+__contract__(+  requires(memory_no_alias(a, MLKEM_POLYBYTES))+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT))+);+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */+++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_poly_frommsg MLK_NAMESPACE(poly_frommsg)+/**+ * Convert a 32-byte message to a polynomial.+ *+ * @spec{Implements `Decompress_1 (ByteDecode_1 (a))`: ByteDecode_d+ * @[FIPS203, Algorithm 6], Decompress_d @[FIPS203, Eq (4.8)], extended to+ * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays].+ * `Decompress_1 (ByteDecode_1 (w))` appears in @[FIPS203, Algorithm 15+ * (K-PKE.Encrypt), L20].}+ *+ * @param[out] r   Output polynomial.+ * @param[in]  msg Input message.+ */+MLK_INTERNAL_API+void mlk_poly_frommsg(mlk_poly *r, const uint8_t msg[MLKEM_INDCPA_MSGBYTES])+__contract__(+  requires(memory_no_alias(msg, MLKEM_INDCPA_MSGBYTES))+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+);+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++#if !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_poly_tomsg MLK_NAMESPACE(poly_tomsg)+/**+ * Convert a polynomial to a 32-byte message.+ *+ * @spec{Implements `ByteEncode_1 (Compress_1 (a))`: ByteEncode_d+ * @[FIPS203, Algorithm 5], Compress_d @[FIPS203, Eq (4.7)], extended to+ * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays].+ * `ByteEncode_1 (Compress_1 (w))` appears in @[FIPS203, Algorithm 14+ * (K-PKE.Decrypt), L7].}+ *+ * @param[out] msg Output message.+ * @param[in]  r   Input polynomial. Coefficients must be unsigned canonical.+ */+MLK_INTERNAL_API+void mlk_poly_tomsg(uint8_t msg[MLKEM_INDCPA_MSGBYTES], const mlk_poly *r)+__contract__(+  requires(memory_no_alias(msg, MLKEM_INDCPA_MSGBYTES))+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  requires(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(msg, MLKEM_INDCPA_MSGBYTES))+);+#endif /* !MLK_CONFIG_NO_DECAPS_API */++#endif /* !MLK_COMPRESS_H */
+ cbits/mlkem/src/context.h view
@@ -0,0 +1,51 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLK_CONTEXT_H+#define MLK_CONTEXT_H++/* This header is included by common.h once the configuration has been pulled+ * in; it is not meant to be included directly. */++/*+ * If the integration wants to provide a context parameter for use in+ * platform-specific hooks, then it should define this parameter.+ *+ * The MLK_CONTEXT_PARAMETERS_n macros are intended to be used with macros+ * defining the function names and expand to either pass or discard the context+ * argument as required by the current build.  If there is no context parameter+ * requested then these are removed from the prototypes and from all calls.+ */+#ifdef MLK_CONFIG_CONTEXT_PARAMETER+#define MLK_CONTEXT_PARAMETERS_0(context) (context)+#define MLK_CONTEXT_PARAMETERS_1(arg0, context) (arg0, context)+#define MLK_CONTEXT_PARAMETERS_2(arg0, arg1, context) (arg0, arg1, context)+#define MLK_CONTEXT_PARAMETERS_3(arg0, arg1, arg2, context) \+  (arg0, arg1, arg2, context)+#define MLK_CONTEXT_PARAMETERS_4(arg0, arg1, arg2, arg3, context) \+  (arg0, arg1, arg2, arg3, context)+#else /* MLK_CONFIG_CONTEXT_PARAMETER */+#define MLK_CONTEXT_PARAMETERS_0(context) ()+#define MLK_CONTEXT_PARAMETERS_1(arg0, context) (arg0)+#define MLK_CONTEXT_PARAMETERS_2(arg0, arg1, context) (arg0, arg1)+#define MLK_CONTEXT_PARAMETERS_3(arg0, arg1, arg2, context) (arg0, arg1, arg2)+#define MLK_CONTEXT_PARAMETERS_4(arg0, arg1, arg2, arg3, context) \+  (arg0, arg1, arg2, arg3)+#endif /* !MLK_CONFIG_CONTEXT_PARAMETER */++/* Consume a context parameter carried only for the integration's benefit,+ * avoiding -Wunused-parameter; expands to nothing when no context is+ * configured. */+#if defined(MLK_CONFIG_CONTEXT_PARAMETER)+#define MLK_CONTEXT_UNUSED(context) ((void)(context))+#else+#define MLK_CONTEXT_UNUSED(context) ((void)0)+#endif++#if defined(MLK_CONFIG_CONTEXT_PARAMETER_TYPE) != \+    defined(MLK_CONFIG_CONTEXT_PARAMETER)+#error MLK_CONFIG_CONTEXT_PARAMETER_TYPE must be defined if and only if MLK_CONFIG_CONTEXT_PARAMETER is defined+#endif++#endif /* !MLK_CONTEXT_H */
+ cbits/mlkem/src/debug.c view
@@ -0,0 +1,64 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* NOTE: You can remove this file unless you compile with MLKEM_DEBUG. */++#include "common.h"++#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && defined(MLKEM_DEBUG)+++#include <stdio.h>+#include <stdlib.h>+#include "debug.h"++#define MLK_DEBUG_ERROR_HEADER "[ERROR:%s:%04d] "++void mlk_debug_check_assert(const char *file, int line, const int val)+{+  if (val == 0)+  {+    fprintf(stderr, MLK_DEBUG_ERROR_HEADER "Assertion failed (value %d)\n",+            file, line, val);+    exit(1);+  }+}++void mlk_debug_check_bounds(const char *file, int line, const int16_t *ptr,+                            unsigned len, int lower_bound_exclusive,+                            int upper_bound_exclusive)+{+  int err = 0;+  unsigned i;+  for (i = 0; i < len; i++)+  {+    int16_t val = ptr[i];+    if (!(val > lower_bound_exclusive && val < upper_bound_exclusive))+    {+      fprintf(+          stderr,+          MLK_DEBUG_ERROR_HEADER+          "Bounds assertion failed: Index %u, value %d out of bounds (%d,%d)\n",+          file, line, i, (int)val, lower_bound_exclusive,+          upper_bound_exclusive);+      err = 1;+    }+  }++  if (err == 1)+  {+    exit(1);+  }+}++#else /* !MLK_CONFIG_MULTILEVEL_NO_SHARED && MLKEM_DEBUG */++MLK_EMPTY_CU(debug)++#endif /* !(!MLK_CONFIG_MULTILEVEL_NO_SHARED && MLKEM_DEBUG) */++/* To facilitate single-compilation-unit (SCU) builds, undefine all macros.+ * Don't modify by hand -- this is auto-generated by scripts/autogen. */+#undef MLK_DEBUG_ERROR_HEADER
+ cbits/mlkem/src/debug.h view
@@ -0,0 +1,121 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLK_DEBUG_H+#define MLK_DEBUG_H+#include "common.h"++#if defined(MLKEM_DEBUG)++/**+ * Check debug assertion.+ *+ * Prints an error message to stderr and calls exit(1) on failure.+ *+ * @param[in] file Filename.+ * @param     line Line number.+ * @param     val  Value asserted to be non-zero.+ */+#define mlk_debug_check_assert MLK_NAMESPACE(mlkem_debug_assert)+void mlk_debug_check_assert(const char *file, int line, const int val);++/**+ * Check whether values in an array of int16_t are within specified bounds.+ *+ * Prints an error message to stderr and calls exit(1) on failure.+ *+ * @param[in] file                  Filename.+ * @param     line                  Line number.+ * @param[in] ptr                   Base of array to be checked.+ * @param     len                   Number of int16_t in @p ptr.+ * @param     lower_bound_exclusive Exclusive lower bound.+ * @param     upper_bound_exclusive Exclusive upper bound.+ */+#define mlk_debug_check_bounds MLK_NAMESPACE(mlkem_debug_check_bounds)+void mlk_debug_check_bounds(const char *file, int line, const int16_t *ptr,+                            unsigned len, int lower_bound_exclusive,+                            int upper_bound_exclusive);++/* Check assertion, calling exit() upon failure+ *+ * val: Value that's asserted to be non-zero+ */+#define mlk_assert(val) mlk_debug_check_assert(__FILE__, __LINE__, (val))++/* Check bounds in array of int16_t's+ * ptr: Base of int16_t array; will be explicitly cast to int16_t*,+ *      so you may pass a byte-compatible type such as mlk_poly or mlk_polyvec.+ * len: Number of int16_t in array+ * value_lb: Inclusive lower value bound+ * value_ub: Exclusive upper value bound */+#define mlk_assert_bound(ptr, len, value_lb, value_ub)                      \+  mlk_debug_check_bounds(__FILE__, __LINE__, (const int16_t *)(ptr), (len), \+                         (value_lb) - 1, (value_ub))++/* Check absolute bounds in array of int16_t's+ * ptr: Base of array, expression of type int16_t*+ * len: Number of int16_t in array+ * value_abs_bd: Exclusive absolute upper bound */+#define mlk_assert_abs_bound(ptr, len, value_abs_bd) \+  mlk_assert_bound((ptr), (len), (-(value_abs_bd) + 1), (value_abs_bd))++/* Version of bounds assertions for 2-dimensional arrays */+#define mlk_assert_bound_2d(ptr, len0, len1, value_lb, value_ub) \+  mlk_assert_bound((ptr), ((len0) * (len1)), (value_lb), (value_ub))++#define mlk_assert_abs_bound_2d(ptr, len0, len1, value_abs_bd) \+  mlk_assert_abs_bound((ptr), ((len0) * (len1)), (value_abs_bd))++/* When running CBMC, convert debug assertions into proof obligations */+#elif defined(CBMC)+#include "cbmc.h"++#define mlk_assert(val) cassert(val)++#define mlk_assert_bound(ptr, len, value_lb, value_ub) \+  cassert(array_bound(((int16_t *)(ptr)), 0, (len), (value_lb), (value_ub)))++#define mlk_assert_abs_bound(ptr, len, value_abs_bd) \+  cassert(array_abs_bound(((int16_t *)(ptr)), 0, (len), (value_abs_bd)))++/* Because of https://github.com/diffblue/cbmc/issues/8570, we can't+ * just use a single flattened array_bound(...) here. */+#define mlk_assert_bound_2d(ptr, M, N, value_lb, value_ub)              \+  cassert(forall(kN, 0, (M),                                            \+                 array_bound(&((int16_t (*)[(N)])(ptr))[kN][0], 0, (N), \+                             (value_lb), (value_ub))))++#define mlk_assert_abs_bound_2d(ptr, M, N, value_abs_bd)                    \+  cassert(forall(kN, 0, (M),                                                \+                 array_abs_bound(&((int16_t (*)[(N)])(ptr))[kN][0], 0, (N), \+                                 (value_abs_bd))))++#else /* !MLKEM_DEBUG && CBMC */++#define mlk_assert(val) \+  do                    \+  {                     \+  } while (0)+#define mlk_assert_bound(ptr, len, value_lb, value_ub) \+  do                                                   \+  {                                                    \+  } while (0)+#define mlk_assert_abs_bound(ptr, len, value_abs_bd) \+  do                                                 \+  {                                                  \+  } while (0)++#define mlk_assert_bound_2d(ptr, len0, len1, value_lb, value_ub) \+  do                                                             \+  {                                                              \+  } while (0)++#define mlk_assert_abs_bound_2d(ptr, len0, len1, value_abs_bd) \+  do                                                           \+  {                                                            \+  } while (0)+++#endif /* !MLKEM_DEBUG && !CBMC */+#endif /* !MLK_DEBUG_H */
+ cbits/mlkem/src/fips202/fips202.c view
@@ -0,0 +1,249 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ *+ * - [mupq]+ *   Common files for pqm4, pqm3, pqriscv+ *   Kannwischer, Petri, Rijneveld, Schwabe, Stoffelen+ *   https://github.com/mupq/mupq+ *+ * - [supercop]+ *   SUPERCOP benchmarking framework+ *   Daniel J. Bernstein+ *   http://bench.cr.yp.to/supercop.html+ *+ * - [tweetfips]+ *   'tweetfips202' FIPS202 implementation+ *   Van Assche, Bernstein, Schwabe+ *   https://keccak.team/2015/tweetfips202.html+ */++/* Based on the CC0 implementation from @[mupq] and the public domain+ * implementation @[supercop, crypto_hash/keccakc512/simple/]+ * by Ronny Van Keer, and the public domain @[tweetfips] implementation. */++#include "../common.h"+#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)+++#include "../verify.h"+#include "fips202.h"+#include "keccakf1600.h"++/**+ * Absorb step of Keccak; non-incremental, starts by zeroeing the state.+ *+ * @warning Must only be called once.+ *+ * @param[out] s    Pointer to (uninitialized) output Keccak state.+ * @param      r    Rate in bytes (e.g., 168 for SHAKE128).+ * @param[in]  m    Input to be absorbed into @p s.+ * @param      mlen Length of input in bytes.+ * @param      p    Domain-separation byte for different Keccak-derived+ *                  functions.+ */+static void mlk_keccak_absorb_once(uint64_t *s, unsigned r, const uint8_t *m,+                                   size_t mlen, uint8_t p)+__contract__(+    requires(mlen <= MLK_MAX_BUFFER_SIZE)+    requires(r > 0)+    requires(r <= sizeof(uint64_t) * MLK_KECCAK_LANES)+    requires(memory_no_alias(s, sizeof(uint64_t) * MLK_KECCAK_LANES))+    requires(memory_no_alias(m, mlen))+    assigns(memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES)))+{+  /* Initialize state */+  size_t i;+  for (i = 0; i < 25; ++i)+  __loop__(invariant(i <= 25)+           decreases(25 - i))+  {+    s[i] = 0;+  }++  while (mlen >= r)+  __loop__(+    assigns(mlen, m, memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES))+    invariant(mlen <= loop_entry(mlen))+    invariant(m == loop_entry(m) + (loop_entry(mlen) - mlen))+    decreases(mlen))+  {+    mlk_keccakf1600_xor_bytes(s, m, 0, r);+    mlk_keccakf1600_permute(s);+    mlen -= r;+    m += r;+  }++  /* At this point, mlen < r, so the truncations to unsigned are safe below. */++  if (mlen > 0)+  {+    mlk_keccakf1600_xor_bytes(s, m, 0, (unsigned int)mlen);+  }++  if (mlen == r - 1)+  {+    p |= 128;+    mlk_keccakf1600_xor_bytes(s, &p, (unsigned int)mlen, 1);+  }+  else+  {+    mlk_keccakf1600_xor_bytes(s, &p, (unsigned int)mlen, 1);+    p = 128;+    mlk_keccakf1600_xor_bytes(s, &p, r - 1, 1);+  }+}++/**+ * Block-level Keccak squeeze.+ *+ * @param[out]    h       Output bytes.+ * @param         nblocks Number of blocks to be squeezed.+ * @param[in,out] s       Input/output state.+ * @param         r       Rate in bytes (e.g., 168 for SHAKE128).+ */+static void mlk_keccak_squeezeblocks(uint8_t *h, size_t nblocks, uint64_t *s,+                                     unsigned r)+__contract__(+    requires(r <= sizeof(uint64_t) * MLK_KECCAK_LANES)+    requires(nblocks <= 8 /* somewhat arbitrary bound */)+    requires(memory_no_alias(s, sizeof(uint64_t) * MLK_KECCAK_LANES))+    requires(memory_no_alias(h, nblocks * r))+    assigns(memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES))+    assigns(memory_slice(h, nblocks * r)))+{+  while (nblocks > 0)+  __loop__(+    assigns(h, nblocks,+      memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES),+      memory_slice(h, nblocks * r))+    invariant(nblocks <= loop_entry(nblocks) &&+      h == loop_entry(h) + r * (loop_entry(nblocks) - nblocks))+    decreases(nblocks))+  {+    mlk_keccakf1600_permute(s);+    mlk_keccakf1600_extract_bytes(s, h, 0, r);+    h += r;+    nblocks--;+  }+}++/**+ * Keccak squeeze; can be called on byte-level.+ *+ * @warning Must only be called once.+ *+ * @param[out]    h      Output bytes.+ * @param         outlen Number of bytes to be squeezed.+ * @param[in,out] s      Keccak state.+ * @param         r      Rate in bytes (e.g., 168 for SHAKE128).+ */+static void mlk_keccak_squeeze_once(uint8_t *h, size_t outlen, uint64_t *s,+                                    unsigned r)+__contract__(+    requires(outlen <= MLK_MAX_BUFFER_SIZE)+    requires(r > 0)+    requires(r <= sizeof(uint64_t) * MLK_KECCAK_LANES)+    requires(memory_no_alias(s, sizeof(uint64_t) * MLK_KECCAK_LANES))+    requires(memory_no_alias(h, outlen))+    assigns(memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES))+    assigns(memory_slice(h, outlen)))+{+  size_t len;+  while (outlen > 0)+  __loop__(+    assigns(len, h, outlen,+      memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES),+      memory_slice(h, outlen))+    invariant(outlen <= loop_entry(outlen) &&+      h == loop_entry(h) + (loop_entry(outlen) - outlen))+    decreases(outlen))+  {+    mlk_keccakf1600_permute(s);++    if (outlen < r)+    {+      len = outlen;+    }+    else+    {+      len = r;+    }+    mlk_keccakf1600_extract_bytes(s, h, 0, (unsigned int)len);+    h += len;+    outlen -= len;+  }+}++void mlk_shake128_absorb_once(mlk_shake128ctx *state, const uint8_t *input,+                              size_t inlen)+{+  mlk_keccak_absorb_once(state->ctx, SHAKE128_RATE, input, inlen, 0x1F);+}++void mlk_shake128_squeezeblocks(uint8_t *output, size_t nblocks,+                                mlk_shake128ctx *state)+{+  mlk_keccak_squeezeblocks(output, nblocks, state->ctx, SHAKE128_RATE);+}++void mlk_shake128_init(mlk_shake128ctx *state) { (void)state; }+void mlk_shake128_release(mlk_shake128ctx *state)+{+  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  mlk_zeroize(state, sizeof(mlk_shake128ctx));+}++typedef mlk_shake128ctx mlk_shake256ctx;+void mlk_shake256(uint8_t *output, size_t outlen, const uint8_t *input,+                  size_t inlen)+{+  mlk_shake256ctx state;+  /* Absorb input */+  mlk_keccak_absorb_once(state.ctx, SHAKE256_RATE, input, inlen, 0x1F);+  /* Squeeze output */+  mlk_keccak_squeeze_once(output, outlen, state.ctx, SHAKE256_RATE);+  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  mlk_zeroize(&state, sizeof(state));+}++void mlk_sha3_256(uint8_t *output, const uint8_t *input, size_t inlen)+{+  uint64_t ctx[25];+  /* Absorb input */+  mlk_keccak_absorb_once(ctx, SHA3_256_RATE, input, inlen, 0x06);+  /* Squeeze output */+  mlk_keccak_squeeze_once(output, 32, ctx, SHA3_256_RATE);+  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  mlk_zeroize(ctx, sizeof(ctx));+}++void mlk_sha3_512(uint8_t *output, const uint8_t *input, size_t inlen)+{+  uint64_t ctx[25];+  /* Absorb input */+  mlk_keccak_absorb_once(ctx, SHA3_512_RATE, input, inlen, 0x06);+  /* Squeeze output */+  mlk_keccak_squeeze_once(output, 64, ctx, SHA3_512_RATE);+  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  mlk_zeroize(ctx, sizeof(ctx));+}++#else /* !MLK_CONFIG_MULTILEVEL_NO_SHARED */++MLK_EMPTY_CU(fips202)++#endif /* MLK_CONFIG_MULTILEVEL_NO_SHARED */
+ cbits/mlkem/src/fips202/fips202.h view
@@ -0,0 +1,144 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLK_FIPS202_FIPS202_H+#define MLK_FIPS202_FIPS202_H++#include "../cbmc.h"+#include "../common.h"++#define SHAKE128_RATE 168+#define SHAKE256_RATE 136+#define SHA3_256_RATE 136+#define SHA3_384_RATE 104+#define SHA3_512_RATE 72++/** Context for the non-incremental SHAKE128 API. */+typedef struct+{+  uint64_t ctx[25]; /**< Keccak state. */+} MLK_ALIGN mlk_shake128ctx;++#define mlk_shake128_absorb_once MLK_NAMESPACE(shake128_absorb_once)+/**+ * One-shot absorb step of the SHAKE128 XOF.+ *+ * For call-sites (in mlkem-native):+ * - This function MUST ONLY be called straight after mlk_shake128_init().+ * - This function MUST ONLY be called once.+ *+ * Consequently, for providers of custom FIPS202 code to be used with+ * mlkem-native:+ * - You may assume that the input context is freshly initialized via+ *   mlk_shake128_init().+ * - You may assume that this function is called exactly once.+ *+ * @param[in,out] state SHAKE128 context.+ * @param[in]     input Input to be absorbed into the state.+ * @param         inlen Length of input in bytes.+ */+void mlk_shake128_absorb_once(mlk_shake128ctx *state, const uint8_t *input,+                              size_t inlen)+__contract__(+  requires(inlen <= MLK_MAX_BUFFER_SIZE)+  requires(memory_no_alias(state, sizeof(mlk_shake128ctx)))+  requires(memory_no_alias(input, inlen))+  assigns(memory_slice(state, sizeof(mlk_shake128ctx)))+);++#define mlk_shake128_squeezeblocks MLK_NAMESPACE(shake128_squeezeblocks)+/**+ * Squeeze step of SHAKE128 XOF. Squeezes full blocks of SHAKE128_RATE bytes+ * each. Modifies the state. Can be called multiple times to keep squeezing,+ * i.e., is incremental.+ *+ * @param[out]    output  Output blocks.+ * @param         nblocks Number of blocks to be squeezed (written to output).+ * @param[in,out] state   Keccak state.+ */+void mlk_shake128_squeezeblocks(uint8_t *output, size_t nblocks,+                                mlk_shake128ctx *state)+__contract__(+  requires(nblocks <= 8 /* somewhat arbitrary bound */)+  requires(memory_no_alias(state, sizeof(mlk_shake128ctx)))+  requires(memory_no_alias(output, nblocks * SHAKE128_RATE))+  assigns(memory_slice(output, nblocks * SHAKE128_RATE), memory_slice(state, sizeof(mlk_shake128ctx)))+);++#define mlk_shake128_init MLK_NAMESPACE(shake128_init)+void mlk_shake128_init(mlk_shake128ctx *state);++#define mlk_shake128_release MLK_NAMESPACE(shake128_release)+void mlk_shake128_release(mlk_shake128ctx *state);++/* One-stop SHAKE256 call. Aliasing between input and+ * output is not permitted */+#define mlk_shake256 MLK_NAMESPACE(shake256)+/**+ * SHAKE256 XOF with non-incremental API.+ *+ * @param[out] output Output buffer.+ * @param      outlen Requested output length in bytes.+ * @param[in]  input  Input buffer.+ * @param      inlen  Length of input in bytes.+ */+void mlk_shake256(uint8_t *output, size_t outlen, const uint8_t *input,+                  size_t inlen)+__contract__(+  requires(inlen <= MLK_MAX_BUFFER_SIZE)+  requires(outlen <= MLK_MAX_BUFFER_SIZE)+  requires(memory_no_alias(input, inlen))+  requires(memory_no_alias(output, outlen))+  assigns(memory_slice(output, outlen))+);++/* One-stop SHA3_256 call. Aliasing between input and+ * output is not permitted */+#define SHA3_256_HASHBYTES 32+#define mlk_sha3_256 MLK_NAMESPACE(sha3_256)+/**+ * SHA3-256 with non-incremental API.+ *+ * @param[out] output Output buffer.+ * @param[in]  input  Input buffer.+ * @param      inlen  Length of input in bytes.+ */+void mlk_sha3_256(uint8_t *output, const uint8_t *input, size_t inlen)+__contract__(+  requires(inlen <= MLK_MAX_BUFFER_SIZE)+  requires(memory_no_alias(input, inlen))+  requires(memory_no_alias(output, SHA3_256_HASHBYTES))+  assigns(memory_slice(output, SHA3_256_HASHBYTES))+);++/* One-stop SHA3_512 call. Aliasing between input and+ * output is not permitted */+#define SHA3_512_HASHBYTES 64+#define mlk_sha3_512 MLK_NAMESPACE(sha3_512)+/**+ * SHA3-512 with non-incremental API.+ *+ * @param[out] output Output buffer.+ * @param[in]  input  Input buffer.+ * @param      inlen  Length of input in bytes.+ */+void mlk_sha3_512(uint8_t *output, const uint8_t *input, size_t inlen)+__contract__(+  requires(inlen <= MLK_MAX_BUFFER_SIZE)+  requires(memory_no_alias(input, inlen))+  requires(memory_no_alias(output, SHA3_512_HASHBYTES))+  assigns(memory_slice(output, SHA3_512_HASHBYTES))+);++#if !defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202) || \+    !defined(MLK_USE_NATIVE_FIPS202_X4)+/* If you provide your own FIPS-202 implementation where the x4-+ * Keccak-f1600-x4 implementation falls back to 4-fold Keccak-f1600,+ * set this to gain a small speedup. */+#define FIPS202_X4_DEFAULT_IMPLEMENTATION+#endif /* !MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 || !MLK_USE_NATIVE_FIPS202_X4 \+        */+++#endif /* !MLK_FIPS202_FIPS202_H */
+ cbits/mlkem/src/fips202/fips202x4.c view
@@ -0,0 +1,207 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ */++#include "../common.h"+#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++#include "../verify.h"+#include "fips202.h"+#include "fips202x4.h"+#include "keccakf1600.h"++typedef mlk_shake128x4ctx mlk_shake256x4_ctx;++static void mlk_keccak_absorb_once_x4(uint64_t *s, unsigned r,+                                      const uint8_t *in0, const uint8_t *in1,+                                      const uint8_t *in2, const uint8_t *in3,+                                      size_t inlen, uint8_t p)+__contract__(+  requires(inlen <= MLK_MAX_BUFFER_SIZE)+  requires(memory_no_alias(s, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY))+  requires(r > 0)+  requires(r <= sizeof(uint64_t) * MLK_KECCAK_LANES)+  requires(memory_no_alias(in0, inlen))+  requires(memory_no_alias(in1, inlen))+  requires(memory_no_alias(in2, inlen))+  requires(memory_no_alias(in3, inlen))+  assigns(memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY)))+{+  while (inlen >= r)+  __loop__(+    assigns(inlen, in0, in1, in2, in3, memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY))+    invariant(inlen <= loop_entry(inlen))+    invariant(in0 == loop_entry(in0) + (loop_entry(inlen) - inlen))+    invariant(in1 == loop_entry(in1) + (loop_entry(inlen) - inlen))+    invariant(in2 == loop_entry(in2) + (loop_entry(inlen) - inlen))+    invariant(in3 == loop_entry(in3) + (loop_entry(inlen) - inlen))+    decreases(inlen))+  {+    mlk_keccakf1600x4_xor_bytes(s, in0, in1, in2, in3, 0, r);+    mlk_keccakf1600x4_permute(s);++    in0 += r;+    in1 += r;+    in2 += r;+    in3 += r;+    inlen -= r;+  }++  /* At this point, inlen < r, so the truncations to unsigned are safe below. */++  if (inlen > 0)+  {+    mlk_keccakf1600x4_xor_bytes(s, in0, in1, in2, in3, 0, (unsigned int)inlen);+  }++  if (inlen == r - 1)+  {+    p |= 128;+    mlk_keccakf1600x4_xor_bytes(s, &p, &p, &p, &p, (unsigned int)inlen, 1);+  }+  else+  {+    mlk_keccakf1600x4_xor_bytes(s, &p, &p, &p, &p, (unsigned int)inlen, 1);+    p = 128;+    mlk_keccakf1600x4_xor_bytes(s, &p, &p, &p, &p, r - 1, 1);+  }+}++static void mlk_keccak_squeezeblocks_x4(uint8_t *out0, uint8_t *out1,+                                        uint8_t *out2, uint8_t *out3,+                                        size_t nblocks, uint64_t *s, unsigned r)+__contract__(+    requires(r <= sizeof(uint64_t) * MLK_KECCAK_LANES)+    requires(r == SHAKE128_RATE || r == SHAKE256_RATE)+    requires(nblocks <= (MLK_MAX_BUFFER_SIZE / SHAKE256_RATE))+    requires(memory_no_alias(s, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY))+    requires(memory_no_alias(out0, nblocks * r))+    requires(memory_no_alias(out1, nblocks * r))+    requires(memory_no_alias(out2, nblocks * r))+    requires(memory_no_alias(out3, nblocks * r))+    assigns(memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY))+    assigns(memory_slice(out0, nblocks * r))+    assigns(memory_slice(out1, nblocks * r))+    assigns(memory_slice(out2, nblocks * r))+    assigns(memory_slice(out3, nblocks * r)))+{+  size_t current_offset = 0;+  while (nblocks > 0)+  __loop__(+    assigns(nblocks, current_offset,+            memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY),+            memory_slice(out0, nblocks * r),+            memory_slice(out1, nblocks * r),+            memory_slice(out2, nblocks * r),+            memory_slice(out3, nblocks * r))+    invariant(nblocks <= loop_entry(nblocks))+    invariant(current_offset == (loop_entry(nblocks) - nblocks) * r)+    decreases(nblocks))+  {+    mlk_keccakf1600x4_permute(s);+    mlk_keccakf1600x4_extract_bytes(+        s, &out0[current_offset], &out1[current_offset], &out2[current_offset],+        &out3[current_offset], 0, r);+    current_offset += r;+    nblocks--;+  }+}++void mlk_shake128x4_absorb_once(mlk_shake128x4ctx *state, const uint8_t *in0,+                                const uint8_t *in1, const uint8_t *in2,+                                const uint8_t *in3, size_t inlen)+{+  mlk_memset(state, 0, sizeof(mlk_shake128x4ctx));+  mlk_keccak_absorb_once_x4(state->ctx, SHAKE128_RATE, in0, in1, in2, in3,+                            inlen, 0x1F);+}++void mlk_shake128x4_squeezeblocks(uint8_t *out0, uint8_t *out1, uint8_t *out2,+                                  uint8_t *out3, size_t nblocks,+                                  mlk_shake128x4ctx *state)+{+  mlk_keccak_squeezeblocks_x4(out0, out1, out2, out3, nblocks, state->ctx,+                              SHAKE128_RATE);+}++void mlk_shake128x4_init(mlk_shake128x4ctx *state) { (void)state; }+void mlk_shake128x4_release(mlk_shake128x4ctx *state)+{+  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  mlk_zeroize(state, sizeof(mlk_shake128x4ctx));+}++static void mlk_shake256x4_absorb_once(mlk_shake256x4_ctx *state,+                                       const uint8_t *in0, const uint8_t *in1,+                                       const uint8_t *in2, const uint8_t *in3,+                                       size_t inlen)+{+  mlk_memset(state, 0, sizeof(mlk_shake128x4ctx));+  mlk_keccak_absorb_once_x4(state->ctx, SHAKE256_RATE, in0, in1, in2, in3,+                            inlen, 0x1F);+}++static void mlk_shake256x4_squeezeblocks(uint8_t *out0, uint8_t *out1,+                                         uint8_t *out2, uint8_t *out3,+                                         size_t nblocks,+                                         mlk_shake256x4_ctx *state)+{+  mlk_keccak_squeezeblocks_x4(out0, out1, out2, out3, nblocks, state->ctx,+                              SHAKE256_RATE);+}++void mlk_shake256x4(uint8_t *out0, uint8_t *out1, uint8_t *out2, uint8_t *out3,+                    size_t outlen, const uint8_t *in0, const uint8_t *in1,+                    const uint8_t *in2, const uint8_t *in3, size_t inlen)+{+  mlk_shake256x4_ctx statex;+  size_t nblocks = outlen / SHAKE256_RATE;+  uint8_t tmp0[SHAKE256_RATE];+  uint8_t tmp1[SHAKE256_RATE];+  uint8_t tmp2[SHAKE256_RATE];+  uint8_t tmp3[SHAKE256_RATE];++  mlk_shake256x4_absorb_once(&statex, in0, in1, in2, in3, inlen);+  mlk_shake256x4_squeezeblocks(out0, out1, out2, out3, nblocks, &statex);++  out0 += nblocks * SHAKE256_RATE;+  out1 += nblocks * SHAKE256_RATE;+  out2 += nblocks * SHAKE256_RATE;+  out3 += nblocks * SHAKE256_RATE;++  outlen -= nblocks * SHAKE256_RATE;++  if (outlen)+  {+    mlk_shake256x4_squeezeblocks(tmp0, tmp1, tmp2, tmp3, 1, &statex);+    mlk_memcpy(out0, tmp0, outlen);+    mlk_memcpy(out1, tmp1, outlen);+    mlk_memcpy(out2, tmp2, outlen);+    mlk_memcpy(out3, tmp3, outlen);+  }++  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  mlk_zeroize(&statex, sizeof(statex));+  mlk_zeroize(tmp0, sizeof(tmp0));+  mlk_zeroize(tmp1, sizeof(tmp1));+  mlk_zeroize(tmp2, sizeof(tmp2));+  mlk_zeroize(tmp3, sizeof(tmp3));+}++#else /* !MLK_CONFIG_MULTILEVEL_NO_SHARED */++MLK_EMPTY_CU(fips202x4)++#endif /* MLK_CONFIG_MULTILEVEL_NO_SHARED */
+ cbits/mlkem/src/fips202/fips202x4.h view
@@ -0,0 +1,81 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLK_FIPS202_FIPS202X4_H+#define MLK_FIPS202_FIPS202X4_H+++#include "../cbmc.h"+#include "../common.h"++#include "fips202.h"+#include "keccakf1600.h"++/** Context for the non-incremental 4-way SHAKE128 API. */+typedef struct+{+  uint64_t ctx[MLK_KECCAK_LANES *+               MLK_KECCAK_WAY]; /**< 4-way Keccak state, stored sequentially. */+} MLK_ALIGN mlk_shake128x4ctx;++#define mlk_shake128x4_absorb_once MLK_NAMESPACE(shake128x4_absorb_once)+void mlk_shake128x4_absorb_once(mlk_shake128x4ctx *state, const uint8_t *in0,+                                const uint8_t *in1, const uint8_t *in2,+                                const uint8_t *in3, size_t inlen)+__contract__(+  requires(inlen <= MLK_MAX_BUFFER_SIZE)+  requires(memory_no_alias(state, sizeof(mlk_shake128x4ctx)))+  requires(memory_no_alias(in0, inlen))+  requires(memory_no_alias(in1, inlen))+  requires(memory_no_alias(in2, inlen))+  requires(memory_no_alias(in3, inlen))+  assigns(memory_slice(state, sizeof(mlk_shake128x4ctx)))+);++#define mlk_shake128x4_squeezeblocks MLK_NAMESPACE(shake128x4_squeezeblocks)+void mlk_shake128x4_squeezeblocks(uint8_t *out0, uint8_t *out1, uint8_t *out2,+                                  uint8_t *out3, size_t nblocks,+                                  mlk_shake128x4ctx *state)+__contract__(+  requires(nblocks <= 8 /* somewhat arbitrary bound */)+  requires(memory_no_alias(state, sizeof(mlk_shake128x4ctx)))+  requires(memory_no_alias(out0, nblocks * SHAKE128_RATE))+  requires(memory_no_alias(out1, nblocks * SHAKE128_RATE))+  requires(memory_no_alias(out2, nblocks * SHAKE128_RATE))+  requires(memory_no_alias(out3, nblocks * SHAKE128_RATE))+  assigns(memory_slice(out0, nblocks * SHAKE128_RATE),+    memory_slice(out1, nblocks * SHAKE128_RATE),+    memory_slice(out2, nblocks * SHAKE128_RATE),+    memory_slice(out3, nblocks * SHAKE128_RATE),+    memory_slice(state, sizeof(mlk_shake128x4ctx)))+);++#define mlk_shake128x4_init MLK_NAMESPACE(shake128x4_init)+void mlk_shake128x4_init(mlk_shake128x4ctx *state);++#define mlk_shake128x4_release MLK_NAMESPACE(shake128x4_release)+void mlk_shake128x4_release(mlk_shake128x4ctx *state);++#define mlk_shake256x4 MLK_NAMESPACE(shake256x4)+void mlk_shake256x4(uint8_t *out0, uint8_t *out1, uint8_t *out2, uint8_t *out3,+                    size_t outlen, const uint8_t *in0, const uint8_t *in1,+                    const uint8_t *in2, const uint8_t *in3, size_t inlen)+__contract__(+  requires(inlen <= MLK_MAX_BUFFER_SIZE)+  requires(outlen <= MLK_MAX_BUFFER_SIZE)+  requires(memory_no_alias(in0, inlen))+  requires(memory_no_alias(in1, inlen))+  requires(memory_no_alias(in2, inlen))+  requires(memory_no_alias(in3, inlen))+  requires(memory_no_alias(out0, outlen))+  requires(memory_no_alias(out1, outlen))+  requires(memory_no_alias(out2, outlen))+  requires(memory_no_alias(out3, outlen))+  assigns(memory_slice(out0, outlen))+  assigns(memory_slice(out1, outlen))+  assigns(memory_slice(out2, outlen))+  assigns(memory_slice(out3, outlen))+);++#endif /* !MLK_FIPS202_FIPS202X4_H */
+ cbits/mlkem/src/fips202/keccakf1600.c view
@@ -0,0 +1,499 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [mupq]+ *   Common files for pqm4, pqm3, pqriscv+ *   Kannwischer, Petri, Rijneveld, Schwabe, Stoffelen+ *   https://github.com/mupq/mupq+ *+ * - [supercop]+ *   SUPERCOP benchmarking framework+ *   Daniel J. Bernstein+ *   http://bench.cr.yp.to/supercop.html+ *+ * - [tweetfips]+ *   'tweetfips202' FIPS202 implementation+ *   Van Assche, Bernstein, Schwabe+ *   https://keccak.team/2015/tweetfips202.html+ */++/* Based on the CC0 implementation from @[mupq] and the public domain+ * implementation @[supercop, crypto_hash/keccakc512/simple/]+ * by Ronny Van Keer, and the public domain @[tweetfips] implementation. */+++#include "keccakf1600.h"+#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++#define MLK_KECCAK_NROUNDS 24+#define MLK_KECCAK_ROL(a, offset) (((a) << (offset)) ^ ((a) >> (64 - (offset))))++void mlk_keccakf1600_extract_bytes(uint64_t *state, unsigned char *data,+                                   unsigned offset, unsigned length)+{+  unsigned i;+#if defined(MLK_SYS_LITTLE_ENDIAN)+  uint8_t *state_ptr = (uint8_t *)state + offset;+  for (i = 0; i < length; i++)+  __loop__(invariant(i <= length)+           decreases(length - i))+  {+    data[i] = state_ptr[i];+  }+#else  /* MLK_SYS_LITTLE_ENDIAN */+  /* Portable version */+  for (i = 0; i < length; i++)+  __loop__(invariant(i <= length)+           decreases(length - i))+  {+    data[i] = (state[(offset + i) >> 3] >> (8 * ((offset + i) & 0x07))) & 0xFF;+  }+#endif /* !MLK_SYS_LITTLE_ENDIAN */+}++void mlk_keccakf1600_xor_bytes(uint64_t *state, const unsigned char *data,+                               unsigned offset, unsigned length)+{+  unsigned i;+#if defined(MLK_SYS_LITTLE_ENDIAN)+  uint8_t *state_ptr = (uint8_t *)state + offset;+  for (i = 0; i < length; i++)+  __loop__(invariant(i <= length)+           decreases(length - i))+  {+    state_ptr[i] ^= data[i];+  }+#else  /* MLK_SYS_LITTLE_ENDIAN */+  /* Portable version */+  for (i = 0; i < length; i++)+  __loop__(invariant(i <= length)+           decreases(length - i))+  {+    state[(offset + i) >> 3] ^= (uint64_t)data[i]+                                << (8 * ((offset + i) & 0x07));+  }+#endif /* !MLK_SYS_LITTLE_ENDIAN */+}++static void mlk_keccakf1600x4_extract_bytes_c(uint64_t *state,+                                              unsigned char *data0,+                                              unsigned char *data1,+                                              unsigned char *data2,+                                              unsigned char *data3,+                                              unsigned offset, unsigned length)+__contract__(+    requires(0 <= offset && offset <= MLK_KECCAK_LANES * sizeof(uint64_t) &&+         0 <= length && length <= MLK_KECCAK_LANES * sizeof(uint64_t) - offset)+    requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY))+    requires(memory_no_alias(data0, length))+    requires(memory_no_alias(data1, length))+    requires(memory_no_alias(data2, length))+    requires(memory_no_alias(data3, length))+    assigns(memory_slice(data0, length))+    assigns(memory_slice(data1, length))+    assigns(memory_slice(data2, length))+    assigns(memory_slice(data3, length))+)+{+  mlk_keccakf1600_extract_bytes(state + MLK_KECCAK_LANES * 0, data0, offset,+                                length);+  mlk_keccakf1600_extract_bytes(state + MLK_KECCAK_LANES * 1, data1, offset,+                                length);+  mlk_keccakf1600_extract_bytes(state + MLK_KECCAK_LANES * 2, data2, offset,+                                length);+  mlk_keccakf1600_extract_bytes(state + MLK_KECCAK_LANES * 3, data3, offset,+                                length);+}++void mlk_keccakf1600x4_extract_bytes(uint64_t *state, unsigned char *data0,+                                     unsigned char *data1, unsigned char *data2,+                                     unsigned char *data3, unsigned offset,+                                     unsigned length)+{+#if defined(MLK_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES)+  if (mlk_keccakf1600_extract_bytes_x4_native(state, data0, data1, data2, data3,+                                              offset, length) ==+      MLK_NATIVE_FUNC_SUCCESS)+  {+    return;+  }+#endif /* MLK_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES */+  mlk_keccakf1600x4_extract_bytes_c(state, data0, data1, data2, data3, offset,+                                    length);+}++static void mlk_keccakf1600x4_xor_bytes_c(uint64_t *state,+                                          const unsigned char *data0,+                                          const unsigned char *data1,+                                          const unsigned char *data2,+                                          const unsigned char *data3,+                                          unsigned offset, unsigned length)+__contract__(+    requires(0 <= offset && offset <= MLK_KECCAK_LANES * sizeof(uint64_t) &&+         0 <= length && length <= MLK_KECCAK_LANES * sizeof(uint64_t) - offset)+    requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY))+    requires(memory_no_alias(data0, length))+    /* Case 1: all input buffers are distinct; Case 2: All input buffers are the same */+    requires((data0 == data1 &&+              data0 == data2 &&+              data0 == data3) ||+         (memory_no_alias(data1, length) &&+              memory_no_alias(data2, length) &&+              memory_no_alias(data3, length)))+    assigns(memory_slice(state, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY))+)+{+  mlk_keccakf1600_xor_bytes(state + MLK_KECCAK_LANES * 0, data0, offset,+                            length);+  mlk_keccakf1600_xor_bytes(state + MLK_KECCAK_LANES * 1, data1, offset,+                            length);+  mlk_keccakf1600_xor_bytes(state + MLK_KECCAK_LANES * 2, data2, offset,+                            length);+  mlk_keccakf1600_xor_bytes(state + MLK_KECCAK_LANES * 3, data3, offset,+                            length);+}++void mlk_keccakf1600x4_xor_bytes(uint64_t *state, const unsigned char *data0,+                                 const unsigned char *data1,+                                 const unsigned char *data2,+                                 const unsigned char *data3, unsigned offset,+                                 unsigned length)+{+#if defined(MLK_USE_NATIVE_FIPS202_X4_XOR_BYTES)+  if (mlk_keccakf1600_xor_bytes_x4_native(state, data0, data1, data2, data3,+                                          offset,+                                          length) == MLK_NATIVE_FUNC_SUCCESS)+  {+    return;+  }+#endif /* MLK_USE_NATIVE_FIPS202_X4_XOR_BYTES */+  mlk_keccakf1600x4_xor_bytes_c(state, data0, data1, data2, data3, offset,+                                length);+}++void mlk_keccakf1600x4_permute(uint64_t *state)+{+#if defined(MLK_USE_NATIVE_FIPS202_X4)+  if (mlk_keccak_f1600_x4_native(state) == MLK_NATIVE_FUNC_SUCCESS)+  {+    return;+  }+#endif /* MLK_USE_NATIVE_FIPS202_X4 */+  mlk_keccakf1600_permute(state + MLK_KECCAK_LANES * 0);+  mlk_keccakf1600_permute(state + MLK_KECCAK_LANES * 1);+  mlk_keccakf1600_permute(state + MLK_KECCAK_LANES * 2);+  mlk_keccakf1600_permute(state + MLK_KECCAK_LANES * 3);+}++static const uint64_t mlk_KeccakF_RoundConstants[MLK_KECCAK_NROUNDS] = {+    (uint64_t)0x0000000000000001ULL, (uint64_t)0x0000000000008082ULL,+    (uint64_t)0x800000000000808aULL, (uint64_t)0x8000000080008000ULL,+    (uint64_t)0x000000000000808bULL, (uint64_t)0x0000000080000001ULL,+    (uint64_t)0x8000000080008081ULL, (uint64_t)0x8000000000008009ULL,+    (uint64_t)0x000000000000008aULL, (uint64_t)0x0000000000000088ULL,+    (uint64_t)0x0000000080008009ULL, (uint64_t)0x000000008000000aULL,+    (uint64_t)0x000000008000808bULL, (uint64_t)0x800000000000008bULL,+    (uint64_t)0x8000000000008089ULL, (uint64_t)0x8000000000008003ULL,+    (uint64_t)0x8000000000008002ULL, (uint64_t)0x8000000000000080ULL,+    (uint64_t)0x000000000000800aULL, (uint64_t)0x800000008000000aULL,+    (uint64_t)0x8000000080008081ULL, (uint64_t)0x8000000000008080ULL,+    (uint64_t)0x0000000080000001ULL, (uint64_t)0x8000000080008008ULL};++MLK_STATIC_TESTABLE+void mlk_keccakf1600_permute_c(uint64_t *state)+__contract__(+    requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES))+    assigns(memory_slice(state, sizeof(uint64_t) * MLK_KECCAK_LANES))+)+{+  unsigned round;++  uint64_t Aba, Abe, Abi, Abo, Abu;+  uint64_t Aga, Age, Agi, Ago, Agu;+  uint64_t Aka, Ake, Aki, Ako, Aku;+  uint64_t Ama, Ame, Ami, Amo, Amu;+  uint64_t Asa, Ase, Asi, Aso, Asu;+  uint64_t BCa, BCe, BCi, BCo, BCu;+  uint64_t Da, De, Di, Do, Du;+  uint64_t Eba, Ebe, Ebi, Ebo, Ebu;+  uint64_t Ega, Ege, Egi, Ego, Egu;+  uint64_t Eka, Eke, Eki, Eko, Eku;+  uint64_t Ema, Eme, Emi, Emo, Emu;+  uint64_t Esa, Ese, Esi, Eso, Esu;++  /* copyFromState(A, state) */+  Aba = state[0];+  Abe = state[1];+  Abi = state[2];+  Abo = state[3];+  Abu = state[4];+  Aga = state[5];+  Age = state[6];+  Agi = state[7];+  Ago = state[8];+  Agu = state[9];+  Aka = state[10];+  Ake = state[11];+  Aki = state[12];+  Ako = state[13];+  Aku = state[14];+  Ama = state[15];+  Ame = state[16];+  Ami = state[17];+  Amo = state[18];+  Amu = state[19];+  Asa = state[20];+  Ase = state[21];+  Asi = state[22];+  Aso = state[23];+  Asu = state[24];++  for (round = 0; round < MLK_KECCAK_NROUNDS; round += 2)+  __loop__(invariant(round <= MLK_KECCAK_NROUNDS && round % 2 == 0)+           decreases(MLK_KECCAK_NROUNDS - round))+  {+    /* prepareTheta */+    BCa = Aba ^ Aga ^ Aka ^ Ama ^ Asa;+    BCe = Abe ^ Age ^ Ake ^ Ame ^ Ase;+    BCi = Abi ^ Agi ^ Aki ^ Ami ^ Asi;+    BCo = Abo ^ Ago ^ Ako ^ Amo ^ Aso;+    BCu = Abu ^ Agu ^ Aku ^ Amu ^ Asu;++    /* thetaRhoPiChiIotaPrepareTheta(round, A, E) */+    Da = BCu ^ MLK_KECCAK_ROL(BCe, 1);+    De = BCa ^ MLK_KECCAK_ROL(BCi, 1);+    Di = BCe ^ MLK_KECCAK_ROL(BCo, 1);+    Do = BCi ^ MLK_KECCAK_ROL(BCu, 1);+    Du = BCo ^ MLK_KECCAK_ROL(BCa, 1);++    Aba ^= Da;+    BCa = Aba;+    Age ^= De;+    BCe = MLK_KECCAK_ROL(Age, 44);+    Aki ^= Di;+    BCi = MLK_KECCAK_ROL(Aki, 43);+    Amo ^= Do;+    BCo = MLK_KECCAK_ROL(Amo, 21);+    Asu ^= Du;+    BCu = MLK_KECCAK_ROL(Asu, 14);+    Eba = BCa ^ ((~BCe) & BCi);+    Eba ^= (uint64_t)mlk_KeccakF_RoundConstants[round];+    Ebe = BCe ^ ((~BCi) & BCo);+    Ebi = BCi ^ ((~BCo) & BCu);+    Ebo = BCo ^ ((~BCu) & BCa);+    Ebu = BCu ^ ((~BCa) & BCe);++    Abo ^= Do;+    BCa = MLK_KECCAK_ROL(Abo, 28);+    Agu ^= Du;+    BCe = MLK_KECCAK_ROL(Agu, 20);+    Aka ^= Da;+    BCi = MLK_KECCAK_ROL(Aka, 3);+    Ame ^= De;+    BCo = MLK_KECCAK_ROL(Ame, 45);+    Asi ^= Di;+    BCu = MLK_KECCAK_ROL(Asi, 61);+    Ega = BCa ^ ((~BCe) & BCi);+    Ege = BCe ^ ((~BCi) & BCo);+    Egi = BCi ^ ((~BCo) & BCu);+    Ego = BCo ^ ((~BCu) & BCa);+    Egu = BCu ^ ((~BCa) & BCe);++    Abe ^= De;+    BCa = MLK_KECCAK_ROL(Abe, 1);+    Agi ^= Di;+    BCe = MLK_KECCAK_ROL(Agi, 6);+    Ako ^= Do;+    BCi = MLK_KECCAK_ROL(Ako, 25);+    Amu ^= Du;+    BCo = MLK_KECCAK_ROL(Amu, 8);+    Asa ^= Da;+    BCu = MLK_KECCAK_ROL(Asa, 18);+    Eka = BCa ^ ((~BCe) & BCi);+    Eke = BCe ^ ((~BCi) & BCo);+    Eki = BCi ^ ((~BCo) & BCu);+    Eko = BCo ^ ((~BCu) & BCa);+    Eku = BCu ^ ((~BCa) & BCe);++    Abu ^= Du;+    BCa = MLK_KECCAK_ROL(Abu, 27);+    Aga ^= Da;+    BCe = MLK_KECCAK_ROL(Aga, 36);+    Ake ^= De;+    BCi = MLK_KECCAK_ROL(Ake, 10);+    Ami ^= Di;+    BCo = MLK_KECCAK_ROL(Ami, 15);+    Aso ^= Do;+    BCu = MLK_KECCAK_ROL(Aso, 56);+    Ema = BCa ^ ((~BCe) & BCi);+    Eme = BCe ^ ((~BCi) & BCo);+    Emi = BCi ^ ((~BCo) & BCu);+    Emo = BCo ^ ((~BCu) & BCa);+    Emu = BCu ^ ((~BCa) & BCe);++    Abi ^= Di;+    BCa = MLK_KECCAK_ROL(Abi, 62);+    Ago ^= Do;+    BCe = MLK_KECCAK_ROL(Ago, 55);+    Aku ^= Du;+    BCi = MLK_KECCAK_ROL(Aku, 39);+    Ama ^= Da;+    BCo = MLK_KECCAK_ROL(Ama, 41);+    Ase ^= De;+    BCu = MLK_KECCAK_ROL(Ase, 2);+    Esa = BCa ^ ((~BCe) & BCi);+    Ese = BCe ^ ((~BCi) & BCo);+    Esi = BCi ^ ((~BCo) & BCu);+    Eso = BCo ^ ((~BCu) & BCa);+    Esu = BCu ^ ((~BCa) & BCe);++    /* prepareTheta */+    BCa = Eba ^ Ega ^ Eka ^ Ema ^ Esa;+    BCe = Ebe ^ Ege ^ Eke ^ Eme ^ Ese;+    BCi = Ebi ^ Egi ^ Eki ^ Emi ^ Esi;+    BCo = Ebo ^ Ego ^ Eko ^ Emo ^ Eso;+    BCu = Ebu ^ Egu ^ Eku ^ Emu ^ Esu;++    /* thetaRhoPiChiIotaPrepareTheta(round+1, E, A) */+    Da = BCu ^ MLK_KECCAK_ROL(BCe, 1);+    De = BCa ^ MLK_KECCAK_ROL(BCi, 1);+    Di = BCe ^ MLK_KECCAK_ROL(BCo, 1);+    Do = BCi ^ MLK_KECCAK_ROL(BCu, 1);+    Du = BCo ^ MLK_KECCAK_ROL(BCa, 1);++    Eba ^= Da;+    BCa = Eba;+    Ege ^= De;+    BCe = MLK_KECCAK_ROL(Ege, 44);+    Eki ^= Di;+    BCi = MLK_KECCAK_ROL(Eki, 43);+    Emo ^= Do;+    BCo = MLK_KECCAK_ROL(Emo, 21);+    Esu ^= Du;+    BCu = MLK_KECCAK_ROL(Esu, 14);+    Aba = BCa ^ ((~BCe) & BCi);+    Aba ^= (uint64_t)mlk_KeccakF_RoundConstants[round + 1];+    Abe = BCe ^ ((~BCi) & BCo);+    Abi = BCi ^ ((~BCo) & BCu);+    Abo = BCo ^ ((~BCu) & BCa);+    Abu = BCu ^ ((~BCa) & BCe);++    Ebo ^= Do;+    BCa = MLK_KECCAK_ROL(Ebo, 28);+    Egu ^= Du;+    BCe = MLK_KECCAK_ROL(Egu, 20);+    Eka ^= Da;+    BCi = MLK_KECCAK_ROL(Eka, 3);+    Eme ^= De;+    BCo = MLK_KECCAK_ROL(Eme, 45);+    Esi ^= Di;+    BCu = MLK_KECCAK_ROL(Esi, 61);+    Aga = BCa ^ ((~BCe) & BCi);+    Age = BCe ^ ((~BCi) & BCo);+    Agi = BCi ^ ((~BCo) & BCu);+    Ago = BCo ^ ((~BCu) & BCa);+    Agu = BCu ^ ((~BCa) & BCe);++    Ebe ^= De;+    BCa = MLK_KECCAK_ROL(Ebe, 1);+    Egi ^= Di;+    BCe = MLK_KECCAK_ROL(Egi, 6);+    Eko ^= Do;+    BCi = MLK_KECCAK_ROL(Eko, 25);+    Emu ^= Du;+    BCo = MLK_KECCAK_ROL(Emu, 8);+    Esa ^= Da;+    BCu = MLK_KECCAK_ROL(Esa, 18);+    Aka = BCa ^ ((~BCe) & BCi);+    Ake = BCe ^ ((~BCi) & BCo);+    Aki = BCi ^ ((~BCo) & BCu);+    Ako = BCo ^ ((~BCu) & BCa);+    Aku = BCu ^ ((~BCa) & BCe);++    Ebu ^= Du;+    BCa = MLK_KECCAK_ROL(Ebu, 27);+    Ega ^= Da;+    BCe = MLK_KECCAK_ROL(Ega, 36);+    Eke ^= De;+    BCi = MLK_KECCAK_ROL(Eke, 10);+    Emi ^= Di;+    BCo = MLK_KECCAK_ROL(Emi, 15);+    Eso ^= Do;+    BCu = MLK_KECCAK_ROL(Eso, 56);+    Ama = BCa ^ ((~BCe) & BCi);+    Ame = BCe ^ ((~BCi) & BCo);+    Ami = BCi ^ ((~BCo) & BCu);+    Amo = BCo ^ ((~BCu) & BCa);+    Amu = BCu ^ ((~BCa) & BCe);++    Ebi ^= Di;+    BCa = MLK_KECCAK_ROL(Ebi, 62);+    Ego ^= Do;+    BCe = MLK_KECCAK_ROL(Ego, 55);+    Eku ^= Du;+    BCi = MLK_KECCAK_ROL(Eku, 39);+    Ema ^= Da;+    BCo = MLK_KECCAK_ROL(Ema, 41);+    Ese ^= De;+    BCu = MLK_KECCAK_ROL(Ese, 2);+    Asa = BCa ^ ((~BCe) & BCi);+    Ase = BCe ^ ((~BCi) & BCo);+    Asi = BCi ^ ((~BCo) & BCu);+    Aso = BCo ^ ((~BCu) & BCa);+    Asu = BCu ^ ((~BCa) & BCe);+  }++  /* copyToState(state, A) */+  state[0] = Aba;+  state[1] = Abe;+  state[2] = Abi;+  state[3] = Abo;+  state[4] = Abu;+  state[5] = Aga;+  state[6] = Age;+  state[7] = Agi;+  state[8] = Ago;+  state[9] = Agu;+  state[10] = Aka;+  state[11] = Ake;+  state[12] = Aki;+  state[13] = Ako;+  state[14] = Aku;+  state[15] = Ama;+  state[16] = Ame;+  state[17] = Ami;+  state[18] = Amo;+  state[19] = Amu;+  state[20] = Asa;+  state[21] = Ase;+  state[22] = Asi;+  state[23] = Aso;+  state[24] = Asu;+}++void mlk_keccakf1600_permute(uint64_t *state)+{+#if defined(MLK_USE_NATIVE_FIPS202_X1)+  if (mlk_keccak_f1600_x1_native(state) == MLK_NATIVE_FUNC_SUCCESS)+  {+    return;+  }+#endif /* MLK_USE_NATIVE_FIPS202_X1 */+  mlk_keccakf1600_permute_c(state);+}++#else /* !MLK_CONFIG_MULTILEVEL_NO_SHARED */++MLK_EMPTY_CU(keccakf1600)++#endif /* MLK_CONFIG_MULTILEVEL_NO_SHARED */++/* To facilitate single-compilation-unit (SCU) builds, undefine all macros.+ * Don't modify by hand -- this is auto-generated by scripts/autogen. */+#undef MLK_KECCAK_NROUNDS+#undef MLK_KECCAK_ROL
+ cbits/mlkem/src/fips202/keccakf1600.h view
@@ -0,0 +1,98 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLK_FIPS202_KECCAKF1600_H+#define MLK_FIPS202_KECCAKF1600_H+#include "../cbmc.h"+#include "../common.h"++#define MLK_KECCAK_LANES 25+#define MLK_KECCAK_WAY 4++/*+ * WARNING:+ * The contents of this structure, including the placement+ * and interleaving of Keccak lanes, are IMPLEMENTATION-DEFINED.+ * The struct is only exposed here to allow its construction on the stack.+ */++#define mlk_keccakf1600_extract_bytes MLK_NAMESPACE(keccakf1600_extract_bytes)+void mlk_keccakf1600_extract_bytes(uint64_t *state, unsigned char *data,+                                   unsigned offset, unsigned length)+__contract__(+    requires(0 <= offset && offset <= MLK_KECCAK_LANES * sizeof(uint64_t) &&+             0 <= length && length <= MLK_KECCAK_LANES * sizeof(uint64_t) - offset)+    requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES))+    requires(memory_no_alias(data, length))+    assigns(memory_slice(data, length))+);++#define mlk_keccakf1600_xor_bytes MLK_NAMESPACE(keccakf1600_xor_bytes)+void mlk_keccakf1600_xor_bytes(uint64_t *state, const unsigned char *data,+                               unsigned offset, unsigned length)+__contract__(+    requires(0 <= offset && offset <= MLK_KECCAK_LANES * sizeof(uint64_t) &&+             0 <= length && length <= MLK_KECCAK_LANES * sizeof(uint64_t) - offset)+    requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES))+    requires(memory_no_alias(data, length))+    assigns(memory_slice(state, sizeof(uint64_t) * MLK_KECCAK_LANES))+);++#define mlk_keccakf1600x4_extract_bytes \+  MLK_NAMESPACE(keccakf1600x4_extract_bytes)+void mlk_keccakf1600x4_extract_bytes(uint64_t *state, unsigned char *data0,+                                     unsigned char *data1, unsigned char *data2,+                                     unsigned char *data3, unsigned offset,+                                     unsigned length)+__contract__(+    requires(0 <= offset && offset <= MLK_KECCAK_LANES * sizeof(uint64_t) &&+             0 <= length && length <= MLK_KECCAK_LANES * sizeof(uint64_t) - offset)+    requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY))+    requires(memory_no_alias(data0, length))+    requires(memory_no_alias(data1, length))+    requires(memory_no_alias(data2, length))+    requires(memory_no_alias(data3, length))+    assigns(memory_slice(data0, length))+    assigns(memory_slice(data1, length))+    assigns(memory_slice(data2, length))+    assigns(memory_slice(data3, length))+);++#define mlk_keccakf1600x4_xor_bytes MLK_NAMESPACE(keccakf1600x4_xor_bytes)+void mlk_keccakf1600x4_xor_bytes(uint64_t *state, const unsigned char *data0,+                                 const unsigned char *data1,+                                 const unsigned char *data2,+                                 const unsigned char *data3, unsigned offset,+                                 unsigned length)+__contract__(+    requires(0 <= offset && offset <= MLK_KECCAK_LANES * sizeof(uint64_t) &&+             0 <= length && length <= MLK_KECCAK_LANES * sizeof(uint64_t) - offset)+    requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY))+    requires(memory_no_alias(data0, length))+    /* Case 1: all input buffers are distinct; Case 2: All input buffers are the same */+    requires((data0 == data1 &&+              data0 == data2 &&+              data0 == data3) ||+             (memory_no_alias(data1, length) &&+              memory_no_alias(data2, length) &&+              memory_no_alias(data3, length)))+    assigns(memory_slice(state, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY))+);+++#define mlk_keccakf1600x4_permute MLK_NAMESPACE(keccakf1600x4_permute)+void mlk_keccakf1600x4_permute(uint64_t *state)+__contract__(+    requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY))+    assigns(memory_slice(state, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY))+);++#define mlk_keccakf1600_permute MLK_NAMESPACE(keccakf1600_permute)+void mlk_keccakf1600_permute(uint64_t *state)+__contract__(+    requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES))+    assigns(memory_slice(state, sizeof(uint64_t) * MLK_KECCAK_LANES))+);++#endif /* !MLK_FIPS202_KECCAKF1600_H */
+ cbits/mlkem/src/fips202/native/aarch64/auto.h view
@@ -0,0 +1,78 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [HYBRID]+ *   Hybrid scalar/vector implementations of Keccak and SPHINCS+ on AArch64+ *   Becker, Kannwischer+ *   https://eprint.iacr.org/2022/1243+ */++#ifndef MLK_FIPS202_NATIVE_AARCH64_AUTO_H+#define MLK_FIPS202_NATIVE_AARCH64_AUTO_H+/* Default FIPS202 assembly profile for AArch64 systems */++/*+ * Default logic to decide which implementation to use.+ *+ */++/*+ * Keccak-f1600+ *+ * - On Arm-based Apple CPUs, or if MLK_SYS_AARCH64_FAST_SHA3 is set,+ *   we pick a pure Neon implementation.+ * - Otherwise, unless MLK_SYS_AARCH64_SLOW_BARREL_SHIFTER is set,+ *   we use lazy-rotation scalar assembly from @[HYBRID].+ * - Otherwise, if MLK_SYS_AARCH64_SLOW_BARREL_SHIFTER is set, we+ *   fall back to the standard C implementation.+ */+#if defined(__ARM_FEATURE_SHA3) && \+    (defined(__APPLE__) || defined(MLK_SYS_AARCH64_FAST_SHA3))+#include "x1_v84a.h"+#elif !defined(MLK_SYS_AARCH64_SLOW_BARREL_SHIFTER)+#include "x1_scalar.h"+#endif++/* Batched, SIMD-based Keccak-f1600 implementations. */+#if defined(MLK_SYS_AARCH64_NEON)++/*+ * Keccak-f1600x2/x4+ *+ * The optimal implementation is highly CPU-specific; see @[HYBRID].+ *+ * For now, if v8.4-A is not implemented, we fall back to Keccak-f1600.+ * If v8.4-A is implemented and we are on an Apple CPU or+ * MLK_SYS_AARCH64_FAST_SHA3 is set, we use a plain Neon-based+ * implementation.+ * Otherwise, if v8.4-A is implemented, we use a scalar/Neon/Neon hybrid.+ * The reason for this distinction is that Apple CPUs (and CPUs flagged with+ * MLK_SYS_AARCH64_FAST_SHA3) implement the SHA3 instructions on all SIMD+ * units, while Arm CPUs prior to Cortex-X4 don't, and ordinary Neon+ * instructions are still needed.+ */+#if defined(__ARM_FEATURE_SHA3)+/*+ * For Apple-M cores (and CPUs flagged with MLK_SYS_AARCH64_FAST_SHA3), we+ * use a plain implementation leveraging SHA3 instructions only.+ */+#if defined(__APPLE__) || defined(MLK_SYS_AARCH64_FAST_SHA3)+#include "x2_v84a.h"+#else+#include "x4_v8a_v84a_scalar.h"+#endif++#else /* __ARM_FEATURE_SHA3 */++#include "x4_v8a_scalar.h"++#endif /* !__ARM_FEATURE_SHA3 */++#endif /* MLK_SYS_AARCH64_NEON */++#endif /* !MLK_FIPS202_NATIVE_AARCH64_AUTO_H */
+ cbits/mlkem/src/fips202/native/aarch64/src/fips202_native_aarch64.h view
@@ -0,0 +1,80 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLK_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H+#define MLK_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H++#include "../../../../cbmc.h"+#include "../../../../common.h"++#define mlk_keccakf1600_round_constants \+  MLK_NAMESPACE(keccakf1600_round_constants)+MLK_INTERNAL_DATA_DECLARATION const uint64_t+    mlk_keccakf1600_round_constants[24];++#define mlk_keccak_f1600_x1_scalar_aarch64_asm \+  MLK_NAMESPACE(keccak_f1600_x1_scalar_aarch64_asm)+void mlk_keccak_f1600_x1_scalar_aarch64_asm(uint64_t state[25],+                                            const uint64_t rc[24])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/keccak_f1600_x1_scalar_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 1))+  requires(rc == mlk_keccakf1600_round_constants)+  assigns(memory_slice(state, sizeof(uint64_t) * 25 * 1))+);++#define mlk_keccak_f1600_x1_v84a_aarch64_asm \+  MLK_NAMESPACE(keccak_f1600_x1_v84a_aarch64_asm)+void mlk_keccak_f1600_x1_v84a_aarch64_asm(uint64_t state[25],+                                          const uint64_t rc[24])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/keccak_f1600_x1_v84a_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 1))+  requires(rc == mlk_keccakf1600_round_constants)+  assigns(memory_slice(state, sizeof(uint64_t) * 25 * 1))+);++#define mlk_keccak_f1600_x2_v84a_aarch64_asm \+  MLK_NAMESPACE(keccak_f1600_x2_v84a_aarch64_asm)+void mlk_keccak_f1600_x2_v84a_aarch64_asm(uint64_t state[50],+                                          const uint64_t rc[24])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/keccak_f1600_x2_v84a_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 2))+  requires(rc == mlk_keccakf1600_round_constants)+  assigns(memory_slice(state, sizeof(uint64_t) * 25 * 2))+);++#define mlk_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm \+  MLK_NAMESPACE(keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm)+void mlk_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm(uint64_t state[100],+                                                       const uint64_t rc[24])+/* This must be kept in sync with the HOL-Light specification+ * in+ * proofs/hol_light/aarch64/proofs/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.ml+ */+__contract__(+  requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4))+  requires(rc == mlk_keccakf1600_round_constants)+  assigns(memory_slice(state, sizeof(uint64_t) * 25 * 4))+);++#define mlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm \+  MLK_NAMESPACE(keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm)+void mlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm(+    uint64_t state[100], const uint64_t rc[24])+/* This must be kept in sync with the HOL-Light specification+ * in+ * proofs/hol_light/aarch64/proofs/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.ml+ */+__contract__(+  requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4))+  requires(rc == mlk_keccakf1600_round_constants)+  assigns(memory_slice(state, sizeof(uint64_t) * 25 * 4))+);++#endif /* !MLK_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H */
+ cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x1_scalar_aarch64_asm.S view
@@ -0,0 +1,377 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) 2021-2022 Arm Limited+ * Copyright (c) 2022 Matthias Kannwischer+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++// Author: Hanno Becker <hanno.becker@arm.com>+// Author: Matthias Kannwischer <matthias@kannwischer.eu>++/*yaml+  Name: keccak_f1600_x1_scalar_aarch64_asm+  Description: AArch64 scalar implementation of Keccak-f[1600] permutation for single state+  Signature: void mlk_keccak_f1600_x1_scalar_aarch64_asm(uint64_t state[25], const uint64_t rc[24])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    x0:+      type: buffer+      size_bytes: 200+      permissions: read/write+      c_parameter: uint64_t state[25]+      description: Keccak state (25 x uint64_t)+    x1:+      type: buffer+      size_bytes: 192+      permissions: read-only+      c_parameter: uint64_t const *rc+      description: Round constants (24 x uint64_t)+  Stack:+    bytes: 128+    description: register preservation and temporary storage+*/++#include "../../../../common.h"+#if defined(MLK_FIPS202_AARCH64_NEED_X1_SCALAR) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/fips202/aarch64/src/keccak_f1600_x1_scalar_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(keccak_f1600_x1_scalar_aarch64_asm)+MLK_ASM_FN_SYMBOL(keccak_f1600_x1_scalar_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0x80+        .cfi_adjust_cfa_offset 0x80+        stp x19, x20, [sp, #0x20]+        .cfi_rel_offset x19, 0x20+        .cfi_rel_offset x20, 0x28+        stp x21, x22, [sp, #0x30]+        .cfi_rel_offset x21, 0x30+        .cfi_rel_offset x22, 0x38+        stp x23, x24, [sp, #0x40]+        .cfi_rel_offset x23, 0x40+        .cfi_rel_offset x24, 0x48+        stp x25, x26, [sp, #0x50]+        .cfi_rel_offset x25, 0x50+        .cfi_rel_offset x26, 0x58+        stp x27, x28, [sp, #0x60]+        .cfi_rel_offset x27, 0x60+        .cfi_rel_offset x28, 0x68+        stp x29, x30, [sp, #0x70]+        .cfi_rel_offset x29, 0x70+        .cfi_rel_offset x30, 0x78++Lmlk_keccak_f1600_x1_scalar_initial:+        mov x26, x1+        str x1, [sp, #0x8]+        ldp x1, x6, [x0]+        ldp x11, x16, [x0, #0x10]+        ldp x21, x2, [x0, #0x20]+        ldp x7, x12, [x0, #0x30]+        ldp x17, x22, [x0, #0x40]+        ldp x3, x8, [x0, #0x50]+        ldp x13, x28, [x0, #0x60]+        ldp x23, x4, [x0, #0x70]+        ldp x9, x14, [x0, #0x80]+        ldp x19, x24, [x0, #0x90]+        ldp x5, x10, [x0, #0xa0]+        ldp x15, x20, [x0, #0xb0]+        ldr x25, [x0, #0xc0]+        str x0, [sp]+        eor x30, x24, x25+        eor x27, x9, x10+        eor x0, x30, x21+        eor x26, x27, x6+        eor x27, x26, x7+        eor x29, x0, x22+        eor x26, x29, x23+        eor x29, x4, x5+        eor x30, x29, x1+        eor x0, x27, x8+        eor x29, x30, x2+        eor x30, x19, x20+        eor x30, x30, x16+        eor x27, x26, x0, ror #63+        eor x4, x4, x27+        eor x30, x30, x17+        eor x30, x30, x28+        eor x29, x29, x3+        eor x0, x0, x30, ror #63+        eor x30, x30, x29, ror #63+        eor x22, x22, x30+        eor x23, x23, x30+        str x23, [sp, #0x18]+        eor x23, x14, x15+        eor x14, x14, x0+        eor x23, x23, x11+        eor x15, x15, x0+        eor x1, x1, x27+        eor x23, x23, x12+        eor x23, x23, x13+        eor x11, x11, x0+        eor x29, x29, x23, ror #63+        eor x23, x23, x26, ror #63+        eor x26, x13, x0+        eor x13, x28, x23+        eor x28, x24, x30+        eor x24, x16, x23+        eor x16, x21, x30+        eor x21, x25, x30+        eor x30, x19, x23+        eor x19, x20, x23+        eor x20, x17, x23+        eor x17, x12, x0+        eor x0, x2, x27+        eor x2, x6, x29+        eor x6, x8, x29+        bic x8, x28, x13, ror #47+        eor x12, x3, x27+        bic x3, x13, x17, ror #19+        eor x5, x5, x27+        ldr x27, [sp, #0x18]+        bic x25, x17, x2, ror #5+        eor x9, x9, x29+        eor x23, x25, x5, ror #52+        eor x3, x3, x2, ror #24+        eor x8, x8, x17, ror #2+        eor x17, x10, x29+        bic x25, x12, x22, ror #47+        eor x29, x7, x29+        bic x10, x4, x27, ror #2+        bic x7, x5, x28, ror #10+        eor x10, x10, x20, ror #50+        eor x13, x7, x13, ror #57+        bic x7, x2, x5, ror #47+        eor x2, x25, x24, ror #39+        bic x25, x20, x11, ror #57+        bic x5, x17, x4, ror #25+        eor x25, x25, x17, ror #53+        bic x17, x11, x17, ror #60+        eor x28, x7, x28, ror #57+        bic x7, x9, x12, ror #42+        eor x7, x7, x22, ror #25+        bic x22, x22, x24, ror #56+        bic x24, x24, x15, ror #31+        eor x22, x22, x15, ror #23+        bic x20, x27, x20, ror #48+        bic x15, x15, x9, ror #16+        eor x12, x15, x12, ror #58+        eor x15, x5, x27, ror #27+        eor x5, x20, x11, ror #41+        ldr x11, [sp, #0x8]+        eor x20, x17, x4, ror #21+        eor x17, x24, x9, ror #47+        mov x24, #0x1               // =1+        bic x9, x0, x16, ror #9+        str x24, [sp, #0x10]+        bic x24, x29, x1, ror #44+        bic x27, x1, x21, ror #50+        bic x4, x26, x29, ror #63+        eor x1, x1, x4, ror #21+        ldr x11, [x11]+        bic x4, x21, x30, ror #57+        eor x21, x24, x21, ror #30+        eor x24, x9, x19, ror #44+        bic x9, x14, x6, ror #5+        eor x9, x9, x0, ror #43+        bic x0, x6, x0, ror #38+        eor x1, x1, x11+        eor x11, x4, x26, ror #35+        eor x4, x0, x16, ror #47+        bic x0, x16, x19, ror #35+        eor x16, x27, x30, ror #43+        bic x27, x30, x26, ror #42+        bic x26, x19, x14, ror #41+        eor x19, x0, x14, ror #12+        eor x14, x26, x6, ror #46+        eor x6, x27, x29, ror #41++Lmlk_keccak_f1600_x1_scalar_loop:+        eor x0, x15, x11, ror #52+        eor x0, x0, x13, ror #48+        eor x26, x8, x9, ror #57+        eor x27, x0, x14, ror #10+        eor x29, x16, x28, ror #63+        eor x26, x26, x6, ror #51+        eor x30, x23, x22, ror #50+        eor x0, x26, x10, ror #31+        eor x29, x29, x19, ror #37+        eor x27, x27, x12, ror #5+        eor x30, x30, x24, ror #34+        eor x0, x0, x7, ror #27+        eor x26, x30, x21, ror #26+        eor x26, x26, x25, ror #15+        ror x30, x27, #0x3e+        eor x30, x30, x26, ror #57+        ror x26, x26, #0x3a+        eor x16, x30, x16+        eor x28, x30, x28, ror #63+        str x28, [sp, #0x18]+        eor x29, x29, x17, ror #36+        eor x28, x1, x2, ror #61+        eor x19, x30, x19, ror #37+        eor x29, x29, x20, ror #2+        eor x28, x28, x4, ror #54+        eor x26, x26, x0, ror #55+        eor x28, x28, x3, ror #39+        eor x28, x28, x5, ror #25+        ror x0, x0, #0x38+        eor x0, x0, x29, ror #63+        eor x27, x28, x27, ror #61+        eor x13, x0, x13, ror #46+        eor x28, x29, x28, ror #63+        eor x29, x30, x20, ror #2+        eor x20, x26, x3, ror #39+        eor x11, x0, x11, ror #50+        eor x25, x28, x25, ror #9+        eor x3, x28, x21, ror #20+        eor x21, x26, x1+        eor x9, x27, x9, ror #49+        eor x24, x28, x24, ror #28+        eor x1, x30, x17, ror #36+        eor x14, x0, x14, ror #8+        eor x22, x28, x22, ror #44+        eor x8, x27, x8, ror #56+        eor x17, x27, x7, ror #19+        eor x15, x0, x15, ror #62+        bic x7, x20, x22, ror #47+        eor x4, x26, x4, ror #54+        eor x0, x0, x12, ror #3+        eor x28, x28, x23, ror #58+        eor x23, x26, x2, ror #61+        eor x26, x26, x5, ror #25+        eor x2, x7, x16, ror #39+        bic x7, x9, x20, ror #42+        bic x30, x15, x9, ror #16+        eor x7, x7, x22, ror #25+        eor x12, x30, x20, ror #58+        bic x20, x22, x16, ror #56+        eor x30, x27, x6, ror #43+        eor x22, x20, x15, ror #23+        bic x6, x19, x13, ror #42+        eor x6, x6, x17, ror #41+        bic x5, x13, x17, ror #63+        eor x5, x21, x5, ror #21+        bic x17, x17, x21, ror #44+        eor x27, x27, x10, ror #23+        bic x21, x21, x25, ror #50+        bic x20, x27, x4, ror #25+        bic x10, x16, x15, ror #31+        eor x16, x21, x19, ror #43+        eor x21, x17, x25, ror #30+        bic x19, x25, x19, ror #57+        ldr x25, [sp, #0x10]+        eor x17, x10, x9, ror #47+        ldr x9, [sp, #0x8]+        eor x15, x20, x28, ror #27+        bic x20, x4, x28, ror #2+        eor x10, x20, x1, ror #50+        bic x20, x11, x27, ror #60+        eor x20, x20, x4, ror #21+        bic x4, x28, x1, ror #48+        bic x1, x1, x11, ror #57+        ldr x28, [x9, x25, lsl #3]+        ldr x9, [sp, #0x18]+        add x25, x25, #0x1+        str x25, [sp, #0x10]+        cmp x25, #0x17+        eor x25, x1, x27, ror #53+        bic x27, x30, x26, ror #47+        eor x1, x5, x28+        eor x5, x4, x11, ror #41+        eor x11, x19, x13, ror #35+        bic x13, x26, x24, ror #10+        eor x28, x27, x24, ror #57+        bic x27, x24, x9, ror #47+        bic x19, x23, x3, ror #9+        bic x4, x29, x14, ror #41+        eor x24, x19, x29, ror #44+        bic x29, x3, x29, ror #35+        eor x13, x13, x9, ror #57+        eor x19, x29, x14, ror #12+        bic x29, x9, x0, ror #19+        bic x14, x14, x8, ror #5+        eor x9, x14, x23, ror #43+        eor x14, x4, x8, ror #46+        bic x23, x8, x23, ror #38+        eor x8, x27, x0, ror #2+        eor x4, x23, x3, ror #47+        bic x3, x0, x30, ror #5+        eor x23, x3, x26, ror #52+        eor x3, x29, x30, ror #24+        b.le Lmlk_keccak_f1600_x1_scalar_loop+        ror x6, x6, #0x2b+        ror x11, x11, #0x32+        ror x21, x21, #0x14+        ror x2, x2, #0x3d+        ror x7, x7, #0x13+        ror x12, x12, #0x3+        ror x17, x17, #0x24+        ror x22, x22, #0x2c+        ror x3, x3, #0x27+        ror x8, x8, #0x38+        ror x13, x13, #0x2e+        ror x28, x28, #0x3f+        ror x23, x23, #0x3a+        ror x4, x4, #0x36+        ror x9, x9, #0x31+        ror x14, x14, #0x8+        ror x19, x19, #0x25+        ror x24, x24, #0x1c+        ror x5, x5, #0x19+        ror x10, x10, #0x17+        ror x15, x15, #0x3e+        ror x20, x20, #0x2+        ror x25, x25, #0x9+        ldr x0, [sp]+        stp x1, x6, [x0]+        stp x11, x16, [x0, #0x10]+        stp x21, x2, [x0, #0x20]+        stp x7, x12, [x0, #0x30]+        stp x17, x22, [x0, #0x40]+        stp x3, x8, [x0, #0x50]+        stp x13, x28, [x0, #0x60]+        stp x23, x4, [x0, #0x70]+        stp x9, x14, [x0, #0x80]+        stp x19, x24, [x0, #0x90]+        stp x5, x10, [x0, #0xa0]+        stp x15, x20, [x0, #0xb0]+        str x25, [x0, #0xc0]+        ldp x19, x20, [sp, #0x20]+        .cfi_restore x19+        .cfi_restore x20+        ldp x21, x22, [sp, #0x30]+        .cfi_restore x21+        .cfi_restore x22+        ldp x23, x24, [sp, #0x40]+        .cfi_restore x23+        .cfi_restore x24+        ldp x25, x26, [sp, #0x50]+        .cfi_restore x25+        .cfi_restore x26+        ldp x27, x28, [sp, #0x60]+        .cfi_restore x27+        .cfi_restore x28+        ldp x29, x30, [sp, #0x70]+        .cfi_restore x29+        .cfi_restore x30+        add sp, sp, #0x80+        .cfi_adjust_cfa_offset -0x80+        ret+        .cfi_endproc++MLK_ASM_FN_SIZE(keccak_f1600_x1_scalar_aarch64_asm)++#endif /* MLK_FIPS202_AARCH64_NEED_X1_SCALAR && \+          !MLK_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x1_v84a_aarch64_asm.S view
@@ -0,0 +1,206 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) 2021-2022 Arm Limited+ * Copyright (c) 2022 Matthias Kannwischer+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [HYBRID]+ *   Hybrid scalar/vector implementations of Keccak and SPHINCS+ on AArch64+ *   Becker, Kannwischer+ *   https://eprint.iacr.org/2022/1243+ */++/*yaml+  Name: keccak_f1600_x1_v84a_aarch64_asm+  Description: AArch64 ARMv8.4-A implementation of Keccak-f[1600] permutation for single state+  Signature: void mlk_keccak_f1600_x1_v84a_aarch64_asm(uint64_t state[25], const uint64_t rc[24])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON, SHA3]+    x0:+      type: buffer+      size_bytes: 200+      permissions: read/write+      c_parameter: uint64_t state[25]+      description: Keccak state (25 x uint64_t)+    x1:+      type: buffer+      size_bytes: 192+      permissions: read-only+      c_parameter: const uint64_t rc[24]+      description: Round constants (24 x uint64_t)+  Stack:+    bytes: 64+    description: register preservation+*/++//+// Author: Hanno Becker <hanno.becker@arm.com>+// Author: Matthias Kannwischer <matthias@kannwischer.eu>+//+// This implementation is essentially from the paper @[HYBRID].+// The only difference is interleaving/deinterleaving of Keccak state+// during load and store, so that the caller need not do this.+//++#include "../../../../common.h"+#if defined(MLK_FIPS202_AARCH64_NEED_X1_V84A) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++#if defined(__ARM_FEATURE_SHA3)++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/fips202/aarch64/src/keccak_f1600_x1_v84a_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(keccak_f1600_x1_v84a_aarch64_asm)+MLK_ASM_FN_SYMBOL(keccak_f1600_x1_v84a_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0x40+        .cfi_adjust_cfa_offset 0x40+        stp d8, d9, [sp]+        .cfi_rel_offset d8, 0x0+        .cfi_rel_offset d9, 0x8+        stp d10, d11, [sp, #0x10]+        .cfi_rel_offset d10, 0x10+        .cfi_rel_offset d11, 0x18+        stp d12, d13, [sp, #0x20]+        .cfi_rel_offset d12, 0x20+        .cfi_rel_offset d13, 0x28+        stp d14, d15, [sp, #0x30]+        .cfi_rel_offset d14, 0x30+        .cfi_rel_offset d15, 0x38+        ldp d0, d1, [x0]+        ldp d2, d3, [x0, #0x10]+        ldp d4, d5, [x0, #0x20]+        ldp d6, d7, [x0, #0x30]+        ldp d8, d9, [x0, #0x40]+        ldp d10, d11, [x0, #0x50]+        ldp d12, d13, [x0, #0x60]+        ldp d14, d15, [x0, #0x70]+        ldp d16, d17, [x0, #0x80]+        ldp d18, d19, [x0, #0x90]+        ldp d20, d21, [x0, #0xa0]+        ldp d22, d23, [x0, #0xb0]+        ldr d24, [x0, #0xc0]+        mov x2, #0x18               // =24++Lmlk_keccak_f1600_x1_v84a_loop:+        eor3 v30.16b, v0.16b, v5.16b, v10.16b+        eor3 v29.16b, v1.16b, v6.16b, v11.16b+        eor3 v28.16b, v2.16b, v7.16b, v12.16b+        eor3 v27.16b, v3.16b, v8.16b, v13.16b+        eor3 v26.16b, v4.16b, v9.16b, v14.16b+        eor3 v30.16b, v30.16b, v15.16b, v20.16b+        eor3 v29.16b, v29.16b, v16.16b, v21.16b+        eor3 v28.16b, v28.16b, v17.16b, v22.16b+        eor3 v27.16b, v27.16b, v18.16b, v23.16b+        eor3 v26.16b, v26.16b, v19.16b, v24.16b+        rax1 v25.2d, v30.2d, v28.2d+        rax1 v28.2d, v28.2d, v26.2d+        rax1 v26.2d, v26.2d, v29.2d+        rax1 v29.2d, v29.2d, v27.2d+        rax1 v27.2d, v27.2d, v30.2d+        eor v30.16b, v0.16b, v26.16b+        xar v0.2d, v2.2d, v29.2d, #0x2+        xar v2.2d, v12.2d, v29.2d, #0x15+        xar v12.2d, v13.2d, v28.2d, #0x27+        xar v13.2d, v19.2d, v27.2d, #0x38+        xar v19.2d, v23.2d, v28.2d, #0x8+        xar v23.2d, v15.2d, v26.2d, #0x17+        xar v15.2d, v1.2d, v25.2d, #0x3f+        xar v1.2d, v8.2d, v28.2d, #0x9+        xar v8.2d, v16.2d, v25.2d, #0x13+        xar v16.2d, v7.2d, v29.2d, #0x3a+        xar v7.2d, v10.2d, v26.2d, #0x3d+        xar v10.2d, v3.2d, v28.2d, #0x24+        xar v3.2d, v18.2d, v28.2d, #0x2b+        xar v18.2d, v17.2d, v29.2d, #0x31+        xar v17.2d, v11.2d, v25.2d, #0x36+        xar v11.2d, v9.2d, v27.2d, #0x2c+        xar v9.2d, v22.2d, v29.2d, #0x3+        xar v22.2d, v14.2d, v27.2d, #0x19+        xar v14.2d, v20.2d, v26.2d, #0x2e+        xar v20.2d, v4.2d, v27.2d, #0x25+        xar v4.2d, v24.2d, v27.2d, #0x32+        xar v24.2d, v21.2d, v25.2d, #0x3e+        xar v21.2d, v5.2d, v26.2d, #0x1c+        xar v27.2d, v6.2d, v25.2d, #0x14+        ld1r { v31.2d }, [x1], #8+        bcax v5.16b, v10.16b, v7.16b, v11.16b+        bcax v6.16b, v11.16b, v8.16b, v7.16b+        bcax v7.16b, v7.16b, v9.16b, v8.16b+        bcax v8.16b, v8.16b, v10.16b, v9.16b+        bcax v9.16b, v9.16b, v11.16b, v10.16b+        bcax v10.16b, v15.16b, v12.16b, v16.16b+        bcax v11.16b, v16.16b, v13.16b, v12.16b+        bcax v12.16b, v12.16b, v14.16b, v13.16b+        bcax v13.16b, v13.16b, v15.16b, v14.16b+        bcax v14.16b, v14.16b, v16.16b, v15.16b+        bcax v15.16b, v20.16b, v17.16b, v21.16b+        bcax v16.16b, v21.16b, v18.16b, v17.16b+        bcax v17.16b, v17.16b, v19.16b, v18.16b+        bcax v18.16b, v18.16b, v20.16b, v19.16b+        bcax v19.16b, v19.16b, v21.16b, v20.16b+        bcax v20.16b, v0.16b, v22.16b, v1.16b+        bcax v21.16b, v1.16b, v23.16b, v22.16b+        bcax v22.16b, v22.16b, v24.16b, v23.16b+        bcax v23.16b, v23.16b, v0.16b, v24.16b+        bcax v24.16b, v24.16b, v1.16b, v0.16b+        bcax v0.16b, v30.16b, v2.16b, v27.16b+        bcax v1.16b, v27.16b, v3.16b, v2.16b+        bcax v2.16b, v2.16b, v4.16b, v3.16b+        bcax v3.16b, v3.16b, v30.16b, v4.16b+        bcax v4.16b, v4.16b, v27.16b, v30.16b+        eor v0.16b, v0.16b, v31.16b+        sub x2, x2, #0x1+        cbnz x2, Lmlk_keccak_f1600_x1_v84a_loop+        stp d0, d1, [x0]+        stp d2, d3, [x0, #0x10]+        stp d4, d5, [x0, #0x20]+        stp d6, d7, [x0, #0x30]+        stp d8, d9, [x0, #0x40]+        stp d10, d11, [x0, #0x50]+        stp d12, d13, [x0, #0x60]+        stp d14, d15, [x0, #0x70]+        stp d16, d17, [x0, #0x80]+        stp d18, d19, [x0, #0x90]+        stp d20, d21, [x0, #0xa0]+        stp d22, d23, [x0, #0xb0]+        str d24, [x0, #0xc0]+        ldp d8, d9, [sp]+        .cfi_restore d8+        .cfi_restore d9+        ldp d10, d11, [sp, #0x10]+        .cfi_restore d10+        .cfi_restore d11+        ldp d12, d13, [sp, #0x20]+        .cfi_restore d12+        .cfi_restore d13+        ldp d14, d15, [sp, #0x30]+        .cfi_restore d14+        .cfi_restore d15+        add sp, sp, #0x40+        .cfi_adjust_cfa_offset -0x40+        ret+        .cfi_endproc++MLK_ASM_FN_SIZE(keccak_f1600_x1_v84a_aarch64_asm)++#endif /* __ARM_FEATURE_SHA3 */++#endif /* MLK_FIPS202_AARCH64_NEED_X1_V84A && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x2_v84a_aarch64_asm.S view
@@ -0,0 +1,261 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) 2021-2022 Arm Limited+ * Copyright (c) 2022 Matthias Kannwischer+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [HYBRID]+ *   Hybrid scalar/vector implementations of Keccak and SPHINCS+ on AArch64+ *   Becker, Kannwischer+ *   https://eprint.iacr.org/2022/1243+ */++/*yaml+  Name: keccak_f1600_x2_v84a_aarch64_asm+  Description: AArch64 ARMv8.4-A implementation of Keccak-f[1600] permutation for two sequential states+  Signature: void mlk_keccak_f1600_x2_v84a_aarch64_asm(uint64_t state[50], const uint64_t rc[24])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON, SHA3]+    x0:+      type: buffer+      size_bytes: 400+      permissions: read/write+      c_parameter: uint64_t state[50]+      description: Two sequential Keccak states (state0[25], state1[25])+    x1:+      type: buffer+      size_bytes: 192+      permissions: read-only+      c_parameter: const uint64_t rc[24]+      description: Round constants (24 x uint64_t)+  Stack:+    bytes: 64+    description: register preservation+*/++//+// Author: Hanno Becker <hanno.becker@arm.com>+// Author: Matthias Kannwischer <matthias@kannwischer.eu>+//+// This implementation is essentially from the paper @[HYBRID].+// The only difference is interleaving/deinterleaving of Keccak state+// during load and store, so that the caller need not do this.+//++#include "../../../../common.h"+#if defined(MLK_FIPS202_AARCH64_NEED_X2_V84A) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++#if defined(__ARM_FEATURE_SHA3)++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/fips202/aarch64/src/keccak_f1600_x2_v84a_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(keccak_f1600_x2_v84a_aarch64_asm)+MLK_ASM_FN_SYMBOL(keccak_f1600_x2_v84a_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0x40+        .cfi_adjust_cfa_offset 0x40+        stp d8, d9, [sp]+        .cfi_rel_offset d8, 0x0+        .cfi_rel_offset d9, 0x8+        stp d10, d11, [sp, #0x10]+        .cfi_rel_offset d10, 0x10+        .cfi_rel_offset d11, 0x18+        stp d12, d13, [sp, #0x20]+        .cfi_rel_offset d12, 0x20+        .cfi_rel_offset d13, 0x28+        stp d14, d15, [sp, #0x30]+        .cfi_rel_offset d14, 0x30+        .cfi_rel_offset d15, 0x38+        add x2, x0, #0xc8+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x2], #32+        trn1 v0.2d, v25.2d, v27.2d+        trn2 v1.2d, v25.2d, v27.2d+        trn1 v2.2d, v26.2d, v28.2d+        trn2 v3.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x2], #32+        trn1 v4.2d, v25.2d, v27.2d+        trn2 v5.2d, v25.2d, v27.2d+        trn1 v6.2d, v26.2d, v28.2d+        trn2 v7.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x2], #32+        trn1 v8.2d, v25.2d, v27.2d+        trn2 v9.2d, v25.2d, v27.2d+        trn1 v10.2d, v26.2d, v28.2d+        trn2 v11.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x2], #32+        trn1 v12.2d, v25.2d, v27.2d+        trn2 v13.2d, v25.2d, v27.2d+        trn1 v14.2d, v26.2d, v28.2d+        trn2 v15.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x2], #32+        trn1 v16.2d, v25.2d, v27.2d+        trn2 v17.2d, v25.2d, v27.2d+        trn1 v18.2d, v26.2d, v28.2d+        trn2 v19.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x2], #32+        trn1 v20.2d, v25.2d, v27.2d+        trn2 v21.2d, v25.2d, v27.2d+        trn1 v22.2d, v26.2d, v28.2d+        trn2 v23.2d, v26.2d, v28.2d+        ldr d25, [x0]+        ldr d27, [x2]+        trn1 v24.2d, v25.2d, v27.2d+        mov x2, #0x18               // =24++Lmlk_keccak_f1600_x2_v84a_loop:+        eor3 v30.16b, v0.16b, v5.16b, v10.16b+        eor3 v29.16b, v1.16b, v6.16b, v11.16b+        eor3 v28.16b, v2.16b, v7.16b, v12.16b+        eor3 v27.16b, v3.16b, v8.16b, v13.16b+        eor3 v26.16b, v4.16b, v9.16b, v14.16b+        eor3 v30.16b, v30.16b, v15.16b, v20.16b+        eor3 v29.16b, v29.16b, v16.16b, v21.16b+        eor3 v28.16b, v28.16b, v17.16b, v22.16b+        eor3 v27.16b, v27.16b, v18.16b, v23.16b+        eor3 v26.16b, v26.16b, v19.16b, v24.16b+        rax1 v25.2d, v30.2d, v28.2d+        rax1 v28.2d, v28.2d, v26.2d+        rax1 v26.2d, v26.2d, v29.2d+        rax1 v29.2d, v29.2d, v27.2d+        rax1 v27.2d, v27.2d, v30.2d+        eor v30.16b, v0.16b, v26.16b+        xar v0.2d, v2.2d, v29.2d, #0x2+        xar v2.2d, v12.2d, v29.2d, #0x15+        xar v12.2d, v13.2d, v28.2d, #0x27+        xar v13.2d, v19.2d, v27.2d, #0x38+        xar v19.2d, v23.2d, v28.2d, #0x8+        xar v23.2d, v15.2d, v26.2d, #0x17+        xar v15.2d, v1.2d, v25.2d, #0x3f+        xar v1.2d, v8.2d, v28.2d, #0x9+        xar v8.2d, v16.2d, v25.2d, #0x13+        xar v16.2d, v7.2d, v29.2d, #0x3a+        xar v7.2d, v10.2d, v26.2d, #0x3d+        xar v10.2d, v3.2d, v28.2d, #0x24+        xar v3.2d, v18.2d, v28.2d, #0x2b+        xar v18.2d, v17.2d, v29.2d, #0x31+        xar v17.2d, v11.2d, v25.2d, #0x36+        xar v11.2d, v9.2d, v27.2d, #0x2c+        xar v9.2d, v22.2d, v29.2d, #0x3+        xar v22.2d, v14.2d, v27.2d, #0x19+        xar v14.2d, v20.2d, v26.2d, #0x2e+        xar v20.2d, v4.2d, v27.2d, #0x25+        xar v4.2d, v24.2d, v27.2d, #0x32+        xar v24.2d, v21.2d, v25.2d, #0x3e+        xar v21.2d, v5.2d, v26.2d, #0x1c+        xar v27.2d, v6.2d, v25.2d, #0x14+        ld1r { v31.2d }, [x1], #8+        bcax v5.16b, v10.16b, v7.16b, v11.16b+        bcax v6.16b, v11.16b, v8.16b, v7.16b+        bcax v7.16b, v7.16b, v9.16b, v8.16b+        bcax v8.16b, v8.16b, v10.16b, v9.16b+        bcax v9.16b, v9.16b, v11.16b, v10.16b+        bcax v10.16b, v15.16b, v12.16b, v16.16b+        bcax v11.16b, v16.16b, v13.16b, v12.16b+        bcax v12.16b, v12.16b, v14.16b, v13.16b+        bcax v13.16b, v13.16b, v15.16b, v14.16b+        bcax v14.16b, v14.16b, v16.16b, v15.16b+        bcax v15.16b, v20.16b, v17.16b, v21.16b+        bcax v16.16b, v21.16b, v18.16b, v17.16b+        bcax v17.16b, v17.16b, v19.16b, v18.16b+        bcax v18.16b, v18.16b, v20.16b, v19.16b+        bcax v19.16b, v19.16b, v21.16b, v20.16b+        bcax v20.16b, v0.16b, v22.16b, v1.16b+        bcax v21.16b, v1.16b, v23.16b, v22.16b+        bcax v22.16b, v22.16b, v24.16b, v23.16b+        bcax v23.16b, v23.16b, v0.16b, v24.16b+        bcax v24.16b, v24.16b, v1.16b, v0.16b+        bcax v0.16b, v30.16b, v2.16b, v27.16b+        bcax v1.16b, v27.16b, v3.16b, v2.16b+        bcax v2.16b, v2.16b, v4.16b, v3.16b+        bcax v3.16b, v3.16b, v30.16b, v4.16b+        bcax v4.16b, v4.16b, v27.16b, v30.16b+        eor v0.16b, v0.16b, v31.16b+        sub x2, x2, #0x1+        cbnz x2, Lmlk_keccak_f1600_x2_v84a_loop+        sub x0, x0, #0xc0+        add x2, x0, #0xc8+        trn1 v25.2d, v0.2d, v1.2d+        trn1 v26.2d, v2.2d, v3.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v0.2d, v1.2d+        trn2 v28.2d, v2.2d, v3.2d+        st1 { v27.2d, v28.2d }, [x2], #32+        trn1 v25.2d, v4.2d, v5.2d+        trn1 v26.2d, v6.2d, v7.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v4.2d, v5.2d+        trn2 v28.2d, v6.2d, v7.2d+        st1 { v27.2d, v28.2d }, [x2], #32+        trn1 v25.2d, v8.2d, v9.2d+        trn1 v26.2d, v10.2d, v11.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v8.2d, v9.2d+        trn2 v28.2d, v10.2d, v11.2d+        st1 { v27.2d, v28.2d }, [x2], #32+        trn1 v25.2d, v12.2d, v13.2d+        trn1 v26.2d, v14.2d, v15.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v12.2d, v13.2d+        trn2 v28.2d, v14.2d, v15.2d+        st1 { v27.2d, v28.2d }, [x2], #32+        trn1 v25.2d, v16.2d, v17.2d+        trn1 v26.2d, v18.2d, v19.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v16.2d, v17.2d+        trn2 v28.2d, v18.2d, v19.2d+        st1 { v27.2d, v28.2d }, [x2], #32+        trn1 v25.2d, v20.2d, v21.2d+        trn1 v26.2d, v22.2d, v23.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v20.2d, v21.2d+        trn2 v28.2d, v22.2d, v23.2d+        st1 { v27.2d, v28.2d }, [x2], #32+        str d24, [x0]+        trn2 v25.2d, v24.2d, v24.2d+        str d25, [x2]+        ldp d8, d9, [sp]+        .cfi_restore d8+        .cfi_restore d9+        ldp d10, d11, [sp, #0x10]+        .cfi_restore d10+        .cfi_restore d11+        ldp d12, d13, [sp, #0x20]+        .cfi_restore d12+        .cfi_restore d13+        ldp d14, d15, [sp, #0x30]+        .cfi_restore d14+        .cfi_restore d15+        add sp, sp, #0x40+        .cfi_adjust_cfa_offset -0x40+        ret+        .cfi_endproc++MLK_ASM_FN_SIZE(keccak_f1600_x2_v84a_aarch64_asm)++#endif /* __ARM_FEATURE_SHA3 */++#endif /* MLK_FIPS202_AARCH64_NEED_X2_V84A && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.S view
@@ -0,0 +1,1079 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) 2021-2022 Arm Limited+ * Copyright (c) 2022 Matthias Kannwischer+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++// Author: Hanno Becker <hannobecker@posteo.de>+// Author: Matthias Kannwischer <matthias@kannwischer.eu>++/*yaml+  Name: keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm+  Description: AArch64 hybrid scalar/vector implementation of Keccak-f[1600] permutation for four sequential states+  Signature: void mlk_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm(uint64_t state[100], const uint64_t rc[24])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 800+      permissions: read/write+      c_parameter: uint64_t state[100]+      description: Four sequential Keccak states (state0[25], state1[25], state2[25], state3[25])+    x1:+      type: buffer+      size_bytes: 192+      permissions: read-only+      c_parameter: const uint64_t rc[24]+      description: Round constants (24 x uint64_t)+  Stack:+    bytes: 224+    description: register preservation and temporary storage+*/++#include "../../../../common.h"+#if defined(MLK_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/fips202/aarch64/src/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm)+MLK_ASM_FN_SYMBOL(keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0xe0+        .cfi_adjust_cfa_offset 0xe0+        stp x19, x20, [sp, #0x30]+        .cfi_rel_offset x19, 0x30+        .cfi_rel_offset x20, 0x38+        stp x21, x22, [sp, #0x40]+        .cfi_rel_offset x21, 0x40+        .cfi_rel_offset x22, 0x48+        stp x23, x24, [sp, #0x50]+        .cfi_rel_offset x23, 0x50+        .cfi_rel_offset x24, 0x58+        stp x25, x26, [sp, #0x60]+        .cfi_rel_offset x25, 0x60+        .cfi_rel_offset x26, 0x68+        stp x27, x28, [sp, #0x70]+        .cfi_rel_offset x27, 0x70+        .cfi_rel_offset x28, 0x78+        stp x29, x30, [sp, #0x80]+        .cfi_rel_offset x29, 0x80+        .cfi_rel_offset x30, 0x88+        stp d8, d9, [sp, #0x90]+        .cfi_rel_offset d8, 0x90+        .cfi_rel_offset d9, 0x98+        stp d10, d11, [sp, #0xa0]+        .cfi_rel_offset d10, 0xa0+        .cfi_rel_offset d11, 0xa8+        stp d12, d13, [sp, #0xb0]+        .cfi_rel_offset d12, 0xb0+        .cfi_rel_offset d13, 0xb8+        stp d14, d15, [sp, #0xc0]+        .cfi_rel_offset d14, 0xc0+        .cfi_rel_offset d15, 0xc8+        mov x29, x1+        mov x30, #0x0               // =0+        str x30, [sp, #0x20]+        str x29, [sp, #0x8]+        str x29, [sp, #0x10]+        str x0, [sp]+        add x4, x0, #0xc8+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v0.2d, v25.2d, v27.2d+        trn2 v1.2d, v25.2d, v27.2d+        trn1 v2.2d, v26.2d, v28.2d+        trn2 v3.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v4.2d, v25.2d, v27.2d+        trn2 v5.2d, v25.2d, v27.2d+        trn1 v6.2d, v26.2d, v28.2d+        trn2 v7.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v8.2d, v25.2d, v27.2d+        trn2 v9.2d, v25.2d, v27.2d+        trn1 v10.2d, v26.2d, v28.2d+        trn2 v11.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v12.2d, v25.2d, v27.2d+        trn2 v13.2d, v25.2d, v27.2d+        trn1 v14.2d, v26.2d, v28.2d+        trn2 v15.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v16.2d, v25.2d, v27.2d+        trn2 v17.2d, v25.2d, v27.2d+        trn1 v18.2d, v26.2d, v28.2d+        trn2 v19.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v20.2d, v25.2d, v27.2d+        trn2 v21.2d, v25.2d, v27.2d+        trn1 v22.2d, v26.2d, v28.2d+        trn2 v23.2d, v26.2d, v28.2d+        ldr d25, [x0]+        ldr d27, [x4]+        trn1 v24.2d, v25.2d, v27.2d+        sub x0, x0, #0xc0+        add x0, x0, #0x190+        ldp x1, x6, [x0]+        ldp x11, x16, [x0, #0x10]+        ldp x21, x2, [x0, #0x20]+        ldp x7, x12, [x0, #0x30]+        ldp x17, x22, [x0, #0x40]+        ldp x3, x8, [x0, #0x50]+        ldp x13, x28, [x0, #0x60]+        ldp x23, x4, [x0, #0x70]+        ldp x9, x14, [x0, #0x80]+        ldp x19, x24, [x0, #0x90]+        ldp x5, x10, [x0, #0xa0]+        ldp x15, x20, [x0, #0xb0]+        ldr x25, [x0, #0xc0]+        sub x0, x0, #0x190++Lmlk_keccak_f1600_x4_v8a_scalar_hybrid_initial:+        eor x30, x24, x25+        eor x27, x9, x10+        eor v30.16b, v0.16b, v5.16b+        eor v30.16b, v30.16b, v10.16b+        eor x0, x30, x21+        eor v30.16b, v30.16b, v15.16b+        eor x26, x27, x6+        eor x27, x26, x7+        eor v30.16b, v30.16b, v20.16b+        eor x29, x0, x22+        eor v29.16b, v1.16b, v6.16b+        eor x26, x29, x23+        eor v29.16b, v29.16b, v11.16b+        eor x29, x4, x5+        eor x30, x29, x1+        eor v29.16b, v29.16b, v16.16b+        eor x0, x27, x8+        eor v29.16b, v29.16b, v21.16b+        eor x29, x30, x2+        eor v28.16b, v2.16b, v7.16b+        eor x30, x19, x20+        eor x30, x30, x16+        eor v28.16b, v28.16b, v12.16b+        eor x27, x26, x0, ror #63+        eor v28.16b, v28.16b, v17.16b+        eor x4, x4, x27+        eor v28.16b, v28.16b, v22.16b+        eor x30, x30, x17+        eor x30, x30, x28+        eor v27.16b, v3.16b, v8.16b+        eor x29, x29, x3+        eor v27.16b, v27.16b, v13.16b+        eor x0, x0, x30, ror #63+        eor v27.16b, v27.16b, v18.16b+        eor x30, x30, x29, ror #63+        eor x22, x22, x30+        eor v27.16b, v27.16b, v23.16b+        eor x23, x23, x30+        eor v26.16b, v4.16b, v9.16b+        str x23, [sp, #0xd0]+        eor v26.16b, v26.16b, v14.16b+        eor x23, x14, x15+        eor x14, x14, x0+        eor v26.16b, v26.16b, v19.16b+        eor x23, x23, x11+        eor v26.16b, v26.16b, v24.16b+        eor x15, x15, x0+        eor x1, x1, x27+        add v31.2d, v28.2d, v28.2d+        eor x23, x23, x12+        sri v31.2d, v28.2d, #0x3f+        eor x23, x23, x13+        eor v25.16b, v31.16b, v30.16b+        eor x11, x11, x0+        eor x29, x29, x23, ror #63+        add v31.2d, v26.2d, v26.2d+        eor x23, x23, x26, ror #63+        sri v31.2d, v26.2d, #0x3f+        eor x26, x13, x0+        eor v28.16b, v31.16b, v28.16b+        eor x13, x28, x23+        eor x28, x24, x30+        add v31.2d, v29.2d, v29.2d+        eor x24, x16, x23+        sri v31.2d, v29.2d, #0x3f+        eor x16, x21, x30+        eor v26.16b, v31.16b, v26.16b+        eor x21, x25, x30+        eor x30, x19, x23+        add v31.2d, v27.2d, v27.2d+        eor x19, x20, x23+        sri v31.2d, v27.2d, #0x3f+        eor x20, x17, x23+        eor v29.16b, v31.16b, v29.16b+        eor x17, x12, x0+        eor x0, x2, x27+        add v31.2d, v30.2d, v30.2d+        eor x2, x6, x29+        sri v31.2d, v30.2d, #0x3f+        eor x6, x8, x29+        eor v27.16b, v31.16b, v27.16b+        bic x8, x28, x13, ror #47+        eor x12, x3, x27+        eor v30.16b, v0.16b, v26.16b+        bic x3, x13, x17, ror #19+        eor v31.16b, v2.16b, v29.16b+        eor x5, x5, x27+        ldr x27, [sp, #0xd0]+        shl v0.2d, v31.2d, #0x3e+        bic x25, x17, x2, ror #5+        sri v0.2d, v31.2d, #0x2+        eor x9, x9, x29+        eor v31.16b, v12.16b, v29.16b+        eor x23, x25, x5, ror #52+        eor x3, x3, x2, ror #24+        shl v2.2d, v31.2d, #0x2b+        eor x8, x8, x17, ror #2+        sri v2.2d, v31.2d, #0x15+        eor x17, x10, x29+        eor v31.16b, v13.16b, v28.16b+        bic x25, x12, x22, ror #47+        eor x29, x7, x29+        shl v12.2d, v31.2d, #0x19+        bic x10, x4, x27, ror #2+        sri v12.2d, v31.2d, #0x27+        bic x7, x5, x28, ror #10+        eor v31.16b, v19.16b, v27.16b+        eor x10, x10, x20, ror #50+        eor x13, x7, x13, ror #57+        shl v13.2d, v31.2d, #0x8+        bic x7, x2, x5, ror #47+        sri v13.2d, v31.2d, #0x38+        eor x2, x25, x24, ror #39+        eor v31.16b, v23.16b, v28.16b+        bic x25, x20, x11, ror #57+        bic x5, x17, x4, ror #25+        shl v19.2d, v31.2d, #0x38+        eor x25, x25, x17, ror #53+        sri v19.2d, v31.2d, #0x8+        bic x17, x11, x17, ror #60+        eor v31.16b, v15.16b, v26.16b+        eor x28, x7, x28, ror #57+        bic x7, x9, x12, ror #42+        shl v23.2d, v31.2d, #0x29+        eor x7, x7, x22, ror #25+        sri v23.2d, v31.2d, #0x17+        bic x22, x22, x24, ror #56+        bic x24, x24, x15, ror #31+        eor v31.16b, v1.16b, v25.16b+        eor x22, x22, x15, ror #23+        shl v15.2d, v31.2d, #0x1+        bic x20, x27, x20, ror #48+        sri v15.2d, v31.2d, #0x3f+        bic x15, x15, x9, ror #16+        eor x12, x15, x12, ror #58+        eor v31.16b, v8.16b, v28.16b+        eor x15, x5, x27, ror #27+        shl v1.2d, v31.2d, #0x37+        eor x5, x20, x11, ror #41+        sri v1.2d, v31.2d, #0x9+        ldr x11, [sp, #0x8]+        eor x20, x17, x4, ror #21+        eor v31.16b, v16.16b, v25.16b+        eor x17, x24, x9, ror #47+        shl v8.2d, v31.2d, #0x2d+        mov x24, #0x1               // =1+        sri v8.2d, v31.2d, #0x13+        bic x9, x0, x16, ror #9+        str x24, [sp, #0x18]+        eor v31.16b, v7.16b, v29.16b+        bic x24, x29, x1, ror #44+        shl v16.2d, v31.2d, #0x6+        bic x27, x1, x21, ror #50+        sri v16.2d, v31.2d, #0x3a+        bic x4, x26, x29, ror #63+        eor x1, x1, x4, ror #21+        eor v31.16b, v10.16b, v26.16b+        ldr x11, [x11]+        shl v7.2d, v31.2d, #0x3+        bic x4, x21, x30, ror #57+        sri v7.2d, v31.2d, #0x3d+        eor x21, x24, x21, ror #30+        eor x24, x9, x19, ror #44+        eor v31.16b, v3.16b, v28.16b+        bic x9, x14, x6, ror #5+        shl v10.2d, v31.2d, #0x1c+        eor x9, x9, x0, ror #43+        sri v10.2d, v31.2d, #0x24+        bic x0, x6, x0, ror #38+        eor x1, x1, x11+        eor v31.16b, v18.16b, v28.16b+        eor x11, x4, x26, ror #35+        shl v3.2d, v31.2d, #0x15+        eor x4, x0, x16, ror #47+        bic x0, x16, x19, ror #35+        sri v3.2d, v31.2d, #0x2b+        eor x16, x27, x30, ror #43+        eor v31.16b, v17.16b, v29.16b+        bic x27, x30, x26, ror #42+        shl v18.2d, v31.2d, #0xf+        bic x26, x19, x14, ror #41+        eor x19, x0, x14, ror #12+        sri v18.2d, v31.2d, #0x31+        eor x14, x26, x6, ror #46+        eor v31.16b, v11.16b, v25.16b+        eor x6, x27, x29, ror #41+        shl v17.2d, v31.2d, #0xa+        eor x0, x15, x11, ror #52+        eor x0, x0, x13, ror #48+        sri v17.2d, v31.2d, #0x36+        eor x26, x8, x9, ror #57+        eor v31.16b, v9.16b, v27.16b+        eor x27, x0, x14, ror #10+        shl v11.2d, v31.2d, #0x14+        eor x29, x16, x28, ror #63+        eor x26, x26, x6, ror #51+        sri v11.2d, v31.2d, #0x2c+        eor x30, x23, x22, ror #50+        eor v31.16b, v22.16b, v29.16b+        eor x0, x26, x10, ror #31+        shl v9.2d, v31.2d, #0x3d+        eor x29, x29, x19, ror #37+        eor x27, x27, x12, ror #5+        sri v9.2d, v31.2d, #0x3+        eor x30, x30, x24, ror #34+        eor v31.16b, v14.16b, v27.16b+        eor x0, x0, x7, ror #27+        shl v22.2d, v31.2d, #0x27+        eor x26, x30, x21, ror #26+        eor x26, x26, x25, ror #15+        sri v22.2d, v31.2d, #0x19+        ror x30, x27, #0x3e+        eor v31.16b, v20.16b, v26.16b+        eor x30, x30, x26, ror #57+        ror x26, x26, #0x3a+        shl v14.2d, v31.2d, #0x12+        eor x16, x30, x16+        sri v14.2d, v31.2d, #0x2e+        eor x28, x30, x28, ror #63+        eor v31.16b, v4.16b, v27.16b+        str x28, [sp, #0xd0]+        eor x29, x29, x17, ror #36+        shl v20.2d, v31.2d, #0x1b+        eor x28, x1, x2, ror #61+        sri v20.2d, v31.2d, #0x25+        eor x19, x30, x19, ror #37+        eor v31.16b, v24.16b, v27.16b+        eor x29, x29, x20, ror #2+        eor x28, x28, x4, ror #54+        shl v4.2d, v31.2d, #0xe+        eor x26, x26, x0, ror #55+        sri v4.2d, v31.2d, #0x32+        eor x28, x28, x3, ror #39+        eor v31.16b, v21.16b, v25.16b+        eor x28, x28, x5, ror #25+        ror x0, x0, #0x38+        shl v24.2d, v31.2d, #0x2+        eor x0, x0, x29, ror #63+        sri v24.2d, v31.2d, #0x3e+        eor x27, x28, x27, ror #61+        eor v31.16b, v5.16b, v26.16b+        eor x13, x0, x13, ror #46+        eor x28, x29, x28, ror #63+        shl v21.2d, v31.2d, #0x24+        eor x29, x30, x20, ror #2+        sri v21.2d, v31.2d, #0x1c+        eor x20, x26, x3, ror #39+        eor v31.16b, v6.16b, v25.16b+        eor x11, x0, x11, ror #50+        eor x25, x28, x25, ror #9+        shl v27.2d, v31.2d, #0x2c+        eor x3, x28, x21, ror #20+        sri v27.2d, v31.2d, #0x14+        eor x21, x26, x1+        eor x9, x27, x9, ror #49+        bic v31.16b, v7.16b, v11.16b+        eor x24, x28, x24, ror #28+        eor v5.16b, v31.16b, v10.16b+        eor x1, x30, x17, ror #36+        bic v31.16b, v8.16b, v7.16b+        eor x14, x0, x14, ror #8+        eor x22, x28, x22, ror #44+        eor v6.16b, v31.16b, v11.16b+        eor x8, x27, x8, ror #56+        bic v31.16b, v9.16b, v8.16b+        eor x17, x27, x7, ror #19+        eor v7.16b, v31.16b, v7.16b+        eor x15, x0, x15, ror #62+        bic x7, x20, x22, ror #47+        bic v31.16b, v10.16b, v9.16b+        eor x4, x26, x4, ror #54+        eor v8.16b, v31.16b, v8.16b+        eor x0, x0, x12, ror #3+        bic v31.16b, v11.16b, v10.16b+        eor x28, x28, x23, ror #58+        eor x23, x26, x2, ror #61+        eor v9.16b, v31.16b, v9.16b+        eor x26, x26, x5, ror #25+        bic v31.16b, v12.16b, v16.16b+        eor x2, x7, x16, ror #39+        eor v10.16b, v31.16b, v15.16b+        bic x7, x9, x20, ror #42+        bic x30, x15, x9, ror #16+        bic v31.16b, v13.16b, v12.16b+        eor x7, x7, x22, ror #25+        eor v11.16b, v31.16b, v16.16b+        eor x12, x30, x20, ror #58+        bic v31.16b, v14.16b, v13.16b+        bic x20, x22, x16, ror #56+        eor x30, x27, x6, ror #43+        eor v12.16b, v31.16b, v12.16b+        eor x22, x20, x15, ror #23+        bic v31.16b, v15.16b, v14.16b+        bic x6, x19, x13, ror #42+        eor v13.16b, v31.16b, v13.16b+        eor x6, x6, x17, ror #41+        bic x5, x13, x17, ror #63+        bic v31.16b, v16.16b, v15.16b+        eor x5, x21, x5, ror #21+        eor v14.16b, v31.16b, v14.16b+        bic x17, x17, x21, ror #44+        eor x27, x27, x10, ror #23+        bic v31.16b, v17.16b, v21.16b+        bic x21, x21, x25, ror #50+        eor v15.16b, v31.16b, v20.16b+        bic x20, x27, x4, ror #25+        bic v31.16b, v18.16b, v17.16b+        bic x10, x16, x15, ror #31+        eor x16, x21, x19, ror #43+        eor v16.16b, v31.16b, v21.16b+        eor x21, x17, x25, ror #30+        bic v31.16b, v19.16b, v18.16b+        bic x19, x25, x19, ror #57+        eor v17.16b, v31.16b, v17.16b+        ldr x25, [sp, #0x18]+        eor x17, x10, x9, ror #47+        bic v31.16b, v20.16b, v19.16b+        ldr x9, [sp, #0x8]+        eor v18.16b, v31.16b, v18.16b+        eor x15, x20, x28, ror #27+        bic v31.16b, v21.16b, v20.16b+        bic x20, x4, x28, ror #2+        eor x10, x20, x1, ror #50+        eor v19.16b, v31.16b, v19.16b+        bic x20, x11, x27, ror #60+        bic v31.16b, v22.16b, v1.16b+        eor x20, x20, x4, ror #21+        eor v20.16b, v31.16b, v0.16b+        bic x4, x28, x1, ror #48+        bic x1, x1, x11, ror #57+        bic v31.16b, v23.16b, v22.16b+        ldr x28, [x9, x25, lsl #3]+        eor v21.16b, v31.16b, v1.16b+        ldr x9, [sp, #0xd0]+        bic v31.16b, v24.16b, v23.16b+        add x25, x25, #0x1+        str x25, [sp, #0x18]+        eor v22.16b, v31.16b, v22.16b+        cmp x25, #0x17+        bic v31.16b, v0.16b, v24.16b+        eor x25, x1, x27, ror #53+        bic x27, x30, x26, ror #47+        eor v23.16b, v31.16b, v23.16b+        eor x1, x5, x28+        bic v31.16b, v1.16b, v0.16b+        eor x5, x4, x11, ror #41+        eor v24.16b, v31.16b, v24.16b+        eor x11, x19, x13, ror #35+        bic x13, x26, x24, ror #10+        bic v31.16b, v2.16b, v27.16b+        eor x28, x27, x24, ror #57+        eor v0.16b, v31.16b, v30.16b+        bic x27, x24, x9, ror #47+        bic v31.16b, v3.16b, v2.16b+        bic x19, x23, x3, ror #9+        bic x4, x29, x14, ror #41+        eor v1.16b, v31.16b, v27.16b+        eor x24, x19, x29, ror #44+        bic v31.16b, v4.16b, v3.16b+        bic x29, x3, x29, ror #35+        eor v2.16b, v31.16b, v2.16b+        eor x13, x13, x9, ror #57+        eor x19, x29, x14, ror #12+        bic v31.16b, v30.16b, v4.16b+        bic x29, x9, x0, ror #19+        eor v3.16b, v31.16b, v3.16b+        bic x14, x14, x8, ror #5+        bic v31.16b, v27.16b, v30.16b+        eor x9, x14, x23, ror #43+        eor x14, x4, x8, ror #46+        eor v4.16b, v31.16b, v4.16b+        bic x23, x8, x23, ror #38+        eor x8, x27, x0, ror #2+        eor x4, x23, x3, ror #47+        bic x3, x0, x30, ror #5+        eor x23, x3, x26, ror #52+        eor x3, x29, x30, ror #24+        ldr x30, [sp, #0x10]+        ld1r { v28.2d }, [x30], #8+        str x30, [sp, #0x10]+        eor v0.16b, v0.16b, v28.16b++Lmlk_keccak_f1600_x4_v8a_scalar_hybrid_loop:+        eor x0, x15, x11, ror #52+        eor x0, x0, x13, ror #48+        eor v30.16b, v0.16b, v5.16b+        eor v30.16b, v30.16b, v10.16b+        eor x26, x8, x9, ror #57+        eor v30.16b, v30.16b, v15.16b+        eor x27, x0, x14, ror #10+        eor x29, x16, x28, ror #63+        eor v30.16b, v30.16b, v20.16b+        eor x26, x26, x6, ror #51+        eor v29.16b, v1.16b, v6.16b+        eor x30, x23, x22, ror #50+        eor v29.16b, v29.16b, v11.16b+        eor x0, x26, x10, ror #31+        eor x29, x29, x19, ror #37+        eor v29.16b, v29.16b, v16.16b+        eor x27, x27, x12, ror #5+        eor v29.16b, v29.16b, v21.16b+        eor x30, x30, x24, ror #34+        eor x0, x0, x7, ror #27+        eor v28.16b, v2.16b, v7.16b+        eor x26, x30, x21, ror #26+        eor v28.16b, v28.16b, v12.16b+        eor x26, x26, x25, ror #15+        eor v28.16b, v28.16b, v17.16b+        ror x30, x27, #0x3e+        eor x30, x30, x26, ror #57+        eor v28.16b, v28.16b, v22.16b+        ror x26, x26, #0x3a+        eor v27.16b, v3.16b, v8.16b+        eor x16, x30, x16+        eor v27.16b, v27.16b, v13.16b+        eor x28, x30, x28, ror #63+        str x28, [sp, #0xd0]+        eor v27.16b, v27.16b, v18.16b+        eor x29, x29, x17, ror #36+        eor v27.16b, v27.16b, v23.16b+        eor x28, x1, x2, ror #61+        eor x19, x30, x19, ror #37+        eor v26.16b, v4.16b, v9.16b+        eor x29, x29, x20, ror #2+        eor v26.16b, v26.16b, v14.16b+        eor x28, x28, x4, ror #54+        eor v26.16b, v26.16b, v19.16b+        eor x26, x26, x0, ror #55+        eor x28, x28, x3, ror #39+        eor v26.16b, v26.16b, v24.16b+        eor x28, x28, x5, ror #25+        add v31.2d, v28.2d, v28.2d+        ror x0, x0, #0x38+        eor x0, x0, x29, ror #63+        sri v31.2d, v28.2d, #0x3f+        eor x27, x28, x27, ror #61+        eor v25.16b, v31.16b, v30.16b+        eor x13, x0, x13, ror #46+        add v31.2d, v26.2d, v26.2d+        eor x28, x29, x28, ror #63+        eor x29, x30, x20, ror #2+        sri v31.2d, v26.2d, #0x3f+        eor x20, x26, x3, ror #39+        eor v28.16b, v31.16b, v28.16b+        eor x11, x0, x11, ror #50+        add v31.2d, v29.2d, v29.2d+        eor x25, x28, x25, ror #9+        eor x3, x28, x21, ror #20+        sri v31.2d, v29.2d, #0x3f+        eor x21, x26, x1+        eor v26.16b, v31.16b, v26.16b+        eor x9, x27, x9, ror #49+        eor x24, x28, x24, ror #28+        add v31.2d, v27.2d, v27.2d+        eor x1, x30, x17, ror #36+        sri v31.2d, v27.2d, #0x3f+        eor x14, x0, x14, ror #8+        eor v29.16b, v31.16b, v29.16b+        eor x22, x28, x22, ror #44+        eor x8, x27, x8, ror #56+        add v31.2d, v30.2d, v30.2d+        eor x17, x27, x7, ror #19+        sri v31.2d, v30.2d, #0x3f+        eor x15, x0, x15, ror #62+        bic x7, x20, x22, ror #47+        eor v27.16b, v31.16b, v27.16b+        eor x4, x26, x4, ror #54+        eor v30.16b, v0.16b, v26.16b+        eor x0, x0, x12, ror #3+        eor v31.16b, v2.16b, v29.16b+        eor x28, x28, x23, ror #58+        eor x23, x26, x2, ror #61+        shl v0.2d, v31.2d, #0x3e+        eor x26, x26, x5, ror #25+        sri v0.2d, v31.2d, #0x2+        eor x2, x7, x16, ror #39+        eor v31.16b, v12.16b, v29.16b+        bic x7, x9, x20, ror #42+        bic x30, x15, x9, ror #16+        shl v2.2d, v31.2d, #0x2b+        eor x7, x7, x22, ror #25+        sri v2.2d, v31.2d, #0x15+        eor x12, x30, x20, ror #58+        bic x20, x22, x16, ror #56+        eor v31.16b, v13.16b, v28.16b+        eor x30, x27, x6, ror #43+        shl v12.2d, v31.2d, #0x19+        eor x22, x20, x15, ror #23+        sri v12.2d, v31.2d, #0x27+        bic x6, x19, x13, ror #42+        eor x6, x6, x17, ror #41+        eor v31.16b, v19.16b, v27.16b+        bic x5, x13, x17, ror #63+        shl v13.2d, v31.2d, #0x8+        eor x5, x21, x5, ror #21+        sri v13.2d, v31.2d, #0x38+        bic x17, x17, x21, ror #44+        eor x27, x27, x10, ror #23+        eor v31.16b, v23.16b, v28.16b+        bic x21, x21, x25, ror #50+        shl v19.2d, v31.2d, #0x38+        bic x20, x27, x4, ror #25+        bic x10, x16, x15, ror #31+        sri v19.2d, v31.2d, #0x8+        eor x16, x21, x19, ror #43+        eor v31.16b, v15.16b, v26.16b+        eor x21, x17, x25, ror #30+        shl v23.2d, v31.2d, #0x29+        bic x19, x25, x19, ror #57+        ldr x25, [sp, #0x18]+        sri v23.2d, v31.2d, #0x17+        eor x17, x10, x9, ror #47+        eor v31.16b, v1.16b, v25.16b+        ldr x9, [sp, #0x8]+        eor x15, x20, x28, ror #27+        shl v15.2d, v31.2d, #0x1+        bic x20, x4, x28, ror #2+        sri v15.2d, v31.2d, #0x3f+        eor x10, x20, x1, ror #50+        eor v31.16b, v8.16b, v28.16b+        bic x20, x11, x27, ror #60+        eor x20, x20, x4, ror #21+        shl v1.2d, v31.2d, #0x37+        bic x4, x28, x1, ror #48+        sri v1.2d, v31.2d, #0x9+        bic x1, x1, x11, ror #57+        eor v31.16b, v16.16b, v25.16b+        ldr x28, [x9, x25, lsl #3]+        ldr x9, [sp, #0xd0]+        shl v8.2d, v31.2d, #0x2d+        add x25, x25, #0x1+        sri v8.2d, v31.2d, #0x13+        str x25, [sp, #0x18]+        cmp x25, #0x17+        eor v31.16b, v7.16b, v29.16b+        eor x25, x1, x27, ror #53+        shl v16.2d, v31.2d, #0x6+        bic x27, x30, x26, ror #47+        sri v16.2d, v31.2d, #0x3a+        eor x1, x5, x28+        eor x5, x4, x11, ror #41+        eor v31.16b, v10.16b, v26.16b+        eor x11, x19, x13, ror #35+        shl v7.2d, v31.2d, #0x3+        bic x13, x26, x24, ror #10+        eor x28, x27, x24, ror #57+        sri v7.2d, v31.2d, #0x3d+        bic x27, x24, x9, ror #47+        eor v31.16b, v3.16b, v28.16b+        bic x19, x23, x3, ror #9+        shl v10.2d, v31.2d, #0x1c+        bic x4, x29, x14, ror #41+        eor x24, x19, x29, ror #44+        sri v10.2d, v31.2d, #0x24+        bic x29, x3, x29, ror #35+        eor v31.16b, v18.16b, v28.16b+        eor x13, x13, x9, ror #57+        shl v3.2d, v31.2d, #0x15+        eor x19, x29, x14, ror #12+        bic x29, x9, x0, ror #19+        sri v3.2d, v31.2d, #0x2b+        bic x14, x14, x8, ror #5+        eor v31.16b, v17.16b, v29.16b+        eor x9, x14, x23, ror #43+        eor x14, x4, x8, ror #46+        shl v18.2d, v31.2d, #0xf+        bic x23, x8, x23, ror #38+        sri v18.2d, v31.2d, #0x31+        eor x8, x27, x0, ror #2+        eor v31.16b, v11.16b, v25.16b+        eor x4, x23, x3, ror #47+        bic x3, x0, x30, ror #5+        shl v17.2d, v31.2d, #0xa+        eor x23, x3, x26, ror #52+        sri v17.2d, v31.2d, #0x36+        eor x3, x29, x30, ror #24+        eor x0, x15, x11, ror #52+        eor v31.16b, v9.16b, v27.16b+        eor x0, x0, x13, ror #48+        shl v11.2d, v31.2d, #0x14+        eor x26, x8, x9, ror #57+        sri v11.2d, v31.2d, #0x2c+        eor x27, x0, x14, ror #10+        eor x29, x16, x28, ror #63+        eor v31.16b, v22.16b, v29.16b+        eor x26, x26, x6, ror #51+        shl v9.2d, v31.2d, #0x3d+        eor x30, x23, x22, ror #50+        sri v9.2d, v31.2d, #0x3+        eor x0, x26, x10, ror #31+        eor x29, x29, x19, ror #37+        eor v31.16b, v14.16b, v27.16b+        eor x27, x27, x12, ror #5+        shl v22.2d, v31.2d, #0x27+        eor x30, x30, x24, ror #34+        eor x0, x0, x7, ror #27+        sri v22.2d, v31.2d, #0x19+        eor x26, x30, x21, ror #26+        eor v31.16b, v20.16b, v26.16b+        eor x26, x26, x25, ror #15+        shl v14.2d, v31.2d, #0x12+        ror x30, x27, #0x3e+        eor x30, x30, x26, ror #57+        sri v14.2d, v31.2d, #0x2e+        ror x26, x26, #0x3a+        eor v31.16b, v4.16b, v27.16b+        eor x16, x30, x16+        shl v20.2d, v31.2d, #0x1b+        eor x28, x30, x28, ror #63+        str x28, [sp, #0xd0]+        sri v20.2d, v31.2d, #0x25+        eor x29, x29, x17, ror #36+        eor v31.16b, v24.16b, v27.16b+        eor x28, x1, x2, ror #61+        eor x19, x30, x19, ror #37+        shl v4.2d, v31.2d, #0xe+        eor x29, x29, x20, ror #2+        sri v4.2d, v31.2d, #0x32+        eor x28, x28, x4, ror #54+        eor v31.16b, v21.16b, v25.16b+        eor x26, x26, x0, ror #55+        eor x28, x28, x3, ror #39+        shl v24.2d, v31.2d, #0x2+        eor x28, x28, x5, ror #25+        sri v24.2d, v31.2d, #0x3e+        ror x0, x0, #0x38+        eor x0, x0, x29, ror #63+        eor v31.16b, v5.16b, v26.16b+        eor x27, x28, x27, ror #61+        shl v21.2d, v31.2d, #0x24+        eor x13, x0, x13, ror #46+        sri v21.2d, v31.2d, #0x1c+        eor x28, x29, x28, ror #63+        eor x29, x30, x20, ror #2+        eor v31.16b, v6.16b, v25.16b+        eor x20, x26, x3, ror #39+        shl v27.2d, v31.2d, #0x2c+        eor x11, x0, x11, ror #50+        sri v27.2d, v31.2d, #0x14+        eor x25, x28, x25, ror #9+        eor x3, x28, x21, ror #20+        bic v31.16b, v7.16b, v11.16b+        eor x21, x26, x1+        eor v5.16b, v31.16b, v10.16b+        eor x9, x27, x9, ror #49+        eor x24, x28, x24, ror #28+        bic v31.16b, v8.16b, v7.16b+        eor x1, x30, x17, ror #36+        eor v6.16b, v31.16b, v11.16b+        eor x14, x0, x14, ror #8+        bic v31.16b, v9.16b, v8.16b+        eor x22, x28, x22, ror #44+        eor x8, x27, x8, ror #56+        eor v7.16b, v31.16b, v7.16b+        eor x17, x27, x7, ror #19+        bic v31.16b, v10.16b, v9.16b+        eor x15, x0, x15, ror #62+        bic x7, x20, x22, ror #47+        eor v8.16b, v31.16b, v8.16b+        eor x4, x26, x4, ror #54+        bic v31.16b, v11.16b, v10.16b+        eor x0, x0, x12, ror #3+        eor v9.16b, v31.16b, v9.16b+        eor x28, x28, x23, ror #58+        eor x23, x26, x2, ror #61+        bic v31.16b, v12.16b, v16.16b+        eor x26, x26, x5, ror #25+        eor v10.16b, v31.16b, v15.16b+        eor x2, x7, x16, ror #39+        bic v31.16b, v13.16b, v12.16b+        bic x7, x9, x20, ror #42+        bic x30, x15, x9, ror #16+        eor v11.16b, v31.16b, v16.16b+        eor x7, x7, x22, ror #25+        bic v31.16b, v14.16b, v13.16b+        eor x12, x30, x20, ror #58+        bic x20, x22, x16, ror #56+        eor v12.16b, v31.16b, v12.16b+        eor x30, x27, x6, ror #43+        bic v31.16b, v15.16b, v14.16b+        eor x22, x20, x15, ror #23+        eor v13.16b, v31.16b, v13.16b+        bic x6, x19, x13, ror #42+        eor x6, x6, x17, ror #41+        bic v31.16b, v16.16b, v15.16b+        bic x5, x13, x17, ror #63+        eor v14.16b, v31.16b, v14.16b+        eor x5, x21, x5, ror #21+        bic v31.16b, v17.16b, v21.16b+        bic x17, x17, x21, ror #44+        eor x27, x27, x10, ror #23+        eor v15.16b, v31.16b, v20.16b+        bic x21, x21, x25, ror #50+        bic v31.16b, v18.16b, v17.16b+        bic x20, x27, x4, ror #25+        bic x10, x16, x15, ror #31+        eor v16.16b, v31.16b, v21.16b+        eor x16, x21, x19, ror #43+        bic v31.16b, v19.16b, v18.16b+        eor x21, x17, x25, ror #30+        eor v17.16b, v31.16b, v17.16b+        bic x19, x25, x19, ror #57+        ldr x25, [sp, #0x18]+        bic v31.16b, v20.16b, v19.16b+        eor x17, x10, x9, ror #47+        eor v18.16b, v31.16b, v18.16b+        ldr x9, [sp, #0x8]+        eor x15, x20, x28, ror #27+        bic v31.16b, v21.16b, v20.16b+        bic x20, x4, x28, ror #2+        eor v19.16b, v31.16b, v19.16b+        eor x10, x20, x1, ror #50+        bic v31.16b, v22.16b, v1.16b+        bic x20, x11, x27, ror #60+        eor x20, x20, x4, ror #21+        eor v20.16b, v31.16b, v0.16b+        bic x4, x28, x1, ror #48+        bic v31.16b, v23.16b, v22.16b+        bic x1, x1, x11, ror #57+        eor v21.16b, v31.16b, v1.16b+        ldr x28, [x9, x25, lsl #3]+        ldr x9, [sp, #0xd0]+        bic v31.16b, v24.16b, v23.16b+        add x25, x25, #0x1+        eor v22.16b, v31.16b, v22.16b+        str x25, [sp, #0x18]+        cmp x25, #0x17+        bic v31.16b, v0.16b, v24.16b+        eor x25, x1, x27, ror #53+        eor v23.16b, v31.16b, v23.16b+        bic x27, x30, x26, ror #47+        bic v31.16b, v1.16b, v0.16b+        eor x1, x5, x28+        eor x5, x4, x11, ror #41+        eor v24.16b, v31.16b, v24.16b+        eor x11, x19, x13, ror #35+        bic v31.16b, v2.16b, v27.16b+        bic x13, x26, x24, ror #10+        eor x28, x27, x24, ror #57+        eor v0.16b, v31.16b, v30.16b+        bic x27, x24, x9, ror #47+        bic v31.16b, v3.16b, v2.16b+        bic x19, x23, x3, ror #9+        eor v1.16b, v31.16b, v27.16b+        bic x4, x29, x14, ror #41+        eor x24, x19, x29, ror #44+        bic v31.16b, v4.16b, v3.16b+        bic x29, x3, x29, ror #35+        eor v2.16b, v31.16b, v2.16b+        eor x13, x13, x9, ror #57+        bic v31.16b, v30.16b, v4.16b+        eor x19, x29, x14, ror #12+        bic x29, x9, x0, ror #19+        eor v3.16b, v31.16b, v3.16b+        bic x14, x14, x8, ror #5+        bic v31.16b, v27.16b, v30.16b+        eor x9, x14, x23, ror #43+        eor x14, x4, x8, ror #46+        eor v4.16b, v31.16b, v4.16b+        bic x23, x8, x23, ror #38+        eor x8, x27, x0, ror #2+        eor x4, x23, x3, ror #47+        bic x3, x0, x30, ror #5+        eor x23, x3, x26, ror #52+        eor x3, x29, x30, ror #24+        ldr x30, [sp, #0x10]+        ld1r { v28.2d }, [x30], #8+        str x30, [sp, #0x10]+        eor v0.16b, v0.16b, v28.16b++Lmlk_keccak_f1600_x4_v8a_scalar_hybrid_loop_end:+        b.le Lmlk_keccak_f1600_x4_v8a_scalar_hybrid_loop+        ror x2, x2, #0x3d+        ror x3, x3, #0x27+        ror x4, x4, #0x36+        ror x5, x5, #0x19+        ror x6, x6, #0x2b+        ror x7, x7, #0x13+        ror x8, x8, #0x38+        ror x9, x9, #0x31+        ror x10, x10, #0x17+        ror x11, x11, #0x32+        ror x12, x12, #0x3+        ror x13, x13, #0x2e+        ror x14, x14, #0x8+        ror x15, x15, #0x3e+        ror x17, x17, #0x24+        ror x28, x28, #0x3f+        ror x19, x19, #0x25+        ror x20, x20, #0x2+        ror x21, x21, #0x14+        ror x22, x22, #0x2c+        ror x23, x23, #0x3a+        ror x24, x24, #0x1c+        ror x25, x25, #0x9+        ldr x30, [sp, #0x20]+        cmp x30, #0x1+        b.eq Lmlk_keccak_f1600_x4_v8a_scalar_hybrid_done+        mov x30, #0x1               // =1+        str x30, [sp, #0x20]+        ldr x0, [sp]+        add x0, x0, #0x190+        stp x1, x6, [x0]+        stp x11, x16, [x0, #0x10]+        stp x21, x2, [x0, #0x20]+        stp x7, x12, [x0, #0x30]+        stp x17, x22, [x0, #0x40]+        stp x3, x8, [x0, #0x50]+        stp x13, x28, [x0, #0x60]+        stp x23, x4, [x0, #0x70]+        stp x9, x14, [x0, #0x80]+        stp x19, x24, [x0, #0x90]+        stp x5, x10, [x0, #0xa0]+        stp x15, x20, [x0, #0xb0]+        str x25, [x0, #0xc0]+        sub x0, x0, #0x190+        add x0, x0, #0x258+        ldp x1, x6, [x0]+        ldp x11, x16, [x0, #0x10]+        ldp x21, x2, [x0, #0x20]+        ldp x7, x12, [x0, #0x30]+        ldp x17, x22, [x0, #0x40]+        ldp x3, x8, [x0, #0x50]+        ldp x13, x28, [x0, #0x60]+        ldp x23, x4, [x0, #0x70]+        ldp x9, x14, [x0, #0x80]+        ldp x19, x24, [x0, #0x90]+        ldp x5, x10, [x0, #0xa0]+        ldp x15, x20, [x0, #0xb0]+        ldr x25, [x0, #0xc0]+        sub x0, x0, #0x258+        b Lmlk_keccak_f1600_x4_v8a_scalar_hybrid_initial++Lmlk_keccak_f1600_x4_v8a_scalar_hybrid_done:+        ldr x0, [sp]+        add x0, x0, #0x258+        stp x1, x6, [x0]+        stp x11, x16, [x0, #0x10]+        stp x21, x2, [x0, #0x20]+        stp x7, x12, [x0, #0x30]+        stp x17, x22, [x0, #0x40]+        stp x3, x8, [x0, #0x50]+        stp x13, x28, [x0, #0x60]+        stp x23, x4, [x0, #0x70]+        stp x9, x14, [x0, #0x80]+        stp x19, x24, [x0, #0x90]+        stp x5, x10, [x0, #0xa0]+        stp x15, x20, [x0, #0xb0]+        str x25, [x0, #0xc0]+        sub x0, x0, #0x258+        add x4, x0, #0xc8+        trn1 v25.2d, v0.2d, v1.2d+        trn1 v26.2d, v2.2d, v3.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v0.2d, v1.2d+        trn2 v28.2d, v2.2d, v3.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v4.2d, v5.2d+        trn1 v26.2d, v6.2d, v7.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v4.2d, v5.2d+        trn2 v28.2d, v6.2d, v7.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v8.2d, v9.2d+        trn1 v26.2d, v10.2d, v11.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v8.2d, v9.2d+        trn2 v28.2d, v10.2d, v11.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v12.2d, v13.2d+        trn1 v26.2d, v14.2d, v15.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v12.2d, v13.2d+        trn2 v28.2d, v14.2d, v15.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v16.2d, v17.2d+        trn1 v26.2d, v18.2d, v19.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v16.2d, v17.2d+        trn2 v28.2d, v18.2d, v19.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v20.2d, v21.2d+        trn1 v26.2d, v22.2d, v23.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v20.2d, v21.2d+        trn2 v28.2d, v22.2d, v23.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        str d24, [x0]+        trn2 v25.2d, v24.2d, v24.2d+        str d25, [x4]+        ldp d8, d9, [sp, #0x90]+        .cfi_restore d8+        .cfi_restore d9+        ldp d10, d11, [sp, #0xa0]+        .cfi_restore d10+        .cfi_restore d11+        ldp d12, d13, [sp, #0xb0]+        .cfi_restore d12+        .cfi_restore d13+        ldp d14, d15, [sp, #0xc0]+        .cfi_restore d14+        .cfi_restore d15+        ldp x19, x20, [sp, #0x30]+        .cfi_restore x19+        .cfi_restore x20+        ldp x21, x22, [sp, #0x40]+        .cfi_restore x21+        .cfi_restore x22+        ldp x23, x24, [sp, #0x50]+        .cfi_restore x23+        .cfi_restore x24+        ldp x25, x26, [sp, #0x60]+        .cfi_restore x25+        .cfi_restore x26+        ldp x27, x28, [sp, #0x70]+        .cfi_restore x27+        .cfi_restore x28+        ldp x29, x30, [sp, #0x80]+        .cfi_restore x29+        .cfi_restore x30+        add sp, sp, #0xe0+        .cfi_adjust_cfa_offset -0xe0+        ret+        .cfi_endproc++MLK_ASM_FN_SIZE(keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm)++#endif /* MLK_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID && \+          !MLK_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.S view
@@ -0,0 +1,989 @@+/*+ * Copyright (c) The mlkem-native project authors+ * Copyright (c) 2021-2022 Arm Limited+ * Copyright (c) 2022 Matthias Kannwischer+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++// Author: Hanno Becker <hannobecker@posteo.de>+// Author: Matthias Kannwischer <matthias@kannwischer.eu>++/*yaml+  Name: keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm+  Description: AArch64 hybrid scalar/vector implementation of Keccak-f[1600] permutation for four sequential states with ARMv8.4-A optimizations+  Signature: void mlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm(uint64_t state[100], const uint64_t rc[24])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON, SHA3]+    x0:+      type: buffer+      size_bytes: 800+      permissions: read/write+      c_parameter: uint64_t state[100]+      description: Four sequential Keccak states (state0[25], state1[25], state2[25], state3[25])+    x1:+      type: buffer+      size_bytes: 192+      permissions: read-only+      c_parameter: const uint64_t rc[24]+      description: Round constants (24 x uint64_t)+  Stack:+    bytes: 224+    description: register preservation and temporary storage+*/++#include "../../../../common.h"+#if defined(MLK_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++#if defined(__ARM_FEATURE_SHA3)++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/fips202/aarch64/src/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm)+MLK_ASM_FN_SYMBOL(keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0xe0+        .cfi_adjust_cfa_offset 0xe0+        stp x19, x20, [sp, #0x30]+        .cfi_rel_offset x19, 0x30+        .cfi_rel_offset x20, 0x38+        stp x21, x22, [sp, #0x40]+        .cfi_rel_offset x21, 0x40+        .cfi_rel_offset x22, 0x48+        stp x23, x24, [sp, #0x50]+        .cfi_rel_offset x23, 0x50+        .cfi_rel_offset x24, 0x58+        stp x25, x26, [sp, #0x60]+        .cfi_rel_offset x25, 0x60+        .cfi_rel_offset x26, 0x68+        stp x27, x28, [sp, #0x70]+        .cfi_rel_offset x27, 0x70+        .cfi_rel_offset x28, 0x78+        stp x29, x30, [sp, #0x80]+        .cfi_rel_offset x29, 0x80+        .cfi_rel_offset x30, 0x88+        stp d8, d9, [sp, #0x90]+        .cfi_rel_offset d8, 0x90+        .cfi_rel_offset d9, 0x98+        stp d10, d11, [sp, #0xa0]+        .cfi_rel_offset d10, 0xa0+        .cfi_rel_offset d11, 0xa8+        stp d12, d13, [sp, #0xb0]+        .cfi_rel_offset d12, 0xb0+        .cfi_rel_offset d13, 0xb8+        stp d14, d15, [sp, #0xc0]+        .cfi_rel_offset d14, 0xc0+        .cfi_rel_offset d15, 0xc8+        mov x29, x1+        mov x30, #0x0               // =0+        str x30, [sp, #0x20]+        str x29, [sp, #0x8]+        str x29, [sp, #0x10]+        str x0, [sp]+        add x4, x0, #0xc8+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v0.2d, v25.2d, v27.2d+        trn2 v1.2d, v25.2d, v27.2d+        trn1 v2.2d, v26.2d, v28.2d+        trn2 v3.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v4.2d, v25.2d, v27.2d+        trn2 v5.2d, v25.2d, v27.2d+        trn1 v6.2d, v26.2d, v28.2d+        trn2 v7.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v8.2d, v25.2d, v27.2d+        trn2 v9.2d, v25.2d, v27.2d+        trn1 v10.2d, v26.2d, v28.2d+        trn2 v11.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v12.2d, v25.2d, v27.2d+        trn2 v13.2d, v25.2d, v27.2d+        trn1 v14.2d, v26.2d, v28.2d+        trn2 v15.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v16.2d, v25.2d, v27.2d+        trn2 v17.2d, v25.2d, v27.2d+        trn1 v18.2d, v26.2d, v28.2d+        trn2 v19.2d, v26.2d, v28.2d+        ldp q25, q26, [x0], #0x20+        ld1 { v27.2d, v28.2d }, [x4], #32+        trn1 v20.2d, v25.2d, v27.2d+        trn2 v21.2d, v25.2d, v27.2d+        trn1 v22.2d, v26.2d, v28.2d+        trn2 v23.2d, v26.2d, v28.2d+        ldr d25, [x0]+        ldr d27, [x4]+        trn1 v24.2d, v25.2d, v27.2d+        sub x0, x0, #0xc0+        add x0, x0, #0x190+        ldp x1, x6, [x0]+        ldp x11, x16, [x0, #0x10]+        ldp x21, x2, [x0, #0x20]+        ldp x7, x12, [x0, #0x30]+        ldp x17, x22, [x0, #0x40]+        ldp x3, x8, [x0, #0x50]+        ldp x13, x28, [x0, #0x60]+        ldp x23, x4, [x0, #0x70]+        ldp x9, x14, [x0, #0x80]+        ldp x19, x24, [x0, #0x90]+        ldp x5, x10, [x0, #0xa0]+        ldp x15, x20, [x0, #0xb0]+        ldr x25, [x0, #0xc0]+        sub x0, x0, #0x190++Lmlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_initial:+        eor x30, x24, x25+        eor x27, x9, x10+        eor3 v30.16b, v0.16b, v5.16b, v10.16b+        eor v30.16b, v30.16b, v15.16b+        eor x0, x30, x21+        eor x26, x27, x6+        eor v30.16b, v30.16b, v20.16b+        eor x27, x26, x7+        eor x29, x0, x22+        eor3 v29.16b, v1.16b, v6.16b, v11.16b+        eor x26, x29, x23+        eor x29, x4, x5+        eor v29.16b, v29.16b, v16.16b+        eor x30, x29, x1+        eor x0, x27, x8+        eor v29.16b, v29.16b, v21.16b+        eor x29, x30, x2+        eor x30, x19, x20+        eor3 v28.16b, v2.16b, v7.16b, v12.16b+        eor x30, x30, x16+        eor x27, x26, x0, ror #63+        eor v28.16b, v28.16b, v17.16b+        eor x4, x4, x27+        eor x30, x30, x17+        eor v28.16b, v28.16b, v22.16b+        eor x30, x30, x28+        eor x29, x29, x3+        eor3 v27.16b, v3.16b, v8.16b, v13.16b+        eor x0, x0, x30, ror #63+        eor x30, x30, x29, ror #63+        eor v27.16b, v27.16b, v18.16b+        eor x22, x22, x30+        eor v27.16b, v27.16b, v23.16b+        eor x23, x23, x30+        str x23, [sp, #0xd0]+        eor3 v26.16b, v4.16b, v9.16b, v14.16b+        eor x23, x14, x15+        eor x14, x14, x0+        eor v26.16b, v26.16b, v19.16b+        eor x23, x23, x11+        eor x15, x15, x0+        eor v26.16b, v26.16b, v24.16b+        eor x1, x1, x27+        eor x23, x23, x12+        rax1 v25.2d, v30.2d, v28.2d+        eor x23, x23, x13+        eor x11, x11, x0+        add v31.2d, v26.2d, v26.2d+        eor x29, x29, x23, ror #63+        eor x23, x23, x26, ror #63+        sri v31.2d, v26.2d, #0x3f+        eor x26, x13, x0+        eor x13, x28, x23+        eor v28.16b, v31.16b, v28.16b+        eor x28, x24, x30+        eor x24, x16, x23+        rax1 v26.2d, v26.2d, v29.2d+        eor x16, x21, x30+        eor x21, x25, x30+        add v31.2d, v27.2d, v27.2d+        eor x30, x19, x23+        sri v31.2d, v27.2d, #0x3f+        eor x19, x20, x23+        eor x20, x17, x23+        eor v29.16b, v31.16b, v29.16b+        eor x17, x12, x0+        eor x0, x2, x27+        rax1 v27.2d, v27.2d, v30.2d+        eor x2, x6, x29+        eor x6, x8, x29+        eor v30.16b, v0.16b, v26.16b+        bic x8, x28, x13, ror #47+        eor x12, x3, x27+        eor v31.16b, v2.16b, v29.16b+        bic x3, x13, x17, ror #19+        eor x5, x5, x27+        shl v0.2d, v31.2d, #0x3e+        ldr x27, [sp, #0xd0]+        bic x25, x17, x2, ror #5+        sri v0.2d, v31.2d, #0x2+        eor x9, x9, x29+        eor x23, x25, x5, ror #52+        xar v2.2d, v12.2d, v29.2d, #0x15+        eor x3, x3, x2, ror #24+        eor x8, x8, x17, ror #2+        eor v31.16b, v13.16b, v28.16b+        eor x17, x10, x29+        bic x25, x12, x22, ror #47+        shl v12.2d, v31.2d, #0x19+        eor x29, x7, x29+        bic x10, x4, x27, ror #2+        sri v12.2d, v31.2d, #0x27+        bic x7, x5, x28, ror #10+        xar v13.2d, v19.2d, v27.2d, #0x38+        eor x10, x10, x20, ror #50+        eor x13, x7, x13, ror #57+        eor v31.16b, v23.16b, v28.16b+        bic x7, x2, x5, ror #47+        eor x2, x25, x24, ror #39+        shl v19.2d, v31.2d, #0x38+        bic x25, x20, x11, ror #57+        bic x5, x17, x4, ror #25+        sri v19.2d, v31.2d, #0x8+        eor x25, x25, x17, ror #53+        bic x17, x11, x17, ror #60+        xar v23.2d, v15.2d, v26.2d, #0x17+        eor x28, x7, x28, ror #57+        bic x7, x9, x12, ror #42+        eor v31.16b, v1.16b, v25.16b+        eor x7, x7, x22, ror #25+        bic x22, x22, x24, ror #56+        shl v15.2d, v31.2d, #0x1+        bic x24, x24, x15, ror #31+        eor x22, x22, x15, ror #23+        sri v15.2d, v31.2d, #0x3f+        bic x20, x27, x20, ror #48+        bic x15, x15, x9, ror #16+        xar v1.2d, v8.2d, v28.2d, #0x9+        eor x12, x15, x12, ror #58+        eor x15, x5, x27, ror #27+        eor v31.16b, v16.16b, v25.16b+        eor x5, x20, x11, ror #41+        shl v8.2d, v31.2d, #0x2d+        ldr x11, [sp, #0x8]+        eor x20, x17, x4, ror #21+        sri v8.2d, v31.2d, #0x13+        eor x17, x24, x9, ror #47+        mov x24, #0x1               // =1+        xar v16.2d, v7.2d, v29.2d, #0x3a+        bic x9, x0, x16, ror #9+        str x24, [sp, #0x18]+        eor v31.16b, v10.16b, v26.16b+        bic x24, x29, x1, ror #44+        bic x27, x1, x21, ror #50+        shl v7.2d, v31.2d, #0x3+        bic x4, x26, x29, ror #63+        eor x1, x1, x4, ror #21+        sri v7.2d, v31.2d, #0x3d+        ldr x11, [x11]+        bic x4, x21, x30, ror #57+        xar v10.2d, v3.2d, v28.2d, #0x24+        eor x21, x24, x21, ror #30+        eor x24, x9, x19, ror #44+        eor v31.16b, v18.16b, v28.16b+        bic x9, x14, x6, ror #5+        eor x9, x9, x0, ror #43+        shl v3.2d, v31.2d, #0x15+        bic x0, x6, x0, ror #38+        eor x1, x1, x11+        sri v3.2d, v31.2d, #0x2b+        eor x11, x4, x26, ror #35+        eor x4, x0, x16, ror #47+        xar v18.2d, v17.2d, v29.2d, #0x31+        bic x0, x16, x19, ror #35+        eor v31.16b, v11.16b, v25.16b+        eor x16, x27, x30, ror #43+        bic x27, x30, x26, ror #42+        shl v17.2d, v31.2d, #0xa+        bic x26, x19, x14, ror #41+        eor x19, x0, x14, ror #12+        sri v17.2d, v31.2d, #0x36+        eor x14, x26, x6, ror #46+        eor x6, x27, x29, ror #41+        xar v11.2d, v9.2d, v27.2d, #0x2c+        eor x0, x15, x11, ror #52+        eor x0, x0, x13, ror #48+        eor v31.16b, v22.16b, v29.16b+        eor x26, x8, x9, ror #57+        eor x27, x0, x14, ror #10+        shl v9.2d, v31.2d, #0x3d+        eor x29, x16, x28, ror #63+        eor x26, x26, x6, ror #51+        sri v9.2d, v31.2d, #0x3+        eor x30, x23, x22, ror #50+        eor x0, x26, x10, ror #31+        xar v22.2d, v14.2d, v27.2d, #0x19+        eor x29, x29, x19, ror #37+        eor x27, x27, x12, ror #5+        eor v31.16b, v20.16b, v26.16b+        eor x30, x30, x24, ror #34+        eor x0, x0, x7, ror #27+        shl v14.2d, v31.2d, #0x12+        eor x26, x30, x21, ror #26+        sri v14.2d, v31.2d, #0x2e+        eor x26, x26, x25, ror #15+        ror x30, x27, #0x3e+        xar v20.2d, v4.2d, v27.2d, #0x25+        eor x30, x30, x26, ror #57+        ror x26, x26, #0x3a+        eor v31.16b, v24.16b, v27.16b+        eor x16, x30, x16+        eor x28, x30, x28, ror #63+        shl v4.2d, v31.2d, #0xe+        str x28, [sp, #0xd0]+        eor x29, x29, x17, ror #36+        sri v4.2d, v31.2d, #0x32+        eor x28, x1, x2, ror #61+        eor x19, x30, x19, ror #37+        xar v24.2d, v21.2d, v25.2d, #0x3e+        eor x29, x29, x20, ror #2+        eor x28, x28, x4, ror #54+        eor v31.16b, v5.16b, v26.16b+        eor x26, x26, x0, ror #55+        eor x28, x28, x3, ror #39+        shl v21.2d, v31.2d, #0x24+        eor x28, x28, x5, ror #25+        ror x0, x0, #0x38+        sri v21.2d, v31.2d, #0x1c+        eor x0, x0, x29, ror #63+        eor x27, x28, x27, ror #61+        xar v27.2d, v6.2d, v25.2d, #0x14+        eor x13, x0, x13, ror #46+        eor x28, x29, x28, ror #63+        bic v31.16b, v7.16b, v11.16b+        eor x29, x30, x20, ror #2+        eor v5.16b, v31.16b, v10.16b+        eor x20, x26, x3, ror #39+        eor x11, x0, x11, ror #50+        bcax v6.16b, v11.16b, v8.16b, v7.16b+        eor x25, x28, x25, ror #9+        eor x3, x28, x21, ror #20+        bic v31.16b, v9.16b, v8.16b+        eor x21, x26, x1+        eor x9, x27, x9, ror #49+        eor v7.16b, v31.16b, v7.16b+        eor x24, x28, x24, ror #28+        eor x1, x30, x17, ror #36+        bcax v8.16b, v8.16b, v10.16b, v9.16b+        eor x14, x0, x14, ror #8+        eor x22, x28, x22, ror #44+        bic v31.16b, v11.16b, v10.16b+        eor x8, x27, x8, ror #56+        eor x17, x27, x7, ror #19+        eor v9.16b, v31.16b, v9.16b+        eor x15, x0, x15, ror #62+        bic x7, x20, x22, ror #47+        bcax v10.16b, v15.16b, v12.16b, v16.16b+        eor x4, x26, x4, ror #54+        eor x0, x0, x12, ror #3+        bic v31.16b, v13.16b, v12.16b+        eor x28, x28, x23, ror #58+        eor x23, x26, x2, ror #61+        eor v11.16b, v31.16b, v16.16b+        eor x26, x26, x5, ror #25+        bcax v12.16b, v12.16b, v14.16b, v13.16b+        eor x2, x7, x16, ror #39+        bic x7, x9, x20, ror #42+        bic v31.16b, v15.16b, v14.16b+        bic x30, x15, x9, ror #16+        eor x7, x7, x22, ror #25+        eor v13.16b, v31.16b, v13.16b+        eor x12, x30, x20, ror #58+        bic x20, x22, x16, ror #56+        bic v31.16b, v16.16b, v15.16b+        eor x30, x27, x6, ror #43+        eor x22, x20, x15, ror #23+        eor v14.16b, v31.16b, v14.16b+        bic x6, x19, x13, ror #42+        eor x6, x6, x17, ror #41+        bcax v15.16b, v20.16b, v17.16b, v21.16b+        bic x5, x13, x17, ror #63+        eor x5, x21, x5, ror #21+        bic v31.16b, v18.16b, v17.16b+        bic x17, x17, x21, ror #44+        eor x27, x27, x10, ror #23+        eor v16.16b, v31.16b, v21.16b+        bic x21, x21, x25, ror #50+        bic x20, x27, x4, ror #25+        bcax v17.16b, v17.16b, v19.16b, v18.16b+        bic x10, x16, x15, ror #31+        eor x16, x21, x19, ror #43+        bic v31.16b, v20.16b, v19.16b+        eor x21, x17, x25, ror #30+        bic x19, x25, x19, ror #57+        eor v18.16b, v31.16b, v18.16b+        ldr x25, [sp, #0x18]+        bcax v19.16b, v19.16b, v21.16b, v20.16b+        eor x17, x10, x9, ror #47+        ldr x9, [sp, #0x8]+        bic v31.16b, v22.16b, v1.16b+        eor x15, x20, x28, ror #27+        bic x20, x4, x28, ror #2+        eor v20.16b, v31.16b, v0.16b+        eor x10, x20, x1, ror #50+        bic x20, x11, x27, ror #60+        bcax v21.16b, v1.16b, v23.16b, v22.16b+        eor x20, x20, x4, ror #21+        bic x4, x28, x1, ror #48+        bic v31.16b, v24.16b, v23.16b+        bic x1, x1, x11, ror #57+        ldr x28, [x9, x25, lsl #3]+        eor v22.16b, v31.16b, v22.16b+        ldr x9, [sp, #0xd0]+        add x25, x25, #0x1+        bcax v23.16b, v23.16b, v0.16b, v24.16b+        str x25, [sp, #0x18]+        cmp x25, #0x17+        bic v31.16b, v1.16b, v0.16b+        eor x25, x1, x27, ror #53+        bic x27, x30, x26, ror #47+        eor v24.16b, v31.16b, v24.16b+        eor x1, x5, x28+        eor x5, x4, x11, ror #41+        bcax v0.16b, v30.16b, v2.16b, v27.16b+        eor x11, x19, x13, ror #35+        bic v31.16b, v3.16b, v2.16b+        bic x13, x26, x24, ror #10+        eor x28, x27, x24, ror #57+        eor v1.16b, v31.16b, v27.16b+        bic x27, x24, x9, ror #47+        bic x19, x23, x3, ror #9+        bcax v2.16b, v2.16b, v4.16b, v3.16b+        bic x4, x29, x14, ror #41+        eor x24, x19, x29, ror #44+        bic v31.16b, v30.16b, v4.16b+        bic x29, x3, x29, ror #35+        eor x13, x13, x9, ror #57+        eor v3.16b, v31.16b, v3.16b+        eor x19, x29, x14, ror #12+        bic x29, x9, x0, ror #19+        bcax v4.16b, v4.16b, v27.16b, v30.16b+        bic x14, x14, x8, ror #5+        eor x9, x14, x23, ror #43+        eor x14, x4, x8, ror #46+        bic x23, x8, x23, ror #38+        eor x8, x27, x0, ror #2+        eor x4, x23, x3, ror #47+        bic x3, x0, x30, ror #5+        eor x23, x3, x26, ror #52+        eor x3, x29, x30, ror #24+        ldr x30, [sp, #0x10]+        ld1r { v28.2d }, [x30], #8+        str x30, [sp, #0x10]+        eor v0.16b, v0.16b, v28.16b++Lmlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_loop:+        eor x0, x15, x11, ror #52+        eor x0, x0, x13, ror #48+        eor3 v30.16b, v0.16b, v5.16b, v10.16b+        eor v30.16b, v30.16b, v15.16b+        eor x26, x8, x9, ror #57+        eor x27, x0, x14, ror #10+        eor v30.16b, v30.16b, v20.16b+        eor x29, x16, x28, ror #63+        eor x26, x26, x6, ror #51+        eor3 v29.16b, v1.16b, v6.16b, v11.16b+        eor x30, x23, x22, ror #50+        eor x0, x26, x10, ror #31+        eor v29.16b, v29.16b, v16.16b+        eor x29, x29, x19, ror #37+        eor x27, x27, x12, ror #5+        eor v29.16b, v29.16b, v21.16b+        eor x30, x30, x24, ror #34+        eor x0, x0, x7, ror #27+        eor3 v28.16b, v2.16b, v7.16b, v12.16b+        eor x26, x30, x21, ror #26+        eor x26, x26, x25, ror #15+        eor v28.16b, v28.16b, v17.16b+        ror x30, x27, #0x3e+        eor x30, x30, x26, ror #57+        eor v28.16b, v28.16b, v22.16b+        ror x26, x26, #0x3a+        eor x16, x30, x16+        eor3 v27.16b, v3.16b, v8.16b, v13.16b+        eor x28, x30, x28, ror #63+        str x28, [sp, #0xd0]+        eor v27.16b, v27.16b, v18.16b+        eor x29, x29, x17, ror #36+        eor x28, x1, x2, ror #61+        eor v27.16b, v27.16b, v23.16b+        eor x19, x30, x19, ror #37+        eor x29, x29, x20, ror #2+        eor3 v26.16b, v4.16b, v9.16b, v14.16b+        eor x28, x28, x4, ror #54+        eor x26, x26, x0, ror #55+        eor v26.16b, v26.16b, v19.16b+        eor x28, x28, x3, ror #39+        eor x28, x28, x5, ror #25+        eor v26.16b, v26.16b, v24.16b+        ror x0, x0, #0x38+        eor x0, x0, x29, ror #63+        rax1 v25.2d, v30.2d, v28.2d+        eor x27, x28, x27, ror #61+        eor x13, x0, x13, ror #46+        add v31.2d, v26.2d, v26.2d+        eor x28, x29, x28, ror #63+        eor x29, x30, x20, ror #2+        sri v31.2d, v26.2d, #0x3f+        eor x20, x26, x3, ror #39+        eor x11, x0, x11, ror #50+        eor v28.16b, v31.16b, v28.16b+        eor x25, x28, x25, ror #9+        eor x3, x28, x21, ror #20+        rax1 v26.2d, v26.2d, v29.2d+        eor x21, x26, x1+        add v31.2d, v27.2d, v27.2d+        eor x9, x27, x9, ror #49+        eor x24, x28, x24, ror #28+        sri v31.2d, v27.2d, #0x3f+        eor x1, x30, x17, ror #36+        eor x14, x0, x14, ror #8+        eor v29.16b, v31.16b, v29.16b+        eor x22, x28, x22, ror #44+        eor x8, x27, x8, ror #56+        rax1 v27.2d, v27.2d, v30.2d+        eor x17, x27, x7, ror #19+        eor x15, x0, x15, ror #62+        eor v30.16b, v0.16b, v26.16b+        bic x7, x20, x22, ror #47+        eor x4, x26, x4, ror #54+        eor v31.16b, v2.16b, v29.16b+        eor x0, x0, x12, ror #3+        eor x28, x28, x23, ror #58+        shl v0.2d, v31.2d, #0x3e+        eor x23, x26, x2, ror #61+        eor x26, x26, x5, ror #25+        sri v0.2d, v31.2d, #0x2+        eor x2, x7, x16, ror #39+        bic x7, x9, x20, ror #42+        xar v2.2d, v12.2d, v29.2d, #0x15+        bic x30, x15, x9, ror #16+        eor x7, x7, x22, ror #25+        eor v31.16b, v13.16b, v28.16b+        eor x12, x30, x20, ror #58+        bic x20, x22, x16, ror #56+        shl v12.2d, v31.2d, #0x19+        eor x30, x27, x6, ror #43+        eor x22, x20, x15, ror #23+        sri v12.2d, v31.2d, #0x27+        bic x6, x19, x13, ror #42+        eor x6, x6, x17, ror #41+        xar v13.2d, v19.2d, v27.2d, #0x38+        bic x5, x13, x17, ror #63+        eor x5, x21, x5, ror #21+        eor v31.16b, v23.16b, v28.16b+        bic x17, x17, x21, ror #44+        eor x27, x27, x10, ror #23+        shl v19.2d, v31.2d, #0x38+        bic x21, x21, x25, ror #50+        bic x20, x27, x4, ror #25+        sri v19.2d, v31.2d, #0x8+        bic x10, x16, x15, ror #31+        eor x16, x21, x19, ror #43+        xar v23.2d, v15.2d, v26.2d, #0x17+        eor x21, x17, x25, ror #30+        bic x19, x25, x19, ror #57+        eor v31.16b, v1.16b, v25.16b+        ldr x25, [sp, #0x18]+        eor x17, x10, x9, ror #47+        shl v15.2d, v31.2d, #0x1+        ldr x9, [sp, #0x8]+        sri v15.2d, v31.2d, #0x3f+        eor x15, x20, x28, ror #27+        bic x20, x4, x28, ror #2+        xar v1.2d, v8.2d, v28.2d, #0x9+        eor x10, x20, x1, ror #50+        bic x20, x11, x27, ror #60+        eor v31.16b, v16.16b, v25.16b+        eor x20, x20, x4, ror #21+        bic x4, x28, x1, ror #48+        shl v8.2d, v31.2d, #0x2d+        bic x1, x1, x11, ror #57+        ldr x28, [x9, x25, lsl #3]+        sri v8.2d, v31.2d, #0x13+        ldr x9, [sp, #0xd0]+        add x25, x25, #0x1+        xar v16.2d, v7.2d, v29.2d, #0x3a+        str x25, [sp, #0x18]+        cmp x25, #0x17+        eor v31.16b, v10.16b, v26.16b+        eor x25, x1, x27, ror #53+        bic x27, x30, x26, ror #47+        shl v7.2d, v31.2d, #0x3+        eor x1, x5, x28+        eor x5, x4, x11, ror #41+        sri v7.2d, v31.2d, #0x3d+        eor x11, x19, x13, ror #35+        bic x13, x26, x24, ror #10+        xar v10.2d, v3.2d, v28.2d, #0x24+        eor x28, x27, x24, ror #57+        bic x27, x24, x9, ror #47+        eor v31.16b, v18.16b, v28.16b+        bic x19, x23, x3, ror #9+        bic x4, x29, x14, ror #41+        shl v3.2d, v31.2d, #0x15+        eor x24, x19, x29, ror #44+        bic x29, x3, x29, ror #35+        sri v3.2d, v31.2d, #0x2b+        eor x13, x13, x9, ror #57+        eor x19, x29, x14, ror #12+        xar v18.2d, v17.2d, v29.2d, #0x31+        bic x29, x9, x0, ror #19+        bic x14, x14, x8, ror #5+        eor v31.16b, v11.16b, v25.16b+        eor x9, x14, x23, ror #43+        eor x14, x4, x8, ror #46+        shl v17.2d, v31.2d, #0xa+        bic x23, x8, x23, ror #38+        eor x8, x27, x0, ror #2+        sri v17.2d, v31.2d, #0x36+        eor x4, x23, x3, ror #47+        bic x3, x0, x30, ror #5+        xar v11.2d, v9.2d, v27.2d, #0x2c+        eor x23, x3, x26, ror #52+        eor x3, x29, x30, ror #24+        eor v31.16b, v22.16b, v29.16b+        eor x0, x15, x11, ror #52+        shl v9.2d, v31.2d, #0x3d+        eor x0, x0, x13, ror #48+        eor x26, x8, x9, ror #57+        sri v9.2d, v31.2d, #0x3+        eor x27, x0, x14, ror #10+        eor x29, x16, x28, ror #63+        xar v22.2d, v14.2d, v27.2d, #0x19+        eor x26, x26, x6, ror #51+        eor x30, x23, x22, ror #50+        eor v31.16b, v20.16b, v26.16b+        eor x0, x26, x10, ror #31+        eor x29, x29, x19, ror #37+        shl v14.2d, v31.2d, #0x12+        eor x27, x27, x12, ror #5+        eor x30, x30, x24, ror #34+        sri v14.2d, v31.2d, #0x2e+        eor x0, x0, x7, ror #27+        eor x26, x30, x21, ror #26+        xar v20.2d, v4.2d, v27.2d, #0x25+        eor x26, x26, x25, ror #15+        ror x30, x27, #0x3e+        eor v31.16b, v24.16b, v27.16b+        eor x30, x30, x26, ror #57+        ror x26, x26, #0x3a+        shl v4.2d, v31.2d, #0xe+        eor x16, x30, x16+        eor x28, x30, x28, ror #63+        sri v4.2d, v31.2d, #0x32+        str x28, [sp, #0xd0]+        eor x29, x29, x17, ror #36+        xar v24.2d, v21.2d, v25.2d, #0x3e+        eor x28, x1, x2, ror #61+        eor x19, x30, x19, ror #37+        eor v31.16b, v5.16b, v26.16b+        eor x29, x29, x20, ror #2+        eor x28, x28, x4, ror #54+        shl v21.2d, v31.2d, #0x24+        eor x26, x26, x0, ror #55+        eor x28, x28, x3, ror #39+        sri v21.2d, v31.2d, #0x1c+        eor x28, x28, x5, ror #25+        ror x0, x0, #0x38+        xar v27.2d, v6.2d, v25.2d, #0x14+        eor x0, x0, x29, ror #63+        eor x27, x28, x27, ror #61+        bic v31.16b, v7.16b, v11.16b+        eor x13, x0, x13, ror #46+        eor x28, x29, x28, ror #63+        eor v5.16b, v31.16b, v10.16b+        eor x29, x30, x20, ror #2+        eor x20, x26, x3, ror #39+        bcax v6.16b, v11.16b, v8.16b, v7.16b+        eor x11, x0, x11, ror #50+        eor x25, x28, x25, ror #9+        bic v31.16b, v9.16b, v8.16b+        eor x3, x28, x21, ror #20+        eor v7.16b, v31.16b, v7.16b+        eor x21, x26, x1+        eor x9, x27, x9, ror #49+        bcax v8.16b, v8.16b, v10.16b, v9.16b+        eor x24, x28, x24, ror #28+        eor x1, x30, x17, ror #36+        bic v31.16b, v11.16b, v10.16b+        eor x14, x0, x14, ror #8+        eor x22, x28, x22, ror #44+        eor v9.16b, v31.16b, v9.16b+        eor x8, x27, x8, ror #56+        eor x17, x27, x7, ror #19+        bcax v10.16b, v15.16b, v12.16b, v16.16b+        eor x15, x0, x15, ror #62+        bic x7, x20, x22, ror #47+        bic v31.16b, v13.16b, v12.16b+        eor x4, x26, x4, ror #54+        eor x0, x0, x12, ror #3+        eor v11.16b, v31.16b, v16.16b+        eor x28, x28, x23, ror #58+        eor x23, x26, x2, ror #61+        bcax v12.16b, v12.16b, v14.16b, v13.16b+        eor x26, x26, x5, ror #25+        eor x2, x7, x16, ror #39+        bic v31.16b, v15.16b, v14.16b+        bic x7, x9, x20, ror #42+        bic x30, x15, x9, ror #16+        eor v13.16b, v31.16b, v13.16b+        eor x7, x7, x22, ror #25+        eor x12, x30, x20, ror #58+        bic v31.16b, v16.16b, v15.16b+        bic x20, x22, x16, ror #56+        eor x30, x27, x6, ror #43+        eor v14.16b, v31.16b, v14.16b+        eor x22, x20, x15, ror #23+        bic x6, x19, x13, ror #42+        bcax v15.16b, v20.16b, v17.16b, v21.16b+        eor x6, x6, x17, ror #41+        bic x5, x13, x17, ror #63+        bic v31.16b, v18.16b, v17.16b+        eor x5, x21, x5, ror #21+        bic x17, x17, x21, ror #44+        eor v16.16b, v31.16b, v21.16b+        eor x27, x27, x10, ror #23+        bic x21, x21, x25, ror #50+        bcax v17.16b, v17.16b, v19.16b, v18.16b+        bic x20, x27, x4, ror #25+        bic x10, x16, x15, ror #31+        bic v31.16b, v20.16b, v19.16b+        eor x16, x21, x19, ror #43+        eor x21, x17, x25, ror #30+        eor v18.16b, v31.16b, v18.16b+        bic x19, x25, x19, ror #57+        ldr x25, [sp, #0x18]+        bcax v19.16b, v19.16b, v21.16b, v20.16b+        eor x17, x10, x9, ror #47+        bic v31.16b, v22.16b, v1.16b+        ldr x9, [sp, #0x8]+        eor x15, x20, x28, ror #27+        eor v20.16b, v31.16b, v0.16b+        bic x20, x4, x28, ror #2+        eor x10, x20, x1, ror #50+        bcax v21.16b, v1.16b, v23.16b, v22.16b+        bic x20, x11, x27, ror #60+        eor x20, x20, x4, ror #21+        bic v31.16b, v24.16b, v23.16b+        bic x4, x28, x1, ror #48+        bic x1, x1, x11, ror #57+        eor v22.16b, v31.16b, v22.16b+        ldr x28, [x9, x25, lsl #3]+        ldr x9, [sp, #0xd0]+        bcax v23.16b, v23.16b, v0.16b, v24.16b+        add x25, x25, #0x1+        str x25, [sp, #0x18]+        bic v31.16b, v1.16b, v0.16b+        cmp x25, #0x17+        eor x25, x1, x27, ror #53+        eor v24.16b, v31.16b, v24.16b+        bic x27, x30, x26, ror #47+        eor x1, x5, x28+        bcax v0.16b, v30.16b, v2.16b, v27.16b+        eor x5, x4, x11, ror #41+        eor x11, x19, x13, ror #35+        bic v31.16b, v3.16b, v2.16b+        bic x13, x26, x24, ror #10+        eor x28, x27, x24, ror #57+        eor v1.16b, v31.16b, v27.16b+        bic x27, x24, x9, ror #47+        bic x19, x23, x3, ror #9+        bcax v2.16b, v2.16b, v4.16b, v3.16b+        bic x4, x29, x14, ror #41+        eor x24, x19, x29, ror #44+        bic v31.16b, v30.16b, v4.16b+        bic x29, x3, x29, ror #35+        eor x13, x13, x9, ror #57+        eor v3.16b, v31.16b, v3.16b+        eor x19, x29, x14, ror #12+        bic x29, x9, x0, ror #19+        bcax v4.16b, v4.16b, v27.16b, v30.16b+        bic x14, x14, x8, ror #5+        eor x9, x14, x23, ror #43+        eor x14, x4, x8, ror #46+        bic x23, x8, x23, ror #38+        eor x8, x27, x0, ror #2+        eor x4, x23, x3, ror #47+        bic x3, x0, x30, ror #5+        eor x23, x3, x26, ror #52+        eor x3, x29, x30, ror #24+        ldr x30, [sp, #0x10]+        ld1r { v28.2d }, [x30], #8+        str x30, [sp, #0x10]+        eor v0.16b, v0.16b, v28.16b++Lmlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_loop_end:+        b.le Lmlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_loop+        ror x2, x2, #0x3d+        ror x3, x3, #0x27+        ror x4, x4, #0x36+        ror x5, x5, #0x19+        ror x6, x6, #0x2b+        ror x7, x7, #0x13+        ror x8, x8, #0x38+        ror x9, x9, #0x31+        ror x10, x10, #0x17+        ror x11, x11, #0x32+        ror x12, x12, #0x3+        ror x13, x13, #0x2e+        ror x14, x14, #0x8+        ror x15, x15, #0x3e+        ror x17, x17, #0x24+        ror x28, x28, #0x3f+        ror x19, x19, #0x25+        ror x20, x20, #0x2+        ror x21, x21, #0x14+        ror x22, x22, #0x2c+        ror x23, x23, #0x3a+        ror x24, x24, #0x1c+        ror x25, x25, #0x9+        ldr x30, [sp, #0x20]+        cmp x30, #0x1+        b.eq Lmlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_done+        mov x30, #0x1               // =1+        str x30, [sp, #0x20]+        ldr x0, [sp]+        add x0, x0, #0x190+        stp x1, x6, [x0]+        stp x11, x16, [x0, #0x10]+        stp x21, x2, [x0, #0x20]+        stp x7, x12, [x0, #0x30]+        stp x17, x22, [x0, #0x40]+        stp x3, x8, [x0, #0x50]+        stp x13, x28, [x0, #0x60]+        stp x23, x4, [x0, #0x70]+        stp x9, x14, [x0, #0x80]+        stp x19, x24, [x0, #0x90]+        stp x5, x10, [x0, #0xa0]+        stp x15, x20, [x0, #0xb0]+        str x25, [x0, #0xc0]+        sub x0, x0, #0x190+        add x0, x0, #0x258+        ldp x1, x6, [x0]+        ldp x11, x16, [x0, #0x10]+        ldp x21, x2, [x0, #0x20]+        ldp x7, x12, [x0, #0x30]+        ldp x17, x22, [x0, #0x40]+        ldp x3, x8, [x0, #0x50]+        ldp x13, x28, [x0, #0x60]+        ldp x23, x4, [x0, #0x70]+        ldp x9, x14, [x0, #0x80]+        ldp x19, x24, [x0, #0x90]+        ldp x5, x10, [x0, #0xa0]+        ldp x15, x20, [x0, #0xb0]+        ldr x25, [x0, #0xc0]+        sub x0, x0, #0x258+        b Lmlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_initial++Lmlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_done:+        ldr x0, [sp]+        add x0, x0, #0x258+        stp x1, x6, [x0]+        stp x11, x16, [x0, #0x10]+        stp x21, x2, [x0, #0x20]+        stp x7, x12, [x0, #0x30]+        stp x17, x22, [x0, #0x40]+        stp x3, x8, [x0, #0x50]+        stp x13, x28, [x0, #0x60]+        stp x23, x4, [x0, #0x70]+        stp x9, x14, [x0, #0x80]+        stp x19, x24, [x0, #0x90]+        stp x5, x10, [x0, #0xa0]+        stp x15, x20, [x0, #0xb0]+        str x25, [x0, #0xc0]+        sub x0, x0, #0x258+        add x4, x0, #0xc8+        trn1 v25.2d, v0.2d, v1.2d+        trn1 v26.2d, v2.2d, v3.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v0.2d, v1.2d+        trn2 v28.2d, v2.2d, v3.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v4.2d, v5.2d+        trn1 v26.2d, v6.2d, v7.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v4.2d, v5.2d+        trn2 v28.2d, v6.2d, v7.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v8.2d, v9.2d+        trn1 v26.2d, v10.2d, v11.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v8.2d, v9.2d+        trn2 v28.2d, v10.2d, v11.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v12.2d, v13.2d+        trn1 v26.2d, v14.2d, v15.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v12.2d, v13.2d+        trn2 v28.2d, v14.2d, v15.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v16.2d, v17.2d+        trn1 v26.2d, v18.2d, v19.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v16.2d, v17.2d+        trn2 v28.2d, v18.2d, v19.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        trn1 v25.2d, v20.2d, v21.2d+        trn1 v26.2d, v22.2d, v23.2d+        stp q25, q26, [x0], #0x20+        trn2 v27.2d, v20.2d, v21.2d+        trn2 v28.2d, v22.2d, v23.2d+        st1 { v27.2d, v28.2d }, [x4], #32+        str d24, [x0]+        trn2 v25.2d, v24.2d, v24.2d+        str d25, [x4]+        ldp d8, d9, [sp, #0x90]+        .cfi_restore d8+        .cfi_restore d9+        ldp d10, d11, [sp, #0xa0]+        .cfi_restore d10+        .cfi_restore d11+        ldp d12, d13, [sp, #0xb0]+        .cfi_restore d12+        .cfi_restore d13+        ldp d14, d15, [sp, #0xc0]+        .cfi_restore d14+        .cfi_restore d15+        ldp x19, x20, [sp, #0x30]+        .cfi_restore x19+        .cfi_restore x20+        ldp x21, x22, [sp, #0x40]+        .cfi_restore x21+        .cfi_restore x22+        ldp x23, x24, [sp, #0x50]+        .cfi_restore x23+        .cfi_restore x24+        ldp x25, x26, [sp, #0x60]+        .cfi_restore x25+        .cfi_restore x26+        ldp x27, x28, [sp, #0x70]+        .cfi_restore x27+        .cfi_restore x28+        ldp x29, x30, [sp, #0x80]+        .cfi_restore x29+        .cfi_restore x30+        add sp, sp, #0xe0+        .cfi_adjust_cfa_offset -0xe0+        ret+        .cfi_endproc++MLK_ASM_FN_SIZE(keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm)++#endif /* __ARM_FEATURE_SHA3 */++#endif /* MLK_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID && \+          !MLK_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/fips202/native/aarch64/src/keccakf1600_round_constants.c view
@@ -0,0 +1,47 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mlkem-native repository.+ *          Do not modify it directly.+ */++#include "../../../../common.h"++#if (defined(MLK_FIPS202_AARCH64_NEED_X1_SCALAR) ||                  \+     defined(MLK_FIPS202_AARCH64_NEED_X1_V84A) ||                    \+     defined(MLK_FIPS202_AARCH64_NEED_X2_V84A) ||                    \+     defined(MLK_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID) ||       \+     defined(MLK_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID)) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++#include "fips202_native_aarch64.h"++MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint64_t+    mlk_keccakf1600_round_constants[24] = {+        0x0000000000000001, 0x0000000000008082, 0x800000000000808a,+        0x8000000080008000, 0x000000000000808b, 0x0000000080000001,+        0x8000000080008081, 0x8000000000008009, 0x000000000000008a,+        0x0000000000000088, 0x0000000080008009, 0x000000008000000a,+        0x000000008000808b, 0x800000000000008b, 0x8000000000008089,+        0x8000000000008003, 0x8000000000008002, 0x8000000000000080,+        0x000000000000800a, 0x800000008000000a, 0x8000000080008081,+        0x8000000000008080, 0x0000000080000001, 0x8000000080008008,+};++#else /* (MLK_FIPS202_AARCH64_NEED_X1_SCALAR ||                               \+         MLK_FIPS202_AARCH64_NEED_X1_V84A || MLK_FIPS202_AARCH64_NEED_X2_V84A \+         || MLK_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID ||                  \+         MLK_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID) &&               \+         !MLK_CONFIG_MULTILEVEL_NO_SHARED */++MLK_EMPTY_CU(fips202_aarch64_round_constants)++#endif /* !((MLK_FIPS202_AARCH64_NEED_X1_SCALAR ||                             \+          MLK_FIPS202_AARCH64_NEED_X1_V84A || MLK_FIPS202_AARCH64_NEED_X2_V84A \+          || MLK_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID ||                  \+          MLK_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID) &&               \+          !MLK_CONFIG_MULTILEVEL_NO_SHARED) */
+ cbits/mlkem/src/fips202/native/aarch64/x1_scalar.h view
@@ -0,0 +1,26 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLK_FIPS202_NATIVE_AARCH64_X1_SCALAR_H+#define MLK_FIPS202_NATIVE_AARCH64_X1_SCALAR_H++/* Part of backend API */+#define MLK_USE_NATIVE_FIPS202_X1+/* Guard for assembly file */+#define MLK_FIPS202_AARCH64_NEED_X1_SCALAR++#if !defined(__ASSEMBLER__)+#include "../api.h"+#include "src/fips202_native_aarch64.h"+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_keccak_f1600_x1_native(uint64_t *state)+{+  mlk_keccak_f1600_x1_scalar_aarch64_asm(state,+                                         mlk_keccakf1600_round_constants);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* !__ASSEMBLER__ */++#endif /* !MLK_FIPS202_NATIVE_AARCH64_X1_SCALAR_H */
+ cbits/mlkem/src/fips202/native/aarch64/x1_v84a.h view
@@ -0,0 +1,35 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLK_FIPS202_NATIVE_AARCH64_X1_V84A_H+#define MLK_FIPS202_NATIVE_AARCH64_X1_V84A_H++#if !defined(__ARM_FEATURE_SHA3)+#error This backend can only be used if SHA3 extensions are available.+#endif++/* Part of backend API */+#define MLK_USE_NATIVE_FIPS202_X1+/* Guard for assembly file */+#define MLK_FIPS202_AARCH64_NEED_X1_V84A++#if !defined(__ASSEMBLER__)+#include "../api.h"+#include "src/fips202_native_aarch64.h"+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_keccak_f1600_x1_native(uint64_t *state)+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON) ||+      !mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_SHA3))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_keccak_f1600_x1_v84a_aarch64_asm(state, mlk_keccakf1600_round_constants);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* !__ASSEMBLER__ */++#endif /* !MLK_FIPS202_NATIVE_AARCH64_X1_V84A_H */
+ cbits/mlkem/src/fips202/native/aarch64/x2_v84a.h view
@@ -0,0 +1,38 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLK_FIPS202_NATIVE_AARCH64_X2_V84A_H+#define MLK_FIPS202_NATIVE_AARCH64_X2_V84A_H++#if !defined(__ARM_FEATURE_SHA3)+#error This backend can only be used if SHA3 extensions are available.+#endif++/* Part of backend API */+#define MLK_USE_NATIVE_FIPS202_X4+/* Guard for assembly file */+#define MLK_FIPS202_AARCH64_NEED_X2_V84A++#if !defined(__ASSEMBLER__)+#include "../api.h"+#include "src/fips202_native_aarch64.h"+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_keccak_f1600_x4_native(uint64_t *state)+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON) ||+      !mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_SHA3))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_keccak_f1600_x2_v84a_aarch64_asm(state + 0 * 25,+                                       mlk_keccakf1600_round_constants);+  mlk_keccak_f1600_x2_v84a_aarch64_asm(state + 2 * 25,+                                       mlk_keccakf1600_round_constants);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* !__ASSEMBLER__ */++#endif /* !MLK_FIPS202_NATIVE_AARCH64_X2_V84A_H */
+ cbits/mlkem/src/fips202/native/aarch64/x4_v8a_scalar.h view
@@ -0,0 +1,31 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLK_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H+#define MLK_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H++/* Part of backend API */+#define MLK_USE_NATIVE_FIPS202_X4+/* Guard for assembly file */+#define MLK_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID++#if !defined(__ASSEMBLER__)+#include "../api.h"+#include "src/fips202_native_aarch64.h"+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_keccak_f1600_x4_native(uint64_t *state)+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm(+      state, mlk_keccakf1600_round_constants);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* !__ASSEMBLER__ */++#endif /* !MLK_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H */
+ cbits/mlkem/src/fips202/native/aarch64/x4_v8a_v84a_scalar.h view
@@ -0,0 +1,36 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLK_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H+#define MLK_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H++#if !defined(__ARM_FEATURE_SHA3)+#error This backend can only be used if SHA3 extensions are available.+#endif++/* Part of backend API */+#define MLK_USE_NATIVE_FIPS202_X4+/* Guard for assembly file */+#define MLK_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID++#if !defined(__ASSEMBLER__)+#include "../api.h"+#include "src/fips202_native_aarch64.h"+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_keccak_f1600_x4_native(uint64_t *state)+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON) ||+      !mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_SHA3))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm(+      state, mlk_keccakf1600_round_constants);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* !__ASSEMBLER__ */++#endif /* !MLK_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H */
+ cbits/mlkem/src/fips202/native/api.h view
@@ -0,0 +1,117 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLK_FIPS202_NATIVE_API_H+#define MLK_FIPS202_NATIVE_API_H+/*+ * FIPS-202 native interface+ *+ * This header is primarily for documentation purposes.+ * It should not be included by backend implementations.+ */++#include "../../cbmc.h"++/* Backends must return MLK_NATIVE_FUNC_SUCCESS upon success. */+#define MLK_NATIVE_FUNC_SUCCESS (0)+/* Backends may return MLK_NATIVE_FUNC_FALLBACK to signal to the frontend that+ * the target/parameters are unsupported; typically, this would be because of+ * dependencies on CPU features not detected on the host CPU. In this case,+ * the frontend falls back to the default C implementation. */+#define MLK_NATIVE_FUNC_FALLBACK (-1)++/*+ * This is the C<->native interface allowing for the drop-in+ * of custom Keccak-F1600 implementations.+ *+ * A _backend_ is a specific implementation of parts of this interface.+ *+ * You can replace 1-fold or 4-fold batched Keccak-F1600.+ * To enable, set MLK_USE_NATIVE_FIPS202_X1 or MLK_USE_NATIVE_FIPS202_X4+ * in your backend, and define the inline wrappers mlk_keccak_f1600_x1_native()+ * and/or mlk_keccak_f1600_x4_native(), respectively, to forward to your+ * implementation.+ */++#if defined(MLK_USE_NATIVE_FIPS202_X1)+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_keccak_f1600_x1_native(uint64_t *state)+__contract__(+  requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 1))+  assigns(memory_slice(state, sizeof(uint64_t) * 25 * 1))+  ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLK_NATIVE_FUNC_FALLBACK) ==> array_unchanged_u64(state, 25 * 1)));+#endif /* MLK_USE_NATIVE_FIPS202_X1 */+#if defined(MLK_USE_NATIVE_FIPS202_X4)+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_keccak_f1600_x4_native(uint64_t *state)+__contract__(+  requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4))+  assigns(memory_slice(state, sizeof(uint64_t) * 25 * 4))+  ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLK_NATIVE_FUNC_FALLBACK) ==> array_unchanged_u64(state, 25 * 4)));+#endif /* MLK_USE_NATIVE_FIPS202_X4 */++/*+ * Native x4 XOR bytes and extract bytes interface.+ *+ * These functions allow backends to provide optimized implementations for+ * XORing input data into the state and extracting output data from the state.+ * This is particularly useful for backends that use a different internal state+ * representation (e.g., bit-interleaved), as conversion can happen during+ * XOR/extract rather than before/after each permutation.+ *+ * NOTE: We assume that the custom representation of the zero state is the+ * all-zero state.+ *+ * MLK_USE_NATIVE_FIPS202_X4_XOR_BYTES: Backend provides native XOR bytes+ * MLK_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES: Backend provides native extract+ * bytes+ */++#if defined(MLK_USE_NATIVE_FIPS202_X4_XOR_BYTES)+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_keccakf1600_xor_bytes_x4_native(+    uint64_t *state, const unsigned char *data0, const unsigned char *data1,+    const unsigned char *data2, const unsigned char *data3, unsigned offset,+    unsigned length)+__contract__(+  requires(0 <= offset && offset <= 25 * sizeof(uint64_t) &&+           0 <= length && length <= 25 * sizeof(uint64_t) - offset)+  requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4))+  requires(memory_no_alias(data0, length))+  requires((data0 == data1 &&+            data0 == data2 &&+            data0 == data3) ||+           (memory_no_alias(data1, length) &&+            memory_no_alias(data2, length) &&+            memory_no_alias(data3, length)))+  assigns(memory_slice(state, sizeof(uint64_t) * 25 * 4))+  ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLK_NATIVE_FUNC_FALLBACK) ==> array_unchanged_u64(state, 25 * 4)));+#endif /* MLK_USE_NATIVE_FIPS202_X4_XOR_BYTES */++#if defined(MLK_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES)+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_keccakf1600_extract_bytes_x4_native(+    uint64_t *state, unsigned char *data0, unsigned char *data1,+    unsigned char *data2, unsigned char *data3, unsigned offset,+    unsigned length)+__contract__(+  requires(0 <= offset && offset <= 25 * sizeof(uint64_t) &&+           0 <= length && length <= 25 * sizeof(uint64_t) - offset)+  requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4))+  requires(memory_no_alias(data0, length))+  requires(memory_no_alias(data1, length))+  requires(memory_no_alias(data2, length))+  requires(memory_no_alias(data3, length))+  assigns(memory_slice(data0, length))+  assigns(memory_slice(data1, length))+  assigns(memory_slice(data2, length))+  assigns(memory_slice(data3, length))+  ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS));+#endif /* MLK_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES */++#endif /* !MLK_FIPS202_NATIVE_API_H */
+ cbits/mlkem/src/fips202/native/auto.h view
@@ -0,0 +1,29 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLK_FIPS202_NATIVE_AUTO_H+#define MLK_FIPS202_NATIVE_AUTO_H++/*+ * Default FIPS202 backend+ */+#include "../../sys.h"++#if defined(MLK_SYS_AARCH64)+#include "aarch64/auto.h"+#endif++/* The x86_64 backend requires toolchain support for the SysV ABI */+#if defined(MLK_SYS_X86_64_AVX2) && defined(MLK_SYSV_ABI_SUPPORTED)+#include "x86_64/keccak_f1600_x4_avx2.h"+#endif++/* We do not yet include the FIPS202 backend for Armv8.1-M+MVE by default+ * as it is still experimental and undergoing review. */+/* #if defined(MLK_SYS_ARMV81M_MVE) */+/* #include "armv81m/mve.h" */+/* #endif */++#endif /* !MLK_FIPS202_NATIVE_AUTO_H */
+ cbits/mlkem/src/fips202/native/x86_64/keccak_f1600_x4_avx2.h view
@@ -0,0 +1,33 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLK_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H+#define MLK_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H++#include "../../../common.h"++#define MLK_FIPS202_X86_64_NEED_X4_AVX2++/* Part of backend API */+#define MLK_USE_NATIVE_FIPS202_X4++#if !defined(__ASSEMBLER__)+#include "../api.h"+#include "src/fips202_native_x86_64.h"+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_keccak_f1600_x4_native(uint64_t *state)+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_keccak_f1600_x4_avx2_asm(state, mlk_keccakf1600_round_constants,+                               mlk_keccak_rho8, mlk_keccak_rho56);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* !__ASSEMBLER__ */++#endif /* !MLK_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H */
+ cbits/mlkem/src/fips202/native/x86_64/src/fips202_native_x86_64.h view
@@ -0,0 +1,44 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLK_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H+#define MLK_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H++#include "../../../../cbmc.h"+#include "../../../../common.h"++/* TODO: Reconsider whether this check is needed -- x86_64 is always+ * little-endian, so the backend selection already implies this. */+#ifndef MLK_SYS_LITTLE_ENDIAN+#error Expecting a little-endian platform+#endif++#define mlk_keccakf1600_round_constants \+  MLK_NAMESPACE(keccakf1600_round_constants)+MLK_INTERNAL_DATA_DECLARATION const uint64_t+    mlk_keccakf1600_round_constants[24];++#define mlk_keccak_rho8 MLK_NAMESPACE(keccak_rho8)+MLK_INTERNAL_DATA_DECLARATION const uint64_t mlk_keccak_rho8[4];++#define mlk_keccak_rho56 MLK_NAMESPACE(keccak_rho56)+MLK_INTERNAL_DATA_DECLARATION const uint64_t mlk_keccak_rho56[4];++#define mlk_keccak_f1600_x4_avx2_asm MLK_NAMESPACE(keccak_f1600_x4_avx2_asm)+MLK_SYSV_ABI+void mlk_keccak_f1600_x4_avx2_asm(uint64_t states[100], const uint64_t rc[24],+                                  const uint64_t rho8[4],+                                  const uint64_t rho56[4])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/keccak_f1600_x4_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(states, sizeof(uint64_t) * 25 * 4))+  requires(rc == mlk_keccakf1600_round_constants)+  requires(rho8 == mlk_keccak_rho8)+  requires(rho56 == mlk_keccak_rho56)+  assigns(memory_slice(states, sizeof(uint64_t) * 25 * 4))+);++#endif /* !MLK_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H */
+ cbits/mlkem/src/fips202/native/x86_64/src/keccak_f1600_x4_avx2_asm.S view
@@ -0,0 +1,487 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#include "../../../../common.h"++#if defined(MLK_FIPS202_X86_64_NEED_X4_AVX2) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)+/*yaml+  Name: keccak_f1600_x4_avx2_asm+  Description: x86_64 AVX2 Keccak-f[1600] permutation for four sequential states+  Signature: void mlk_keccak_f1600_x4_avx2_asm(uint64_t states[100], const uint64_t rc[24], const uint64_t rho8[4], const uint64_t rho56[4])+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 800+      permissions: read/write+      c_parameter: uint64_t states[100]+      description: Four sequential Keccak states (4 x 25 x uint64_t)+    rsi:+      type: buffer+      size_bytes: 192+      permissions: read-only+      c_parameter: const uint64_t rc[24]+      description: Round constants (24 x uint64_t)+    rdx:+      type: buffer+      size_bytes: 32+      permissions: read-only+      c_parameter: const uint64_t rho8[4]+      description: Rotation constant rho8 (4 x uint64_t)+    rcx:+      type: buffer+      size_bytes: 32+      permissions: read-only+      c_parameter: const uint64_t rho56[4]+      description: Rotation constant rho56 (4 x uint64_t)+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/fips202/x86_64/src/keccak_f1600_x4_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(keccak_f1600_x4_avx2_asm)+MLK_ASM_FN_SYMBOL(keccak_f1600_x4_avx2_asm)++        .cfi_startproc+        movq %rsp, %r11+        .cfi_def_cfa_register %r11+        andq $-0x20, %rsp+        subq $0x300, %rsp            # imm = 0x300+        vmovdqu (%rdi), %ymm0+        vmovdqu 0xc8(%rdi), %ymm3+        vmovdqu 0x190(%rdi), %ymm1+        vmovdqu 0x258(%rdi), %ymm4+        vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2]+        vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3]+        vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2]+        vperm2i128 $0x20, %ymm3, %ymm2, %ymm7 # ymm7 = ymm2[0,1],ymm3[0,1]+        vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3]+        vperm2i128 $0x31, %ymm3, %ymm2, %ymm3 # ymm3 = ymm2[2,3],ymm3[2,3]+        vmovdqu 0x278(%rdi), %ymm4+        vmovdqu %ymm3, 0x40(%rsp)+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm3 # ymm3 = ymm0[2,3],ymm1[2,3]+        vmovdqu %ymm7, (%rsp)+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1]+        vmovdqu 0x20(%rdi), %ymm0+        vmovdqu 0x1b0(%rdi), %ymm1+        vmovdqu %ymm3, 0x60(%rsp)+        vmovdqu 0xe8(%rdi), %ymm3+        vmovdqu %ymm7, 0x20(%rsp)+        vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2]+        vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3]+        vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2]+        vperm2i128 $0x20, %ymm3, %ymm2, %ymm7 # ymm7 = ymm2[0,1],ymm3[0,1]+        vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3]+        vperm2i128 $0x31, %ymm3, %ymm2, %ymm3 # ymm3 = ymm2[2,3],ymm3[2,3]+        vmovdqu 0x298(%rdi), %ymm4+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm14 # ymm14 = ymm0[2,3],ymm1[2,3]+        vmovdqu %ymm7, 0x80(%rsp)+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1]+        vmovdqu 0x40(%rdi), %ymm0+        vmovdqu 0x1d0(%rdi), %ymm1+        vmovdqu %ymm3, 0xc0(%rsp)+        vmovdqu 0x108(%rdi), %ymm3+        vmovdqu %ymm14, %ymm10+        vmovdqu %ymm7, 0xa0(%rsp)+        vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2]+        vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3]+        vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3]+        vperm2i128 $0x20, %ymm3, %ymm2, %ymm11 # ymm11 = ymm2[0,1],ymm3[0,1]+        vperm2i128 $0x31, %ymm3, %ymm2, %ymm3 # ymm3 = ymm2[2,3],ymm3[2,3]+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1]+        vmovdqu %ymm3, 0x100(%rsp)+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm8 # ymm8 = ymm0[2,3],ymm1[2,3]+        vmovdqu 0x128(%rdi), %ymm3+        vmovdqu 0x60(%rdi), %ymm0+        vmovdqu 0x1f0(%rdi), %ymm1+        vmovdqu %ymm7, 0xe0(%rsp)+        vmovdqu %ymm11, %ymm14+        vmovdqu 0x2b8(%rdi), %ymm4+        vmovdqu 0x2f8(%rdi), %ymm5+        vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2]+        vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3]+        vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3]+        vmovdqu 0x2d8(%rdi), %ymm4+        vperm2i128 $0x20, %ymm3, %ymm2, %ymm15 # ymm15 = ymm2[0,1],ymm3[0,1]+        vperm2i128 $0x31, %ymm3, %ymm2, %ymm3 # ymm3 = ymm2[2,3],ymm3[2,3]+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1]+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm9 # ymm9 = ymm0[2,3],ymm1[2,3]+        vmovdqu %ymm3, 0x140(%rsp)+        vmovdqu 0x80(%rdi), %ymm0+        vmovdqu 0x148(%rdi), %ymm3+        vmovdqu 0x210(%rdi), %ymm1+        vmovdqu %ymm7, 0x120(%rsp)+        vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2]+        vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3]+        vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3]+        vperm2i128 $0x20, %ymm3, %ymm2, %ymm7 # ymm7 = ymm2[0,1],ymm3[0,1]+        vperm2i128 $0x31, %ymm3, %ymm2, %ymm13 # ymm13 = ymm2[2,3],ymm3[2,3]+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm3 # ymm3 = ymm0[2,3],ymm1[2,3]+        vmovdqu %ymm7, 0x160(%rsp)+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1]+        vmovdqu 0xa0(%rdi), %ymm0+        vmovdqu 0x230(%rdi), %ymm1+        vmovdqu %ymm3, 0x1a0(%rsp)+        vmovdqu 0x168(%rdi), %ymm3+        vpunpcklqdq %ymm5, %ymm1, %ymm4 # ymm4 = ymm1[0],ymm5[0],ymm1[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm5[1],ymm1[3],ymm5[3]+        vmovdqu %ymm7, 0x180(%rsp)+        vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2]+        vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3]+        vperm2i128 $0x20, %ymm4, %ymm2, %ymm12 # ymm12 = ymm2[0,1],ymm4[0,1]+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm3 # ymm3 = ymm0[0,1],ymm1[0,1]+        vperm2i128 $0x31, %ymm4, %ymm2, %ymm7 # ymm7 = ymm2[2,3],ymm4[2,3]+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm4 # ymm4 = ymm0[2,3],ymm1[2,3]+        vmovq 0x250(%rdi), %xmm0+        vmovq 0xc0(%rdi), %xmm1+        vmovdqu %ymm12, 0x1c0(%rsp)+        vmovdqu %ymm4, 0x1e0(%rsp)+        vpinsrq $0x1, 0x318(%rdi), %xmm0, %xmm0+        vpinsrq $0x1, 0x188(%rdi), %xmm1, %xmm1+        vinserti128 $0x1, %xmm0, %ymm1, %ymm2+        movq $0x0, %r10++LLmlk_keccak_f1600_x4_avx2:+        vmovdqu 0xa0(%rsp), %ymm4+        vpxor 0x1c0(%rsp), %ymm9, %ymm0+        vmovdqu %ymm9, 0x200(%rsp)+        vmovdqu %ymm10, %ymm9+        vmovdqu 0xc0(%rsp), %ymm11+        vmovdqu 0x160(%rsp), %ymm12+        vmovdqu %ymm3, 0x240(%rsp)+        vpxor 0x100(%rsp), %ymm4, %ymm1+        vmovdqu 0x40(%rsp), %ymm10+        vmovdqu %ymm4, 0x220(%rsp)+        vpxor %ymm3, %ymm12, %ymm12+        vmovdqu 0x20(%rsp), %ymm6+        vmovdqu 0x140(%rsp), %ymm4+        vmovdqu %ymm14, 0x2a0(%rsp)+        vpxor %ymm1, %ymm0, %ymm0+        vpxor %ymm8, %ymm11, %ymm1+        vpxor 0x180(%rsp), %ymm7, %ymm11+        vmovdqu %ymm10, 0x280(%rsp)+        vpxor %ymm1, %ymm12, %ymm12+        vpxor %ymm15, %ymm9, %ymm1+        vmovdqu 0xe0(%rsp), %ymm3+        vmovdqu %ymm8, 0x260(%rsp)+        vpxor %ymm1, %ymm11, %ymm11+        vpxor 0x120(%rsp), %ymm14, %ymm1+        vpxor %ymm6, %ymm12, %ymm12+        vmovdqu 0x60(%rsp), %ymm8+        vpxor %ymm10, %ymm11, %ymm11+        vpxor 0x1e0(%rsp), %ymm13, %ymm10+        vpxor %ymm4, %ymm3, %ymm3+        vmovdqu %ymm4, 0x2c0(%rsp)+        vpsrlq $0x3f, %ymm12, %ymm4+        vpsrlq $0x3f, %ymm11, %ymm5+        vpxor (%rsp), %ymm0, %ymm0+        vpxor %ymm1, %ymm10, %ymm10+        vmovdqu 0x80(%rsp), %ymm1+        vpxor %ymm8, %ymm10, %ymm10+        vmovdqu %ymm1, %ymm14+        vpxor 0x1a0(%rsp), %ymm2, %ymm1+        vmovdqu %ymm14, 0x2e0(%rsp)+        vpxor %ymm3, %ymm1, %ymm1+        vpsllq $0x1, %ymm12, %ymm3+        vpor %ymm4, %ymm3, %ymm3+        vpsllq $0x1, %ymm11, %ymm4+        vpxor %ymm14, %ymm1, %ymm1+        vpor %ymm5, %ymm4, %ymm4+        vpsrlq $0x3f, %ymm10, %ymm14+        vpxor %ymm1, %ymm3, %ymm3+        vpsllq $0x1, %ymm10, %ymm5+        vpxor %ymm0, %ymm4, %ymm4+        vpor %ymm14, %ymm5, %ymm5+        vpxor %ymm6, %ymm4, %ymm6+        vpxor %ymm12, %ymm5, %ymm5+        vpsrlq $0x3f, %ymm1, %ymm12+        vpsllq $0x1, %ymm1, %ymm1+        vpxor %ymm7, %ymm5, %ymm7+        vpxor %ymm9, %ymm5, %ymm9+        vpor %ymm12, %ymm1, %ymm1+        vpxor (%rsp), %ymm3, %ymm12+        vpxor %ymm11, %ymm1, %ymm1+        vpsrlq $0x3f, %ymm0, %ymm11+        vpsllq $0x1, %ymm0, %ymm0+        vpxor %ymm13, %ymm1, %ymm13+        vpxor %ymm8, %ymm1, %ymm8+        vpor %ymm11, %ymm0, %ymm0+        vpxor %ymm10, %ymm0, %ymm0+        vpxor 0xc0(%rsp), %ymm4, %ymm10+        vpxor %ymm2, %ymm0, %ymm2+        vpsrlq $0x14, %ymm10, %ymm11+        vpsllq $0x2c, %ymm10, %ymm10+        vpor %ymm11, %ymm10, %ymm10+        vpxor %ymm15, %ymm5, %ymm11+        vpbroadcastq (%rsi), %ymm15+        vpsrlq $0x15, %ymm11, %ymm14+        vpsllq $0x2b, %ymm11, %ymm11+        vpor %ymm14, %ymm11, %ymm11+        vpandn %ymm11, %ymm10, %ymm14+        vpxor %ymm15, %ymm14, %ymm14+        vpxor %ymm12, %ymm14, %ymm15+        vpsrlq $0x2b, %ymm13, %ymm14+        vpsllq $0x15, %ymm13, %ymm13+        vmovdqu %ymm15, (%rsp)+        vpor %ymm14, %ymm13, %ymm13+        vpandn %ymm13, %ymm11, %ymm14+        vpxor %ymm10, %ymm14, %ymm15+        vpsrlq $0x32, %ymm2, %ymm14+        vpsllq $0xe, %ymm2, %ymm2+        vmovdqu %ymm15, 0x20(%rsp)+        vpor %ymm14, %ymm2, %ymm2+        vpandn %ymm2, %ymm13, %ymm14+        vpxor %ymm11, %ymm14, %ymm11+        vmovdqu %ymm11, 0x40(%rsp)+        vpandn %ymm12, %ymm2, %ymm11+        vpandn %ymm10, %ymm12, %ymm12+        vpxor %ymm13, %ymm11, %ymm11+        vmovdqu %ymm11, 0x60(%rsp)+        vpxor %ymm2, %ymm12, %ymm11+        vpsrlq $0x24, %ymm8, %ymm2+        vpsllq $0x1c, %ymm8, %ymm8+        vmovdqu %ymm11, 0x80(%rsp)+        vpor %ymm2, %ymm8, %ymm8+        vpxor 0xe0(%rsp), %ymm0, %ymm2+        vpsrlq $0x2c, %ymm2, %ymm10+        vpsllq $0x14, %ymm2, %ymm2+        vpor %ymm10, %ymm2, %ymm2+        vpxor 0x100(%rsp), %ymm3, %ymm10+        vpsrlq $0x3d, %ymm10, %ymm11+        vpsllq $0x3, %ymm10, %ymm10+        vpor %ymm11, %ymm10, %ymm10+        vpandn %ymm10, %ymm2, %ymm11+        vpxor %ymm8, %ymm11, %ymm11+        vmovdqu %ymm11, 0xa0(%rsp)+        vpxor 0x160(%rsp), %ymm4, %ymm11+        vpsrlq $0x13, %ymm11, %ymm12+        vpsllq $0x2d, %ymm11, %ymm11+        vpor %ymm12, %ymm11, %ymm11+        vpandn %ymm11, %ymm10, %ymm12+        vpxor %ymm2, %ymm12, %ymm12+        vmovdqu %ymm12, 0xc0(%rsp)+        vpsrlq $0x3, %ymm7, %ymm12+        vpsllq $0x3d, %ymm7, %ymm7+        vpor %ymm12, %ymm7, %ymm7+        vpandn %ymm7, %ymm11, %ymm12+        vpxor %ymm10, %ymm12, %ymm10+        vpandn %ymm8, %ymm7, %ymm12+        vpandn %ymm2, %ymm8, %ymm8+        vpsrlq $0x3f, %ymm6, %ymm2+        vpsllq $0x1, %ymm6, %ymm6+        vpxor %ymm11, %ymm12, %ymm14+        vpor %ymm2, %ymm6, %ymm6+        vpsrlq $0x3a, %ymm9, %ymm2+        vpxor %ymm7, %ymm8, %ymm12+        vpsllq $0x6, %ymm9, %ymm9+        vmovdqu %ymm12, 0xe0(%rsp)+        vpxor 0x1a0(%rsp), %ymm0, %ymm7+        vpor %ymm2, %ymm9, %ymm9+        vpxor 0x120(%rsp), %ymm1, %ymm2+        vpshufb (%rdx), %ymm7, %ymm7+        vpsrlq $0x27, %ymm2, %ymm11+        vpsllq $0x19, %ymm2, %ymm2+        vpor %ymm2, %ymm11, %ymm11+        vpandn %ymm11, %ymm9, %ymm2+        vpandn %ymm7, %ymm11, %ymm8+        vpxor %ymm6, %ymm2, %ymm12+        vpxor 0x1c0(%rsp), %ymm3, %ymm2+        vpxor %ymm9, %ymm8, %ymm8+        vmovdqu %ymm12, 0x100(%rsp)+        vpsrlq $0x2e, %ymm2, %ymm12+        vpsllq $0x12, %ymm2, %ymm2+        vpor %ymm2, %ymm12, %ymm2+        vpandn %ymm2, %ymm7, %ymm12+        vpxor %ymm11, %ymm12, %ymm15+        vpandn %ymm6, %ymm2, %ymm11+        vpandn %ymm9, %ymm6, %ymm6+        vpxor %ymm7, %ymm11, %ymm12+        vmovdqu %ymm12, 0x120(%rsp)+        vpxor %ymm2, %ymm6, %ymm12+        vpxor 0x2e0(%rsp), %ymm0, %ymm6+        vpxor 0x2c0(%rsp), %ymm0, %ymm0+        vmovdqu %ymm12, 0x140(%rsp)+        vpsrlq $0x25, %ymm6, %ymm2+        vpsllq $0x1b, %ymm6, %ymm6+        vpor %ymm6, %ymm2, %ymm2+        vpxor 0x220(%rsp), %ymm3, %ymm6+        vpxor 0x200(%rsp), %ymm3, %ymm3+        vpsrlq $0x1c, %ymm6, %ymm7+        vpsllq $0x24, %ymm6, %ymm6+        vpor %ymm6, %ymm7, %ymm7+        vpxor 0x260(%rsp), %ymm4, %ymm6+        vpxor 0x240(%rsp), %ymm4, %ymm4+        vpsrlq $0x36, %ymm6, %ymm12+        vpsllq $0xa, %ymm6, %ymm6+        vpor %ymm6, %ymm12, %ymm12+        vpxor 0x180(%rsp), %ymm5, %ymm6+        vpxor 0x280(%rsp), %ymm5, %ymm5+        vpandn %ymm12, %ymm7, %ymm9+        vpsrlq $0x31, %ymm6, %ymm11+        vpsllq $0xf, %ymm6, %ymm6+        vpxor %ymm2, %ymm9, %ymm9+        vpor %ymm6, %ymm11, %ymm11+        vpandn %ymm11, %ymm12, %ymm6+        vpxor %ymm7, %ymm6, %ymm6+        vmovdqu %ymm6, 0x160(%rsp)+        vpxor 0x1e0(%rsp), %ymm1, %ymm6+        vpxor 0x2a0(%rsp), %ymm1, %ymm1+        vpshufb (%rcx), %ymm6, %ymm6+        vpandn %ymm6, %ymm11, %ymm13+        vpxor %ymm12, %ymm13, %ymm13+        vmovdqu %ymm13, 0x180(%rsp)+        vpandn %ymm2, %ymm6, %ymm13+        vpandn %ymm7, %ymm2, %ymm2+        vpxor %ymm6, %ymm2, %ymm2+        vpsrlq $0x3e, %ymm4, %ymm6+        vpxor %ymm11, %ymm13, %ymm13+        vmovdqu %ymm2, 0x1a0(%rsp)+        vpsrlq $0x2, %ymm5, %ymm2+        vpsllq $0x3e, %ymm5, %ymm5+        vpor %ymm5, %ymm2, %ymm2+        vpsrlq $0x9, %ymm1, %ymm5+        vpsllq $0x37, %ymm1, %ymm1+        vpsllq $0x2, %ymm4, %ymm4+        vpor %ymm1, %ymm5, %ymm1+        vpsrlq $0x19, %ymm0, %ymm5+        vpor %ymm4, %ymm6, %ymm4+        vpsllq $0x27, %ymm0, %ymm0+        vpor %ymm0, %ymm5, %ymm5+        vpandn %ymm5, %ymm1, %ymm0+        vpxor %ymm2, %ymm0, %ymm0+        vmovdqu %ymm0, 0x1c0(%rsp)+        vpsrlq $0x17, %ymm3, %ymm0+        vpsllq $0x29, %ymm3, %ymm3+        vpor %ymm3, %ymm0, %ymm0+        vpandn %ymm4, %ymm0, %ymm7+        vpandn %ymm0, %ymm5, %ymm3+        vpxor %ymm5, %ymm7, %ymm7+        vpandn %ymm2, %ymm4, %ymm5+        vpandn %ymm1, %ymm2, %ymm2+        vpxor %ymm0, %ymm5, %ymm5+        vpxor %ymm1, %ymm3, %ymm3+        vpxor %ymm4, %ymm2, %ymm2+        vmovdqu %ymm5, 0x1e0(%rsp)+        addq $0x8, %rsi+        addq $0x1, %r10+        cmpq $0x18, %r10+        jne LLmlk_keccak_f1600_x4_avx2+        vmovdqu (%rsp), %ymm4+        vmovdqu 0x40(%rsp), %ymm5+        vmovdqu 0x20(%rsp), %ymm0+        vmovdqu 0x60(%rsp), %ymm1+        vmovdqu 0x1c0(%rsp), %ymm12+        vmovdqu %ymm2, 0x1c0(%rsp)+        vpunpcklqdq %ymm0, %ymm4, %ymm2 # ymm2 = ymm4[0],ymm0[0],ymm4[2],ymm0[2]+        vpunpckhqdq %ymm0, %ymm4, %ymm0 # ymm0 = ymm4[1],ymm0[1],ymm4[3],ymm0[3]+        vpunpcklqdq %ymm1, %ymm5, %ymm4 # ymm4 = ymm5[0],ymm1[0],ymm5[2],ymm1[2]+        vpunpckhqdq %ymm1, %ymm5, %ymm1 # ymm1 = ymm5[1],ymm1[1],ymm5[3],ymm1[3]+        vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1]+        vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3]+        vmovdqu 0x80(%rsp), %ymm4+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm5 # ymm5 = ymm0[0,1],ymm1[0,1]+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm0 # ymm0 = ymm0[2,3],ymm1[2,3]+        vmovdqu %ymm6, (%rdi)+        vmovdqu %ymm5, 0xc8(%rdi)+        vmovdqu %ymm2, 0x190(%rdi)+        vmovdqu %ymm0, 0x258(%rdi)+        vmovdqu 0xa0(%rsp), %ymm0+        vpunpcklqdq %ymm0, %ymm4, %ymm2 # ymm2 = ymm4[0],ymm0[0],ymm4[2],ymm0[2]+        vpunpckhqdq %ymm0, %ymm4, %ymm1 # ymm1 = ymm4[1],ymm0[1],ymm4[3],ymm0[3]+        vmovdqu 0xc0(%rsp), %ymm0+        vpunpcklqdq %ymm10, %ymm0, %ymm4 # ymm4 = ymm0[0],ymm10[0],ymm0[2],ymm10[2]+        vpunpckhqdq %ymm10, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm10[1],ymm0[3],ymm10[3]+        vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1]+        vperm2i128 $0x20, %ymm0, %ymm1, %ymm5 # ymm5 = ymm1[0,1],ymm0[0,1]+        vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3]+        vmovdqu 0xe0(%rsp), %ymm4+        vperm2i128 $0x31, %ymm0, %ymm1, %ymm1 # ymm1 = ymm1[2,3],ymm0[2,3]+        vmovdqu 0x100(%rsp), %ymm0+        vmovdqu %ymm2, 0x1b0(%rdi)+        vmovdqu %ymm1, 0x278(%rdi)+        vpunpcklqdq %ymm4, %ymm14, %ymm2 # ymm2 = ymm14[0],ymm4[0],ymm14[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm14, %ymm1 # ymm1 = ymm14[1],ymm4[1],ymm14[3],ymm4[3]+        vpunpcklqdq %ymm8, %ymm0, %ymm4 # ymm4 = ymm0[0],ymm8[0],ymm0[2],ymm8[2]+        vpunpckhqdq %ymm8, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm8[1],ymm0[3],ymm8[3]+        vmovdqu %ymm6, 0x20(%rdi)+        vmovdqu %ymm5, 0xe8(%rdi)+        vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1]+        vperm2i128 $0x20, %ymm0, %ymm1, %ymm5 # ymm5 = ymm1[0,1],ymm0[0,1]+        vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3]+        vperm2i128 $0x31, %ymm0, %ymm1, %ymm1 # ymm1 = ymm1[2,3],ymm0[2,3]+        vmovdqu 0x120(%rsp), %ymm4+        vmovdqu 0x140(%rsp), %ymm0+        vmovdqu %ymm2, 0x1d0(%rdi)+        vmovdqu %ymm1, 0x298(%rdi)+        vpunpcklqdq %ymm4, %ymm15, %ymm2 # ymm2 = ymm15[0],ymm4[0],ymm15[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm15, %ymm1 # ymm1 = ymm15[1],ymm4[1],ymm15[3],ymm4[3]+        vpunpcklqdq %ymm9, %ymm0, %ymm4 # ymm4 = ymm0[0],ymm9[0],ymm0[2],ymm9[2]+        vmovdqu %ymm5, 0x108(%rdi)+        vpunpckhqdq %ymm9, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm9[1],ymm0[3],ymm9[3]+        vmovdqu %ymm6, 0x40(%rdi)+        vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1]+        vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3]+        vperm2i128 $0x20, %ymm0, %ymm1, %ymm5 # ymm5 = ymm1[0,1],ymm0[0,1]+        vmovdqu 0x160(%rsp), %ymm4+        vperm2i128 $0x31, %ymm0, %ymm1, %ymm1 # ymm1 = ymm1[2,3],ymm0[2,3]+        vmovdqu 0x180(%rsp), %ymm0+        vmovdqu %ymm5, 0x128(%rdi)+        vmovdqu 0x1a0(%rsp), %ymm5+        vmovdqu %ymm2, 0x1f0(%rdi)+        vpunpcklqdq %ymm0, %ymm4, %ymm2 # ymm2 = ymm4[0],ymm0[0],ymm4[2],ymm0[2]+        vpunpckhqdq %ymm0, %ymm4, %ymm0 # ymm0 = ymm4[1],ymm0[1],ymm4[3],ymm0[3]+        vpunpcklqdq %ymm5, %ymm13, %ymm4 # ymm4 = ymm13[0],ymm5[0],ymm13[2],ymm5[2]+        vmovdqu %ymm6, 0x60(%rdi)+        vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1]+        vmovdqu %ymm1, 0x2b8(%rdi)+        vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3]+        vpunpckhqdq %ymm5, %ymm13, %ymm1 # ymm1 = ymm13[1],ymm5[1],ymm13[3],ymm5[3]+        vmovdqu %ymm6, 0x80(%rdi)+        vmovdqu 0x1e0(%rsp), %ymm4+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm5 # ymm5 = ymm0[0,1],ymm1[0,1]+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm0 # ymm0 = ymm0[2,3],ymm1[2,3]+        vmovdqu %ymm2, 0x210(%rdi)+        vpunpcklqdq %ymm3, %ymm12, %ymm2 # ymm2 = ymm12[0],ymm3[0],ymm12[2],ymm3[2]+        vmovdqu %ymm0, 0x2d8(%rdi)+        vpunpckhqdq %ymm3, %ymm12, %ymm0 # ymm0 = ymm12[1],ymm3[1],ymm12[3],ymm3[3]+        vpunpcklqdq %ymm4, %ymm7, %ymm3 # ymm3 = ymm7[0],ymm4[0],ymm7[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm7, %ymm1 # ymm1 = ymm7[1],ymm4[1],ymm7[3],ymm4[3]+        vmovdqu %ymm5, 0x148(%rdi)+        vperm2i128 $0x20, %ymm3, %ymm2, %ymm5 # ymm5 = ymm2[0,1],ymm3[0,1]+        vperm2i128 $0x31, %ymm3, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm3[2,3]+        vmovdqu 0x1c0(%rsp), %ymm3+        vperm2i128 $0x20, %ymm1, %ymm0, %ymm4 # ymm4 = ymm0[0,1],ymm1[0,1]+        vperm2i128 $0x31, %ymm1, %ymm0, %ymm0 # ymm0 = ymm0[2,3],ymm1[2,3]+        vmovdqu %ymm5, 0xa0(%rdi)+        vextracti128 $0x1, %ymm3, %xmm15+        vmovdqu %ymm4, 0x168(%rdi)+        vmovdqu %ymm2, 0x230(%rdi)+        vmovdqu %ymm0, 0x2f8(%rdi)+        vmovq %xmm3, 0xc0(%rdi)+        vmovhpd %xmm3, 0x188(%rdi)+        vmovq %xmm15, 0x250(%rdi)+        vmovhpd %xmm15, 0x318(%rdi)+        movq %r11, %rsp+        .cfi_def_cfa_register %rsp+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(keccak_f1600_x4_avx2_asm)++#endif /* MLK_FIPS202_X86_64_NEED_X4_AVX2 && !MLK_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/fips202/native/x86_64/src/keccakf1600_constants.c view
@@ -0,0 +1,52 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mlkem-native repository.+ *          Do not modify it directly.+ */++#include "../../../../common.h"+#if defined(MLK_FIPS202_X86_64_NEED_X4_AVX2) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++#include <stdint.h>++#include "fips202_native_x86_64.h"++MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint64_t+    mlk_keccakf1600_round_constants[24] = {+        0x0000000000000001, 0x0000000000008082, 0x800000000000808a,+        0x8000000080008000, 0x000000000000808b, 0x0000000080000001,+        0x8000000080008081, 0x8000000000008009, 0x000000000000008a,+        0x0000000000000088, 0x0000000080008009, 0x000000008000000a,+        0x000000008000808b, 0x800000000000008b, 0x8000000000008089,+        0x8000000000008003, 0x8000000000008002, 0x8000000000000080,+        0x000000000000800a, 0x800000008000000a, 0x8000000080008081,+        0x8000000000008080, 0x0000000080000001, 0x8000000080008008,+};++MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint64_t mlk_keccak_rho8[4] = {+    0x0605040302010007,+    0x0e0d0c0b0a09080f,+    0x1615141312111017,+    0x1e1d1c1b1a19181f,+};++MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint64_t mlk_keccak_rho56[4] = {+    0x0007060504030201,+    0x080f0e0d0c0b0a09,+    0x1017161514131211,+    0x181f1e1d1c1b1a19,+};++#else /* MLK_FIPS202_X86_64_NEED_X4_AVX2 && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+       */++MLK_EMPTY_CU(fips202_x86_64_constants)++#endif /* !(MLK_FIPS202_X86_64_NEED_X4_AVX2 && \+          !MLK_CONFIG_MULTILEVEL_NO_SHARED) */
+ cbits/mlkem/src/indcpa.c view
@@ -0,0 +1,675 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ *+ * - [REF]+ *   CRYSTALS-Kyber C reference implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/ref+ */++#include "indcpa.h"++#include "debug.h"+#include "randombytes.h"+#include "sampling.h"+#include "symmetric.h"+#include "verify.h"++/* Parameter set namespacing+ * This is to facilitate building multiple instances+ * of mlkem-native (e.g. with varying parameter sets)+ * within a single compilation unit. */+#define mlk_pack_pk MLK_ADD_PARAM_SET(mlk_pack_pk)+#define mlk_unpack_pk MLK_ADD_PARAM_SET(mlk_unpack_pk)+#define mlk_pack_sk MLK_ADD_PARAM_SET(mlk_pack_sk)+#define mlk_unpack_sk MLK_ADD_PARAM_SET(mlk_unpack_sk)+#define mlk_pack_ciphertext MLK_ADD_PARAM_SET(mlk_pack_ciphertext)+#define mlk_unpack_ciphertext MLK_ADD_PARAM_SET(mlk_unpack_ciphertext)+#define mlk_matvec_mul MLK_ADD_PARAM_SET(mlk_matvec_mul)+#define mlk_polyvec_permute_bitrev_to_custom \+  MLK_ADD_PARAM_SET(mlk_polyvec_permute_bitrev_to_custom)+#define mlk_polymat_permute_bitrev_to_custom \+  MLK_ADD_PARAM_SET(mlk_polymat_permute_bitrev_to_custom)+#define mlk_keypair_getnoise_eta1 MLK_ADD_PARAM_SET(mlk_keypair_getnoise_eta1)+#define mlk_enc_getnoise_eta1_eta2 MLK_ADD_PARAM_SET(mlk_enc_getnoise_eta1_eta2)+/* End of parameter set namespacing */++/**+ * Serialize the public key as the concatenation of the serialized vector of+ * polynomials pk and the public seed used to generate the matrix A.+ *+ * @spec{Implements @[FIPS203, Algorithm 13 (K-PKE.KeyGen), L19].}+ *+ * @param[out] r    Output serialized public key.+ * @param[in]  pk   Input public-key polyvec. Must have coefficients within+ *                  [0,..,MLKEM_Q-1].+ * @param[in]  seed Input public seed.+ */+#if !defined(MLK_CONFIG_NO_KEYPAIR_API)+static void mlk_pack_pk(uint8_t r[MLKEM_INDCPA_PUBLICKEYBYTES],+                        const mlk_polyvec *pk,+                        const uint8_t seed[MLKEM_SYMBYTES])+{+  mlk_assert_bound_2d(pk->vec, MLKEM_K, MLKEM_N, 0, MLKEM_Q);+  mlk_polyvec_tobytes(r, pk);+  mlk_memcpy(r + MLKEM_POLYVECBYTES, seed, MLKEM_SYMBYTES);+}+#endif /* !MLK_CONFIG_NO_KEYPAIR_API */++/**+ * De-serialize public key from a byte array; approximate inverse of+ * mlk_pack_pk.+ *+ * @spec{Implements @[FIPS203, Algorithm 14 (K-PKE.Encrypt), L2-3].}+ *+ * @param[out] pk       Output public-key polynomial vector. Coefficients+ *                      will be normalized to [0,1,..,MLKEM_Q-1].+ * @param[out] seed     Output seed to generate matrix A.+ * @param[in]  packedpk Input serialized public key.+ */+#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+static void mlk_unpack_pk(mlk_polyvec *pk, uint8_t seed[MLKEM_SYMBYTES],+                          const uint8_t packedpk[MLKEM_INDCPA_PUBLICKEYBYTES])+{+  mlk_polyvec_frombytes(pk, packedpk);+  mlk_memcpy(seed, packedpk + MLKEM_POLYVECBYTES, MLKEM_SYMBYTES);++  /* NOTE: If a modulus check was conducted on the PK, we know at this+   * point that the coefficients of `pk` are unsigned canonical. The+   * specifications and proofs, however, do _not_ assume this, and instead+   * work with the easily provable bound by MLKEM_UINT12_LIMIT. */+}+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++/**+ * Serialize the secret key.+ *+ * @spec{Implements @[FIPS203, Algorithm 13 (K-PKE.KeyGen), L20].}+ *+ * @param[out] r  Output serialized secret key.+ * @param[in]  sk Input vector of polynomials (secret key).+ */+#if !defined(MLK_CONFIG_NO_KEYPAIR_API)+static void mlk_pack_sk(uint8_t r[MLKEM_INDCPA_SECRETKEYBYTES],+                        const mlk_polyvec *sk)+{+  mlk_assert_bound_2d(sk->vec, MLKEM_K, MLKEM_N, 0, MLKEM_Q);+  mlk_polyvec_tobytes(r, sk);+}+#endif /* !MLK_CONFIG_NO_KEYPAIR_API */++/**+ * De-serialize the secret key; inverse of mlk_pack_sk.+ *+ * @spec{Implements @[FIPS203, Algorithm 15 (K-PKE.Decrypt), L5].}+ *+ * @param[out] sk       Output vector of polynomials (secret key).+ * @param[in]  packedsk Input serialized secret key.+ */+#if !defined(MLK_CONFIG_NO_DECAPS_API)+static void mlk_unpack_sk(mlk_polyvec *sk,+                          const uint8_t packedsk[MLKEM_INDCPA_SECRETKEYBYTES])+{+  mlk_polyvec_frombytes(sk, packedsk);+}+#endif /* !MLK_CONFIG_NO_DECAPS_API */++/**+ * Serialize the ciphertext as the concatenation of the compressed and+ * serialized vector of polynomials b and the compressed and serialized+ * polynomial v.+ *+ * @spec{Implements @[FIPS203, Algorithm 14 (K-PKE.Encrypt), L22-23].}+ *+ * @param[out] r Output serialized ciphertext.+ * @param[in]  b Input vector of polynomials b.+ * @param[in]  v Input polynomial v.+ */+#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+static void mlk_pack_ciphertext(uint8_t r[MLKEM_INDCPA_BYTES],+                                const mlk_polyvec *b, mlk_poly *v)+{+  mlk_polyvec_compress_du(r, b);+  mlk_poly_compress_dv(r + MLKEM_POLYVECCOMPRESSEDBYTES_DU, v);+}+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++/**+ * De-serialize and decompress ciphertext from a byte array; approximate+ * inverse of mlk_pack_ciphertext.+ *+ * @spec{Implements @[FIPS203, Algorithm 15 (K-PKE.Decrypt), L1-4].}+ *+ * @param[out] b Output vector of polynomials b.+ * @param[out] v Output polynomial v.+ * @param[in]  c Input serialized ciphertext.+ */+#if !defined(MLK_CONFIG_NO_DECAPS_API)+static void mlk_unpack_ciphertext(mlk_polyvec *b, mlk_poly *v,+                                  const uint8_t c[MLKEM_INDCPA_BYTES])+{+  mlk_polyvec_decompress_du(b, c);+  mlk_poly_decompress_dv(v, c + MLKEM_POLYVECCOMPRESSEDBYTES_DU);+}+#endif /* !MLK_CONFIG_NO_DECAPS_API */++/* Helper function to ensure that the polynomial entries in the output+ * of gen_matrix use the standard (bitreversed) ordering of coefficients.+ * No-op unless a native backend with a custom ordering is used.+ *+ * We don't inline this into gen_matrix to avoid having to split the CBMC+ * proof for gen_matrix based on MLK_USE_NATIVE_NTT_CUSTOM_ORDER. */+static void mlk_polyvec_permute_bitrev_to_custom(mlk_polyvec *v)+__contract__(+  /* We don't specify that this should be a permutation, but only+   * that it does not change the bound established at the end of mlk_gen_matrix. */+  requires(memory_no_alias(v, sizeof(mlk_polyvec)))+  requires(forall(x, 0, MLKEM_K,+    array_bound(v->vec[x].coeffs, 0, MLKEM_N, 0, MLKEM_Q)))+  assigns(memory_slice(v, sizeof(mlk_polyvec)))+  ensures(forall(x, 0, MLKEM_K,+    array_bound(v->vec[x].coeffs, 0, MLKEM_N, 0, MLKEM_Q))))+{+#if defined(MLK_USE_NATIVE_NTT_CUSTOM_ORDER)+  unsigned i;+  for (i = 0; i < MLKEM_K; i++)+  __loop__(+     assigns(i, memory_slice(v, sizeof(mlk_polyvec)))+     invariant(i <= MLKEM_K)+     invariant(forall(x, 0, MLKEM_K,+       array_bound(v->vec[x].coeffs, 0, MLKEM_N, 0, MLKEM_Q)))+     decreases(MLKEM_K - i))+  {+    mlk_poly_permute_bitrev_to_custom(v->vec[i].coeffs);+  }+#else  /* MLK_USE_NATIVE_NTT_CUSTOM_ORDER */+  /* Nothing to do */+  (void)v;+#endif /* !MLK_USE_NATIVE_NTT_CUSTOM_ORDER */+}++static void mlk_polymat_permute_bitrev_to_custom(mlk_polymat *a)+__contract__(+  /* We don't specify that this should be a permutation, but only+   * that it does not change the bound established at the end of mlk_gen_matrix. */+  requires(memory_no_alias(a, sizeof(mlk_polymat)))+  requires(forall(x, 0, MLKEM_K, forall(y, 0, MLKEM_K,+    array_bound(a->vec[x].vec[y].coeffs, 0, MLKEM_N, 0, MLKEM_Q))))+  assigns(memory_slice(a, sizeof(mlk_polymat)))+  ensures(forall(x, 0, MLKEM_K, forall(y, 0, MLKEM_K,+    array_bound(a->vec[x].vec[y].coeffs, 0, MLKEM_N, 0, MLKEM_Q)))))+{+  unsigned i;+  for (i = 0; i < MLKEM_K; i++)+  __loop__(+     assigns(i, memory_slice(a, sizeof(mlk_polymat)))+     invariant(i <= MLKEM_K)+     invariant(forall(x, 0, MLKEM_K, forall(y, 0, MLKEM_K,+       array_bound(a->vec[x].vec[y].coeffs, 0, MLKEM_N, 0, MLKEM_Q))))+     decreases(MLKEM_K - i))+  {+    mlk_polyvec_permute_bitrev_to_custom(&a->vec[i]);+  }+}++/* Reference: `gen_matrix()` in the reference implementation @[REF].+ *            - We use a special subroutine to generate 4 polynomials+ *              at a time, to be able to leverage batched Keccak-f1600+ *              implementations. The reference implementation generates+ *              one matrix entry a time.+ *+ * Not static for benchmarking */+MLK_INTERNAL_API+void mlk_gen_matrix(mlk_polymat *a, const uint8_t seed[MLKEM_SYMBYTES],+                    int transposed)+{+  unsigned i, j;+  MLK_ALIGN uint8_t seed_ext[4][MLK_ALIGN_UP(MLKEM_SYMBYTES + 2)];++  for (j = 0; j < 4; j++)+  {+    mlk_memcpy(seed_ext[j], seed, MLKEM_SYMBYTES);+  }++#if !defined(MLK_CONFIG_SERIAL_FIPS202_ONLY)+  /* Sample 4 matrix entries a time. */+  for (i = 0; i < (MLKEM_K * MLKEM_K / 4) * 4; i += 4)+  {+    for (j = 0; j < 4; j++)+    {+      uint8_t x, y;+      /* MLKEM_K <= 4, so the values fit in uint8_t. */+      x = (uint8_t)((i + j) / MLKEM_K);+      y = (uint8_t)((i + j) % MLKEM_K);+      if (transposed)+      {+        seed_ext[j][MLKEM_SYMBYTES + 0] = x;+        seed_ext[j][MLKEM_SYMBYTES + 1] = y;+      }+      else+      {+        seed_ext[j][MLKEM_SYMBYTES + 0] = y;+        seed_ext[j][MLKEM_SYMBYTES + 1] = x;+      }+    }++    mlk_poly_rej_uniform_x4(&a->vec[i / MLKEM_K].vec[i % MLKEM_K],+                            &a->vec[(i + 1) / MLKEM_K].vec[(i + 1) % MLKEM_K],+                            &a->vec[(i + 2) / MLKEM_K].vec[(i + 2) % MLKEM_K],+                            &a->vec[(i + 3) / MLKEM_K].vec[(i + 3) % MLKEM_K],+                            seed_ext);+  }+#else  /* !MLK_CONFIG_SERIAL_FIPS202_ONLY */+  /* When using serial FIPS202, sample all entries individually. */+  i = 0;+#endif /* MLK_CONFIG_SERIAL_FIPS202_ONLY */++  /* For MLKEM_K == 3, sample the last entry individually.+   * When MLK_CONFIG_SERIAL_FIPS202_ONLY is set, sample all entries+   * individually. */+  for (; i < MLKEM_K * MLKEM_K; i++)+  {+    uint8_t x, y;+    /* MLKEM_K <= 4, so the values fit in uint8_t. */+    x = (uint8_t)(i / MLKEM_K);+    y = (uint8_t)(i % MLKEM_K);++    if (transposed)+    {+      seed_ext[0][MLKEM_SYMBYTES + 0] = x;+      seed_ext[0][MLKEM_SYMBYTES + 1] = y;+    }+    else+    {+      seed_ext[0][MLKEM_SYMBYTES + 0] = y;+      seed_ext[0][MLKEM_SYMBYTES + 1] = x;+    }++    mlk_poly_rej_uniform(&a->vec[i / MLKEM_K].vec[i % MLKEM_K], seed_ext[0]);+  }++  mlk_assert(i == MLKEM_K * MLKEM_K);++  /*+   * The public matrix is generated in NTT domain. If the native backend+   * uses a custom order in NTT domain, permute A accordingly.+   */+  mlk_polymat_permute_bitrev_to_custom(a);++  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  mlk_zeroize(seed_ext, sizeof(seed_ext));+}++/**+ * Compute matrix-vector product in NTT domain, via Montgomery multiplication.+ *+ * @spec{Implements @[FIPS203, Section 2.4.7, Eq (2.12), (2.13)].}+ *+ * @param[out] out Output polynomial vector.+ * @param[in]  a   Input matrix. Must be in NTT domain and have coefficients+ *                 of absolute value < 4096.+ * @param[in]  v   Input polynomial vector. Must be in NTT domain.+ * @param[in]  vc  Mulcache for @p v, computed via+ *                 mlk_polyvec_mulcache_compute().+ */+static void mlk_matvec_mul(mlk_polyvec *out, const mlk_polymat *a,+                           const mlk_polyvec *v, const mlk_polyvec_mulcache *vc)+__contract__(+  requires(memory_no_alias(out, sizeof(mlk_polyvec)))+  requires(memory_no_alias(a, sizeof(mlk_polymat)))+  requires(memory_no_alias(v, sizeof(mlk_polyvec)))+  requires(memory_no_alias(vc, sizeof(mlk_polyvec_mulcache)))+  requires(forall(k0, 0, MLKEM_K,+    forall(k1, 0, MLKEM_K,+      array_bound(a->vec[k0].vec[k1].coeffs, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT))))+  assigns(memory_slice(out, sizeof(mlk_polyvec))))+{+  unsigned i;+  for (i = 0; i < MLKEM_K; i++)+  __loop__(+    assigns(i, memory_slice(out, sizeof(mlk_polyvec)))+    invariant(i <= MLKEM_K)+    decreases(MLKEM_K - i))+  {+    mlk_polyvec_basemul_acc_montgomery_cached(&out->vec[i], &a->vec[i], v, vc);+  }+}++/**+ * Compute and fill the pv and e polyvec structures needed by+ * mlk_keypair_derand(). Uses x4-batched versions of `poly_getnoise` to+ * leverage batched Keccak-f1600.+ *+ * @spec{Implements @[FIPS203, Algorithm 13 (K-PKE.KeyGen)] steps 8-15.}+ *+ * @param[out] pv   Output polynomial vector.+ * @param[out] e    Output polynomial vector.+ * @param[in]  seed Seed bytes for sampling.+ */+#if !defined(MLK_CONFIG_NO_KEYPAIR_API)+static void mlk_keypair_getnoise_eta1(mlk_polyvec *pv, mlk_polyvec *e,+                                      const uint8_t seed[MLKEM_SYMBYTES])+__contract__(+  requires(memory_no_alias(pv, sizeof(mlk_polyvec)))+  requires(memory_no_alias(e, sizeof(mlk_polyvec)))+  requires(memory_no_alias(seed, MLKEM_SYMBYTES))+  assigns(memory_slice(pv, sizeof(mlk_polyvec)))+  assigns(memory_slice(e, sizeof(mlk_polyvec)))+  ensures(forall(k0, 0, MLKEM_K, array_abs_bound(pv->vec[k0].coeffs, 0, MLKEM_N, MLKEM_ETA1 + 1)))+  ensures(forall(k1, 0, MLKEM_K, array_abs_bound(e->vec[k1].coeffs, 0, MLKEM_N, MLKEM_ETA1 + 1)))+)+{+#if MLKEM_K == 2+  mlk_poly_getnoise_eta1_4x(&pv->vec[0], &pv->vec[1], /* Fill elements of pv */+                            &e->vec[0], &e->vec[1], /* and two elements of e */+                            seed, 0, 1, 2, 3);+#elif MLKEM_K == 3+  /*+   * Only the first three output buffers are needed, so we pass NULL as+   * the fourth parameter, and 0xFF as its dummy nonce.+   */+  mlk_poly_getnoise_eta1_4x(&pv->vec[0], &pv->vec[1], &pv->vec[2], NULL, seed,+                            0, 1, 2, 0xFF);+  /* Same here */+  mlk_poly_getnoise_eta1_4x(&e->vec[0], &e->vec[1], &e->vec[2], NULL, seed, 3,+                            4, 5, 0xFF);+#elif MLKEM_K == 4+  mlk_poly_getnoise_eta1_4x(&pv->vec[0], &pv->vec[1], &pv->vec[2], &pv->vec[3],+                            seed, 0, 1, 2, 3);+  mlk_poly_getnoise_eta1_4x(&e->vec[0], &e->vec[1], &e->vec[2], &e->vec[3],+                            seed, 4, 5, 6, 7);+#endif /* MLKEM_K == 4 */+}+#endif /* !MLK_CONFIG_NO_KEYPAIR_API */++/**+ * Compute and fill the sp, ep, and epp polynomial structures needed by+ * mlk_indcpa_enc(). Uses x4-batched versions of `poly_getnoise` to leverage+ * batched Keccak-f1600.+ *+ * @spec{Implements @[FIPS203, Algorithm 14 (K-PKE.Encrypt)] steps 9-16.}+ *+ * @param[out] sp    Output polynomial vector.+ * @param[out] ep    Output polynomial vector.+ * @param[out] epp   Output polynomial.+ * @param[in]  coins Seed bytes for sampling.+ */+#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+static void mlk_enc_getnoise_eta1_eta2(mlk_polyvec *sp, mlk_polyvec *ep,+                                       mlk_poly *epp,+                                       const uint8_t coins[MLKEM_SYMBYTES])+__contract__(+  requires(memory_no_alias(sp, sizeof(mlk_polyvec)))+  requires(memory_no_alias(ep, sizeof(mlk_polyvec)))+  requires(memory_no_alias(epp, sizeof(mlk_poly)))+  requires(memory_no_alias(coins, MLKEM_SYMBYTES))+  assigns(memory_slice(sp, sizeof(mlk_polyvec)))+  assigns(memory_slice(ep, sizeof(mlk_polyvec)))+  assigns(memory_slice(epp, sizeof(mlk_poly)))+  ensures(forall(k0, 0, MLKEM_K, array_abs_bound(sp->vec[k0].coeffs, 0, MLKEM_N, MLKEM_ETA1 + 1)))+  ensures(forall(k1, 0, MLKEM_K, array_abs_bound(ep->vec[k1].coeffs, 0, MLKEM_N, MLKEM_ETA2 + 1)))+  ensures(array_abs_bound(epp->coeffs, 0, MLKEM_N, MLKEM_ETA2 + 1))+)+{+#if MLKEM_K == 2+  mlk_poly_getnoise_eta1122_4x(&sp->vec[0], &sp->vec[1], &ep->vec[0],+                               &ep->vec[1], coins, 0, 1, 2, 3);+  mlk_poly_getnoise_eta2(epp, coins, 4);+#elif MLKEM_K == 3+  /*+   * In this call, only the first three output buffers are needed,+   * so we pass NULL as the fourth parameter, and 0xFF as its dummy nonce.+   */+  mlk_poly_getnoise_eta1_4x(&sp->vec[0], &sp->vec[1], &sp->vec[2], NULL, coins,+                            0, 1, 2, 0xFF /* irrelevant */);+  /* The fourth output buffer in this call _is_ used. */+  mlk_poly_getnoise_eta2_4x(&ep->vec[0], &ep->vec[1], &ep->vec[2], epp, coins,+                            3, 4, 5, 6);+#elif MLKEM_K == 4+  mlk_poly_getnoise_eta1_4x(&sp->vec[0], &sp->vec[1], &sp->vec[2], &sp->vec[3],+                            coins, 0, 1, 2, 3);+  mlk_poly_getnoise_eta2_4x(&ep->vec[0], &ep->vec[1], &ep->vec[2], &ep->vec[3],+                            coins, 4, 5, 6, 7);+  mlk_poly_getnoise_eta2(epp, coins, 8);+#endif /* MLKEM_K == 4 */+}+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */+++/* Reference: `indcpa_keypair_derand()` in the reference implementation @[REF].+ *            - We use a different implementation of `gen_matrix()` which+ *              uses x4-batched Keccak-f1600 (see `mlk_gen_matrix()` above).+ *            - We use a mulcache to speed up matrix-vector multiplication.+ *            - We include buffer zeroization.+ */+#if !defined(MLK_CONFIG_NO_KEYPAIR_API)+MLK_INTERNAL_API+int mlk_indcpa_keypair_derand(uint8_t pk[MLKEM_INDCPA_PUBLICKEYBYTES],+                              uint8_t sk[MLKEM_INDCPA_SECRETKEYBYTES],+                              const uint8_t coins[MLKEM_SYMBYTES],+                              MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  int ret = 0;+  const uint8_t *publicseed;+  const uint8_t *noiseseed;+  MLK_ALLOC(buf, uint8_t, 2 * MLKEM_SYMBYTES, context);+  MLK_ALLOC(coins_with_domain_separator, uint8_t, MLKEM_SYMBYTES + 1, context);+  MLK_ALLOC(a, mlk_polymat, 1, context);+  MLK_ALLOC(e, mlk_polyvec, 1, context);+  MLK_ALLOC(pkpv, mlk_polyvec, 1, context);+  MLK_ALLOC(skpv, mlk_polyvec, 1, context);+  MLK_ALLOC(skpv_cache, mlk_polyvec_mulcache, 1, context);++  if (buf == NULL || coins_with_domain_separator == NULL || a == NULL ||+      e == NULL || pkpv == NULL || skpv == NULL || skpv_cache == NULL)+  {+    ret = MLK_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  publicseed = buf;+  noiseseed = buf + MLKEM_SYMBYTES;++  /* Concatenate coins with MLKEM_K for domain separation of security levels */+  mlk_memcpy(coins_with_domain_separator, coins, MLKEM_SYMBYTES);+  coins_with_domain_separator[MLKEM_SYMBYTES] = MLKEM_K;++  mlk_hash_g(buf, coins_with_domain_separator, MLKEM_SYMBYTES + 1);++  /*+   * Declassify the public seed.+   * Required to use it in conditional-branches in rejection sampling.+   * This is needed because all output of randombytes is marked as secret+   * (=undefined)+   */+  MLK_CT_TESTING_DECLASSIFY(publicseed, MLKEM_SYMBYTES);++  mlk_gen_matrix(a, publicseed, 0 /* no transpose */);++  mlk_keypair_getnoise_eta1(skpv, e, noiseseed);++  mlk_polyvec_ntt(skpv);+  mlk_polyvec_ntt(e);++  mlk_polyvec_mulcache_compute(skpv_cache, skpv);+  mlk_matvec_mul(pkpv, a, skpv, skpv_cache);+  mlk_polyvec_tomont(pkpv);++  mlk_polyvec_add(pkpv, e);+  mlk_polyvec_reduce(pkpv);+  mlk_polyvec_reduce(skpv);++  mlk_pack_sk(sk, skpv);+  mlk_pack_pk(pk, pkpv, publicseed);++cleanup:+  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  MLK_FREE(skpv_cache, mlk_polyvec_mulcache, 1, context);+  MLK_FREE(skpv, mlk_polyvec, 1, context);+  MLK_FREE(pkpv, mlk_polyvec, 1, context);+  MLK_FREE(e, mlk_polyvec, 1, context);+  MLK_FREE(a, mlk_polymat, 1, context);+  MLK_FREE(coins_with_domain_separator, uint8_t, MLKEM_SYMBYTES + 1, context);+  MLK_FREE(buf, uint8_t, 2 * MLKEM_SYMBYTES, context);+  return ret;+}+#endif /* !MLK_CONFIG_NO_KEYPAIR_API */++/* Reference: `indcpa_enc()` in the reference implementation @[REF].+ *            - We use x4-batched versions of `poly_getnoise` to leverage+ *              batched x4-batched Keccak-f1600.+ *            - We use a different implementation of `gen_matrix()` which+ *              uses x4-batched Keccak-f1600 (see `mlk_gen_matrix()` above).+ *            - We use a mulcache to speed up matrix-vector multiplication.+ *            - We include buffer zeroization.+ */+#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+MLK_INTERNAL_API+int mlk_indcpa_enc(uint8_t c[MLKEM_INDCPA_BYTES],+                   const uint8_t m[MLKEM_INDCPA_MSGBYTES],+                   const uint8_t pk[MLKEM_INDCPA_PUBLICKEYBYTES],+                   const uint8_t coins[MLKEM_SYMBYTES],+                   MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  int ret = 0;+  MLK_ALLOC(seed, uint8_t, MLKEM_SYMBYTES, context);+  MLK_ALLOC(at, mlk_polymat, 1, context);+  MLK_ALLOC(sp, mlk_polyvec, 1, context);+  MLK_ALLOC(pkpv, mlk_polyvec, 1, context);+  MLK_ALLOC(ep, mlk_polyvec, 1, context);+  MLK_ALLOC(b, mlk_polyvec, 1, context);+  MLK_ALLOC(v, mlk_poly, 1, context);+  MLK_ALLOC(k, mlk_poly, 1, context);+  MLK_ALLOC(epp, mlk_poly, 1, context);+  MLK_ALLOC(sp_cache, mlk_polyvec_mulcache, 1, context);++  if (seed == NULL || at == NULL || sp == NULL || pkpv == NULL || ep == NULL ||+      b == NULL || v == NULL || k == NULL || epp == NULL || sp_cache == NULL)+  {+    ret = MLK_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  mlk_unpack_pk(pkpv, seed, pk);+  mlk_poly_frommsg(k, m);++  /*+   * Declassify the public seed.+   * Required to use it in conditional-branches in rejection sampling.+   * This is needed because in re-encryption the publicseed originated from sk+   * which is marked undefined.+   */+  MLK_CT_TESTING_DECLASSIFY(seed, MLKEM_SYMBYTES);++  mlk_gen_matrix(at, seed, 1 /* transpose */);++  mlk_enc_getnoise_eta1_eta2(sp, ep, epp, coins);++  mlk_polyvec_ntt(sp);++  mlk_polyvec_mulcache_compute(sp_cache, sp);+  mlk_matvec_mul(b, at, sp, sp_cache);+  mlk_polyvec_basemul_acc_montgomery_cached(v, pkpv, sp, sp_cache);++  mlk_polyvec_invntt_tomont(b);+  mlk_poly_invntt_tomont(v);++  mlk_polyvec_add(b, ep);+  mlk_poly_add(v, epp);+  mlk_poly_add(v, k);++  mlk_polyvec_reduce(b);+  mlk_poly_reduce(v);++  mlk_pack_ciphertext(c, b, v);++cleanup:+  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  MLK_FREE(sp_cache, mlk_polyvec_mulcache, 1, context);+  MLK_FREE(epp, mlk_poly, 1, context);+  MLK_FREE(k, mlk_poly, 1, context);+  MLK_FREE(v, mlk_poly, 1, context);+  MLK_FREE(b, mlk_polyvec, 1, context);+  MLK_FREE(ep, mlk_polyvec, 1, context);+  MLK_FREE(pkpv, mlk_polyvec, 1, context);+  MLK_FREE(sp, mlk_polyvec, 1, context);+  MLK_FREE(at, mlk_polymat, 1, context);+  MLK_FREE(seed, uint8_t, MLKEM_SYMBYTES, context);+  return ret;+}+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++/* Reference: `indcpa_dec()` in the reference implementation @[REF].+ *            - We use a mulcache for the scalar product.+ *            - We include buffer zeroization. */+#if !defined(MLK_CONFIG_NO_DECAPS_API)+MLK_INTERNAL_API+int mlk_indcpa_dec(uint8_t m[MLKEM_INDCPA_MSGBYTES],+                   const uint8_t c[MLKEM_INDCPA_BYTES],+                   const uint8_t sk[MLKEM_INDCPA_SECRETKEYBYTES],+                   MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  int ret = 0;+  MLK_ALLOC(b, mlk_polyvec, 1, context);+  MLK_ALLOC(skpv, mlk_polyvec, 1, context);+  MLK_ALLOC(v, mlk_poly, 1, context);+  MLK_ALLOC(sb, mlk_poly, 1, context);+  MLK_ALLOC(b_cache, mlk_polyvec_mulcache, 1, context);++  if (b == NULL || skpv == NULL || v == NULL || sb == NULL || b_cache == NULL)+  {+    ret = MLK_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  mlk_unpack_ciphertext(b, v, c);+  mlk_unpack_sk(skpv, sk);++  mlk_polyvec_ntt(b);+  mlk_polyvec_mulcache_compute(b_cache, b);+  mlk_polyvec_basemul_acc_montgomery_cached(sb, skpv, b, b_cache);+  mlk_poly_invntt_tomont(sb);++  mlk_poly_sub(v, sb);+  mlk_poly_reduce(v);++  mlk_poly_tomsg(m, v);++cleanup:+  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  MLK_FREE(b_cache, mlk_polyvec_mulcache, 1, context);+  MLK_FREE(sb, mlk_poly, 1, context);+  MLK_FREE(v, mlk_poly, 1, context);+  MLK_FREE(skpv, mlk_polyvec, 1, context);+  MLK_FREE(b, mlk_polyvec, 1, context);+  return ret;+}+#endif /* !MLK_CONFIG_NO_DECAPS_API */++/* To facilitate single-compilation-unit (SCU) builds, undefine all macros.+ * Don't modify by hand -- this is auto-generated by scripts/autogen. */+#undef mlk_pack_pk+#undef mlk_unpack_pk+#undef mlk_pack_sk+#undef mlk_unpack_sk+#undef mlk_pack_ciphertext+#undef mlk_unpack_ciphertext+#undef mlk_matvec_mul+#undef mlk_polyvec_permute_bitrev_to_custom+#undef mlk_polymat_permute_bitrev_to_custom+#undef mlk_keypair_getnoise_eta1+#undef mlk_enc_getnoise_eta1_eta2
+ cbits/mlkem/src/indcpa.h view
@@ -0,0 +1,172 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ */++#ifndef MLK_INDCPA_H+#define MLK_INDCPA_H++#include "cbmc.h"+#include "common.h"+#include "poly_k.h"++#define mlk_gen_matrix MLK_NAMESPACE_K(gen_matrix)+/**+ * Deterministically generate matrix A (or the transpose of A) from a seed.+ * Entries of the matrix are polynomials that look uniformly random.+ * Performs rejection sampling on the output of an XOF.+ *+ * @spec{Implements @[FIPS203, Algorithm 13 (K-PKE.KeyGen), L3-7] and+ * @[FIPS203, Algorithm 14 (K-PKE.Encrypt), L4-8]. The @p transposed+ * parameter only affects internal presentation.}+ *+ * @param[out] a          Output matrix A.+ * @param[in]  seed       Input seed.+ * @param      transposed Boolean deciding whether A or A^T is generated.+ */+MLK_INTERNAL_API+void mlk_gen_matrix(mlk_polymat *a, const uint8_t seed[MLKEM_SYMBYTES],+                    int transposed)+__contract__(+  requires(memory_no_alias(a, sizeof(mlk_polymat)))+  requires(memory_no_alias(seed, MLKEM_SYMBYTES))+  requires(transposed == 0 || transposed == 1)+  assigns(memory_slice(a, sizeof(mlk_polymat)))+  ensures(forall(x, 0, MLKEM_K, forall(y, 0, MLKEM_K,+  array_bound(a->vec[x].vec[y].coeffs, 0, MLKEM_N, 0, MLKEM_Q))))+);++#if !defined(MLK_CONFIG_NO_KEYPAIR_API)+#define mlk_indcpa_keypair_derand \+  MLK_NAMESPACE_K(indcpa_keypair_derand) MLK_CONTEXT_PARAMETERS_3+/**+ * Generate public and private key for the CPA-secure public-key encryption+ * scheme underlying ML-KEM.+ *+ * @spec{Implements @[FIPS203, Algorithm 13 (K-PKE.KeyGen)].}+ *+ * @param[out] pk      Output public key+ *                     (length MLKEM_INDCPA_PUBLICKEYBYTES bytes).+ * @param[out] sk      Output private key+ *                     (length MLKEM_INDCPA_SECRETKEYBYTES bytes).+ * @param[in]  coins   Input randomness (length MLKEM_SYMBYTES bytes).+ * @param      context Application context. Only present when+ *                     MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ */+MLK_INTERNAL_API+MLK_MUST_CHECK_RETURN_VALUE+int mlk_indcpa_keypair_derand(uint8_t pk[MLKEM_INDCPA_PUBLICKEYBYTES],+                              uint8_t sk[MLKEM_INDCPA_SECRETKEYBYTES],+                              const uint8_t coins[MLKEM_SYMBYTES],+                              MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(pk, MLKEM_INDCPA_PUBLICKEYBYTES))+  requires(memory_no_alias(sk, MLKEM_INDCPA_SECRETKEYBYTES))+  requires(memory_no_alias(coins, MLKEM_SYMBYTES))+  assigns(memory_slice(pk, MLKEM_INDCPA_PUBLICKEYBYTES))+  assigns(memory_slice(sk, MLKEM_INDCPA_SECRETKEYBYTES))+  ensures(return_value == 0 || return_value == MLK_ERR_OUT_OF_MEMORY)+  /* Output buffers on error, per API-CONVENTIONS.md */+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(pk, MLKEM_INDCPA_PUBLICKEYBYTES))+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(sk, MLKEM_INDCPA_SECRETKEYBYTES))+);+#endif /* !MLK_CONFIG_NO_KEYPAIR_API */++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_indcpa_enc MLK_NAMESPACE_K(indcpa_enc) MLK_CONTEXT_PARAMETERS_4+/**+ * Encryption function of the CPA-secure public-key encryption scheme+ * underlying ML-KEM.+ *+ * @spec{Implements @[FIPS203, Algorithm 14 (K-PKE.Encrypt)].}+ *+ * @param[out] c       Output ciphertext (length MLKEM_INDCPA_BYTES bytes).+ * @param[in]  m       Input message (length MLKEM_INDCPA_MSGBYTES bytes).+ * @param[in]  pk      Input public key+ *                     (length MLKEM_INDCPA_PUBLICKEYBYTES bytes).+ * @param[in]  coins   Input random coins used as seed (length MLKEM_SYMBYTES+ *                     bytes) to deterministically generate all randomness.+ * @param      context Application context. Only present when+ *                     MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ */+MLK_INTERNAL_API+MLK_MUST_CHECK_RETURN_VALUE+int mlk_indcpa_enc(uint8_t c[MLKEM_INDCPA_BYTES],+                   const uint8_t m[MLKEM_INDCPA_MSGBYTES],+                   const uint8_t pk[MLKEM_INDCPA_PUBLICKEYBYTES],+                   const uint8_t coins[MLKEM_SYMBYTES],+                   MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(c, MLKEM_INDCPA_BYTES))+  requires(memory_no_alias(m, MLKEM_INDCPA_MSGBYTES))+  requires(memory_no_alias(pk, MLKEM_INDCPA_PUBLICKEYBYTES))+  requires(memory_no_alias(coins, MLKEM_SYMBYTES))+  assigns(memory_slice(c, MLKEM_INDCPA_BYTES))+  ensures(return_value == 0 || return_value == MLK_ERR_OUT_OF_MEMORY)+  /* Output buffers on error, per API-CONVENTIONS.md */+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(c, MLKEM_INDCPA_BYTES))+);+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++#if !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_indcpa_dec MLK_NAMESPACE_K(indcpa_dec) MLK_CONTEXT_PARAMETERS_3+/**+ * Decryption function of the CPA-secure public-key encryption scheme+ * underlying ML-KEM.+ *+ * @spec{Implements @[FIPS203, Algorithm 15 (K-PKE.Decrypt)].}+ *+ * @param[out] m       Output decrypted message+ *                     (length MLKEM_INDCPA_MSGBYTES bytes).+ * @param[in]  c       Input ciphertext (length MLKEM_INDCPA_BYTES bytes).+ * @param[in]  sk      Input secret key+ *                     (length MLKEM_INDCPA_SECRETKEYBYTES bytes).+ * @param      context Application context. Only present when+ *                     MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ */+MLK_INTERNAL_API+MLK_MUST_CHECK_RETURN_VALUE+int mlk_indcpa_dec(uint8_t m[MLKEM_INDCPA_MSGBYTES],+                   const uint8_t c[MLKEM_INDCPA_BYTES],+                   const uint8_t sk[MLKEM_INDCPA_SECRETKEYBYTES],+                   MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(c, MLKEM_INDCPA_BYTES))+  requires(memory_no_alias(m, MLKEM_INDCPA_MSGBYTES))+  requires(memory_no_alias(sk, MLKEM_INDCPA_SECRETKEYBYTES))+  assigns(memory_slice(m, MLKEM_INDCPA_MSGBYTES))+  ensures(return_value == 0 || return_value == MLK_ERR_OUT_OF_MEMORY)+  /* Output buffers on error, per API-CONVENTIONS.md */+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(m, MLKEM_INDCPA_MSGBYTES))+);+#endif /* !MLK_CONFIG_NO_DECAPS_API */++#endif /* !MLK_INDCPA_H */
+ cbits/mlkem/src/kem.c view
@@ -0,0 +1,474 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS140_3_IG]+ *   Implementation Guidance for FIPS 140-3 and the Cryptographic Module+ *   Validation Program+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/projects/cryptographic-module-validation-program/fips-140-3-ig-announcements+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ *+ * - [REF]+ *   CRYSTALS-Kyber C reference implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/ref+ */++#include "kem.h"++#include "indcpa.h"+#include "randombytes.h"+#include "symmetric.h"+#include "verify.h"++/* Parameter set namespacing+ * This is to facilitate building multiple instances+ * of mlkem-native (e.g. with varying security levels)+ * within a single compilation unit. */+#define mlk_check_pct MLK_ADD_PARAM_SET(mlk_check_pct) MLK_CONTEXT_PARAMETERS_2+/* End of parameter set namespacing */++#if !defined(MLK_CONFIG_NO_ENCAPS_API)+/* Reference: Not implemented in the reference implementation @[REF]. */+MLK_EXTERNAL_API+MLK_MUST_CHECK_RETURN_VALUE+int mlk_kem_check_pk(const uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES],+                     MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  int ret = 0;+  MLK_ALLOC(p, mlk_polyvec, 1, context);+  MLK_ALLOC(p_reencoded, uint8_t, MLKEM_POLYVECBYTES, context);++  if (p == NULL || p_reencoded == NULL)+  {+    ret = MLK_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  mlk_polyvec_frombytes(p, pk);+  mlk_polyvec_reduce(p);+  mlk_polyvec_tobytes(p_reencoded, p);++  /* We use a constant-time memcmp here to avoid having to+   * declassify the PK before the PCT has succeeded. */+  ret = mlk_ct_memcmp(pk, p_reencoded, MLKEM_POLYVECBYTES) ? MLK_ERR_INVALID_PK+                                                           : 0;++cleanup:+  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  MLK_FREE(p_reencoded, uint8_t, MLKEM_POLYVECBYTES, context);+  MLK_FREE(p, mlk_polyvec, 1, context);+  return ret;+}+#endif /* !MLK_CONFIG_NO_ENCAPS_API */+++#if !defined(MLK_CONFIG_NO_DECAPS_API)+/* Reference: Not implemented in the reference implementation @[REF]. */+MLK_EXTERNAL_API+MLK_MUST_CHECK_RETURN_VALUE+int mlk_kem_check_sk(const uint8_t sk[MLKEM_INDCCA_SECRETKEYBYTES],+                     MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  int ret = 0;+  MLK_ALLOC(test, uint8_t, MLKEM_SYMBYTES, context);++  if (test == NULL)+  {+    ret = MLK_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  /*+   * The parts of `sk` being hashed and compared here are public, so+   * no public information is leaked through the runtime or the return value+   * of this function.+   */++  /* Declassify the public part of the secret key */+  MLK_CT_TESTING_DECLASSIFY(sk + MLKEM_INDCPA_SECRETKEYBYTES,+                            MLKEM_INDCCA_PUBLICKEYBYTES);+  MLK_CT_TESTING_DECLASSIFY(+      sk + MLKEM_INDCCA_SECRETKEYBYTES - 2 * MLKEM_SYMBYTES, MLKEM_SYMBYTES);++  mlk_hash_h(test, sk + MLKEM_INDCPA_SECRETKEYBYTES,+             MLKEM_INDCCA_PUBLICKEYBYTES);+  /* This doesn't have to be a constant-time memcmp, but it's the only place+   * in the library where a normal memcmp would be used otherwise, so for sake+   * of minimizing stdlib dependency, we use our constant-time one anyway. */+  ret = mlk_ct_memcmp(sk + MLKEM_INDCCA_SECRETKEYBYTES - 2 * MLKEM_SYMBYTES,+                      test, MLKEM_SYMBYTES)+            ? MLK_ERR_INVALID_SK+            : 0;++cleanup:+  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  MLK_FREE(test, uint8_t, MLKEM_SYMBYTES, context);+  return ret;+}+#endif /* !MLK_CONFIG_NO_DECAPS_API */++#if !defined(MLK_CONFIG_NO_KEYPAIR_API)+MLK_MUST_CHECK_RETURN_VALUE+static int mlk_check_pct(uint8_t const pk[MLKEM_INDCCA_PUBLICKEYBYTES],+                         uint8_t const sk[MLKEM_INDCCA_SECRETKEYBYTES],+                         MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(pk, MLKEM_INDCCA_PUBLICKEYBYTES))+  requires(memory_no_alias(sk, MLKEM_INDCCA_SECRETKEYBYTES))+  ensures(return_value == 0 || return_value == MLK_ERR_OUT_OF_MEMORY ||+          return_value == MLK_ERR_RNG_FAIL ||+          return_value == MLK_ERR_PCT_FAIL)+);++#if defined(MLK_CONFIG_KEYGEN_PCT)+/* Specification:+ * Partially implements 'Pairwise Consistency Test' @[FIPS140_3_IG, p.87] and+ * @[FIPS203, Section 7.1, Pairwise Consistency]. */++/* Reference: Not implemented in the reference implementation @[REF].+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ * @retval MLK_ERR_RNG_FAIL      Random number generation failed.+ * @retval MLK_ERR_PCT_FAIL      The consistency check failed. */+MLK_MUST_CHECK_RETURN_VALUE+static int mlk_check_pct(uint8_t const pk[MLKEM_INDCCA_PUBLICKEYBYTES],+                         uint8_t const sk[MLKEM_INDCCA_SECRETKEYBYTES],+                         MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  int ret = 0;+  MLK_ALLOC(ct, uint8_t, MLKEM_INDCCA_CIPHERTEXTBYTES, context);+  MLK_ALLOC(ss_enc, uint8_t, MLKEM_SSBYTES, context);+  MLK_ALLOC(ss_dec, uint8_t, MLKEM_SSBYTES, context);++  if (ct == NULL || ss_enc == NULL || ss_dec == NULL)+  {+    ret = MLK_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  ret = mlk_kem_enc(ct, ss_enc, pk, context);+  if (ret != 0)+  {+    goto cleanup;+  }++  ret = mlk_kem_dec(ss_dec, ct, sk, context);+  if (ret != 0)+  {+    goto cleanup;+  }++#if defined(MLK_CONFIG_KEYGEN_PCT_BREAKAGE_TEST)+  /* Deliberately break PCT for testing purposes */+  if (mlk_break_pct())+  {+    ss_enc[0] = ~ss_enc[0];+  }+#endif /* MLK_CONFIG_KEYGEN_PCT_BREAKAGE_TEST */++  ret = mlk_ct_memcmp(ss_enc, ss_dec, MLKEM_SSBYTES);+  /* The result of the PCT is public. */+  MLK_CT_TESTING_DECLASSIFY(&ret, sizeof(ret));++  if (ret != 0)+  {+    ret = MLK_ERR_PCT_FAIL;+  }++cleanup:++  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  MLK_FREE(ss_dec, uint8_t, MLKEM_SSBYTES, context);+  MLK_FREE(ss_enc, uint8_t, MLKEM_SSBYTES, context);+  MLK_FREE(ct, uint8_t, MLKEM_INDCCA_CIPHERTEXTBYTES, context);++  /* The key pair being tested was just generated by this library, so a key+   * check rejecting it hints at a faulty implementation rather than at bad+   * input. Report it as a PCT failure. */+  if (ret == MLK_ERR_INVALID_PK || ret == MLK_ERR_INVALID_SK)+  {+    ret = MLK_ERR_PCT_FAIL;+  }++  /* Other error codes, e.g. platform failures like out of memory or+   * randomness failure, are passed on unmodified. */++  return ret;+}+#else  /* MLK_CONFIG_KEYGEN_PCT */+MLK_MUST_CHECK_RETURN_VALUE+static int mlk_check_pct(uint8_t const pk[MLKEM_INDCCA_PUBLICKEYBYTES],+                         uint8_t const sk[MLKEM_INDCCA_SECRETKEYBYTES],+                         MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  /* Skip PCT */+  ((void)pk);+  ((void)sk);+  MLK_CONTEXT_UNUSED(context);+  return 0;+}+#endif /* !MLK_CONFIG_KEYGEN_PCT */++/* Reference: `crypto_kem_keypair_derand()` in the reference implementation+ *            @[REF].+ *            - We optionally include PCT which is not present in+ *              the reference code. */+MLK_EXTERNAL_API+MLK_MUST_CHECK_RETURN_VALUE+int mlk_kem_keypair_derand(uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES],+                           uint8_t sk[MLKEM_INDCCA_SECRETKEYBYTES],+                           const uint8_t coins[2 * MLKEM_SYMBYTES],+                           MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  int ret;++  ret = mlk_indcpa_keypair_derand(pk, sk, coins, context);+  if (ret != 0)+  {+    goto cleanup;+  }++  mlk_memcpy(sk + MLKEM_INDCPA_SECRETKEYBYTES, pk, MLKEM_INDCCA_PUBLICKEYBYTES);+  mlk_hash_h(sk + MLKEM_INDCCA_SECRETKEYBYTES - 2 * MLKEM_SYMBYTES, pk,+             MLKEM_INDCCA_PUBLICKEYBYTES);+  /* Value z for pseudo-random output on reject */+  mlk_memcpy(sk + MLKEM_INDCCA_SECRETKEYBYTES - MLKEM_SYMBYTES,+             coins + MLKEM_SYMBYTES, MLKEM_SYMBYTES);++  /* Declassify public key */+  MLK_CT_TESTING_DECLASSIFY(pk, MLKEM_INDCCA_PUBLICKEYBYTES);++  /* Pairwise Consistency Test (PCT) @[FIPS140_3_IG, p.87] */+  ret = mlk_check_pct(pk, sk, context);+  if (ret != 0)+  {+    goto cleanup;+  }++cleanup:+  if (ret != 0)+  {+    mlk_zeroize(pk, MLKEM_INDCCA_PUBLICKEYBYTES);+    mlk_zeroize(sk, MLKEM_INDCCA_SECRETKEYBYTES);+  }++  return ret;+}++#if !defined(MLK_CONFIG_NO_RANDOMIZED_API)+/* Reference: `crypto_kem_keypair()` in the reference implementation @[REF]+ *            - We zeroize the stack buffer */+MLK_EXTERNAL_API+int mlk_kem_keypair(uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES],+                    uint8_t sk[MLKEM_INDCCA_SECRETKEYBYTES],+                    MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  int ret = 0;+  MLK_ALLOC(coins, uint8_t, 2 * MLKEM_SYMBYTES, context);++  if (coins == NULL)+  {+    ret = MLK_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  /* Acquire necessary randomness, and mark it as secret. */+  if (mlk_randombytes(coins, 2 * MLKEM_SYMBYTES) != 0)+  {+    ret = MLK_ERR_RNG_FAIL;+    goto cleanup;+  }++  MLK_CT_TESTING_SECRET(coins, 2 * MLKEM_SYMBYTES);++  ret = mlk_kem_keypair_derand(pk, sk, coins, context);++cleanup:+  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  MLK_FREE(coins, uint8_t, 2 * MLKEM_SYMBYTES, context);+  return ret;+}+#endif /* !MLK_CONFIG_NO_RANDOMIZED_API */+#endif /* !MLK_CONFIG_NO_KEYPAIR_API */++#if !defined(MLK_CONFIG_NO_ENCAPS_API)+/* Reference: `crypto_kem_enc_derand()` in the reference implementation @[REF]+ *            - We include public key check+ *            - We include stack buffer zeroization */+MLK_EXTERNAL_API+int mlk_kem_enc_derand(uint8_t ct[MLKEM_INDCCA_CIPHERTEXTBYTES],+                       uint8_t ss[MLKEM_SSBYTES],+                       const uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES],+                       const uint8_t coins[MLKEM_SYMBYTES],+                       MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  int ret = 0;+  MLK_ALLOC(buf, uint8_t, 2 * MLKEM_SYMBYTES, context);+  MLK_ALLOC(kr, uint8_t, 2 * MLKEM_SYMBYTES, context);++  if (buf == NULL || kr == NULL)+  {+    ret = MLK_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  /* Specification: Implements @[FIPS203, Section 7.2, Modulus check] */+  ret = mlk_kem_check_pk(pk, context);+  if (ret != 0)+  {+    goto cleanup;+  }++  mlk_memcpy(buf, coins, MLKEM_SYMBYTES);++  /* Multitarget countermeasure for coins + contributory KEM */+  mlk_hash_h(buf + MLKEM_SYMBYTES, pk, MLKEM_INDCCA_PUBLICKEYBYTES);+  mlk_hash_g(kr, buf, 2 * MLKEM_SYMBYTES);++  /* coins are in kr+MLKEM_SYMBYTES */+  ret = mlk_indcpa_enc(ct, buf, pk, kr + MLKEM_SYMBYTES, context);+  if (ret != 0)+  {+    goto cleanup;+  }++  mlk_memcpy(ss, kr, MLKEM_SYMBYTES);++cleanup:+  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  MLK_FREE(kr, uint8_t, 2 * MLKEM_SYMBYTES, context);+  MLK_FREE(buf, uint8_t, 2 * MLKEM_SYMBYTES, context);+  return ret;+}++#if !defined(MLK_CONFIG_NO_RANDOMIZED_API)+/* Reference: `crypto_kem_enc()` in the reference implementation @[REF]+ *            - We include stack buffer zeroization */+MLK_EXTERNAL_API+int mlk_kem_enc(uint8_t ct[MLKEM_INDCCA_CIPHERTEXTBYTES],+                uint8_t ss[MLKEM_SSBYTES],+                const uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES],+                MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  int ret = 0;+  MLK_ALLOC(coins, uint8_t, MLKEM_SYMBYTES, context);++  if (coins == NULL)+  {+    ret = MLK_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  if (mlk_randombytes(coins, MLKEM_SYMBYTES) != 0)+  {+    ret = MLK_ERR_RNG_FAIL;+    goto cleanup;+  }++  MLK_CT_TESTING_SECRET(coins, MLKEM_SYMBYTES);++  ret = mlk_kem_enc_derand(ct, ss, pk, coins, context);++cleanup:+  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  MLK_FREE(coins, uint8_t, MLKEM_SYMBYTES, context);+  return ret;+}+#endif /* !MLK_CONFIG_NO_RANDOMIZED_API */+#endif /* !MLK_CONFIG_NO_ENCAPS_API */++#if !defined(MLK_CONFIG_NO_DECAPS_API)+/* Reference: `crypto_kem_dec()` in the reference implementation @[REF]+ *            - We include secret key check+ *            - We include stack buffer zeroization */+MLK_EXTERNAL_API+int mlk_kem_dec(uint8_t ss[MLKEM_SSBYTES],+                const uint8_t ct[MLKEM_INDCCA_CIPHERTEXTBYTES],+                const uint8_t sk[MLKEM_INDCCA_SECRETKEYBYTES],+                MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+{+  int ret = 0;+  uint8_t fail;+  const uint8_t *pk = sk + MLKEM_INDCPA_SECRETKEYBYTES;+  MLK_ALLOC(buf, uint8_t, 2 * MLKEM_SYMBYTES, context);+  MLK_ALLOC(kr, uint8_t, 2 * MLKEM_SYMBYTES, context);+  MLK_ALLOC(tmp, uint8_t, MLKEM_SYMBYTES + MLKEM_INDCCA_CIPHERTEXTBYTES,+            context);++  if (buf == NULL || kr == NULL || tmp == NULL)+  {+    ret = MLK_ERR_OUT_OF_MEMORY;+    goto cleanup;+  }++  /* Specification: Implements @[FIPS203, Section 7.3, Hash check] */+  ret = mlk_kem_check_sk(sk, context);+  if (ret != 0)+  {+    goto cleanup;+  }++  ret = mlk_indcpa_dec(buf, ct, sk, context);+  if (ret != 0)+  {+    goto cleanup;+  }++  /* Multitarget countermeasure for coins + contributory KEM */+  mlk_memcpy(buf + MLKEM_SYMBYTES,+             sk + MLKEM_INDCCA_SECRETKEYBYTES - 2 * MLKEM_SYMBYTES,+             MLKEM_SYMBYTES);+  mlk_hash_g(kr, buf, 2 * MLKEM_SYMBYTES);++  /* Recompute and compare ciphertext */+  /* coins are in kr+MLKEM_SYMBYTES */+  ret = mlk_indcpa_enc(tmp, buf, pk, kr + MLKEM_SYMBYTES, context);+  if (ret != 0)+  {+    goto cleanup;+  }++  fail = mlk_ct_memcmp(ct, tmp, MLKEM_INDCCA_CIPHERTEXTBYTES);++  /* Compute rejection key */+  mlk_memcpy(tmp, sk + MLKEM_INDCCA_SECRETKEYBYTES - MLKEM_SYMBYTES,+             MLKEM_SYMBYTES);+  mlk_memcpy(tmp + MLKEM_SYMBYTES, ct, MLKEM_INDCCA_CIPHERTEXTBYTES);+  mlk_hash_j(ss, tmp, MLKEM_SYMBYTES + MLKEM_INDCCA_CIPHERTEXTBYTES);++  /* Copy true key to return buffer if fail is 0 */+  mlk_ct_cmov_zero(ss, kr, MLKEM_SYMBYTES, fail);++cleanup:+  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  MLK_FREE(tmp, uint8_t, MLKEM_SYMBYTES + MLKEM_INDCCA_CIPHERTEXTBYTES,+           context);+  MLK_FREE(kr, uint8_t, 2 * MLKEM_SYMBYTES, context);+  MLK_FREE(buf, uint8_t, 2 * MLKEM_SYMBYTES, context);++  return ret;+}+#endif /* !MLK_CONFIG_NO_DECAPS_API */++/* To facilitate single-compilation-unit (SCU) builds, undefine all macros.+ * Don't modify by hand -- this is auto-generated by scripts/autogen. */+#undef mlk_check_pct
+ cbits/mlkem/src/kem.h view
@@ -0,0 +1,353 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ *+ * - [REF]+ *   CRYSTALS-Kyber C reference implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/ref+ */++#ifndef MLK_KEM_H+#define MLK_KEM_H++#include "cbmc.h"+#include "common.h"+#include "sys.h"++#if defined(MLK_CHECK_APIS)+/* Include to ensure consistency between internal kem.h+ * and external mlkem_native.h. */+#include "mlkem_native.h"++#if MLKEM_INDCCA_SECRETKEYBYTES != \+    MLKEM_SECRETKEYBYTES(MLK_CONFIG_PARAMETER_SET)+#error Mismatch for SECRETKEYBYTES between kem.h and mlkem_native.h+#endif++#if MLKEM_INDCCA_PUBLICKEYBYTES != \+    MLKEM_PUBLICKEYBYTES(MLK_CONFIG_PARAMETER_SET)+#error Mismatch for PUBLICKEYBYTES between kem.h and mlkem_native.h+#endif++#if MLKEM_INDCCA_CIPHERTEXTBYTES != \+    MLKEM_CIPHERTEXTBYTES(MLK_CONFIG_PARAMETER_SET)+#error Mismatch for CIPHERTEXTBYTES between kem.h and mlkem_native.h+#endif++#endif /* MLK_CHECK_APIS */++#if !defined(MLK_CONFIG_NO_KEYPAIR_API)+#define mlk_kem_keypair_derand \+  MLK_NAMESPACE_K(keypair_derand) MLK_CONTEXT_PARAMETERS_3+#if !defined(MLK_CONFIG_NO_RANDOMIZED_API)+#define mlk_kem_keypair MLK_NAMESPACE_K(keypair) MLK_CONTEXT_PARAMETERS_2+#endif+#endif /* !MLK_CONFIG_NO_KEYPAIR_API */+#if !defined(MLK_CONFIG_NO_ENCAPS_API)+#define mlk_kem_enc_derand MLK_NAMESPACE_K(enc_derand) MLK_CONTEXT_PARAMETERS_4+#if !defined(MLK_CONFIG_NO_RANDOMIZED_API)+#define mlk_kem_enc MLK_NAMESPACE_K(enc) MLK_CONTEXT_PARAMETERS_3+#endif+#define mlk_kem_check_pk MLK_NAMESPACE_K(check_pk) MLK_CONTEXT_PARAMETERS_1+#endif /* !MLK_CONFIG_NO_ENCAPS_API */+#if !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_kem_dec MLK_NAMESPACE_K(dec) MLK_CONTEXT_PARAMETERS_3+#define mlk_kem_check_sk MLK_NAMESPACE_K(check_sk) MLK_CONTEXT_PARAMETERS_1+#endif++/**+ * Implements modulus check mandated by FIPS 203, i.e., ensures that+ * coefficients are in [0,q-1].+ *+ * @spec{Implements @[FIPS203, Section 7.2, 'modulus check'].}+ *+ * @reference{Not implemented in the reference implementation @[REF].}+ *+ * @param[in] pk      Input public key (an already allocated array of+ *                    MLKEM_INDCCA_PUBLICKEYBYTES bytes).+ * @param     context Application context. Only present when+ *                    MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                    MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_INVALID_PK    Modulus check failed.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ */+#if !defined(MLK_CONFIG_NO_ENCAPS_API)+MLK_EXTERNAL_API+MLK_MUST_CHECK_RETURN_VALUE+int mlk_kem_check_pk(const uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES],+                     MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(pk, MLKEM_INDCCA_PUBLICKEYBYTES))+  ensures(return_value == 0 || return_value == MLK_ERR_INVALID_PK ||+          return_value == MLK_ERR_OUT_OF_MEMORY)+);+#endif /* !MLK_CONFIG_NO_ENCAPS_API */+++/**+ * Implements public key hash check mandated by FIPS 203, i.e., ensures that+ * sk[768𝑘+32 ∶ 768𝑘+64] = H(pk) = H(sk[384𝑘 : 768𝑘+32]).+ *+ * @spec{Implements @[FIPS203, Section 7.3, 'hash check'].}+ *+ * @reference{Not implemented in the reference implementation @[REF].}+ *+ * @param[in] sk      Input private key (an already allocated array of+ *                    MLKEM_INDCCA_SECRETKEYBYTES bytes).+ * @param     context Application context. Only present when+ *                    MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                    MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_INVALID_SK    Public key hash check failed.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ */+#if !defined(MLK_CONFIG_NO_DECAPS_API)+MLK_EXTERNAL_API+MLK_MUST_CHECK_RETURN_VALUE+int mlk_kem_check_sk(const uint8_t sk[MLKEM_INDCCA_SECRETKEYBYTES],+                     MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(sk, MLKEM_INDCCA_SECRETKEYBYTES))+  ensures(return_value == 0 || return_value == MLK_ERR_INVALID_SK ||+          return_value == MLK_ERR_OUT_OF_MEMORY)+);+#endif /* !MLK_CONFIG_NO_DECAPS_API */++#if !defined(MLK_CONFIG_NO_KEYPAIR_API)+/**+ * Generate a public/private keypair for the ML-KEM key encapsulation mechanism.+ *+ * @spec{Implements @[FIPS203, Algorithm 16, ML-KEM.KeyGen_Internal].}+ *+ * @param[out] pk      Output public key (an already allocated array of+ *                     MLKEM_INDCCA_PUBLICKEYBYTES bytes).+ * @param[out] sk      Output private key (an already allocated array of+ *                     MLKEM_INDCCA_SECRETKEYBYTES bytes).+ * @param[in]  coins   Input randomness (an already allocated array filled+ *                     with 2*MLKEM_SYMBYTES random bytes).+ * @param      context Application context. Only present when+ *                     MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_PCT_FAIL      MLK_CONFIG_KEYGEN_PCT enabled and PCT failed.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ * @retval MLK_ERR_RNG_FAIL      MLK_CONFIG_KEYGEN_PCT enabled and random+ *                               number generation failed within the PCT.+ */+MLK_EXTERNAL_API+MLK_MUST_CHECK_RETURN_VALUE+int mlk_kem_keypair_derand(uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES],+                           uint8_t sk[MLKEM_INDCCA_SECRETKEYBYTES],+                           const uint8_t coins[2 * MLKEM_SYMBYTES],+                           MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(pk, MLKEM_INDCCA_PUBLICKEYBYTES))+  requires(memory_no_alias(sk, MLKEM_INDCCA_SECRETKEYBYTES))+  requires(memory_no_alias(coins, 2 * MLKEM_SYMBYTES))+  assigns(memory_slice(pk, MLKEM_INDCCA_PUBLICKEYBYTES))+  assigns(memory_slice(sk, MLKEM_INDCCA_SECRETKEYBYTES))+  ensures(return_value == 0 || return_value == MLK_ERR_PCT_FAIL ||+          return_value == MLK_ERR_OUT_OF_MEMORY ||+          return_value == MLK_ERR_RNG_FAIL)+  /* Output buffers on error, per API-CONVENTIONS.md */+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(pk, MLKEM_INDCCA_PUBLICKEYBYTES))+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(sk, MLKEM_INDCCA_SECRETKEYBYTES))+);++#if !defined(MLK_CONFIG_NO_RANDOMIZED_API)+/**+ * Generate a public/private keypair for the ML-KEM key encapsulation mechanism.+ *+ * @spec{Implements @[FIPS203, Algorithm 19, ML-KEM.KeyGen].}+ *+ * @param[out] pk      Output public key (an already allocated array of+ *                     MLKEM_INDCCA_PUBLICKEYBYTES bytes).+ * @param[out] sk      Output private key (an already allocated array of+ *                     MLKEM_INDCCA_SECRETKEYBYTES bytes).+ * @param      context Application context. Only present when+ *                     MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ * @retval MLK_ERR_RNG_FAIL      Random number generation failed.+ * @retval MLK_ERR_PCT_FAIL      MLK_CONFIG_KEYGEN_PCT enabled and PCT failed.+ */+MLK_EXTERNAL_API+MLK_MUST_CHECK_RETURN_VALUE+int mlk_kem_keypair(uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES],+                    uint8_t sk[MLKEM_INDCCA_SECRETKEYBYTES],+                    MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(pk, MLKEM_INDCCA_PUBLICKEYBYTES))+  requires(memory_no_alias(sk, MLKEM_INDCCA_SECRETKEYBYTES))+  assigns(memory_slice(pk, MLKEM_INDCCA_PUBLICKEYBYTES))+  assigns(memory_slice(sk, MLKEM_INDCCA_SECRETKEYBYTES))+  ensures(return_value == 0 || return_value == MLK_ERR_PCT_FAIL ||+          return_value == MLK_ERR_OUT_OF_MEMORY ||+          return_value == MLK_ERR_RNG_FAIL)+  /* Output buffers on error, per API-CONVENTIONS.md */+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(pk, MLKEM_INDCCA_PUBLICKEYBYTES))+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(sk, MLKEM_INDCCA_SECRETKEYBYTES))+);+#endif /* !MLK_CONFIG_NO_RANDOMIZED_API */+#endif /* !MLK_CONFIG_NO_KEYPAIR_API */++/**+ * Generate ciphertext and shared secret for a given public key.+ *+ * @spec{Implements @[FIPS203, Algorithm 17, ML-KEM.Encaps_Internal].}+ *+ * @param[out] ct      Output ciphertext (an already allocated array of+ *                     MLKEM_INDCCA_CIPHERTEXTBYTES bytes).+ * @param[out] ss      Output shared secret (an already allocated array of+ *                     MLKEM_SSBYTES bytes).+ * @param[in]  pk      Input public key (an already allocated array of+ *                     MLKEM_INDCCA_PUBLICKEYBYTES bytes).+ * @param[in]  coins   Input randomness (an already allocated array filled+ *                     with MLKEM_SYMBYTES random bytes).+ * @param      context Application context. Only present when+ *                     MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_INVALID_PK    The 'modulus check' @[FIPS203, Section 7.2]+ *                               for the public key failed.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ */+#if !defined(MLK_CONFIG_NO_ENCAPS_API)+MLK_EXTERNAL_API+MLK_MUST_CHECK_RETURN_VALUE+int mlk_kem_enc_derand(uint8_t ct[MLKEM_INDCCA_CIPHERTEXTBYTES],+                       uint8_t ss[MLKEM_SSBYTES],+                       const uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES],+                       const uint8_t coins[MLKEM_SYMBYTES],+                       MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(ct, MLKEM_INDCCA_CIPHERTEXTBYTES))+  requires(memory_no_alias(ss, MLKEM_SSBYTES))+  requires(memory_no_alias(pk, MLKEM_INDCCA_PUBLICKEYBYTES))+  requires(memory_no_alias(coins, MLKEM_SYMBYTES))+  assigns(memory_slice(ct, MLKEM_INDCCA_CIPHERTEXTBYTES))+  assigns(memory_slice(ss, MLKEM_SSBYTES))+  ensures(return_value == 0 || return_value == MLK_ERR_INVALID_PK ||+          return_value == MLK_ERR_OUT_OF_MEMORY)+  /* Output buffers on error, per API-CONVENTIONS.md */+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(ct, MLKEM_INDCCA_CIPHERTEXTBYTES))+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(ss, MLKEM_SSBYTES))+);++#if !defined(MLK_CONFIG_NO_RANDOMIZED_API)+/**+ * Generate ciphertext and shared secret for a given public key.+ *+ * @spec{Implements @[FIPS203, Algorithm 20, ML-KEM.Encaps].}+ *+ * @param[out] ct      Output ciphertext (an already allocated array of+ *                     MLKEM_INDCCA_CIPHERTEXTBYTES bytes).+ * @param[out] ss      Output shared secret (an already allocated array of+ *                     MLKEM_SSBYTES bytes).+ * @param[in]  pk      Input public key (an already allocated array of+ *                     MLKEM_INDCCA_PUBLICKEYBYTES bytes).+ * @param      context Application context. Only present when+ *                     MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ * @retval MLK_ERR_RNG_FAIL      Random number generation failed.+ * @retval MLK_ERR_INVALID_PK    The 'modulus check' @[FIPS203, Section 7.2]+ *                               for the public key failed.+ */+MLK_EXTERNAL_API+MLK_MUST_CHECK_RETURN_VALUE+int mlk_kem_enc(uint8_t ct[MLKEM_INDCCA_CIPHERTEXTBYTES],+                uint8_t ss[MLKEM_SSBYTES],+                const uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES],+                MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(ct, MLKEM_INDCCA_CIPHERTEXTBYTES))+  requires(memory_no_alias(ss, MLKEM_SSBYTES))+  requires(memory_no_alias(pk, MLKEM_INDCCA_PUBLICKEYBYTES))+  assigns(memory_slice(ct, MLKEM_INDCCA_CIPHERTEXTBYTES))+  assigns(memory_slice(ss, MLKEM_SSBYTES))+  ensures(return_value == 0 || return_value == MLK_ERR_INVALID_PK ||+          return_value == MLK_ERR_OUT_OF_MEMORY ||+          return_value == MLK_ERR_RNG_FAIL)+  /* Output buffers on error, per API-CONVENTIONS.md */+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(ct, MLKEM_INDCCA_CIPHERTEXTBYTES))+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(ss, MLKEM_SSBYTES))+);+#endif /* !MLK_CONFIG_NO_RANDOMIZED_API */+#endif /* !MLK_CONFIG_NO_ENCAPS_API */++/**+ * Generate shared secret for a given ciphertext and private key.+ *+ * @spec{Implements @[FIPS203, Algorithm 21, ML-KEM.Decaps].}+ *+ * @param[out] ss      Output shared secret (an already allocated array of+ *                     MLKEM_SSBYTES bytes).+ * @param[in]  ct      Input ciphertext (an already allocated array of+ *                     MLKEM_INDCCA_CIPHERTEXTBYTES bytes).+ * @param[in]  sk      Input private key (an already allocated array of+ *                     MLKEM_INDCCA_SECRETKEYBYTES bytes).+ * @param      context Application context. Only present when+ *                     MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by+ *                     MLK_CONFIG_CONTEXT_PARAMETER_TYPE.+ *+ * @retval 0                     Success.+ * @retval MLK_ERR_INVALID_SK    The 'hash check' @[FIPS203, Section 7.3]+ *                               for the secret key failed.+ * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and+ *                               MLK_CUSTOM_ALLOC returned NULL.+ */+#if !defined(MLK_CONFIG_NO_DECAPS_API)+MLK_EXTERNAL_API+MLK_MUST_CHECK_RETURN_VALUE+int mlk_kem_dec(uint8_t ss[MLKEM_SSBYTES],+                const uint8_t ct[MLKEM_INDCCA_CIPHERTEXTBYTES],+                const uint8_t sk[MLKEM_INDCCA_SECRETKEYBYTES],+                MLK_CONFIG_CONTEXT_PARAMETER_TYPE context)+__contract__(+  requires(memory_no_alias(ss, MLKEM_SSBYTES))+  requires(memory_no_alias(ct, MLKEM_INDCCA_CIPHERTEXTBYTES))+  requires(memory_no_alias(sk, MLKEM_INDCCA_SECRETKEYBYTES))+  assigns(memory_slice(ss, MLKEM_SSBYTES))+  ensures(return_value == 0 || return_value == MLK_ERR_INVALID_SK ||+          return_value == MLK_ERR_OUT_OF_MEMORY)+  /* Output buffers on error, per API-CONVENTIONS.md */+  ensures(return_value != 0 ==>+          array_unchanged_or_zeroized_u8(ss, MLKEM_SSBYTES))+);+#endif /* !MLK_CONFIG_NO_DECAPS_API */++#endif /* !MLK_KEM_H */
+ cbits/mlkem/src/native/aarch64/README.md view
@@ -0,0 +1,16 @@+[//]: # (SPDX-License-Identifier: CC-BY-4.0)++# AArch64 backend (little endian)++This directory contains a native backend for little endian AArch64 systems. It is derived from [^NeonNTT] [^SLOTHY_Paper].++The code in this directory is auto-generated from the 'clean' assembly in [dev/aarch64_clean](../../../../dev/aarch64_clean)+in a two-step fashion: First, it is superoptimized using the [SLOTHY](https://github.com/slothy-optimizer/slothy) superoptimizer,+giving the assembly in [dev/aarch64_opt](../../../../dev/aarch64_opt). Then, it is stripped of remaining register aliases, macros+and most preprocessor directives by [`scripts/simpasm`](../../../../scripts/simpasm).++If you want to understand how the assembly works, and/or make changes to it, consult [dev/](../../../../dev).++<!--- bibliography --->+[^NeonNTT]: Becker, Hwang, Kannwischer, Yang, Yang: Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1, [https://eprint.iacr.org/2021/986](https://eprint.iacr.org/2021/986)+[^SLOTHY_Paper]: Abdulrahman, Becker, Kannwischer, Klein: Fast and Clean: Auditable high-performance assembly via constraint solving, [https://eprint.iacr.org/2022/1303](https://eprint.iacr.org/2022/1303)
+ cbits/mlkem/src/native/aarch64/meta.h view
@@ -0,0 +1,166 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLK_NATIVE_AARCH64_META_H+#define MLK_NATIVE_AARCH64_META_H++/* Set of primitives that this backend replaces */+#define MLK_USE_NATIVE_NTT+#define MLK_USE_NATIVE_INTT+#define MLK_USE_NATIVE_POLY_REDUCE+#define MLK_USE_NATIVE_POLY_TOMONT+#define MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE+#define MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED+#define MLK_USE_NATIVE_POLY_TOBYTES+#define MLK_USE_NATIVE_REJ_UNIFORM++/* Identifier for this backend so that source and assembly files+ * in the build can be appropriately guarded. */+#define MLK_ARITH_BACKEND_AARCH64+++#if !defined(__ASSEMBLER__)+#include "../api.h"+#include "src/arith_native_aarch64.h"++MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_ntt_native(int16_t data[MLKEM_N])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }+  mlk_ntt_aarch64_asm(data, mlk_aarch64_ntt_zetas_layer12345,+                      mlk_aarch64_ntt_zetas_layer67);+  return MLK_NATIVE_FUNC_SUCCESS;+}++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_intt_native(int16_t data[MLKEM_N])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }+  mlk_intt_aarch64_asm(data, mlk_aarch64_invntt_zetas_layer12345,+                       mlk_aarch64_invntt_zetas_layer67);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_reduce_native(int16_t data[MLKEM_N])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }+  mlk_poly_reduce_aarch64_asm(data);+  return MLK_NATIVE_FUNC_SUCCESS;+}++#if !defined(MLK_CONFIG_NO_KEYPAIR_API)+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_tomont_native(int16_t data[MLKEM_N])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }+  mlk_poly_tomont_aarch64_asm(data);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* !MLK_CONFIG_NO_KEYPAIR_API */++MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_mulcache_compute_native(int16_t x[MLKEM_N / 2],+                                                       const int16_t y[MLKEM_N])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }+  mlk_poly_mulcache_compute_aarch64_asm(+      x, y, mlk_aarch64_zetas_mulcache_native,+      mlk_aarch64_zetas_mulcache_twisted_native);+  return MLK_NATIVE_FUNC_SUCCESS;+}++#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k2_native(+    int16_t r[MLKEM_N], const int16_t a[2 * MLKEM_N],+    const int16_t b[2 * MLKEM_N], const int16_t b_cache[2 * (MLKEM_N / 2)])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }+  mlk_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm(r, a, b, b_cache);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 */++#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 3+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k3_native(+    int16_t r[MLKEM_N], const int16_t a[3 * MLKEM_N],+    const int16_t b[3 * MLKEM_N], const int16_t b_cache[3 * (MLKEM_N / 2)])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }+  mlk_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm(r, a, b, b_cache);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 3 */++#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k4_native(+    int16_t r[MLKEM_N], const int16_t a[4 * MLKEM_N],+    const int16_t b[4 * MLKEM_N], const int16_t b_cache[4 * (MLKEM_N / 2)])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }+  mlk_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm(r, a, b, b_cache);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4 */++#if !defined(MLK_CONFIG_NO_KEYPAIR_API) || !defined(MLK_CONFIG_NO_ENCAPS_API)+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_tobytes_native(uint8_t r[MLKEM_POLYBYTES],+                                              const int16_t a[MLKEM_N])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }+  mlk_poly_tobytes_aarch64_asm(r, a);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* !MLK_CONFIG_NO_KEYPAIR_API || !MLK_CONFIG_NO_ENCAPS_API */++MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_rej_uniform_native(int16_t *r, unsigned len,+                                             const uint8_t *buf,+                                             unsigned buflen)+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON) || len != MLKEM_N ||+      buflen % 24 != 0)+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }+  return (int)mlk_rej_uniform_aarch64_asm(r, buf, buflen,+                                          mlk_rej_uniform_table);+}+#endif /* !__ASSEMBLER__ */++#endif /* !MLK_NATIVE_AARCH64_META_H */
+ cbits/mlkem/src/native/aarch64/src/aarch64_zetas.c view
@@ -0,0 +1,184 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mlkem-native repository.+ *          Do not modify it directly.+ */++#include "../../../common.h"++#if defined(MLK_ARITH_BACKEND_AARCH64) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++#include "arith_native_aarch64.h"++/*+ * Table of zeta values used in the AArch64 forward NTT+ * See autogen for details.+ */+MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const int16_t+    mlk_aarch64_ntt_zetas_layer12345[80] = {+        -1600, -15749, -749,  -7373,  -40,   -394,   -687, -6762, 630,  6201,+        -1432, -14095, 848,   8347,   0,     0,      1062, 10453, 296,  2914,+        -882,  -8682,  0,     0,      -1410, -13879, 1339, 13180, 1476, 14529,+        0,     0,      193,   1900,   -283,  -2786,  56,   551,   0,    0,+        797,   7845,   -1089, -10719, 1333,  13121,  0,    0,     -543, -5345,+        1426,  14036,  -1235, -12156, 0,     0,      -69,  -679,  535,  5266,+        -447,  -4400,  0,     0,      569,   5601,   -936, -9213, -450, -4429,+        0,     0,      -1583, -15582, -1355, -13338, 821,  8081,  0,    0,+};++MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const int16_t+    mlk_aarch64_ntt_zetas_layer67[384] = {+        289,    289,    331,    331,    -76,    -76,    -1573,  -1573,  2845,+        2845,   3258,   3258,   -748,   -748,   -15483, -15483, 17,     17,+        583,    583,    1637,   1637,   -1041,  -1041,  167,    167,    5739,+        5739,   16113,  16113,  -10247, -10247, -568,   -568,   -680,   -680,+        723,    723,    1100,   1100,   -5591,  -5591,  -6693,  -6693,  7117,+        7117,   10828,  10828,  1197,   1197,   -1025,  -1025,  -1052,  -1052,+        -1274,  -1274,  11782,  11782,  -10089, -10089, -10355, -10355, -12540,+        -12540, 1409,   1409,   -48,    -48,    756,    756,    -314,   -314,+        13869,  13869,  -472,   -472,   7441,   7441,   -3091,  -3091,  -667,+        -667,   233,    233,    -1173,  -1173,  -279,   -279,   -6565,  -6565,+        2293,   2293,   -11546, -11546, -2746,  -2746,  650,    650,    -1352,+        -1352,  -816,   -816,   632,    632,    6398,   6398,   -13308, -13308,+        -8032,  -8032,  6221,   6221,   -1626,  -1626,  -540,   -540,   -1482,+        -1482,  1461,   1461,   -16005, -16005, -5315,  -5315,  -14588, -14588,+        14381,  14381,  1651,   1651,   -1540,  -1540,  952,    952,    -642,+        -642,   16251,  16251,  -15159, -15159, 9371,   9371,   -6319,  -6319,+        -464,   -464,   33,     33,     1320,   1320,   -1414,  -1414,  -4567,+        -4567,  325,    325,    12993,  12993,  -13918, -13918, 939,    939,+        -892,   -892,   733,    733,    268,    268,    9243,   9243,   -8780,+        -8780,  7215,   7215,   2638,   2638,   -1021,  -1021,  -941,   -941,+        -992,   -992,   641,    641,    -10050, -10050, -9262,  -9262,  -9764,+        -9764,  6309,   6309,   -1010,  -1010,  1435,   1435,   807,    807,+        452,    452,    -9942,  -9942,  14125,  14125,  7943,   7943,   4449,+        4449,   1584,   1584,   -1292,  -1292,  375,    375,    -1239,  -1239,+        15592,  15592,  -12717, -12717, 3691,   3691,   -12196, -12196, -1031,+        -1031,  -109,   -109,   -780,   -780,   1645,   1645,   -10148, -10148,+        -1073,  -1073,  -7678,  -7678,  16192,  16192,  1438,   1438,   -461,+        -461,   1534,   1534,   -927,   -927,   14155,  14155,  -4538,  -4538,+        15099,  15099,  -9125,  -9125,  1063,   1063,   -556,   -556,   -1230,+        -1230,  -863,   -863,   10463,  10463,  -5473,  -5473,  -12107, -12107,+        -8495,  -8495,  319,    319,    757,    757,    561,    561,    -735,+        -735,   3140,   3140,   7451,   7451,   5522,   5522,   -7235,  -7235,+        -682,   -682,   -712,   -712,   1481,   1481,   648,    648,    -6713,+        -6713,  -7008,  -7008,  14578,  14578,  6378,   6378,   -525,   -525,+        403,    403,    1143,   1143,   -554,   -554,   -5168,  -5168,  3967,+        3967,   11251,  11251,  -5453,  -5453,  1092,   1092,   1026,   1026,+        -1179,  -1179,  886,    886,    10749,  10749,  10099,  10099,  -11605,+        -11605, 8721,   8721,   -855,   -855,   -219,   -219,   1227,   1227,+        910,    910,    -8416,  -8416,  -2156,  -2156,  12078,  12078,  8957,+        8957,   -1607,  -1607,  -1455,  -1455,  -1219,  -1219,  885,    885,+        -15818, -15818, -14322, -14322, -11999, -11999, 8711,   8711,   1212,+        1212,   1029,   1029,   -394,   -394,   -1175,  -1175,  11930,  11930,+        10129,  10129,  -3878,  -3878,  -11566, -11566,+};++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const int16_t+    mlk_aarch64_invntt_zetas_layer12345[80] = {+        1583,  15582, -821,   -8081,  1355,  13338, 0,      0,     -569,+        -5601, 450,   4429,   936,    9213,  0,     0,      69,    679,+        447,   4400,  -535,   -5266,  0,     0,     543,    5345,  1235,+        12156, -1426, -14036, 0,      0,     -797,  -7845,  -1333, -13121,+        1089,  10719, 0,      0,      -193,  -1900, -56,    -551,  283,+        2786,  0,     0,      1410,   13879, -1476, -14529, -1339, -13180,+        0,     0,     -1062,  -10453, 882,   8682,  -296,   -2914, 0,+        0,     1600,  15749,  40,     394,   749,   7373,   -848,  -8347,+        1432,  14095, -630,   -6201,  687,   6762,  0,      0,+};++MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const int16_t+    mlk_aarch64_invntt_zetas_layer67[384] = {+        -910,   -910,   -1227,  -1227,  219,    219,    855,    855,    -8957,+        -8957,  -12078, -12078, 2156,   2156,   8416,   8416,   1175,   1175,+        394,    394,    -1029,  -1029,  -1212,  -1212,  11566,  11566,  3878,+        3878,   -10129, -10129, -11930, -11930, -885,   -885,   1219,   1219,+        1455,   1455,   1607,   1607,   -8711,  -8711,  11999,  11999,  14322,+        14322,  15818,  15818,  -648,   -648,   -1481,  -1481,  712,    712,+        682,    682,    -6378,  -6378,  -14578, -14578, 7008,   7008,   6713,+        6713,   -886,   -886,   1179,   1179,   -1026,  -1026,  -1092,  -1092,+        -8721,  -8721,  11605,  11605,  -10099, -10099, -10749, -10749, 554,+        554,    -1143,  -1143,  -403,   -403,   525,    525,    5453,   5453,+        -11251, -11251, -3967,  -3967,  5168,   5168,   927,    927,    -1534,+        -1534,  461,    461,    -1438,  -1438,  9125,   9125,   -15099, -15099,+        4538,   4538,   -14155, -14155, 735,    735,    -561,   -561,   -757,+        -757,   -319,   -319,   7235,   7235,   -5522,  -5522,  -7451,  -7451,+        -3140,  -3140,  863,    863,    1230,   1230,   556,    556,    -1063,+        -1063,  8495,   8495,   12107,  12107,  5473,   5473,   -10463, -10463,+        -452,   -452,   -807,   -807,   -1435,  -1435,  1010,   1010,   -4449,+        -4449,  -7943,  -7943,  -14125, -14125, 9942,   9942,   -1645,  -1645,+        780,    780,    109,    109,    1031,   1031,   -16192, -16192, 7678,+        7678,   1073,   1073,   10148,  10148,  1239,   1239,   -375,   -375,+        1292,   1292,   -1584,  -1584,  12196,  12196,  -3691,  -3691,  12717,+        12717,  -15592, -15592, 1414,   1414,   -1320,  -1320,  -33,    -33,+        464,    464,    13918,  13918,  -12993, -12993, -325,   -325,   4567,+        4567,   -641,   -641,   992,    992,    941,    941,    1021,   1021,+        -6309,  -6309,  9764,   9764,   9262,   9262,   10050,  10050,  -268,+        -268,   -733,   -733,   892,    892,    -939,   -939,   -2638,  -2638,+        -7215,  -7215,  8780,   8780,   -9243,  -9243,  -632,   -632,   816,+        816,    1352,   1352,   -650,   -650,   -6221,  -6221,  8032,   8032,+        13308,  13308,  -6398,  -6398,  642,    642,    -952,   -952,   1540,+        1540,   -1651,  -1651,  6319,   6319,   -9371,  -9371,  15159,  15159,+        -16251, -16251, -1461,  -1461,  1482,   1482,   540,    540,    1626,+        1626,   -14381, -14381, 14588,  14588,  5315,   5315,   16005,  16005,+        1274,   1274,   1052,   1052,   1025,   1025,   -1197,  -1197,  12540,+        12540,  10355,  10355,  10089,  10089,  -11782, -11782, 279,    279,+        1173,   1173,   -233,   -233,   667,    667,    2746,   2746,   11546,+        11546,  -2293,  -2293,  6565,   6565,   314,    314,    -756,   -756,+        48,     48,     -1409,  -1409,  3091,   3091,   -7441,  -7441,  472,+        472,    -13869, -13869, 1573,   1573,   76,     76,     -331,   -331,+        -289,   -289,   15483,  15483,  748,    748,    -3258,  -3258,  -2845,+        -2845,  -1100,  -1100,  -723,   -723,   680,    680,    568,    568,+        -10828, -10828, -7117,  -7117,  6693,   6693,   5591,   5591,   1041,+        1041,   -1637,  -1637,  -583,   -583,   -17,    -17,    10247,  10247,+        -16113, -16113, -5739,  -5739,  -167,   -167,+};+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const int16_t+    mlk_aarch64_zetas_mulcache_native[128] = {+        17,    -17,   -568,  568,   583,  -583,  -680,  680,   1637,  -1637,+        723,   -723,  -1041, 1041,  1100, -1100, 1409,  -1409, -667,  667,+        -48,   48,    233,   -233,  756,  -756,  -1173, 1173,  -314,  314,+        -279,  279,   -1626, 1626,  1651, -1651, -540,  540,   -1540, 1540,+        -1482, 1482,  952,   -952,  1461, -1461, -642,  642,   939,   -939,+        -1021, 1021,  -892,  892,   -941, 941,   733,   -733,  -992,  992,+        268,   -268,  641,   -641,  1584, -1584, -1031, 1031,  -1292, 1292,+        -109,  109,   375,   -375,  -780, 780,   -1239, 1239,  1645,  -1645,+        1063,  -1063, 319,   -319,  -556, 556,   757,   -757,  -1230, 1230,+        561,   -561,  -863,  863,   -735, 735,   -525,  525,   1092,  -1092,+        403,   -403,  1026,  -1026, 1143, -1143, -1179, 1179,  -554,  554,+        886,   -886,  -1607, 1607,  1212, -1212, -1455, 1455,  1029,  -1029,+        -1219, 1219,  -394,  394,   885,  -885,  -1175, 1175,+};++MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const int16_t+    mlk_aarch64_zetas_mulcache_twisted_native[128] = {+        167,    -167,  -5591,  5591,   5739,   -5739,  -6693,  6693,   16113,+        -16113, 7117,  -7117,  -10247, 10247,  10828,  -10828, 13869,  -13869,+        -6565,  6565,  -472,   472,    2293,   -2293,  7441,   -7441,  -11546,+        11546,  -3091, 3091,   -2746,  2746,   -16005, 16005,  16251,  -16251,+        -5315,  5315,  -15159, 15159,  -14588, 14588,  9371,   -9371,  14381,+        -14381, -6319, 6319,   9243,   -9243,  -10050, 10050,  -8780,  8780,+        -9262,  9262,  7215,   -7215,  -9764,  9764,   2638,   -2638,  6309,+        -6309,  15592, -15592, -10148, 10148,  -12717, 12717,  -1073,  1073,+        3691,   -3691, -7678,  7678,   -12196, 12196,  16192,  -16192, 10463,+        -10463, 3140,  -3140,  -5473,  5473,   7451,   -7451,  -12107, 12107,+        5522,   -5522, -8495,  8495,   -7235,  7235,   -5168,  5168,   10749,+        -10749, 3967,  -3967,  10099,  -10099, 11251,  -11251, -11605, 11605,+        -5453,  5453,  8721,   -8721,  -15818, 15818,  11930,  -11930, -14322,+        14322,  10129, -10129, -11999, 11999,  -3878,  3878,   8711,   -8711,+        -11566, 11566,+};++#else /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED */++MLK_EMPTY_CU(aarch64_zetas)++#endif /* !(MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED) */
+ cbits/mlkem/src/native/aarch64/src/arith_native_aarch64.h view
@@ -0,0 +1,184 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLK_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H+#define MLK_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H++#include "../../../cbmc.h"+#include "../../../common.h"++#define mlk_aarch64_ntt_zetas_layer12345 \+  MLK_NAMESPACE(aarch64_ntt_zetas_layer12345)+#define mlk_aarch64_ntt_zetas_layer67 MLK_NAMESPACE(aarch64_ntt_zetas_layer67)+#define mlk_aarch64_invntt_zetas_layer12345 \+  MLK_NAMESPACE(aarch64_invntt_zetas_layer12345)+#define mlk_aarch64_invntt_zetas_layer67 \+  MLK_NAMESPACE(aarch64_invntt_zetas_layer67)+#define mlk_aarch64_zetas_mulcache_native \+  MLK_NAMESPACE(aarch64_zetas_mulcache_native)+#define mlk_aarch64_zetas_mulcache_twisted_native \+  MLK_NAMESPACE(aarch64_zetas_mulcache_twisted_native)+#define mlk_rej_uniform_table MLK_NAMESPACE(rej_uniform_table)++MLK_INTERNAL_DATA_DECLARATION const int16_t+    mlk_aarch64_ntt_zetas_layer12345[80];+MLK_INTERNAL_DATA_DECLARATION const int16_t mlk_aarch64_ntt_zetas_layer67[384];+MLK_INTERNAL_DATA_DECLARATION const int16_t+    mlk_aarch64_invntt_zetas_layer12345[80];+MLK_INTERNAL_DATA_DECLARATION const int16_t+    mlk_aarch64_invntt_zetas_layer67[384];+MLK_INTERNAL_DATA_DECLARATION const int16_t+    mlk_aarch64_zetas_mulcache_native[128];+MLK_INTERNAL_DATA_DECLARATION const int16_t+    mlk_aarch64_zetas_mulcache_twisted_native[128];+MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_rej_uniform_table[4096];++#define mlk_ntt_aarch64_asm MLK_NAMESPACE(ntt_aarch64_asm)+void mlk_ntt_aarch64_asm(int16_t p[256], const int16_t twiddles12345[80],+                         const int16_t twiddles56[384])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mlkem_ntt_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N))+  requires(array_abs_bound(p, 0, MLKEM_N, 8192))+  requires(twiddles12345 == mlk_aarch64_ntt_zetas_layer12345)+  requires(twiddles56 == mlk_aarch64_ntt_zetas_layer67)+  assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N))+  /* check-magic: off */+  ensures(array_abs_bound(p, 0, MLKEM_N, 23595))+  /* check-magic: on */+);++#define mlk_intt_aarch64_asm MLK_NAMESPACE(intt_aarch64_asm)+void mlk_intt_aarch64_asm(int16_t p[256], const int16_t twiddles12345[80],+                          const int16_t twiddles56[384])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mlkem_intt_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N))+  requires(twiddles12345 == mlk_aarch64_invntt_zetas_layer12345)+  requires(twiddles56 == mlk_aarch64_invntt_zetas_layer67)+  assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N))+  /* check-magic: off */+  ensures(array_abs_bound(p, 0, MLKEM_N, 26625))+  /* check-magic: on */+);++#define mlk_poly_reduce_aarch64_asm MLK_NAMESPACE(poly_reduce_aarch64_asm)+void mlk_poly_reduce_aarch64_asm(int16_t p[256])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mlkem_poly_reduce_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N))+  assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N))+  ensures(array_bound(p, 0, MLKEM_N, 0, MLKEM_Q))+);++#define mlk_poly_tomont_aarch64_asm MLK_NAMESPACE(poly_tomont_aarch64_asm)+void mlk_poly_tomont_aarch64_asm(int16_t p[256])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mlkem_poly_tomont_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N))+  assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N))+  ensures(array_abs_bound(p, 0, MLKEM_N, MLKEM_Q))+);++#define mlk_poly_mulcache_compute_aarch64_asm \+  MLK_NAMESPACE(poly_mulcache_compute_aarch64_asm)+void mlk_poly_mulcache_compute_aarch64_asm(int16_t cache[128],+                                           const int16_t mlk_poly[256],+                                           const int16_t zetas[128],+                                           const int16_t zetas_twisted[128])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mlkem_poly_mulcache_compute_aarch64_asm.ml+ */+__contract__(+  requires(memory_no_alias(cache, sizeof(int16_t) * (MLKEM_N / 2)))+  requires(memory_no_alias(mlk_poly, sizeof(int16_t) * MLKEM_N))+  requires(zetas == mlk_aarch64_zetas_mulcache_native)+  requires(zetas_twisted == mlk_aarch64_zetas_mulcache_twisted_native)+  assigns(memory_slice(cache, sizeof(int16_t) * (MLKEM_N / 2)))+  ensures(array_abs_bound(cache, 0, MLKEM_N/2, MLKEM_Q))+);++#define mlk_poly_tobytes_aarch64_asm MLK_NAMESPACE(poly_tobytes_aarch64_asm)+void mlk_poly_tobytes_aarch64_asm(uint8_t r[384], const int16_t a[256])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mlkem_poly_tobytes_aarch64_asm.ml */+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYBYTES))+  requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N))+  requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT))+  assigns(memory_slice(r, MLKEM_POLYBYTES))+);++#define mlk_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm \+  MLK_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm)+void mlk_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm(+    int16_t r[256], const int16_t a[512], const int16_t b[512],+    const int16_t b_cache[256])+/* This must be kept in sync with the HOL-Light specification in+ * proofs/hol_light/aarch64/proofs/mlkem_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm.ml.+ */+__contract__(+    requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+    requires(memory_no_alias(a, sizeof(int16_t) * 2 * MLKEM_N))+    requires(memory_no_alias(b, sizeof(int16_t) * 2 * MLKEM_N))+    requires(memory_no_alias(b_cache, sizeof(int16_t) * 2 * (MLKEM_N / 2)))+    requires(array_abs_bound(a, 0, 2 * MLKEM_N, MLKEM_UINT12_LIMIT + 1))+    assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+);++#define mlk_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm \+  MLK_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm)+void mlk_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm(+    int16_t r[256], const int16_t a[768], const int16_t b[768],+    const int16_t b_cache[384])+/* This must be kept in sync with the HOL-Light specification in+ * proofs/hol_light/aarch64/proofs/mlkem_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm.ml.+ */+__contract__(+    requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+    requires(memory_no_alias(a, sizeof(int16_t) * 3 * MLKEM_N))+    requires(memory_no_alias(b, sizeof(int16_t) * 3 * MLKEM_N))+    requires(memory_no_alias(b_cache, sizeof(int16_t) * 3 * (MLKEM_N / 2)))+    requires(array_abs_bound(a, 0, 3 * MLKEM_N, MLKEM_UINT12_LIMIT + 1))+    assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+);++#define mlk_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm \+  MLK_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm)+void mlk_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm(+    int16_t r[256], const int16_t a[1024], const int16_t b[1024],+    const int16_t b_cache[512])+/* This must be kept in sync with the HOL-Light specification in+ * proofs/hol_light/aarch64/proofs/mlkem_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm.ml.+ */+__contract__(+    requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+    requires(memory_no_alias(a, sizeof(int16_t) * 4 * MLKEM_N))+    requires(memory_no_alias(b, sizeof(int16_t) * 4 * MLKEM_N))+    requires(memory_no_alias(b_cache, sizeof(int16_t) * 4 * (MLKEM_N / 2)))+    requires(array_abs_bound(a, 0, 4 * MLKEM_N, MLKEM_UINT12_LIMIT + 1))+    assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+);++#define mlk_rej_uniform_aarch64_asm MLK_NAMESPACE(rej_uniform_aarch64_asm)+MLK_MUST_CHECK_RETURN_VALUE+uint64_t mlk_rej_uniform_aarch64_asm(int16_t r[256], const uint8_t *buf,+                                     unsigned buflen, const uint8_t table[4096])+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/aarch64/proofs/mlkem_rej_uniform_aarch64_asm.ml. */+__contract__(+    requires(buflen % 24 == 0)+    requires(memory_no_alias(buf, buflen))+    requires(table == mlk_rej_uniform_table)+    requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+    assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+    ensures(return_value <= MLKEM_N)+    ensures(array_bound(r, 0, (unsigned) return_value, 0, MLKEM_Q))+);++#endif /* !MLK_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H */
+ cbits/mlkem/src/native/aarch64/src/mlkem_intt_aarch64_asm.S view
@@ -0,0 +1,635 @@+/* Copyright (c) 2022 Arm Limited+ * Copyright (c) 2022 Hanno Becker+ * Copyright (c) 2023 Amin Abdulrahman, Matthias Kannwischer+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [NeonNTT]+ *   Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1+ *   Becker, Hwang, Kannwischer, Yang, Yang+ *   https://eprint.iacr.org/2021/986+ *+ * - [SLOTHY_Paper]+ *   Fast and Clean: Auditable high-performance assembly via constraint solving+ *   Abdulrahman, Becker, Kannwischer, Klein+ *   https://eprint.iacr.org/2022/1303+ */++/*yaml+  Name: intt_aarch64_asm+  Description: AArch64 ML-KEM inverse NTT following @[NeonNTT] and @[SLOTHY_Paper]+  Signature: void mlk_intt_aarch64_asm(int16_t p[256], const int16_t twiddles12345[80], const int16_t twiddles56[384])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 512+      permissions: read/write+      c_parameter: int16_t p[256]+      description: Input/output polynomial+    x1:+      type: buffer+      size_bytes: 160+      permissions: read-only+      c_parameter: const int16_t twiddles12345[80]+      description: Twiddle factors for layers 1-5+    x2:+      type: buffer+      size_bytes: 768+      permissions: read-only+      c_parameter: const int16_t twiddles56[384]+      description: Twiddle factors for layers 6-7+  Stack:+    bytes: 64+    description: saving callee-saved Neon registers+*/++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_AARCH64) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    (!defined(MLK_CONFIG_NO_ENCAPS_API) || \+     !defined(MLK_CONFIG_NO_DECAPS_API))++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/aarch64_opt/src/mlkem_intt_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(intt_aarch64_asm)+MLK_ASM_FN_SYMBOL(intt_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0x40+        .cfi_adjust_cfa_offset 0x40+        stp d8, d9, [sp]+        .cfi_rel_offset d8, 0x0+        .cfi_rel_offset d9, 0x8+        stp d10, d11, [sp, #0x10]+        .cfi_rel_offset d10, 0x10+        .cfi_rel_offset d11, 0x18+        stp d12, d13, [sp, #0x20]+        .cfi_rel_offset d12, 0x20+        .cfi_rel_offset d13, 0x28+        stp d14, d15, [sp, #0x30]+        .cfi_rel_offset d14, 0x30+        .cfi_rel_offset d15, 0x38+        mov w5, #0xd01              // =3329+        mov v7.h[0], w5+        mov w5, #0x4ebf             // =20159+        mov v7.h[1], w5+        mov w5, #0x200              // =512+        dup v29.8h, w5+        mov w5, #0x13b0             // =5040+        dup v30.8h, w5+        mov x3, x0+        mov x4, #0x8                // =8+        ldr q13, [x3, #0x20]+        ldr q8, [x3, #0x30]+        ldr q6, [x3]+        ldr q16, [x3, #0x10]+        ldr q4, [x3, #0x50]+        ldr q11, [x3, #0x40]+        ldr q3, [x3, #0x70]+        trn1 v23.4s, v13.4s, v8.4s+        ldr q0, [x3, #0x60]+        trn2 v19.4s, v6.4s, v16.4s+        trn2 v21.4s, v13.4s, v8.4s+        trn1 v6.4s, v6.4s, v16.4s+        ldr q24, [x2, #0x20]+        trn1 v10.2d, v19.2d, v21.2d+        ldr q16, [x2], #0x60+        trn1 v5.2d, v6.2d, v23.2d+        trn1 v28.4s, v0.4s, v3.4s+        trn2 v18.2d, v6.2d, v23.2d+        mul v31.8h, v10.8h, v29.8h+        trn2 v13.4s, v0.4s, v3.4s+        ldur q14, [x2, #-0x50]+        sqrdmulh v26.8h, v18.8h, v30.8h+        ldur q20, [x2, #-0x20]+        mul v17.8h, v18.8h, v29.8h+        trn2 v18.2d, v19.2d, v21.2d+        mul v9.8h, v18.8h, v29.8h+        trn1 v12.4s, v11.4s, v4.4s+        sqrdmulh v22.8h, v18.8h, v30.8h+        sqrdmulh v3.8h, v10.8h, v30.8h+        sqrdmulh v25.8h, v5.8h, v30.8h+        mls v9.8h, v22.8h, v7.h[0]+        mls v17.8h, v26.8h, v7.h[0]+        trn2 v26.4s, v11.4s, v4.4s+        mul v8.8h, v5.8h, v29.8h+        trn1 v10.2d, v26.2d, v13.2d+        ldur q11, [x2, #-0x10]+        mls v31.8h, v3.8h, v7.h[0]+        trn1 v6.2d, v12.2d, v28.2d+        trn2 v3.2d, v26.2d, v13.2d+        ldur q4, [x2, #-0x30]+        mls v8.8h, v25.8h, v7.h[0]+        sub v19.8h, v17.8h, v9.8h+        trn2 v13.2d, v12.2d, v28.2d+        sqrdmulh v1.8h, v3.8h, v30.8h+        add v9.8h, v17.8h, v9.8h+        mul v18.8h, v19.8h, v20.8h+        add v28.8h, v8.8h, v31.8h+        sqrdmulh v20.8h, v19.8h, v11.8h+        sub v12.8h, v28.8h, v9.8h+        sub v23.8h, v8.8h, v31.8h+        sqrdmulh v11.8h, v13.8h, v30.8h+        sqrdmulh v5.8h, v23.8h, v4.8h+        mul v0.8h, v23.8h, v24.8h+        mul v2.8h, v13.8h, v29.8h+        mls v0.8h, v5.8h, v7.h[0]+        add v24.8h, v28.8h, v9.8h+        mls v18.8h, v20.8h, v7.h[0]+        sqrdmulh v15.8h, v6.8h, v30.8h+        sqrdmulh v25.8h, v12.8h, v14.8h+        mul v21.8h, v12.8h, v16.8h+        sub v23.8h, v0.8h, v18.8h+        sqrdmulh v8.8h, v23.8h, v14.8h+        mul v23.8h, v23.8h, v16.8h+        mls v21.8h, v25.8h, v7.h[0]+        mls v23.8h, v8.8h, v7.h[0]+        mul v14.8h, v3.8h, v29.8h+        add v3.8h, v0.8h, v18.8h+        trn2 v4.4s, v24.4s, v3.4s+        mls v14.8h, v1.8h, v7.h[0]+        trn1 v9.4s, v24.4s, v3.4s+        trn2 v12.4s, v21.4s, v23.4s+        mls v2.8h, v11.8h, v7.h[0]+        trn1 v28.4s, v21.4s, v23.4s+        ldr q11, [x1], #0x10+        mul v31.8h, v10.8h, v29.8h+        trn1 v25.2d, v4.2d, v12.2d+        trn1 v20.2d, v9.2d, v28.2d+        ldr q23, [x2, #0x50]+        trn2 v13.2d, v4.2d, v12.2d+        sqrdmulh v21.8h, v10.8h, v30.8h+        trn2 v4.2d, v9.2d, v28.2d+        ldr q9, [x2, #0x40]+        mul v27.8h, v6.8h, v29.8h+        add v26.8h, v20.8h, v25.8h+        sub v3.8h, v2.8h, v14.8h+        sqdmulh v12.8h, v26.8h, v7.h[1]+        add v5.8h, v4.8h, v13.8h+        sub v8.8h, v4.8h, v13.8h+        add v10.8h, v2.8h, v14.8h+        sqdmulh v6.8h, v5.8h, v7.h[1]+        ldr q2, [x2, #0x10]+        mls v27.8h, v15.8h, v7.h[0]+        ldr q15, [x2, #0x20]+        srshr v12.8h, v12.8h, #0xb+        mls v31.8h, v21.8h, v7.h[0]+        srshr v6.8h, v6.8h, #0xb+        sqrdmulh v23.8h, v3.8h, v23.8h+        mls v26.8h, v12.8h, v7.h[0]+        add v21.8h, v27.8h, v31.8h+        mls v5.8h, v6.8h, v7.h[0]+        sub v6.8h, v27.8h, v31.8h+        sub v14.8h, v21.8h, v10.8h+        ldr q27, [x2], #0x60+        mul v3.8h, v3.8h, v9.8h+        mls v3.8h, v23.8h, v7.h[0]+        ldur q13, [x2, #-0x30]+        sub v12.8h, v26.8h, v5.8h+        add v5.8h, v26.8h, v5.8h+        sqrdmulh v31.8h, v8.8h, v11.h[5]+        sqrdmulh v19.8h, v12.8h, v11.h[1]+        mul v24.8h, v12.8h, v11.h[0]+        sqrdmulh v13.8h, v6.8h, v13.8h+        mls v24.8h, v19.8h, v7.h[0]+        sub x4, x4, #0x2++Lmlk_intt_layer4567_start:+        add v16.8h, v21.8h, v10.8h+        mul v18.8h, v6.8h, v15.8h+        sub v19.8h, v20.8h, v25.8h+        ldr q21, [x3, #0xa0]+        str q5, [x3], #0x40+        mls v18.8h, v13.8h, v7.h[0]+        sqrdmulh v15.8h, v14.8h, v2.8h+        ldr q10, [x3, #0x50]+        ldr q12, [x3, #0x40]+        stur q24, [x3, #-0x20]+        mul v5.8h, v8.8h, v11.h[4]+        sub v0.8h, v18.8h, v3.8h+        ldr q24, [x3, #0x70]+        mls v5.8h, v31.8h, v7.h[0]+        ldr q26, [x2, #0x50]+        trn2 v1.4s, v12.4s, v10.4s+        add v6.8h, v18.8h, v3.8h+        sqrdmulh v20.8h, v0.8h, v2.8h+        trn1 v13.4s, v12.4s, v10.4s+        trn1 v18.4s, v16.4s, v6.4s+        mul v22.8h, v0.8h, v27.8h+        trn1 v17.4s, v21.4s, v24.4s+        sqrdmulh v0.8h, v19.8h, v11.h[3]+        trn1 v25.2d, v13.2d, v17.2d+        mls v22.8h, v20.8h, v7.h[0]+        trn2 v21.4s, v21.4s, v24.4s+        mul v24.8h, v25.8h, v29.8h+        trn2 v28.2d, v13.2d, v17.2d+        sqrdmulh v4.8h, v25.8h, v30.8h+        trn2 v3.2d, v1.2d, v21.2d+        mul v17.8h, v28.8h, v29.8h+        sqrdmulh v31.8h, v28.8h, v30.8h+        ldr q2, [x2, #0x10]+        mls v24.8h, v4.8h, v7.h[0]+        mul v4.8h, v19.8h, v11.h[2]+        ldr q19, [x2, #0x40]+        mls v4.8h, v0.8h, v7.h[0]+        mul v0.8h, v14.8h, v27.8h+        mls v0.8h, v15.8h, v7.h[0]+        sub v8.8h, v4.8h, v5.8h+        mul v12.8h, v3.8h, v29.8h+        mul v23.8h, v8.8h, v11.h[0]+        trn2 v28.4s, v16.4s, v6.4s+        sqrdmulh v10.8h, v8.8h, v11.h[1]+        trn1 v9.4s, v0.4s, v22.4s+        trn2 v22.4s, v0.4s, v22.4s+        ldr q11, [x1], #0x10+        mls v17.8h, v31.8h, v7.h[0]+        trn1 v20.2d, v18.2d, v9.2d+        trn2 v14.2d, v18.2d, v9.2d+        ldr q15, [x2, #0x20]+        trn1 v6.2d, v1.2d, v21.2d+        sqrdmulh v9.8h, v3.8h, v30.8h+        trn1 v25.2d, v28.2d, v22.2d+        trn2 v16.2d, v28.2d, v22.2d+        mls v23.8h, v10.8h, v7.h[0]+        add v1.8h, v20.8h, v25.8h+        sqrdmulh v21.8h, v6.8h, v30.8h+        add v8.8h, v14.8h, v16.8h+        ldr q27, [x2], #0x60+        sqdmulh v28.8h, v8.8h, v7.h[1]+        mls v12.8h, v9.8h, v7.h[0]+        sqdmulh v31.8h, v1.8h, v7.h[1]+        mul v0.8h, v6.8h, v29.8h+        sub v10.8h, v17.8h, v12.8h+        mls v0.8h, v21.8h, v7.h[0]+        srshr v21.8h, v28.8h, #0xb+        srshr v13.8h, v31.8h, #0xb+        sqrdmulh v22.8h, v10.8h, v26.8h+        mls v8.8h, v21.8h, v7.h[0]+        mls v1.8h, v13.8h, v7.h[0]+        add v21.8h, v24.8h, v0.8h+        stur q23, [x3, #-0x10]+        sub v6.8h, v24.8h, v0.8h+        mul v3.8h, v10.8h, v19.8h+        add v0.8h, v4.8h, v5.8h+        sqdmulh v13.8h, v0.8h, v7.h[1]+        ldur q10, [x2, #-0x30]+        add v5.8h, v1.8h, v8.8h+        mls v3.8h, v22.8h, v7.h[0]+        sub v8.8h, v1.8h, v8.8h+        mul v24.8h, v8.8h, v11.h[0]+        sqrdmulh v8.8h, v8.8h, v11.h[1]+        srshr v1.8h, v13.8h, #0xb+        sqrdmulh v13.8h, v6.8h, v10.8h+        mls v0.8h, v1.8h, v7.h[0]+        add v10.8h, v17.8h, v12.8h+        mls v24.8h, v8.8h, v7.h[0]+        sub v8.8h, v14.8h, v16.8h+        sqrdmulh v31.8h, v8.8h, v11.h[5]+        sub v14.8h, v21.8h, v10.8h+        stur q0, [x3, #-0x30]+        sub x4, x4, #0x1+        cbnz x4, Lmlk_intt_layer4567_start+        mul v15.8h, v6.8h, v15.8h+        sub v22.8h, v20.8h, v25.8h+        add v4.8h, v21.8h, v10.8h+        str q24, [x3, #0x20]+        mls v15.8h, v13.8h, v7.h[0]+        str q5, [x3], #0x40+        ldr q9, [x1], #0x10+        sqrdmulh v28.8h, v14.8h, v2.8h+        mul v16.8h, v14.8h, v27.8h+        sub v18.8h, v15.8h, v3.8h+        add v15.8h, v15.8h, v3.8h+        sqrdmulh v0.8h, v18.8h, v2.8h+        trn2 v24.4s, v4.4s, v15.4s+        trn1 v2.4s, v4.4s, v15.4s+        mul v18.8h, v18.8h, v27.8h+        mls v16.8h, v28.8h, v7.h[0]+        mls v18.8h, v0.8h, v7.h[0]+        mul v23.8h, v8.8h, v11.h[4]+        sqrdmulh v12.8h, v22.8h, v11.h[3]+        trn1 v17.4s, v16.4s, v18.4s+        trn2 v4.4s, v16.4s, v18.4s+        mls v23.8h, v31.8h, v7.h[0]+        trn2 v3.2d, v2.2d, v17.2d+        trn2 v6.2d, v24.2d, v4.2d+        mul v26.8h, v22.8h, v11.h[2]+        trn1 v28.2d, v2.2d, v17.2d+        mls v26.8h, v12.8h, v7.h[0]+        add v25.8h, v3.8h, v6.8h+        sub v18.8h, v3.8h, v6.8h+        trn1 v24.2d, v24.2d, v4.2d+        sqdmulh v1.8h, v25.8h, v7.h[1]+        sub v27.8h, v28.8h, v24.8h+        sqrdmulh v2.8h, v18.8h, v9.h[5]+        add v28.8h, v28.8h, v24.8h+        mul v24.8h, v27.8h, v9.h[2]+        sqdmulh v12.8h, v28.8h, v7.h[1]+        mul v20.8h, v18.8h, v9.h[4]+        mls v20.8h, v2.8h, v7.h[0]+        srshr v1.8h, v1.8h, #0xb+        sqrdmulh v19.8h, v27.8h, v9.h[3]+        srshr v15.8h, v12.8h, #0xb+        mls v25.8h, v1.8h, v7.h[0]+        add v8.8h, v26.8h, v23.8h+        sub v4.8h, v26.8h, v23.8h+        mls v28.8h, v15.8h, v7.h[0]+        mls v24.8h, v19.8h, v7.h[0]+        mul v2.8h, v4.8h, v11.h[0]+        sub v19.8h, v28.8h, v25.8h+        sqrdmulh v15.8h, v4.8h, v11.h[1]+        add v25.8h, v28.8h, v25.8h+        sub v10.8h, v24.8h, v20.8h+        str q25, [x3], #0x40+        sqrdmulh v22.8h, v19.8h, v9.h[1]+        add v28.8h, v24.8h, v20.8h+        sqrdmulh v25.8h, v10.8h, v9.h[1]+        mul v27.8h, v19.8h, v9.h[0]+        mul v26.8h, v10.8h, v9.h[0]+        sqdmulh v20.8h, v28.8h, v7.h[1]+        sqdmulh v16.8h, v8.8h, v7.h[1]+        mls v26.8h, v25.8h, v7.h[0]+        mls v2.8h, v15.8h, v7.h[0]+        srshr v15.8h, v20.8h, #0xb+        srshr v1.8h, v16.8h, #0xb+        mls v27.8h, v22.8h, v7.h[0]+        mls v28.8h, v15.8h, v7.h[0]+        mls v8.8h, v1.8h, v7.h[0]+        stur q27, [x3, #-0x20]+        stur q2, [x3, #-0x50]+        stur q28, [x3, #-0x30]+        stur q26, [x3, #-0x10]+        stur q8, [x3, #-0x70]+        mov x4, #0x4                // =4+        ldr q0, [x1], #0x20+        ldur q1, [x1, #-0x10]+        ldr q26, [x0]+        ldr q13, [x0, #0x40]+        ldr q28, [x0, #0xc0]+        ldr q2, [x0, #0x140]+        ldr q6, [x0, #0x80]+        ldr q9, [x0, #0x100]+        ldr q29, [x0, #0x1c0]+        ldr q23, [x0, #0x180]+        sub v17.8h, v26.8h, v13.8h+        add v4.8h, v26.8h, v13.8h+        ldr q25, [x0, #0xd0]+        ldr q24, [x0, #0x50]+        add v5.8h, v6.8h, v28.8h+        mul v19.8h, v17.8h, v0.h[6]+        sub v10.8h, v6.8h, v28.8h+        ldr q30, [x0, #0x150]+        sqrdmulh v12.8h, v17.8h, v0.h[7]+        add v17.8h, v9.8h, v2.8h+        sub v28.8h, v9.8h, v2.8h+        ldr q2, [x0, #0x90]+        sub v26.8h, v23.8h, v29.8h+        sqrdmulh v31.8h, v10.8h, v1.h[1]+        add v22.8h, v23.8h, v29.8h+        ldr q3, [x0, #0x110]+        sqrdmulh v9.8h, v28.8h, v1.h[3]+        sub v20.8h, v4.8h, v5.8h+        sub v27.8h, v17.8h, v22.8h+        ldr q29, [x0, #0x10]+        add v16.8h, v4.8h, v5.8h+        sqrdmulh v4.8h, v26.8h, v1.h[5]+        add v6.8h, v17.8h, v22.8h+        ldr q22, [x0, #0x1d0]+        mul v8.8h, v28.8h, v1.h[2]+        sub v21.8h, v2.8h, v25.8h+        sub v5.8h, v16.8h, v6.8h+        mul v17.8h, v26.8h, v1.h[4]+        mul v26.8h, v10.8h, v1.h[0]+        mls v26.8h, v31.8h, v7.h[0]+        mls v17.8h, v4.8h, v7.h[0]+        mls v19.8h, v12.8h, v7.h[0]+        mls v8.8h, v9.8h, v7.h[0]+        sqrdmulh v10.8h, v27.8h, v0.h[5]+        sub v12.8h, v19.8h, v26.8h+        add v9.8h, v19.8h, v26.8h+        sqrdmulh v26.8h, v20.8h, v0.h[3]+        sub v11.8h, v8.8h, v17.8h+        add v14.8h, v8.8h, v17.8h+        sqrdmulh v13.8h, v12.8h, v0.h[3]+        add v23.8h, v9.8h, v14.8h+        sqrdmulh v28.8h, v11.8h, v0.h[5]+        sub v19.8h, v9.8h, v14.8h+        mul v17.8h, v27.8h, v0.h[4]+        str q23, [x0, #0x40]+        mul v14.8h, v20.8h, v0.h[2]+        mul v8.8h, v11.8h, v0.h[4]+        mul v4.8h, v12.8h, v0.h[2]+        mls v14.8h, v26.8h, v7.h[0]+        mls v17.8h, v10.8h, v7.h[0]+        mls v8.8h, v28.8h, v7.h[0]+        mls v4.8h, v13.8h, v7.h[0]+        sub v10.8h, v14.8h, v17.8h+        add v20.8h, v14.8h, v17.8h+        sqrdmulh v28.8h, v5.8h, v0.h[1]+        mul v18.8h, v5.8h, v0.h[0]+        str q20, [x0, #0x80]+        sub v13.8h, v4.8h, v8.8h+        mul v23.8h, v10.8h, v0.h[0]+        mul v17.8h, v19.8h, v0.h[0]+        sqrdmulh v9.8h, v13.8h, v0.h[1]+        mls v18.8h, v28.8h, v7.h[0]+        sqrdmulh v10.8h, v10.8h, v0.h[1]+        sub x4, x4, #0x2++Lmlk_intt_layer123_start:+        sub v12.8h, v3.8h, v30.8h+        mul v11.8h, v21.8h, v1.h[0]+        add v28.8h, v4.8h, v8.8h+        ldr q20, [x0, #0x190]+        add v27.8h, v16.8h, v6.8h+        sqrdmulh v8.8h, v12.8h, v1.h[3]+        add v16.8h, v29.8h, v24.8h+        str q28, [x0, #0xc0]+        mls v23.8h, v10.8h, v7.h[0]+        str q27, [x0], #0x10+        add v15.8h, v20.8h, v22.8h+        str q18, [x0, #0xf0]+        mul v14.8h, v13.8h, v0.h[0]+        add v2.8h, v2.8h, v25.8h+        sub v26.8h, v20.8h, v22.8h+        mul v4.8h, v12.8h, v1.h[2]+        sub v5.8h, v16.8h, v2.8h+        str q23, [x0, #0x170]+        add v20.8h, v3.8h, v30.8h+        sqrdmulh v27.8h, v26.8h, v1.h[5]+        add v16.8h, v16.8h, v2.8h+        mul v18.8h, v26.8h, v1.h[4]+        sub v31.8h, v20.8h, v15.8h+        mls v4.8h, v8.8h, v7.h[0]+        sub v28.8h, v29.8h, v24.8h+        mls v18.8h, v27.8h, v7.h[0]+        ldr q22, [x0, #0x1d0]+        mul v26.8h, v28.8h, v0.h[6]+        mul v2.8h, v5.8h, v0.h[2]+        sub v12.8h, v4.8h, v18.8h+        sqrdmulh v24.8h, v28.8h, v0.h[7]+        mls v14.8h, v9.8h, v7.h[0]+        sqrdmulh v10.8h, v12.8h, v0.h[5]+        mls v26.8h, v24.8h, v7.h[0]+        ldr q24, [x0, #0x50]+        mul v8.8h, v12.8h, v0.h[4]+        str q14, [x0, #0x1b0]+        add v28.8h, v4.8h, v18.8h+        sqrdmulh v5.8h, v5.8h, v0.h[3]+        add v6.8h, v20.8h, v15.8h+        sqrdmulh v3.8h, v19.8h, v0.h[1]+        sub v13.8h, v16.8h, v6.8h+        sqrdmulh v12.8h, v21.8h, v1.h[1]+        sqrdmulh v21.8h, v13.8h, v0.h[1]+        sqrdmulh v27.8h, v31.8h, v0.h[5]+        ldr q25, [x0, #0xd0]+        mls v11.8h, v12.8h, v7.h[0]+        mul v23.8h, v31.8h, v0.h[4]+        mul v18.8h, v13.8h, v0.h[0]+        add v30.8h, v26.8h, v11.8h+        sub v13.8h, v26.8h, v11.8h+        mls v23.8h, v27.8h, v7.h[0]+        add v12.8h, v30.8h, v28.8h+        sub v19.8h, v30.8h, v28.8h+        mls v2.8h, v5.8h, v7.h[0]+        str q12, [x0, #0x40]+        sqrdmulh v26.8h, v13.8h, v0.h[3]+        mls v8.8h, v10.8h, v7.h[0]+        ldr q30, [x0, #0x150]+        sub v20.8h, v2.8h, v23.8h+        mul v4.8h, v13.8h, v0.h[2]+        add v13.8h, v2.8h, v23.8h+        mls v4.8h, v26.8h, v7.h[0]+        ldr q2, [x0, #0x90]+        mul v23.8h, v20.8h, v0.h[0]+        ldr q29, [x0, #0x10]+        sqrdmulh v10.8h, v20.8h, v0.h[1]+        str q13, [x0, #0x80]+        sub v13.8h, v4.8h, v8.8h+        mls v17.8h, v3.8h, v7.h[0]+        ldr q3, [x0, #0x110]+        mls v18.8h, v21.8h, v7.h[0]+        sub v21.8h, v2.8h, v25.8h+        sqrdmulh v9.8h, v13.8h, v0.h[1]+        str q17, [x0, #0x130]+        mul v17.8h, v19.8h, v0.h[0]+        sub x4, x4, #0x1+        cbnz x4, Lmlk_intt_layer123_start+        mls v23.8h, v10.8h, v7.h[0]+        ldr q11, [x0, #0x190]+        str q18, [x0, #0x100]+        add v27.8h, v3.8h, v30.8h+        mul v13.8h, v13.8h, v0.h[0]+        sub v5.8h, v29.8h, v24.8h+        add v14.8h, v16.8h, v6.8h+        mls v13.8h, v9.8h, v7.h[0]+        add v10.8h, v11.8h, v22.8h+        str q23, [x0, #0x180]+        sub v20.8h, v11.8h, v22.8h+        sub v23.8h, v27.8h, v10.8h+        sqrdmulh v16.8h, v21.8h, v1.h[1]+        sqrdmulh v31.8h, v23.8h, v0.h[5]+        str q13, [x0, #0x1c0]+        add v13.8h, v4.8h, v8.8h+        mul v18.8h, v21.8h, v1.h[0]+        str q13, [x0, #0xc0]+        sqrdmulh v13.8h, v19.8h, v0.h[1]+        sqrdmulh v28.8h, v20.8h, v1.h[5]+        str q14, [x0], #0x10+        mul v4.8h, v20.8h, v1.h[4]+        mls v17.8h, v13.8h, v7.h[0]+        sub v13.8h, v3.8h, v30.8h+        sqrdmulh v8.8h, v13.8h, v1.h[3]+        mul v12.8h, v13.8h, v1.h[2]+        mls v4.8h, v28.8h, v7.h[0]+        mls v12.8h, v8.8h, v7.h[0]+        mls v18.8h, v16.8h, v7.h[0]+        str q17, [x0, #0x130]+        sqrdmulh v15.8h, v5.8h, v0.h[7]+        add v11.8h, v27.8h, v10.8h+        mul v16.8h, v5.8h, v0.h[6]+        sub v8.8h, v12.8h, v4.8h+        sqrdmulh v28.8h, v8.8h, v0.h[5]+        add v13.8h, v2.8h, v25.8h+        mls v16.8h, v15.8h, v7.h[0]+        add v26.8h, v12.8h, v4.8h+        mul v8.8h, v8.8h, v0.h[4]+        add v4.8h, v29.8h, v24.8h+        mls v8.8h, v28.8h, v7.h[0]+        sub v20.8h, v4.8h, v13.8h+        add v14.8h, v4.8h, v13.8h+        add v12.8h, v16.8h, v18.8h+        sqrdmulh v22.8h, v20.8h, v0.h[3]+        add v27.8h, v14.8h, v11.8h+        sub v13.8h, v16.8h, v18.8h+        mul v4.8h, v20.8h, v0.h[2]+        str q27, [x0], #0x10+        sub v24.8h, v12.8h, v26.8h+        sqrdmulh v3.8h, v13.8h, v0.h[3]+        mul v13.8h, v13.8h, v0.h[2]+        sqrdmulh v27.8h, v24.8h, v0.h[1]+        mls v13.8h, v3.8h, v7.h[0]+        mul v9.8h, v24.8h, v0.h[0]+        mls v9.8h, v27.8h, v7.h[0]+        add v30.8h, v13.8h, v8.8h+        sub v13.8h, v13.8h, v8.8h+        mls v4.8h, v22.8h, v7.h[0]+        str q30, [x0, #0xb0]+        sqrdmulh v16.8h, v13.8h, v0.h[1]+        str q9, [x0, #0x130]+        mul v9.8h, v13.8h, v0.h[0]+        add v13.8h, v12.8h, v26.8h+        str q13, [x0, #0x30]+        mul v13.8h, v23.8h, v0.h[4]+        sub v23.8h, v14.8h, v11.8h+        mls v13.8h, v31.8h, v7.h[0]+        mls v9.8h, v16.8h, v7.h[0]+        mul v30.8h, v23.8h, v0.h[0]+        sub v24.8h, v4.8h, v13.8h+        add v13.8h, v4.8h, v13.8h+        sqrdmulh v23.8h, v23.8h, v0.h[1]+        str q9, [x0, #0x1b0]+        str q13, [x0, #0x70]+        sqrdmulh v13.8h, v24.8h, v0.h[1]+        mul v21.8h, v24.8h, v0.h[0]+        mls v30.8h, v23.8h, v7.h[0]+        mls v21.8h, v13.8h, v7.h[0]+        str q30, [x0, #0xf0]+        str q21, [x0, #0x170]+        ldp d8, d9, [sp]+        .cfi_restore d8+        .cfi_restore d9+        ldp d10, d11, [sp, #0x10]+        .cfi_restore d10+        .cfi_restore d11+        ldp d12, d13, [sp, #0x20]+        .cfi_restore d12+        .cfi_restore d13+        ldp d14, d15, [sp, #0x30]+        .cfi_restore d14+        .cfi_restore d15+        add sp, sp, #0x40+        .cfi_adjust_cfa_offset -0x40+        ret+        .cfi_endproc++MLK_ASM_FN_SIZE(intt_aarch64_asm)++#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED && \+          (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/aarch64/src/mlkem_ntt_aarch64_asm.S view
@@ -0,0 +1,565 @@+/* Copyright (c) 2022 Arm Limited+ * Copyright (c) 2022 Hanno Becker+ * Copyright (c) 2023 Amin Abdulrahman, Matthias Kannwischer+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [NeonNTT]+ *   Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1+ *   Becker, Hwang, Kannwischer, Yang, Yang+ *   https://eprint.iacr.org/2021/986+ *+ * - [SLOTHY_Paper]+ *   Fast and Clean: Auditable high-performance assembly via constraint solving+ *   Abdulrahman, Becker, Kannwischer, Klein+ *   https://eprint.iacr.org/2022/1303+ */++/*yaml+  Name: ntt_aarch64_asm+  Description: AArch64 ML-KEM forward NTT following @[NeonNTT] and @[SLOTHY_Paper]+  Signature: void mlk_ntt_aarch64_asm(int16_t p[256], const int16_t twiddles12345[80], const int16_t twiddles56[384])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 512+      permissions: read/write+      c_parameter: int16_t p[256]+      description: Input/output polynomial+    x1:+      type: buffer+      size_bytes: 160+      permissions: read-only+      c_parameter: const int16_t twiddles12345[80]+      description: Twiddle factors for layers 1-5+    x2:+      type: buffer+      size_bytes: 768+      permissions: read-only+      c_parameter: const int16_t twiddles56[384]+      description: Twiddle factors for layers 6-7+  Stack:+    bytes: 64+    description: saving callee-saved Neon registers+*/++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_AARCH64) &&  !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/aarch64_opt/src/mlkem_ntt_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(ntt_aarch64_asm)+MLK_ASM_FN_SYMBOL(ntt_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0x40+        .cfi_adjust_cfa_offset 0x40+        stp d8, d9, [sp]+        .cfi_rel_offset d8, 0x0+        .cfi_rel_offset d9, 0x8+        stp d10, d11, [sp, #0x10]+        .cfi_rel_offset d10, 0x10+        .cfi_rel_offset d11, 0x18+        stp d12, d13, [sp, #0x20]+        .cfi_rel_offset d12, 0x20+        .cfi_rel_offset d13, 0x28+        stp d14, d15, [sp, #0x30]+        .cfi_rel_offset d14, 0x30+        .cfi_rel_offset d15, 0x38+        mov w5, #0xd01              // =3329+        mov v7.h[0], w5+        mov w5, #0x4ebf             // =20159+        mov v7.h[1], w5+        mov x3, x0+        mov x4, #0x4                // =4+        ldr q0, [x1], #0x20+        ldur q1, [x1, #-0x10]+        ldr q21, [x0, #0x40]+        ldr q5, [x0, #0x1c0]+        ldr q30, [x0, #0x110]+        ldr q24, [x0, #0x140]+        ldr q12, [x0, #0x80]+        sqrdmulh v9.8h, v5.8h, v0.h[1]+        mul v23.8h, v5.8h, v0.h[0]+        sqrdmulh v17.8h, v24.8h, v0.h[1]+        ldr q13, [x0, #0xc0]+        mls v23.8h, v9.8h, v7.h[0]+        mul v8.8h, v24.8h, v0.h[0]+        mls v8.8h, v17.8h, v7.h[0]+        add v9.8h, v13.8h, v23.8h+        sub v10.8h, v13.8h, v23.8h+        mul v11.8h, v30.8h, v0.h[0]+        ldr q13, [x0, #0x180]+        sqrdmulh v28.8h, v9.8h, v0.h[3]+        sub v29.8h, v21.8h, v8.8h+        mul v26.8h, v9.8h, v0.h[2]+        add v8.8h, v21.8h, v8.8h+        mul v2.8h, v13.8h, v0.h[0]+        mls v26.8h, v28.8h, v7.h[0]+        mul v28.8h, v10.8h, v0.h[4]+        sqrdmulh v23.8h, v10.8h, v0.h[5]+        add v22.8h, v8.8h, v26.8h+        sqrdmulh v10.8h, v13.8h, v0.h[1]+        sqrdmulh v21.8h, v22.8h, v0.h[7]+        ldr q13, [x0, #0x100]+        mul v16.8h, v22.8h, v0.h[6]+        mls v28.8h, v23.8h, v7.h[0]+        mls v2.8h, v10.8h, v7.h[0]+        sqrdmulh v23.8h, v13.8h, v0.h[1]+        sub v10.8h, v29.8h, v28.8h+        add v17.8h, v29.8h, v28.8h+        mls v16.8h, v21.8h, v7.h[0]+        sub v18.8h, v12.8h, v2.8h+        ldr q29, [x0]+        sqrdmulh v14.8h, v17.8h, v1.h[3]+        add v22.8h, v12.8h, v2.8h+        sqrdmulh v9.8h, v18.8h, v0.h[5]+        mul v21.8h, v13.8h, v0.h[0]+        ldr q13, [x0, #0x150]+        mul v5.8h, v18.8h, v0.h[4]+        mls v5.8h, v9.8h, v7.h[0]+        mul v18.8h, v13.8h, v0.h[0]+        mls v21.8h, v23.8h, v7.h[0]+        sqrdmulh v2.8h, v13.8h, v0.h[1]+        mul v13.8h, v17.8h, v1.h[2]+        sub v4.8h, v29.8h, v21.8h+        mls v13.8h, v14.8h, v7.h[0]+        add v25.8h, v29.8h, v21.8h+        add v6.8h, v4.8h, v5.8h+        sqrdmulh v15.8h, v22.8h, v0.h[3]+        sub v21.8h, v4.8h, v5.8h+        sub v5.8h, v8.8h, v26.8h+        mul v23.8h, v22.8h, v0.h[2]+        add v28.8h, v6.8h, v13.8h+        sub v13.8h, v6.8h, v13.8h+        mul v4.8h, v5.8h, v1.h[0]+        sub x4, x4, #0x2++Lmlk_ntt_layer123_start:+        mls v23.8h, v15.8h, v7.h[0]+        ldr q6, [x0, #0x190]+        ldr q15, [x0, #0x90]+        ldr q19, [x0, #0x10]+        mul v22.8h, v10.8h, v1.h[4]+        ldr q24, [x0, #0x50]+        str q13, [x0, #0x140]+        sqrdmulh v13.8h, v6.8h, v0.h[1]+        sub v20.8h, v25.8h, v23.8h+        sqrdmulh v3.8h, v30.8h, v0.h[1]+        str q28, [x0, #0x100]+        ldr q30, [x0, #0x120]+        mul v8.8h, v6.8h, v0.h[0]+        sqrdmulh v27.8h, v10.8h, v1.h[5]+        mls v11.8h, v3.8h, v7.h[0]+        mls v18.8h, v2.8h, v7.h[0]+        ldr q31, [x0, #0x160]+        sqrdmulh v10.8h, v5.8h, v1.h[1]+        mls v8.8h, v13.8h, v7.h[0]+        ldr q13, [x0, #0x1d0]+        sub v14.8h, v24.8h, v18.8h+        add v9.8h, v24.8h, v18.8h+        sqrdmulh v2.8h, v31.8h, v0.h[1]+        mls v4.8h, v10.8h, v7.h[0]+        add v10.8h, v25.8h, v23.8h+        sub v24.8h, v19.8h, v11.8h+        add v25.8h, v19.8h, v11.8h+        sqrdmulh v28.8h, v13.8h, v0.h[1]+        mul v11.8h, v30.8h, v0.h[0]+        mul v17.8h, v13.8h, v0.h[0]+        sub v13.8h, v10.8h, v16.8h+        sub v6.8h, v15.8h, v8.8h+        mls v17.8h, v28.8h, v7.h[0]+        str q13, [x0, #0x40]+        mls v22.8h, v27.8h, v7.h[0]+        ldr q13, [x0, #0xd0]+        add v26.8h, v20.8h, v4.8h+        mul v18.8h, v31.8h, v0.h[0]+        add v27.8h, v10.8h, v16.8h+        str q26, [x0, #0x80]+        sqrdmulh v31.8h, v6.8h, v0.h[5]+        add v3.8h, v21.8h, v22.8h+        str q27, [x0], #0x10+        mul v26.8h, v6.8h, v0.h[4]+        add v6.8h, v13.8h, v17.8h+        sub v5.8h, v13.8h, v17.8h+        str q3, [x0, #0x170]+        sub v17.8h, v21.8h, v22.8h+        sqrdmulh v10.8h, v6.8h, v0.h[3]+        sub v13.8h, v20.8h, v4.8h+        add v20.8h, v15.8h, v8.8h+        sqrdmulh v12.8h, v5.8h, v0.h[5]+        str q13, [x0, #0xb0]+        mul v8.8h, v6.8h, v0.h[2]+        str q17, [x0, #0x1b0]+        mls v8.8h, v10.8h, v7.h[0]+        mul v29.8h, v5.8h, v0.h[4]+        mls v29.8h, v12.8h, v7.h[0]+        sub v5.8h, v9.8h, v8.8h+        add v3.8h, v9.8h, v8.8h+        sqrdmulh v15.8h, v20.8h, v0.h[3]+        mul v4.8h, v5.8h, v1.h[0]+        add v6.8h, v14.8h, v29.8h+        sqrdmulh v9.8h, v3.8h, v0.h[7]+        sqrdmulh v12.8h, v6.8h, v1.h[3]+        sub v10.8h, v14.8h, v29.8h+        mul v23.8h, v6.8h, v1.h[2]+        mls v26.8h, v31.8h, v7.h[0]+        mls v23.8h, v12.8h, v7.h[0]+        mul v16.8h, v3.8h, v0.h[6]+        add v13.8h, v24.8h, v26.8h+        sub v21.8h, v24.8h, v26.8h+        mls v16.8h, v9.8h, v7.h[0]+        add v28.8h, v13.8h, v23.8h+        sub v13.8h, v13.8h, v23.8h+        mul v23.8h, v20.8h, v0.h[2]+        sub x4, x4, #0x1+        cbnz x4, Lmlk_ntt_layer123_start+        sqrdmulh v3.8h, v5.8h, v1.h[1]+        mls v23.8h, v15.8h, v7.h[0]+        ldr q5, [x0, #0x190]+        mul v29.8h, v10.8h, v1.h[4]+        mls v4.8h, v3.8h, v7.h[0]+        sub v19.8h, v25.8h, v23.8h+        sqrdmulh v31.8h, v5.8h, v0.h[1]+        sqrdmulh v6.8h, v30.8h, v0.h[1]+        sub v3.8h, v19.8h, v4.8h+        mul v5.8h, v5.8h, v0.h[0]+        str q3, [x0, #0xc0]+        sqrdmulh v12.8h, v10.8h, v1.h[5]+        mls v18.8h, v2.8h, v7.h[0]+        ldr q3, [x0, #0x1d0]+        mls v5.8h, v31.8h, v7.h[0]+        sqrdmulh v10.8h, v3.8h, v0.h[1]+        mls v11.8h, v6.8h, v7.h[0]+        ldr q31, [x0, #0x90]+        mul v30.8h, v3.8h, v0.h[0]+        mls v30.8h, v10.8h, v7.h[0]+        sub v10.8h, v31.8h, v5.8h+        mls v29.8h, v12.8h, v7.h[0]+        ldr q6, [x0, #0xd0]+        sqrdmulh v15.8h, v10.8h, v0.h[5]+        mul v17.8h, v10.8h, v0.h[4]+        add v10.8h, v6.8h, v30.8h+        sub v6.8h, v6.8h, v30.8h+        sqrdmulh v12.8h, v10.8h, v0.h[3]+        sub v27.8h, v21.8h, v29.8h+        sqrdmulh v3.8h, v6.8h, v0.h[5]+        mul v10.8h, v10.8h, v0.h[2]+        ldr q20, [x0, #0x50]+        mls v10.8h, v12.8h, v7.h[0]+        mul v2.8h, v6.8h, v0.h[4]+        add v6.8h, v20.8h, v18.8h+        add v5.8h, v31.8h, v5.8h+        mls v2.8h, v3.8h, v7.h[0]+        sub v31.8h, v6.8h, v10.8h+        sqrdmulh v12.8h, v5.8h, v0.h[3]+        sub v22.8h, v20.8h, v18.8h+        add v6.8h, v6.8h, v10.8h+        mul v20.8h, v31.8h, v1.h[0]+        add v30.8h, v22.8h, v2.8h+        sqrdmulh v3.8h, v6.8h, v0.h[7]+        sqrdmulh v10.8h, v30.8h, v1.h[3]+        mul v9.8h, v30.8h, v1.h[2]+        ldr q30, [x0, #0x10]+        mls v17.8h, v15.8h, v7.h[0]+        mls v9.8h, v10.8h, v7.h[0]+        mul v15.8h, v6.8h, v0.h[6]+        add v24.8h, v30.8h, v11.8h+        sub v10.8h, v22.8h, v2.8h+        mls v15.8h, v3.8h, v7.h[0]+        add v6.8h, v19.8h, v4.8h+        add v22.8h, v25.8h, v23.8h+        sqrdmulh v3.8h, v10.8h, v1.h[5]+        str q13, [x0, #0x140]+        sub v19.8h, v30.8h, v11.8h+        add v25.8h, v22.8h, v16.8h+        mul v5.8h, v5.8h, v0.h[2]+        sub v13.8h, v22.8h, v16.8h+        str q28, [x0, #0x100]+        mls v5.8h, v12.8h, v7.h[0]+        str q13, [x0, #0x40]+        str q6, [x0, #0x80]+        add v21.8h, v21.8h, v29.8h+        sqrdmulh v13.8h, v31.8h, v1.h[1]+        str q25, [x0], #0x10+        add v12.8h, v19.8h, v17.8h+        sub v31.8h, v19.8h, v17.8h+        mul v30.8h, v10.8h, v1.h[4]+        str q21, [x0, #0x170]+        add v21.8h, v24.8h, v5.8h+        add v6.8h, v12.8h, v9.8h+        mls v30.8h, v3.8h, v7.h[0]+        str q27, [x0, #0x1b0]+        sub v10.8h, v21.8h, v15.8h+        sub v12.8h, v12.8h, v9.8h+        mls v20.8h, v13.8h, v7.h[0]+        str q6, [x0, #0x100]+        str q10, [x0, #0x40]+        sub v13.8h, v24.8h, v5.8h+        add v3.8h, v21.8h, v15.8h+        str q12, [x0, #0x140]+        sub v10.8h, v31.8h, v30.8h+        add v21.8h, v31.8h, v30.8h+        str q3, [x0], #0x10+        add v12.8h, v13.8h, v20.8h+        sub v13.8h, v13.8h, v20.8h+        str q21, [x0, #0x170]+        str q10, [x0, #0x1b0]+        str q12, [x0, #0x70]+        str q13, [x0, #0xb0]+        mov x0, x3+        mov x4, #0x8                // =8+        ldr q2, [x0, #0x20]+        ldr q13, [x1], #0x10+        ldr q30, [x0, #0x30]+        ldr q25, [x2, #0x40]+        ldr q5, [x0]+        ldr q18, [x0, #0x60]+        ldr q12, [x0, #0x70]+        sqrdmulh v17.8h, v2.8h, v13.h[1]+        ldr q4, [x1], #0x10+        ldr q23, [x0, #0x10]+        sqrdmulh v21.8h, v30.8h, v13.h[1]+        ldr q24, [x2, #0x20]+        ldr q9, [x2], #0x60+        mul v10.8h, v30.8h, v13.h[0]+        mul v11.8h, v2.8h, v13.h[0]+        mls v10.8h, v21.8h, v7.h[0]+        sqrdmulh v29.8h, v12.8h, v4.h[1]+        mul v1.8h, v12.8h, v4.h[0]+        add v21.8h, v23.8h, v10.8h+        sub v10.8h, v23.8h, v10.8h+        mul v8.8h, v18.8h, v4.h[0]+        sqrdmulh v23.8h, v21.8h, v13.h[3]+        mul v2.8h, v21.8h, v13.h[2]+        mls v1.8h, v29.8h, v7.h[0]+        mls v2.8h, v23.8h, v7.h[0]+        ldur q15, [x2, #-0x50]+        sqrdmulh v0.8h, v10.8h, v13.h[5]+        mls v11.8h, v17.8h, v7.h[0]+        ldr q29, [x0, #0x50]+        mul v23.8h, v10.8h, v13.h[4]+        mls v23.8h, v0.8h, v7.h[0]+        sub v16.8h, v29.8h, v1.8h+        add v3.8h, v5.8h, v11.8h+        sub v31.8h, v5.8h, v11.8h+        sqrdmulh v22.8h, v16.8h, v4.h[5]+        add v30.8h, v3.8h, v2.8h+        sub v0.8h, v3.8h, v2.8h+        sqrdmulh v28.8h, v18.8h, v4.h[1]+        add v21.8h, v31.8h, v23.8h+        sub v19.8h, v31.8h, v23.8h+        mul v26.8h, v16.8h, v4.h[4]+        trn2 v3.4s, v30.4s, v0.4s+        ldur q23, [x2, #-0x10]+        trn2 v18.4s, v21.4s, v19.4s+        mls v26.8h, v22.8h, v7.h[0]+        trn1 v13.4s, v30.4s, v0.4s+        mls v8.8h, v28.8h, v7.h[0]+        trn2 v31.2d, v3.2d, v18.2d+        trn1 v11.4s, v21.4s, v19.4s+        add v27.8h, v29.8h, v1.8h+        sqrdmulh v6.8h, v31.8h, v15.8h+        trn1 v2.2d, v13.2d, v11.2d+        trn2 v13.2d, v13.2d, v11.2d+        mul v1.8h, v31.8h, v9.8h+        ldr q11, [x0, #0x40]+        sqrdmulh v29.8h, v13.8h, v15.8h+        mls v1.8h, v6.8h, v7.h[0]+        trn1 v6.2d, v3.2d, v18.2d+        mul v17.8h, v13.8h, v9.8h+        sub v13.8h, v11.8h, v8.8h+        sqrdmulh v10.8h, v27.8h, v4.h[3]+        sub v12.8h, v13.8h, v26.8h+        sub v18.8h, v6.8h, v1.8h+        mls v17.8h, v29.8h, v7.h[0]+        add v30.8h, v6.8h, v1.8h+        add v6.8h, v13.8h, v26.8h+        ldur q13, [x2, #-0x30]+        sqrdmulh v16.8h, v18.8h, v23.8h+        trn1 v28.4s, v6.4s, v12.4s+        mul v23.8h, v18.8h, v25.8h+        ldr q25, [x2, #0x10]+        add v20.8h, v2.8h, v17.8h+        mul v0.8h, v30.8h, v24.8h+        sqrdmulh v29.8h, v30.8h, v13.8h+        sub v30.8h, v2.8h, v17.8h+        mls v23.8h, v16.8h, v7.h[0]+        sub x4, x4, #0x2++Lmlk_ntt_layer4567_start:+        ldr q19, [x2, #0x50]+        sub v31.8h, v30.8h, v23.8h+        mls v0.8h, v29.8h, v7.h[0]+        add v16.8h, v11.8h, v8.8h+        ldr q18, [x0, #0xa0]+        trn2 v14.4s, v6.4s, v12.4s+        mul v26.8h, v27.8h, v4.h[2]+        ldr q4, [x1], #0x10+        ldr q24, [x2, #0x40]+        ldr q21, [x0, #0xb0]+        mls v26.8h, v10.8h, v7.h[0]+        add v23.8h, v30.8h, v23.8h+        sub v15.8h, v20.8h, v0.8h+        ldr q9, [x0, #0x90]+        add v10.8h, v20.8h, v0.8h+        mul v8.8h, v18.8h, v4.h[0]+        ldr q1, [x2], #0x60+        trn1 v27.4s, v23.4s, v31.4s+        sqrdmulh v12.8h, v18.8h, v4.h[1]+        trn1 v5.4s, v10.4s, v15.4s+        sub v30.8h, v16.8h, v26.8h+        trn2 v13.2d, v5.2d, v27.2d+        sqrdmulh v2.8h, v21.8h, v4.h[1]+        add v29.8h, v16.8h, v26.8h+        mul v0.8h, v21.8h, v4.h[0]+        str q13, [x0, #0x20]+        trn1 v11.4s, v29.4s, v30.4s+        mls v8.8h, v12.8h, v7.h[0]+        trn2 v26.4s, v29.4s, v30.4s+        trn2 v6.2d, v11.2d, v28.2d+        mls v0.8h, v2.8h, v7.h[0]+        trn2 v16.2d, v26.2d, v14.2d+        trn1 v26.2d, v26.2d, v14.2d+        trn1 v20.2d, v5.2d, v27.2d+        sqrdmulh v29.8h, v6.8h, v25.8h+        trn2 v15.4s, v10.4s, v15.4s+        sqrdmulh v13.8h, v16.8h, v25.8h+        str q20, [x0], #0x40+        sub v30.8h, v9.8h, v0.8h+        add v27.8h, v9.8h, v0.8h+        mul v17.8h, v6.8h, v1.8h+        sqrdmulh v22.8h, v30.8h, v4.h[5]+        mul v18.8h, v16.8h, v1.8h+        mls v18.8h, v13.8h, v7.h[0]+        mul v2.8h, v30.8h, v4.h[4]+        mls v2.8h, v22.8h, v7.h[0]+        trn2 v22.4s, v23.4s, v31.4s+        sub v3.8h, v26.8h, v18.8h+        ldur q25, [x2, #-0x30]+        mls v17.8h, v29.8h, v7.h[0]+        trn2 v31.2d, v15.2d, v22.2d+        trn1 v20.2d, v15.2d, v22.2d+        add v16.8h, v26.8h, v18.8h+        sqrdmulh v26.8h, v3.8h, v19.8h+        trn1 v21.2d, v11.2d, v28.2d+        ldr q11, [x0, #0x40]+        sqrdmulh v29.8h, v16.8h, v25.8h+        stur q20, [x0, #-0x30]+        add v20.8h, v21.8h, v17.8h+        stur q31, [x0, #-0x10]+        mul v23.8h, v3.8h, v24.8h+        ldr q25, [x2, #0x10]+        sub v13.8h, v11.8h, v8.8h+        mls v23.8h, v26.8h, v7.h[0]+        ldur q1, [x2, #-0x40]+        sub v12.8h, v13.8h, v2.8h+        add v6.8h, v13.8h, v2.8h+        sqrdmulh v10.8h, v27.8h, v4.h[3]+        sub v30.8h, v21.8h, v17.8h+        mul v0.8h, v16.8h, v1.8h+        trn1 v28.4s, v6.4s, v12.4s+        sub x4, x4, #0x1+        cbnz x4, Lmlk_ntt_layer4567_start+        add v22.8h, v11.8h, v8.8h+        mul v27.8h, v27.8h, v4.h[2]+        trn2 v17.4s, v6.4s, v12.4s+        ldr q15, [x2], #0x60+        mls v27.8h, v10.8h, v7.h[0]+        add v4.8h, v30.8h, v23.8h+        sub v18.8h, v30.8h, v23.8h+        ldur q6, [x2, #-0x30]+        mls v0.8h, v29.8h, v7.h[0]+        ldur q12, [x2, #-0x40]+        ldur q24, [x2, #-0x20]+        ldur q2, [x2, #-0x10]+        trn1 v9.4s, v4.4s, v18.4s+        add v10.8h, v22.8h, v27.8h+        sub v13.8h, v22.8h, v27.8h+        sub v1.8h, v20.8h, v0.8h+        trn2 v21.4s, v10.4s, v13.4s+        add v27.8h, v20.8h, v0.8h+        trn2 v3.2d, v21.2d, v17.2d+        trn1 v13.4s, v10.4s, v13.4s+        trn1 v31.4s, v27.4s, v1.4s+        sqrdmulh v10.8h, v3.8h, v25.8h+        trn2 v5.2d, v13.2d, v28.2d+        trn1 v13.2d, v13.2d, v28.2d+        trn1 v21.2d, v21.2d, v17.2d+        sqrdmulh v17.8h, v5.8h, v25.8h+        trn2 v30.2d, v31.2d, v9.2d+        mul v25.8h, v3.8h, v15.8h+        str q30, [x0, #0x20]+        trn2 v30.4s, v4.4s, v18.4s+        mls v25.8h, v10.8h, v7.h[0]+        trn2 v3.4s, v27.4s, v1.4s+        mul v20.8h, v5.8h, v15.8h+        trn2 v10.2d, v3.2d, v30.2d+        mls v20.8h, v17.8h, v7.h[0]+        str q10, [x0, #0x30]+        sub v18.8h, v21.8h, v25.8h+        add v10.8h, v21.8h, v25.8h+        trn1 v3.2d, v3.2d, v30.2d+        sqrdmulh v30.8h, v18.8h, v2.8h+        mul v12.8h, v10.8h, v12.8h+        sqrdmulh v6.8h, v10.8h, v6.8h+        str q3, [x0, #0x10]+        add v21.8h, v13.8h, v20.8h+        mul v10.8h, v18.8h, v24.8h+        sub v13.8h, v13.8h, v20.8h+        mls v10.8h, v30.8h, v7.h[0]+        mls v12.8h, v6.8h, v7.h[0]+        trn1 v30.2d, v31.2d, v9.2d+        sub v3.8h, v13.8h, v10.8h+        add v6.8h, v13.8h, v10.8h+        add v10.8h, v21.8h, v12.8h+        sub v21.8h, v21.8h, v12.8h+        trn2 v13.4s, v6.4s, v3.4s+        trn1 v12.4s, v10.4s, v21.4s+        trn2 v21.4s, v10.4s, v21.4s+        trn1 v3.4s, v6.4s, v3.4s+        str q30, [x0], #0x40+        trn2 v10.2d, v21.2d, v13.2d+        trn1 v13.2d, v21.2d, v13.2d+        trn2 v21.2d, v12.2d, v3.2d+        trn1 v3.2d, v12.2d, v3.2d+        str q10, [x0, #0x30]+        str q13, [x0, #0x10]+        str q3, [x0], #0x40+        stur q21, [x0, #-0x20]+        ldp d8, d9, [sp]+        .cfi_restore d8+        .cfi_restore d9+        ldp d10, d11, [sp, #0x10]+        .cfi_restore d10+        .cfi_restore d11+        ldp d12, d13, [sp, #0x20]+        .cfi_restore d12+        .cfi_restore d13+        ldp d14, d15, [sp, #0x30]+        .cfi_restore d14+        .cfi_restore d15+        add sp, sp, #0x40+        .cfi_adjust_cfa_offset -0x40+        ret+        .cfi_endproc++MLK_ASM_FN_SIZE(ntt_aarch64_asm)++#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/aarch64/src/mlkem_poly_mulcache_compute_aarch64_asm.S view
@@ -0,0 +1,130 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*yaml+  Name: poly_mulcache_compute_aarch64_asm+  Description: Compute multiplication cache for polynomial+  Signature: void mlk_poly_mulcache_compute_aarch64_asm(int16_t cache[128], const int16_t mlk_poly[256], const int16_t zetas[128], const int16_t zetas_twisted[128])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 256+      permissions: write-only+      c_parameter: int16_t cache[128]+      description: Output cache+    x1:+      type: buffer+      size_bytes: 512+      permissions: read-only+      c_parameter: const int16_t mlk_poly[256]+      description: Input polynomial+    x2:+      type: buffer+      size_bytes: 256+      permissions: read-only+      c_parameter: const int16_t zetas[128]+      description: Zeta values+    x3:+      type: buffer+      size_bytes: 256+      permissions: read-only+      c_parameter: const int16_t zetas_twisted[128]+      description: Twisted zeta values+  Stack:+    bytes: 0+*/++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_AARCH64) &&  !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/aarch64_opt/src/mlkem_poly_mulcache_compute_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(poly_mulcache_compute_aarch64_asm)+MLK_ASM_FN_SYMBOL(poly_mulcache_compute_aarch64_asm)++        .cfi_startproc+        mov w5, #0xd01              // =3329+        dup v6.8h, w5+        mov w5, #0x4ebf             // =20159+        dup v7.8h, w5+        mov x4, #0x10               // =16+        ldr q0, [x1], #0x20+        ldur q2, [x1, #-0x10]+        ldr q19, [x1], #0x20+        ldr q29, [x3], #0x10+        ldur q16, [x1, #-0x10]+        ldr q18, [x2], #0x10+        ldr q26, [x1], #0x20+        ldr q25, [x2], #0x10+        uzp2 v5.8h, v0.8h, v2.8h+        ldr q28, [x3], #0x10+        ldur q7, [x1, #-0x10]+        ldr q2, [x1], #0x20+        uzp2 v27.8h, v19.8h, v16.8h+        sqrdmulh v16.8h, v5.8h, v29.8h+        ldr q17, [x3], #0x10+        ldr q19, [x3], #0x10+        mul v5.8h, v5.8h, v18.8h+        uzp2 v29.8h, v26.8h, v7.8h+        mul v26.8h, v27.8h, v25.8h+        sqrdmulh v4.8h, v27.8h, v28.8h+        mls v5.8h, v16.8h, v6.h[0]+        lsr x4, x4, #1+        sub x4, x4, #0x2++Lmlk_poly_mulcache_compute_loop_start:+        str q5, [x0], #0x10+        sqrdmulh v22.8h, v29.8h, v17.8h+        ldr q28, [x2], #0x10+        ldur q24, [x1, #-0x10]+        ldr q0, [x1], #0x20+        mls v26.8h, v4.8h, v6.h[0]+        ldur q16, [x1, #-0x10]+        ldr q17, [x3], #0x10+        mul v5.8h, v29.8h, v28.8h+        uzp2 v23.8h, v2.8h, v24.8h+        ldr q18, [x2], #0x10+        mls v5.8h, v22.8h, v6.h[0]+        uzp2 v29.8h, v0.8h, v16.8h+        sqrdmulh v4.8h, v23.8h, v19.8h+        ldr q2, [x1], #0x20+        ldr q19, [x3], #0x10+        str q26, [x0], #0x10+        mul v26.8h, v23.8h, v18.8h+        subs x4, x4, #0x1+        cbnz x4, Lmlk_poly_mulcache_compute_loop_start+        mls v26.8h, v4.8h, v6.h[0]+        str q5, [x0], #0x10+        ldr q5, [x2], #0x10+        ldur q4, [x1, #-0x10]+        sqrdmulh v16.8h, v29.8h, v17.8h+        ldr q0, [x2], #0x10+        mul v29.8h, v29.8h, v5.8h+        uzp2 v18.8h, v2.8h, v4.8h+        str q26, [x0], #0x10+        sqrdmulh v17.8h, v18.8h, v19.8h+        mls v29.8h, v16.8h, v6.h[0]+        mul v26.8h, v18.8h, v0.8h+        mls v26.8h, v17.8h, v6.h[0]+        str q29, [x0], #0x10+        str q26, [x0], #0x10+        ret+        .cfi_endproc++MLK_ASM_FN_SIZE(poly_mulcache_compute_aarch64_asm)++#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/aarch64/src/mlkem_poly_reduce_aarch64_asm.S view
@@ -0,0 +1,153 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*yaml+  Name: poly_reduce_aarch64_asm+  Description: Barrett reduction of polynomial coefficients+  Signature: void mlk_poly_reduce_aarch64_asm(int16_t p[256])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 512+      permissions: read/write+      c_parameter: int16_t p[256]+      description: Input/output polynomial+  Stack:+    bytes: 0+*/++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_AARCH64) &&  !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/aarch64_opt/src/mlkem_poly_reduce_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(poly_reduce_aarch64_asm)+MLK_ASM_FN_SYMBOL(poly_reduce_aarch64_asm)++        .cfi_startproc+        mov w2, #0xd01              // =3329+        dup v3.8h, w2+        mov w2, #0x4ebf             // =20159+        dup v4.8h, w2+        mov x1, #0x8                // =8+        ldr q21, [x0], #0x40+        ldur q18, [x0, #-0x20]+        ldur q0, [x0, #-0x30]+        ldur q5, [x0, #-0x10]+        ldr q26, [x0], #0x40+        sqdmulh v17.8h, v21.8h, v4.h[0]+        sqdmulh v27.8h, v18.8h, v4.h[0]+        sqdmulh v22.8h, v0.8h, v4.h[0]+        srshr v17.8h, v17.8h, #0xb+        sqdmulh v23.8h, v5.8h, v4.h[0]+        srshr v29.8h, v27.8h, #0xb+        mls v21.8h, v17.8h, v3.h[0]+        srshr v17.8h, v22.8h, #0xb+        mls v18.8h, v29.8h, v3.h[0]+        srshr v22.8h, v23.8h, #0xb+        mls v0.8h, v17.8h, v3.h[0]+        sshr v2.8h, v21.8h, #0xf+        mls v5.8h, v22.8h, v3.h[0]+        sshr v29.8h, v18.8h, #0xf+        and v19.16b, v3.16b, v2.16b+        sqdmulh v2.8h, v26.8h, v4.h[0]+        sshr v31.8h, v0.8h, #0xf+        add v17.8h, v21.8h, v19.8h+        and v21.16b, v3.16b, v29.16b+        and v31.16b, v3.16b, v31.16b+        sub x1, x1, #0x2++Lmlk_poly_reduce_loop_start:+        add v21.8h, v18.8h, v21.8h+        ldur q18, [x0, #-0x20]+        add v25.8h, v0.8h, v31.8h+        ldur q0, [x0, #-0x30]+        stur q21, [x0, #-0x60]+        sshr v28.8h, v5.8h, #0xf+        stur q17, [x0, #-0x80]+        srshr v23.8h, v2.8h, #0xb+        sqdmulh v30.8h, v18.8h, v4.h[0]+        stur q25, [x0, #-0x70]+        and v22.16b, v3.16b, v28.16b+        sqdmulh v7.8h, v0.8h, v4.h[0]+        add v16.8h, v5.8h, v22.8h+        ldur q5, [x0, #-0x10]+        mls v26.8h, v23.8h, v3.h[0]+        stur q16, [x0, #-0x50]+        srshr v6.8h, v30.8h, #0xb+        srshr v1.8h, v7.8h, #0xb+        sqdmulh v19.8h, v5.8h, v4.h[0]+        mls v18.8h, v6.8h, v3.h[0]+        sshr v24.8h, v26.8h, #0xf+        mls v0.8h, v1.8h, v3.h[0]+        and v27.16b, v3.16b, v24.16b+        srshr v29.8h, v19.8h, #0xb+        add v17.8h, v26.8h, v27.8h+        ldr q26, [x0], #0x40+        sshr v1.8h, v18.8h, #0xf+        mls v5.8h, v29.8h, v3.h[0]+        sshr v20.8h, v0.8h, #0xf+        and v21.16b, v3.16b, v1.16b+        and v31.16b, v3.16b, v20.16b+        sqdmulh v2.8h, v26.8h, v4.h[0]+        subs x1, x1, #0x1+        cbnz x1, Lmlk_poly_reduce_loop_start+        add v28.8h, v0.8h, v31.8h+        ldur q29, [x0, #-0x10]+        add v21.8h, v18.8h, v21.8h+        srshr v18.8h, v2.8h, #0xb+        sshr v2.8h, v5.8h, #0xf+        ldur q16, [x0, #-0x20]+        stur q17, [x0, #-0x80]+        ldur q0, [x0, #-0x30]+        and v2.16b, v3.16b, v2.16b+        sqdmulh v24.8h, v29.8h, v4.h[0]+        stur q28, [x0, #-0x70]+        stur q21, [x0, #-0x60]+        add v31.8h, v5.8h, v2.8h+        sqdmulh v6.8h, v16.8h, v4.h[0]+        stur q31, [x0, #-0x50]+        sqdmulh v17.8h, v0.8h, v4.h[0]+        srshr v22.8h, v24.8h, #0xb+        mls v26.8h, v18.8h, v3.h[0]+        srshr v31.8h, v6.8h, #0xb+        mls v29.8h, v22.8h, v3.h[0]+        srshr v19.8h, v17.8h, #0xb+        mls v16.8h, v31.8h, v3.h[0]+        sshr v7.8h, v26.8h, #0xf+        mls v0.8h, v19.8h, v3.h[0]+        and v5.16b, v3.16b, v7.16b+        sshr v22.8h, v29.8h, #0xf+        add v27.8h, v26.8h, v5.8h+        and v26.16b, v3.16b, v22.16b+        sshr v20.8h, v16.8h, #0xf+        stur q27, [x0, #-0x40]+        and v2.16b, v3.16b, v20.16b+        sshr v23.8h, v0.8h, #0xf+        add v18.8h, v29.8h, v26.8h+        add v31.8h, v16.8h, v2.8h+        and v29.16b, v3.16b, v23.16b+        stur q18, [x0, #-0x10]+        add v25.8h, v0.8h, v29.8h+        stur q31, [x0, #-0x20]+        stur q25, [x0, #-0x30]+        ret+        .cfi_endproc++MLK_ASM_FN_SIZE(poly_reduce_aarch64_asm)++#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/aarch64/src/mlkem_poly_tobytes_aarch64_asm.S view
@@ -0,0 +1,124 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*yaml+  Name: poly_tobytes_aarch64_asm+  Description: Convert polynomial to byte representation+  Signature: void mlk_poly_tobytes_aarch64_asm(uint8_t r[384], const int16_t a[256])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 384+      permissions: write-only+      c_parameter: uint8_t r[384]+      description: Output byte array+    x1:+      type: buffer+      size_bytes: 512+      permissions: read-only+      c_parameter: const int16_t a[256]+      description: Input polynomial+  Stack:+    bytes: 0+*/++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_AARCH64) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    (!defined(MLK_CONFIG_NO_KEYPAIR_API) || \+     !defined(MLK_CONFIG_NO_ENCAPS_API))++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/aarch64_opt/src/mlkem_poly_tobytes_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(poly_tobytes_aarch64_asm)+MLK_ASM_FN_SYMBOL(poly_tobytes_aarch64_asm)++        .cfi_startproc+        mov x2, #0x10               // =16+        ldr q5, [x1, #0x10]+        ldr q3, [x1], #0x20+        ldr q29, [x1], #0x20+        ldur q2, [x1, #-0x10]+        ldr q27, [x1, #0x10]+        ldr q23, [x1, #0x30]+        ldr q17, [x1], #0x20+        ldr q16, [x1], #0x20+        uzp2 v26.8h, v3.8h, v5.8h+        uzp1 v19.8h, v3.8h, v5.8h+        uzp2 v0.8h, v29.8h, v2.8h+        uzp1 v1.8h, v29.8h, v2.8h+        xtn v5.8b, v26.8h+        shrn v3.8b, v19.8h, #0x8+        shrn v4.8b, v26.8h, #0x4+        xtn v18.8b, v0.8h+        shrn v30.8b, v0.8h, #0x4+        xtn v28.8b, v1.8h+        shrn v29.8b, v1.8h, #0x8+        sli v3.8b, v5.8b, #0x4+        xtn v2.8b, v19.8h+        sli v29.8b, v18.8b, #0x4+        lsr x2, x2, #1+        sub x2, x2, #0x2++Lmlk_poly_tobytes_loop_start:+        uzp1 v25.8h, v17.8h, v27.8h+        uzp2 v31.8h, v17.8h, v27.8h+        uzp1 v24.8h, v16.8h, v23.8h+        uzp2 v6.8h, v16.8h, v23.8h+        st3 { v2.8b, v3.8b, v4.8b }, [x0], #24+        shrn v3.8b, v25.8h, #0x8+        ldr q17, [x1], #0x20+        shrn v4.8b, v31.8h, #0x4+        xtn v21.8b, v6.8h+        ldr q23, [x1, #0x10]+        st3 { v28.8b, v29.8b, v30.8b }, [x0], #24+        shrn v29.8b, v24.8h, #0x8+        ldur q27, [x1, #-0x10]+        xtn v20.8b, v31.8h+        ldr q16, [x1], #0x20+        sli v29.8b, v21.8b, #0x4+        xtn v2.8b, v25.8h+        sli v3.8b, v20.8b, #0x4+        xtn v28.8b, v24.8h+        shrn v30.8b, v6.8h, #0x4+        subs x2, x2, #0x1+        cbnz x2, Lmlk_poly_tobytes_loop_start+        uzp2 v7.8h, v17.8h, v27.8h+        uzp1 v25.8h, v17.8h, v27.8h+        uzp2 v0.8h, v16.8h, v23.8h+        st3 { v2.8b, v3.8b, v4.8b }, [x0], #24+        st3 { v28.8b, v29.8b, v30.8b }, [x0], #24+        shrn v21.8b, v25.8h, #0x8+        uzp1 v2.8h, v16.8h, v23.8h+        shrn v22.8b, v7.8h, #0x4+        shrn v4.8b, v0.8h, #0x4+        xtn v28.8b, v7.8h+        xtn v27.8b, v0.8h+        shrn v3.8b, v2.8h, #0x8+        sli v21.8b, v28.8b, #0x4+        xtn v2.8b, v2.8h+        sli v3.8b, v27.8b, #0x4+        xtn v20.8b, v25.8h+        st3 { v20.8b, v21.8b, v22.8b }, [x0], #24+        st3 { v2.8b, v3.8b, v4.8b }, [x0], #24+        ret+        .cfi_endproc++MLK_ASM_FN_SIZE(poly_tobytes_aarch64_asm)++#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED && \+          (!MLK_CONFIG_NO_KEYPAIR_API || !MLK_CONFIG_NO_ENCAPS_API) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/aarch64/src/mlkem_poly_tomont_aarch64_asm.S view
@@ -0,0 +1,102 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*yaml+  Name: poly_tomont_aarch64_asm+  Description: Convert polynomial to Montgomery domain+  Signature: void mlk_poly_tomont_aarch64_asm(int16_t p[256])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 512+      permissions: read/write+      c_parameter: int16_t p[256]+      description: Input/output polynomial+  Stack:+    bytes: 0+*/++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_AARCH64) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    !defined(MLK_CONFIG_NO_KEYPAIR_API)++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/aarch64_opt/src/mlkem_poly_tomont_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(poly_tomont_aarch64_asm)+MLK_ASM_FN_SYMBOL(poly_tomont_aarch64_asm)++        .cfi_startproc+        mov w2, #0xd01              // =3329+        dup v4.8h, w2+        mov w2, #-0x414             // =-1044+        dup v2.8h, w2+        mov w2, #-0x2824            // =-10276+        dup v3.8h, w2+        mov x1, #0x8                // =8+        ldr q18, [x0, #0x20]+        ldr q0, [x0, #0x10]+        ldr q16, [x0], #0x40+        sqrdmulh v23.8h, v0.8h, v3.8h+        mul v26.8h, v0.8h, v2.8h+        sqrdmulh v19.8h, v16.8h, v3.8h+        mls v26.8h, v23.8h, v4.h[0]+        mul v29.8h, v16.8h, v2.8h+        ldur q16, [x0, #-0x10]+        mls v29.8h, v19.8h, v4.h[0]+        stur q26, [x0, #-0x30]+        sqrdmulh v26.8h, v18.8h, v3.8h+        mul v18.8h, v18.8h, v2.8h+        stur q29, [x0, #-0x40]+        sqrdmulh v29.8h, v16.8h, v3.8h+        mls v18.8h, v26.8h, v4.h[0]+        sub x1, x1, #0x1++Lmlk_poly_tomont_loop:+        ldr q19, [x0, #0x10]+        mul v26.8h, v16.8h, v2.8h+        ldr q23, [x0, #0x20]+        ldr q17, [x0], #0x40+        mls v26.8h, v29.8h, v4.h[0]+        ldur q16, [x0, #-0x10]+        sqrdmulh v28.8h, v19.8h, v3.8h+        stur q18, [x0, #-0x60]+        mul v0.8h, v19.8h, v2.8h+        stur q26, [x0, #-0x50]+        sqrdmulh v24.8h, v23.8h, v3.8h+        mul v18.8h, v23.8h, v2.8h+        sqrdmulh v22.8h, v17.8h, v3.8h+        mul v26.8h, v17.8h, v2.8h+        mls v0.8h, v28.8h, v4.h[0]+        mls v26.8h, v22.8h, v4.h[0]+        sqrdmulh v29.8h, v16.8h, v3.8h+        stur q0, [x0, #-0x30]+        mls v18.8h, v24.8h, v4.h[0]+        stur q26, [x0, #-0x40]+        sub x1, x1, #0x1+        cbnz x1, Lmlk_poly_tomont_loop+        mul v16.8h, v16.8h, v2.8h+        stur q18, [x0, #-0x20]+        mls v16.8h, v29.8h, v4.h[0]+        stur q16, [x0, #-0x10]+        ret+        .cfi_endproc++MLK_ASM_FN_SIZE(poly_tomont_aarch64_asm)++#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED && \+          !MLK_CONFIG_NO_KEYPAIR_API */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm.S view
@@ -0,0 +1,264 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [NeonNTT]+ *   Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1+ *   Becker, Hwang, Kannwischer, Yang, Yang+ *   https://eprint.iacr.org/2021/986+ */++/*yaml+  Name: polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm+  Description: Re-implementation of asymmetric base multiplication following @[NeonNTT] for k=2+  Signature: void mlk_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm(int16_t r[256], const int16_t a[512], const int16_t b[512], const int16_t b_cache[256])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 512+      permissions: write-only+      c_parameter: int16_t r[256]+      description: Output polynomial+    x1:+      type: buffer+      size_bytes: 1024+      permissions: read-only+      c_parameter: const int16_t a[512]+      description: Input polynomial vector a+    x2:+      type: buffer+      size_bytes: 1024+      permissions: read-only+      c_parameter: const int16_t b[512]+      description: Input polynomial vector b+    x3:+      type: buffer+      size_bytes: 512+      permissions: read-only+      c_parameter: const int16_t b_cache[256]+      description: Cached values for b+  Stack:+    bytes: 64+    description: saving callee-saved Neon registers+*/++/* Re-implementation of asymmetric base multiplication following @[NeonNTT] */++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_AARCH64) &&  !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) &&  (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2)++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/aarch64_opt/src/mlkem_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm)+MLK_ASM_FN_SYMBOL(polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0x40+        .cfi_adjust_cfa_offset 0x40+        stp d8, d9, [sp]+        .cfi_rel_offset d8, 0x0+        .cfi_rel_offset d9, 0x8+        stp d10, d11, [sp, #0x10]+        .cfi_rel_offset d10, 0x10+        .cfi_rel_offset d11, 0x18+        stp d12, d13, [sp, #0x20]+        .cfi_rel_offset d12, 0x20+        .cfi_rel_offset d13, 0x28+        stp d14, d15, [sp, #0x30]+        .cfi_rel_offset d14, 0x30+        .cfi_rel_offset d15, 0x38+        mov w14, #0xd01             // =3329+        dup v0.8h, w14+        mov w14, #0xcff             // =3327+        dup v2.8h, w14+        add x4, x1, #0x200+        add x5, x2, #0x200+        add x6, x3, #0x100+        mov x13, #0x10              // =16+        ldr q12, [x1], #0x20+        ldur q9, [x1, #-0x10]+        ldr q22, [x2], #0x20+        ldur q30, [x2, #-0x10]+        ldr q6, [x5], #0x20+        ldr q7, [x4, #0x10]+        ldr q8, [x4], #0x20+        ldur q23, [x5, #-0x10]+        uzp1 v16.8h, v12.8h, v9.8h+        uzp2 v14.8h, v12.8h, v9.8h+        uzp2 v13.8h, v22.8h, v30.8h+        uzp1 v18.8h, v22.8h, v30.8h+        ld1 { v27.8h }, [x3], #16+        ld1 { v17.8h }, [x6], #16+        smull2 v4.4s, v16.8h, v18.8h+        ldr q31, [x1, #0x10]+        smull v19.4s, v16.4h, v13.4h+        ldr q24, [x1], #0x20+        smlal v19.4s, v14.4h, v18.4h+        ldr q22, [x2], #0x20+        smlal2 v4.4s, v14.8h, v27.8h+        uzp2 v5.8h, v6.8h, v23.8h+        smull2 v29.4s, v16.8h, v13.8h+        uzp2 v26.8h, v8.8h, v7.8h+        smlal2 v29.4s, v14.8h, v18.8h+        uzp1 v30.8h, v24.8h, v31.8h+        uzp1 v8.8h, v8.8h, v7.8h+        smull v11.4s, v16.4h, v18.4h+        smlal v11.4s, v14.4h, v27.4h+        ldur q1, [x2, #-0x10]+        uzp1 v28.8h, v6.8h, v23.8h+        smlal2 v29.4s, v8.8h, v5.8h+        ldr q25, [x5], #0x20+        smlal v19.4s, v8.4h, v5.4h+        ldr q3, [x4, #0x10]+        smlal2 v29.4s, v26.8h, v28.8h+        uzp1 v27.8h, v22.8h, v1.8h+        smlal v19.4s, v26.4h, v28.4h+        ldr q12, [x4], #0x20+        smlal2 v4.4s, v8.8h, v28.8h+        ldur q21, [x5, #-0x10]+        smlal2 v4.4s, v26.8h, v17.8h+        smlal v11.4s, v8.4h, v28.4h+        ld1 { v15.8h }, [x6], #16+        smlal v11.4s, v26.4h, v17.4h+        ld1 { v20.8h }, [x3], #16+        uzp1 v28.8h, v19.8h, v29.8h+        smull2 v23.4s, v30.8h, v27.8h+        smull v26.4s, v30.4h, v27.4h+        uzp2 v16.8h, v22.8h, v1.8h+        mul v28.8h, v28.8h, v2.8h+        uzp1 v10.8h, v11.8h, v4.8h+        smull2 v8.4s, v30.8h, v16.8h+        mul v13.8h, v10.8h, v2.8h+        smlal v19.4s, v28.4h, v0.4h+        smlal2 v29.4s, v28.8h, v0.8h+        smull v18.4s, v30.4h, v16.4h+        uzp1 v30.8h, v25.8h, v21.8h+        smlal v11.4s, v13.4h, v0.4h+        uzp2 v6.8h, v24.8h, v31.8h+        uzp1 v16.8h, v12.8h, v3.8h+        smlal2 v4.4s, v13.8h, v0.8h+        uzp2 v17.8h, v25.8h, v21.8h+        smlal2 v8.4s, v6.8h, v27.8h+        uzp2 v12.8h, v12.8h, v3.8h+        smlal v18.4s, v6.4h, v27.4h+        uzp2 v9.8h, v19.8h, v29.8h+        smlal2 v8.4s, v16.8h, v17.8h+        smlal2 v8.4s, v12.8h, v30.8h+        uzp2 v19.8h, v11.8h, v4.8h+        sub x13, x13, #0x2++Lmlk_polyvec_basemul_acc_montgomery_cached_k2_loop_start:+        smlal v18.4s, v16.4h, v17.4h+        ldr q7, [x4], #0x20+        ldr q10, [x2, #0x10]+        smlal v18.4s, v12.4h, v30.4h+        smlal2 v23.4s, v6.8h, v20.8h+        ldr q14, [x2], #0x20+        smlal2 v23.4s, v16.8h, v30.8h+        zip1 v25.8h, v19.8h, v9.8h+        zip2 v3.8h, v19.8h, v9.8h+        smlal2 v23.4s, v12.8h, v15.8h+        smlal v26.4s, v6.4h, v20.4h+        uzp1 v5.8h, v18.8h, v8.8h+        uzp2 v21.8h, v14.8h, v10.8h+        smlal v26.4s, v16.4h, v30.4h+        str q25, [x0], #0x20+        mul v29.8h, v5.8h, v2.8h+        uzp1 v24.8h, v14.8h, v10.8h+        stur q3, [x0, #-0x10]+        smlal v26.4s, v12.4h, v15.4h+        ld1 { v15.8h }, [x6], #16+        ldr q28, [x1, #0x10]+        ldr q11, [x1], #0x20+        ldr q13, [x5], #0x20+        ldur q27, [x4, #-0x10]+        smlal2 v8.4s, v29.8h, v0.8h+        ldur q22, [x5, #-0x10]+        smlal v18.4s, v29.4h, v0.4h+        uzp1 v4.8h, v26.8h, v23.8h+        uzp1 v1.8h, v11.8h, v28.8h+        uzp2 v6.8h, v11.8h, v28.8h+        uzp1 v16.8h, v7.8h, v27.8h+        mul v31.8h, v4.8h, v2.8h+        uzp2 v17.8h, v13.8h, v22.8h+        ld1 { v20.8h }, [x3], #16+        uzp2 v9.8h, v18.8h, v8.8h+        smull2 v8.4s, v1.8h, v21.8h+        uzp1 v30.8h, v13.8h, v22.8h+        smlal2 v8.4s, v6.8h, v24.8h+        smlal2 v8.4s, v16.8h, v17.8h+        uzp2 v12.8h, v7.8h, v27.8h+        smlal v26.4s, v31.4h, v0.4h+        smlal2 v23.4s, v31.8h, v0.8h+        smull v18.4s, v1.4h, v21.4h+        smlal v18.4s, v6.4h, v24.4h+        smlal2 v8.4s, v12.8h, v30.8h+        uzp2 v19.8h, v26.8h, v23.8h+        smull2 v23.4s, v1.8h, v24.8h+        smull v26.4s, v1.4h, v24.4h+        subs x13, x13, #0x1+        cbnz x13, Lmlk_polyvec_basemul_acc_montgomery_cached_k2_loop_start+        smlal v26.4s, v6.4h, v20.4h+        smlal2 v23.4s, v6.8h, v20.8h+        smlal v26.4s, v16.4h, v30.4h+        smlal2 v23.4s, v16.8h, v30.8h+        smlal v26.4s, v12.4h, v15.4h+        smlal2 v23.4s, v12.8h, v15.8h+        smlal v18.4s, v16.4h, v17.4h+        smlal v18.4s, v12.4h, v30.4h+        zip1 v12.8h, v19.8h, v9.8h+        str q12, [x0], #0x20+        uzp1 v12.8h, v26.8h, v23.8h+        mul v6.8h, v12.8h, v2.8h+        uzp1 v12.8h, v18.8h, v8.8h+        mul v12.8h, v12.8h, v2.8h+        smlal v26.4s, v6.4h, v0.4h+        smlal2 v23.4s, v6.8h, v0.8h+        smlal2 v8.4s, v12.8h, v0.8h+        smlal v18.4s, v12.4h, v0.4h+        zip2 v12.8h, v19.8h, v9.8h+        uzp2 v6.8h, v26.8h, v23.8h+        stur q12, [x0, #-0x10]+        uzp2 v12.8h, v18.8h, v8.8h+        zip2 v1.8h, v6.8h, v12.8h+        zip1 v12.8h, v6.8h, v12.8h+        str q1, [x0, #0x10]+        str q12, [x0], #0x20+        ldp d8, d9, [sp]+        .cfi_restore d8+        .cfi_restore d9+        ldp d10, d11, [sp, #0x10]+        .cfi_restore d10+        .cfi_restore d11+        ldp d12, d13, [sp, #0x20]+        .cfi_restore d12+        .cfi_restore d13+        ldp d14, d15, [sp, #0x30]+        .cfi_restore d14+        .cfi_restore d15+        add sp, sp, #0x40+        .cfi_adjust_cfa_offset -0x40+        ret+        .cfi_endproc++MLK_ASM_FN_SIZE(polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm)+++#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm.S view
@@ -0,0 +1,317 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [NeonNTT]+ *   Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1+ *   Becker, Hwang, Kannwischer, Yang, Yang+ *   https://eprint.iacr.org/2021/986+ */++/*yaml+  Name: polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm+  Description: Re-implementation of asymmetric base multiplication following @[NeonNTT] for k=3+  Signature: void mlk_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm(int16_t r[256], const int16_t a[768], const int16_t b[768], const int16_t b_cache[384])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 512+      permissions: write-only+      c_parameter: int16_t r[256]+      description: Output polynomial+    x1:+      type: buffer+      size_bytes: 1536+      permissions: read-only+      c_parameter: const int16_t a[768]+      description: Input polynomial vector a+    x2:+      type: buffer+      size_bytes: 1536+      permissions: read-only+      c_parameter: const int16_t b[768]+      description: Input polynomial vector b+    x3:+      type: buffer+      size_bytes: 768+      permissions: read-only+      c_parameter: const int16_t b_cache[384]+      description: Cached values for b+  Stack:+    bytes: 64+    description: saving callee-saved Neon registers+*/++/* Re-implementation of asymmetric base multiplication following @[NeonNTT] */++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_AARCH64) &&  !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) &&  (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 3)++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/aarch64_opt/src/mlkem_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm)+MLK_ASM_FN_SYMBOL(polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0x40+        .cfi_adjust_cfa_offset 0x40+        stp d8, d9, [sp]+        .cfi_rel_offset d8, 0x0+        .cfi_rel_offset d9, 0x8+        stp d10, d11, [sp, #0x10]+        .cfi_rel_offset d10, 0x10+        .cfi_rel_offset d11, 0x18+        stp d12, d13, [sp, #0x20]+        .cfi_rel_offset d12, 0x20+        .cfi_rel_offset d13, 0x28+        stp d14, d15, [sp, #0x30]+        .cfi_rel_offset d14, 0x30+        .cfi_rel_offset d15, 0x38+        mov w14, #0xd01             // =3329+        dup v0.8h, w14+        mov w14, #0xcff             // =3327+        dup v2.8h, w14+        add x4, x1, #0x200+        add x5, x2, #0x200+        add x6, x3, #0x100+        add x7, x1, #0x400+        add x8, x2, #0x400+        add x9, x3, #0x200+        mov x13, #0x10              // =16+        ldr q6, [x7], #0x20+        ldr q19, [x2, #0x10]+        ldr q23, [x1], #0x20+        ldur q14, [x1, #-0x10]+        ldr q17, [x2], #0x20+        ldr q11, [x4, #0x10]+        ldur q28, [x7, #-0x10]+        ld1 { v30.8h }, [x3], #16+        ldr q26, [x4], #0x20+        ldr q16, [x8, #0x10]+        uzp1 v8.8h, v23.8h, v14.8h+        ldr q22, [x5, #0x10]+        ldr q18, [x5], #0x20+        uzp1 v20.8h, v17.8h, v19.8h+        uzp2 v24.8h, v23.8h, v14.8h+        ldr q31, [x8], #0x20+        smull2 v4.4s, v8.8h, v20.8h+        uzp1 v25.8h, v26.8h, v11.8h+        smull v13.4s, v8.4h, v20.4h+        ld1 { v23.8h }, [x6], #16+        uzp1 v1.8h, v18.8h, v22.8h+        smlal v13.4s, v24.4h, v30.4h+        smlal2 v4.4s, v24.8h, v30.8h+        uzp2 v5.8h, v26.8h, v11.8h+        smlal2 v4.4s, v25.8h, v1.8h+        uzp1 v29.8h, v6.8h, v28.8h+        smlal2 v4.4s, v5.8h, v23.8h+        ld1 { v7.8h }, [x9], #16+        smlal v13.4s, v25.4h, v1.4h+        uzp2 v17.8h, v17.8h, v19.8h+        uzp1 v27.8h, v31.8h, v16.8h+        smlal v13.4s, v5.4h, v23.4h+        uzp2 v22.8h, v18.8h, v22.8h+        smull v18.4s, v8.4h, v17.4h+        uzp2 v28.8h, v6.8h, v28.8h+        smlal v13.4s, v29.4h, v27.4h+        smlal2 v4.4s, v29.8h, v27.8h+        uzp2 v26.8h, v31.8h, v16.8h+        smlal2 v4.4s, v28.8h, v7.8h+        ldr q3, [x7, #0x10]+        smlal v13.4s, v28.4h, v7.4h+        ldr q7, [x1], #0x20+        smlal v18.4s, v24.4h, v20.4h+        ldr q15, [x2], #0x20+        smlal v18.4s, v25.4h, v22.4h+        smull2 v8.4s, v8.8h, v17.8h+        ldur q17, [x1, #-0x10]+        uzp1 v23.8h, v13.8h, v4.8h+        smlal v18.4s, v5.4h, v1.4h+        smlal2 v8.4s, v24.8h, v20.8h+        ld1 { v16.8h }, [x3], #16+        mul v23.8h, v23.8h, v2.8h+        ldr q19, [x5, #0x10]+        ldr q14, [x4, #0x10]+        ldr q11, [x4], #0x20+        ldur q20, [x2, #-0x10]+        smlal2 v8.4s, v25.8h, v22.8h+        smlal2 v8.4s, v5.8h, v1.8h+        ldr q22, [x5], #0x20+        uzp1 v1.8h, v7.8h, v17.8h+        smlal v18.4s, v29.4h, v26.4h+        smlal v13.4s, v23.4h, v0.4h+        uzp2 v31.8h, v11.8h, v14.8h+        uzp1 v21.8h, v15.8h, v20.8h+        smlal2 v4.4s, v23.8h, v0.8h+        ld1 { v9.8h }, [x6], #16+        smlal v18.4s, v28.4h, v27.4h+        smlal2 v8.4s, v29.8h, v26.8h+        ldr q25, [x7], #0x20+        smull v26.4s, v1.4h, v21.4h+        uzp1 v24.8h, v22.8h, v19.8h+        smlal2 v8.4s, v28.8h, v27.8h+        uzp2 v28.8h, v7.8h, v17.8h+        uzp1 v29.8h, v11.8h, v14.8h+        smull2 v23.4s, v1.8h, v21.8h+        ldr q27, [x8], #0x20+        smlal2 v23.4s, v28.8h, v16.8h+        ldur q11, [x8, #-0x10]+        smlal2 v23.4s, v29.8h, v24.8h+        uzp2 v7.8h, v13.8h, v4.8h+        uzp2 v19.8h, v22.8h, v19.8h+        ld1 { v4.8h }, [x9], #16+        smlal2 v23.4s, v31.8h, v9.8h+        uzp1 v13.8h, v25.8h, v3.8h+        uzp1 v14.8h, v18.8h, v8.8h+        smlal v26.4s, v28.4h, v16.4h+        uzp2 v17.8h, v27.8h, v11.8h+        uzp2 v20.8h, v15.8h, v20.8h+        mul v14.8h, v14.8h, v2.8h+        sub x13, x13, #0x2++Lmlk_polyvec_basemul_acc_montgomery_cached_k3_loop_start:+        uzp1 v6.8h, v27.8h, v11.8h+        smlal v26.4s, v29.4h, v24.4h+        uzp2 v16.8h, v25.8h, v3.8h+        smlal v26.4s, v31.4h, v9.4h+        ldr q3, [x7, #0x10]+        smlal v26.4s, v13.4h, v6.4h+        smlal2 v8.4s, v14.8h, v0.8h+        ldr q27, [x8], #0x20+        smlal v18.4s, v14.4h, v0.4h+        ldr q25, [x7], #0x20+        smlal2 v23.4s, v13.8h, v6.8h+        ldr q11, [x1], #0x20+        smlal2 v23.4s, v16.8h, v4.8h+        smlal v26.4s, v16.4h, v4.4h+        ldur q22, [x1, #-0x10]+        uzp2 v30.8h, v18.8h, v8.8h+        smull v18.4s, v1.4h, v20.4h+        smlal v18.4s, v28.4h, v21.4h+        ldr q14, [x2], #0x20+        smlal v18.4s, v29.4h, v19.4h+        zip1 v5.8h, v7.8h, v30.8h+        uzp1 v4.8h, v26.8h, v23.8h+        smull2 v8.4s, v1.8h, v20.8h+        zip2 v10.8h, v7.8h, v30.8h+        smlal v18.4s, v31.4h, v24.4h+        mul v12.8h, v4.8h, v2.8h+        ldr q4, [x5, #0x10]+        ldr q20, [x4, #0x10]+        ldr q1, [x4], #0x20+        ldur q30, [x2, #-0x10]+        smlal2 v8.4s, v28.8h, v21.8h+        smlal2 v8.4s, v29.8h, v19.8h+        ldr q19, [x5], #0x20+        smlal2 v8.4s, v31.8h, v24.8h+        ld1 { v15.8h }, [x3], #16+        uzp2 v31.8h, v1.8h, v20.8h+        smlal v26.4s, v12.4h, v0.4h+        smlal2 v23.4s, v12.8h, v0.8h+        uzp1 v21.8h, v14.8h, v30.8h+        uzp1 v29.8h, v1.8h, v20.8h+        uzp1 v1.8h, v11.8h, v22.8h+        smlal2 v8.4s, v13.8h, v17.8h+        ld1 { v9.8h }, [x6], #16+        smlal v18.4s, v13.4h, v17.4h+        uzp1 v24.8h, v19.8h, v4.8h+        uzp2 v7.8h, v26.8h, v23.8h+        smull v26.4s, v1.4h, v21.4h+        smlal v18.4s, v16.4h, v6.4h+        uzp2 v19.8h, v19.8h, v4.8h+        smlal2 v8.4s, v16.8h, v6.8h+        uzp2 v28.8h, v11.8h, v22.8h+        smull2 v23.4s, v1.8h, v21.8h+        uzp1 v13.8h, v25.8h, v3.8h+        smlal2 v23.4s, v28.8h, v15.8h+        ldur q11, [x8, #-0x10]+        smlal2 v23.4s, v29.8h, v24.8h+        ld1 { v4.8h }, [x9], #16+        smlal2 v23.4s, v31.8h, v9.8h+        uzp1 v12.8h, v18.8h, v8.8h+        uzp2 v20.8h, v14.8h, v30.8h+        smlal v26.4s, v28.4h, v15.4h+        str q5, [x0], #0x20+        mul v14.8h, v12.8h, v2.8h+        stur q10, [x0, #-0x10]+        uzp2 v17.8h, v27.8h, v11.8h+        subs x13, x13, #0x1+        cbnz x13, Lmlk_polyvec_basemul_acc_montgomery_cached_k3_loop_start+        uzp2 v3.8h, v25.8h, v3.8h+        smull2 v16.4s, v1.8h, v20.8h+        smull v25.4s, v1.4h, v20.4h+        uzp1 v22.8h, v27.8h, v11.8h+        smlal2 v16.4s, v28.8h, v21.8h+        smlal v25.4s, v28.4h, v21.4h+        smlal2 v16.4s, v29.8h, v19.8h+        smlal v25.4s, v29.4h, v19.4h+        smlal2 v16.4s, v31.8h, v24.8h+        smlal v25.4s, v31.4h, v24.4h+        smlal v25.4s, v13.4h, v17.4h+        smlal2 v16.4s, v13.8h, v17.8h+        smlal2 v16.4s, v3.8h, v22.8h+        smlal v25.4s, v3.4h, v22.4h+        smlal2 v23.4s, v13.8h, v22.8h+        smlal v26.4s, v29.4h, v24.4h+        smlal v26.4s, v31.4h, v9.4h+        smlal v26.4s, v13.4h, v22.4h+        uzp1 v10.8h, v25.8h, v16.8h+        smlal2 v23.4s, v3.8h, v4.8h+        smlal v26.4s, v3.4h, v4.4h+        mul v13.8h, v10.8h, v2.8h+        smlal v18.4s, v14.4h, v0.4h+        smlal2 v8.4s, v14.8h, v0.8h+        uzp1 v3.8h, v26.8h, v23.8h+        mul v24.8h, v3.8h, v2.8h+        uzp2 v17.8h, v18.8h, v8.8h+        smlal v25.4s, v13.4h, v0.4h+        smlal2 v16.4s, v13.8h, v0.8h+        zip1 v21.8h, v7.8h, v17.8h+        zip2 v20.8h, v7.8h, v17.8h+        smlal2 v23.4s, v24.8h, v0.8h+        str q21, [x0], #0x20+        smlal v26.4s, v24.4h, v0.4h+        uzp2 v13.8h, v25.8h, v16.8h+        stur q20, [x0, #-0x10]+        uzp2 v23.8h, v26.8h, v23.8h+        zip1 v18.8h, v23.8h, v13.8h+        zip2 v13.8h, v23.8h, v13.8h+        str q18, [x0], #0x20+        stur q13, [x0, #-0x10]+        ldp d8, d9, [sp]+        .cfi_restore d8+        .cfi_restore d9+        ldp d10, d11, [sp, #0x10]+        .cfi_restore d10+        .cfi_restore d11+        ldp d12, d13, [sp, #0x20]+        .cfi_restore d12+        .cfi_restore d13+        ldp d14, d15, [sp, #0x30]+        .cfi_restore d14+        .cfi_restore d15+        add sp, sp, #0x40+        .cfi_adjust_cfa_offset -0x40+        ret+        .cfi_endproc++MLK_ASM_FN_SIZE(polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm)++#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 3) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm.S view
@@ -0,0 +1,371 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [NeonNTT]+ *   Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1+ *   Becker, Hwang, Kannwischer, Yang, Yang+ *   https://eprint.iacr.org/2021/986+ */++/*yaml+  Name: polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm+  Description: Re-implementation of asymmetric base multiplication following @[NeonNTT] for k=4+  Signature: void mlk_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm(int16_t r[256], const int16_t a[1024], const int16_t b[1024], const int16_t b_cache[512])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 512+      permissions: write-only+      c_parameter: int16_t r[256]+      description: Output polynomial+    x1:+      type: buffer+      size_bytes: 2048+      permissions: read-only+      c_parameter: const int16_t a[1024]+      description: Input polynomial vector a+    x2:+      type: buffer+      size_bytes: 2048+      permissions: read-only+      c_parameter: const int16_t b[1024]+      description: Input polynomial vector b+    x3:+      type: buffer+      size_bytes: 1024+      permissions: read-only+      c_parameter: const int16_t b_cache[512]+      description: Cached values for b+  Stack:+    bytes: 64+    description: saving callee-saved Neon registers+*/++/* Re-implementation of asymmetric base multiplication following @[NeonNTT] */++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_AARCH64) &&  !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) &&  (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4)++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/aarch64_opt/src/mlkem_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm)+MLK_ASM_FN_SYMBOL(polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0x40+        .cfi_adjust_cfa_offset 0x40+        stp d8, d9, [sp]+        .cfi_rel_offset d8, 0x0+        .cfi_rel_offset d9, 0x8+        stp d10, d11, [sp, #0x10]+        .cfi_rel_offset d10, 0x10+        .cfi_rel_offset d11, 0x18+        stp d12, d13, [sp, #0x20]+        .cfi_rel_offset d12, 0x20+        .cfi_rel_offset d13, 0x28+        stp d14, d15, [sp, #0x30]+        .cfi_rel_offset d14, 0x30+        .cfi_rel_offset d15, 0x38+        mov w14, #0xd01             // =3329+        dup v0.8h, w14+        mov w14, #0xcff             // =3327+        dup v2.8h, w14+        add x4, x1, #0x200+        add x5, x2, #0x200+        add x6, x3, #0x100+        add x7, x1, #0x400+        add x8, x2, #0x400+        add x9, x3, #0x200+        add x10, x1, #0x600+        add x11, x2, #0x600+        add x12, x3, #0x300+        mov x13, #0x10              // =16+        ldr q28, [x1], #0x20+        ldur q5, [x1, #-0x10]+        ldr q31, [x2], #0x20+        ldur q27, [x2, #-0x10]+        ldr q7, [x5], #0x20+        ldr q10, [x4], #0x20+        ldur q18, [x5, #-0x10]+        ldur q9, [x4, #-0x10]+        uzp1 v11.8h, v28.8h, v5.8h+        uzp2 v19.8h, v28.8h, v5.8h+        uzp2 v4.8h, v31.8h, v27.8h+        uzp1 v1.8h, v31.8h, v27.8h+        ldr q29, [x7], #0x20+        ldr q28, [x8, #0x10]+        uzp1 v24.8h, v10.8h, v9.8h+        uzp1 v17.8h, v7.8h, v18.8h+        uzp2 v7.8h, v7.8h, v18.8h+        ldr q21, [x8], #0x20+        uzp2 v27.8h, v10.8h, v9.8h+        ldur q6, [x7, #-0x10]+        smull v18.4s, v11.4h, v4.4h+        ld1 { v9.8h }, [x3], #16+        smull2 v8.4s, v11.8h, v4.8h+        ldr q16, [x11], #0x20+        smlal2 v8.4s, v19.8h, v1.8h+        ldur q14, [x11, #-0x10]+        smlal v18.4s, v19.4h, v1.4h+        uzp1 v10.8h, v21.8h, v28.8h+        smlal v18.4s, v24.4h, v7.4h+        ldr q4, [x10], #0x20+        smlal2 v8.4s, v24.8h, v7.8h+        ld1 { v12.8h }, [x6], #16+        smull2 v23.4s, v11.8h, v1.8h+        uzp2 v13.8h, v29.8h, v6.8h+        smull v26.4s, v11.4h, v1.4h+        uzp1 v29.8h, v29.8h, v6.8h+        smlal v26.4s, v19.4h, v9.4h+        ldur q15, [x10, #-0x10]+        smlal2 v23.4s, v19.8h, v9.8h+        uzp2 v9.8h, v21.8h, v28.8h+        smlal v18.4s, v27.4h, v17.4h+        uzp2 v6.8h, v16.8h, v14.8h+        uzp1 v21.8h, v16.8h, v14.8h+        smlal2 v8.4s, v27.8h, v17.8h+        smlal2 v8.4s, v29.8h, v9.8h+        uzp1 v30.8h, v4.8h, v15.8h+        uzp2 v16.8h, v4.8h, v15.8h+        smlal v18.4s, v29.4h, v9.4h+        smlal2 v8.4s, v13.8h, v10.8h+        ld1 { v15.8h }, [x9], #16+        smlal v18.4s, v13.4h, v10.4h+        ldr q11, [x4], #0x20+        smlal v18.4s, v30.4h, v6.4h+        ldr q7, [x2], #0x20+        smlal2 v8.4s, v30.8h, v6.8h+        ld1 { v9.8h }, [x12], #16+        smlal2 v23.4s, v24.8h, v17.8h+        ldur q4, [x2, #-0x10]+        smlal v26.4s, v24.4h, v17.4h+        ldur q25, [x4, #-0x10]+        smlal2 v8.4s, v16.8h, v21.8h+        ldr q5, [x5], #0x20+        smlal v18.4s, v16.4h, v21.4h+        ldur q22, [x5, #-0x10]+        smlal v26.4s, v27.4h, v12.4h+        ldr q19, [x1, #0x10]+        smlal v26.4s, v29.4h, v10.4h+        ld1 { v20.8h }, [x3], #16+        smlal v26.4s, v13.4h, v15.4h+        uzp1 v24.8h, v7.8h, v4.8h+        smlal2 v23.4s, v27.8h, v12.8h+        uzp1 v28.8h, v18.8h, v8.8h+        smlal v26.4s, v30.4h, v21.4h+        uzp2 v27.8h, v11.8h, v25.8h+        smlal2 v23.4s, v29.8h, v10.8h+        uzp2 v31.8h, v7.8h, v4.8h+        smlal2 v23.4s, v13.8h, v15.8h+        uzp1 v14.8h, v5.8h, v22.8h+        uzp1 v17.8h, v11.8h, v25.8h+        smlal v26.4s, v16.4h, v9.4h+        mul v29.8h, v28.8h, v2.8h+        sub x13, x13, #0x2++Lmlk_polyvec_basemul_acc_montgomery_cached_k4_loop_start:+        smlal2 v23.4s, v30.8h, v21.8h+        ldr q11, [x1], #0x20+        uzp2 v15.8h, v5.8h, v22.8h+        smlal v18.4s, v29.4h, v0.4h+        ldr q12, [x7], #0x20+        smlal2 v8.4s, v29.8h, v0.8h+        ldur q3, [x7, #-0x10]+        ldr q21, [x8], #0x20+        uzp1 v29.8h, v11.8h, v19.8h+        ldur q13, [x8, #-0x10]+        uzp2 v5.8h, v11.8h, v19.8h+        smlal2 v23.4s, v16.8h, v9.8h+        uzp2 v28.8h, v18.8h, v8.8h+        smull2 v8.4s, v29.8h, v31.8h+        smlal2 v8.4s, v5.8h, v24.8h+        uzp1 v7.8h, v12.8h, v3.8h+        smlal2 v8.4s, v17.8h, v15.8h+        uzp2 v11.8h, v21.8h, v13.8h+        uzp1 v4.8h, v26.8h, v23.8h+        smlal2 v8.4s, v27.8h, v14.8h+        smlal2 v8.4s, v7.8h, v11.8h+        mul v6.8h, v4.8h, v2.8h+        ldr q19, [x11], #0x20+        uzp2 v25.8h, v12.8h, v3.8h+        ldr q12, [x10], #0x20+        smull v18.4s, v29.4h, v31.4h+        ldur q3, [x10, #-0x10]+        smlal v18.4s, v5.4h, v24.4h+        uzp1 v4.8h, v21.8h, v13.8h+        smlal v18.4s, v17.4h, v15.4h+        ldur q13, [x11, #-0x10]+        ld1 { v1.8h }, [x6], #16+        smlal v26.4s, v6.4h, v0.4h+        smlal2 v23.4s, v6.8h, v0.8h+        ld1 { v10.8h }, [x9], #16+        smlal v18.4s, v27.4h, v14.4h+        uzp1 v30.8h, v12.8h, v3.8h+        smlal2 v8.4s, v25.8h, v4.8h+        uzp2 v31.8h, v19.8h, v13.8h+        smlal v18.4s, v7.4h, v11.4h+        ld1 { v9.8h }, [x12], #16+        smlal v18.4s, v25.4h, v4.4h+        uzp1 v21.8h, v19.8h, v13.8h+        uzp2 v16.8h, v12.8h, v3.8h+        smlal v18.4s, v30.4h, v31.4h+        smlal2 v8.4s, v30.8h, v31.8h+        uzp2 v31.8h, v26.8h, v23.8h+        smlal2 v8.4s, v16.8h, v21.8h+        smlal v18.4s, v16.4h, v21.4h+        zip1 v15.8h, v31.8h, v28.8h+        ldr q19, [x1, #0x10]+        smull2 v23.4s, v29.8h, v24.8h+        smull v26.4s, v29.4h, v24.4h+        ldr q3, [x2, #0x10]+        smlal v26.4s, v5.4h, v20.4h+        ldr q11, [x2], #0x20+        uzp1 v6.8h, v18.8h, v8.8h+        smlal v26.4s, v17.4h, v14.4h+        smlal v26.4s, v27.4h, v1.4h+        zip2 v13.8h, v31.8h, v28.8h+        smlal v26.4s, v7.4h, v4.4h+        str q15, [x0], #0x20+        smlal v26.4s, v25.4h, v10.4h+        stur q13, [x0, #-0x10]+        mul v29.8h, v6.8h, v2.8h+        uzp1 v24.8h, v11.8h, v3.8h+        uzp2 v31.8h, v11.8h, v3.8h+        ldr q11, [x4], #0x20+        smlal2 v23.4s, v5.8h, v20.8h+        ldur q28, [x4, #-0x10]+        smlal2 v23.4s, v17.8h, v14.8h+        ldr q5, [x5], #0x20+        smlal2 v23.4s, v27.8h, v1.8h+        ldur q22, [x5, #-0x10]+        smlal v26.4s, v30.4h, v21.4h+        ld1 { v20.8h }, [x3], #16+        smlal v26.4s, v16.4h, v9.4h+        uzp1 v17.8h, v11.8h, v28.8h+        smlal2 v23.4s, v7.8h, v4.8h+        uzp2 v27.8h, v11.8h, v28.8h+        smlal2 v23.4s, v25.8h, v10.8h+        uzp1 v14.8h, v5.8h, v22.8h+        subs x13, x13, #0x1+        cbnz x13, Lmlk_polyvec_basemul_acc_montgomery_cached_k4_loop_start+        smlal v18.4s, v29.4h, v0.4h+        ldr q11, [x1], #0x20+        uzp2 v28.8h, v5.8h, v22.8h+        smlal2 v23.4s, v30.8h, v21.8h+        smlal2 v8.4s, v29.8h, v0.8h+        ldr q15, [x8, #0x10]+        smlal2 v23.4s, v16.8h, v9.8h+        ldr q21, [x8], #0x20+        uzp1 v22.8h, v11.8h, v19.8h+        uzp2 v12.8h, v11.8h, v19.8h+        ldr q1, [x7, #0x10]+        ld1 { v6.8h }, [x6], #16+        uzp2 v3.8h, v18.8h, v8.8h+        smull v9.4s, v22.4h, v31.4h+        smull2 v18.4s, v22.8h, v31.8h+        ldr q16, [x7], #0x20+        smull v19.4s, v22.4h, v24.4h+        uzp1 v30.8h, v21.8h, v15.8h+        uzp2 v25.8h, v21.8h, v15.8h+        smull2 v8.4s, v22.8h, v24.8h+        smlal v19.4s, v12.4h, v20.4h+        ldr q13, [x10, #0x10]+        smlal2 v8.4s, v12.8h, v20.8h+        uzp1 v29.8h, v16.8h, v1.8h+        smlal2 v18.4s, v12.8h, v24.8h+        ldr q5, [x10], #0x20+        smlal v9.4s, v12.4h, v24.4h+        ldr q4, [x11], #0x20+        smlal v9.4s, v17.4h, v28.4h+        ldur q22, [x11, #-0x10]+        smlal2 v18.4s, v17.8h, v28.8h+        uzp2 v16.8h, v16.8h, v1.8h+        smlal v19.4s, v17.4h, v14.4h+        ld1 { v28.8h }, [x9], #16+        smlal2 v8.4s, v17.8h, v14.8h+        uzp1 v7.8h, v5.8h, v13.8h+        smlal v9.4s, v27.4h, v14.4h+        uzp1 v17.8h, v4.8h, v22.8h+        smlal2 v18.4s, v27.8h, v14.8h+        uzp2 v12.8h, v5.8h, v13.8h+        uzp2 v21.8h, v4.8h, v22.8h+        smlal v19.4s, v27.4h, v6.4h+        smlal2 v8.4s, v27.8h, v6.8h+        ld1 { v15.8h }, [x12], #16+        smlal v19.4s, v29.4h, v30.4h+        uzp1 v20.8h, v26.8h, v23.8h+        smlal v9.4s, v29.4h, v25.4h+        smlal2 v18.4s, v29.8h, v25.8h+        smlal2 v8.4s, v29.8h, v30.8h+        smlal v19.4s, v16.4h, v28.4h+        smlal2 v8.4s, v16.8h, v28.8h+        smlal2 v18.4s, v16.8h, v30.8h+        smlal v9.4s, v16.4h, v30.4h+        smlal v9.4s, v7.4h, v21.4h+        smlal2 v18.4s, v7.8h, v21.8h+        smlal2 v8.4s, v7.8h, v17.8h+        smlal v19.4s, v7.4h, v17.4h+        smlal v19.4s, v12.4h, v15.4h+        smlal2 v8.4s, v12.8h, v15.8h+        smlal2 v18.4s, v12.8h, v17.8h+        smlal v9.4s, v12.4h, v17.4h+        mul v6.8h, v20.8h, v2.8h+        uzp1 v4.8h, v19.8h, v8.8h+        mul v17.8h, v4.8h, v2.8h+        uzp1 v12.8h, v9.8h, v18.8h+        smlal v26.4s, v6.4h, v0.4h+        mul v21.8h, v12.8h, v2.8h+        smlal2 v23.4s, v6.8h, v0.8h+        smlal2 v8.4s, v17.8h, v0.8h+        smlal v19.4s, v17.4h, v0.4h+        smlal2 v18.4s, v21.8h, v0.8h+        uzp2 v23.8h, v26.8h, v23.8h+        smlal v9.4s, v21.4h, v0.4h+        zip2 v12.8h, v23.8h, v3.8h+        zip1 v22.8h, v23.8h, v3.8h+        uzp2 v14.8h, v19.8h, v8.8h+        uzp2 v18.8h, v9.8h, v18.8h+        str q12, [x0, #0x10]+        str q22, [x0], #0x20+        zip2 v24.8h, v14.8h, v18.8h+        zip1 v21.8h, v14.8h, v18.8h+        str q24, [x0, #0x10]+        str q21, [x0], #0x20+        ldp d8, d9, [sp]+        .cfi_restore d8+        .cfi_restore d9+        ldp d10, d11, [sp, #0x10]+        .cfi_restore d10+        .cfi_restore d11+        ldp d12, d13, [sp, #0x20]+        .cfi_restore d12+        .cfi_restore d13+        ldp d14, d15, [sp, #0x30]+        .cfi_restore d14+        .cfi_restore d15+        add sp, sp, #0x40+        .cfi_adjust_cfa_offset -0x40+        ret+        .cfi_endproc++MLK_ASM_FN_SIZE(polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm)++#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/aarch64/src/mlkem_rej_uniform_aarch64_asm.S view
@@ -0,0 +1,226 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*yaml+  Name: rej_uniform_aarch64_asm+  Description: Run rejection sampling on uniform random bytes to generate uniform random integers mod q+  Signature: uint64_t mlk_rej_uniform_aarch64_asm(int16_t r[256], const uint8_t *buf, unsigned buflen, const uint8_t table[4096])+  ABI:+    Architecture: aarch64+    CallingConvention: AAPCS64+    Features: [NEON]+    x0:+      type: buffer+      size_bytes: 512+      permissions: write-only+      c_parameter: int16_t r[256]+      description: Output buffer+    x1:+      type: buffer+      size_bytes: x2+      permissions: read-only+      c_parameter: const uint8_t *buf+      description: Input buffer+    x2:+      type: scalar+      c_parameter: unsigned buflen+      description: Length of input buffer (must be multiple of 24)+      test_with: 504  # MLKEM_GEN_MATRIX_NBLOCKS * MLK_XOF_RATE+    x3:+      type: buffer+      size_bytes: 4096+      permissions: read-only+      c_parameter: const uint8_t table[4096]+      description: Lookup table+  Stack:+    bytes: 576+    description: register preservation and temporary storage+*/++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_AARCH64) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/aarch64_opt/src/mlkem_rej_uniform_aarch64_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(rej_uniform_aarch64_asm)+MLK_ASM_FN_SYMBOL(rej_uniform_aarch64_asm)++        .cfi_startproc+        sub sp, sp, #0x240+        .cfi_adjust_cfa_offset 0x240+        mov x7, #0x1                // =1+        movk x7, #0x2, lsl #16+        movk x7, #0x4, lsl #32+        movk x7, #0x8, lsl #48+        mov v31.d[0], x7+        mov x7, #0x10               // =16+        movk x7, #0x20, lsl #16+        movk x7, #0x40, lsl #32+        movk x7, #0x80, lsl #48+        mov v31.d[1], x7+        mov w11, #0xd01             // =3329+        dup v30.8h, w11+        mov x8, sp+        mov x7, x8+        mov x11, #0x0               // =0+        eor v16.16b, v16.16b, v16.16b++Lmlk_rej_uniform_initial_zero:+        str q16, [x7], #0x40+        stur q16, [x7, #-0x30]+        stur q16, [x7, #-0x20]+        stur q16, [x7, #-0x10]+        add x11, x11, #0x20+        cmp x11, #0x100+        b.lt Lmlk_rej_uniform_initial_zero+        mov x7, x8+        mov x9, #0x0                // =0+        mov x4, #0x100              // =256+        cmp x2, #0x30+        b.lo Lmlk_rej_uniform_loop48_end++Lmlk_rej_uniform_loop48:+        cmp x9, x4+        b.hs Lmlk_rej_uniform_memory_copy+        sub x2, x2, #0x30+        ld3 { v0.16b, v1.16b, v2.16b }, [x1], #48+        zip1 v4.16b, v0.16b, v1.16b+        zip2 v5.16b, v0.16b, v1.16b+        zip1 v6.16b, v1.16b, v2.16b+        zip2 v7.16b, v1.16b, v2.16b+        bic v4.8h, #0xf0, lsl #8+        bic v5.8h, #0xf0, lsl #8+        ushr v6.8h, v6.8h, #0x4+        ushr v7.8h, v7.8h, #0x4+        zip1 v16.8h, v4.8h, v6.8h+        zip2 v17.8h, v4.8h, v6.8h+        zip1 v18.8h, v5.8h, v7.8h+        zip2 v19.8h, v5.8h, v7.8h+        cmhi v4.8h, v30.8h, v16.8h+        cmhi v5.8h, v30.8h, v17.8h+        cmhi v6.8h, v30.8h, v18.8h+        cmhi v7.8h, v30.8h, v19.8h+        and v4.16b, v4.16b, v31.16b+        and v5.16b, v5.16b, v31.16b+        and v6.16b, v6.16b, v31.16b+        and v7.16b, v7.16b, v31.16b+        uaddlv s20, v4.8h+        uaddlv s21, v5.8h+        uaddlv s22, v6.8h+        uaddlv s23, v7.8h+        fmov w12, s20+        fmov w13, s21+        fmov w14, s22+        fmov w15, s23+        ldr q24, [x3, x12, lsl #4]+        ldr q25, [x3, x13, lsl #4]+        ldr q26, [x3, x14, lsl #4]+        ldr q27, [x3, x15, lsl #4]+        cnt v4.16b, v4.16b+        cnt v5.16b, v5.16b+        cnt v6.16b, v6.16b+        cnt v7.16b, v7.16b+        uaddlv s20, v4.8h+        uaddlv s21, v5.8h+        uaddlv s22, v6.8h+        uaddlv s23, v7.8h+        fmov w12, s20+        fmov w13, s21+        fmov w14, s22+        fmov w15, s23+        tbl v16.16b, { v16.16b }, v24.16b+        tbl v17.16b, { v17.16b }, v25.16b+        tbl v18.16b, { v18.16b }, v26.16b+        tbl v19.16b, { v19.16b }, v27.16b+        st1 { v16.8h }, [x7]+        add x7, x7, x12, lsl #1+        st1 { v17.8h }, [x7]+        add x7, x7, x13, lsl #1+        st1 { v18.8h }, [x7]+        add x7, x7, x14, lsl #1+        st1 { v19.8h }, [x7]+        add x7, x7, x15, lsl #1+        add x12, x12, x13+        add x14, x14, x15+        add x9, x9, x12+        add x9, x9, x14+        cmp x2, #0x30+        b.hs Lmlk_rej_uniform_loop48++Lmlk_rej_uniform_loop48_end:+        cmp x9, x4+        b.hs Lmlk_rej_uniform_memory_copy+        cmp x2, #0x18+        b.lo Lmlk_rej_uniform_memory_copy+        ld3 { v0.8b, v1.8b, v2.8b }, [x1], #24+        zip1 v4.16b, v0.16b, v1.16b+        zip1 v5.16b, v1.16b, v2.16b+        bic v4.8h, #0xf0, lsl #8+        ushr v5.8h, v5.8h, #0x4+        zip1 v16.8h, v4.8h, v5.8h+        zip2 v17.8h, v4.8h, v5.8h+        cmhi v4.8h, v30.8h, v16.8h+        cmhi v5.8h, v30.8h, v17.8h+        and v4.16b, v4.16b, v31.16b+        and v5.16b, v5.16b, v31.16b+        uaddlv s20, v4.8h+        uaddlv s21, v5.8h+        fmov w12, s20+        fmov w13, s21+        ldr q24, [x3, x12, lsl #4]+        ldr q25, [x3, x13, lsl #4]+        cnt v4.16b, v4.16b+        cnt v5.16b, v5.16b+        uaddlv s20, v4.8h+        uaddlv s21, v5.8h+        fmov w12, s20+        fmov w13, s21+        tbl v16.16b, { v16.16b }, v24.16b+        tbl v17.16b, { v17.16b }, v25.16b+        st1 { v16.8h }, [x7]+        add x7, x7, x12, lsl #1+        st1 { v17.8h }, [x7]+        add x9, x9, x12+        add x9, x9, x13++Lmlk_rej_uniform_memory_copy:+        cmp x9, x4+        csel x9, x9, x4, lo+        mov x11, #0x0               // =0+        mov x7, x8++Lmlk_rej_uniform_final_copy:+        ldr q16, [x7], #0x40+        ldur q17, [x7, #-0x30]+        ldur q18, [x7, #-0x20]+        ldur q19, [x7, #-0x10]+        str q16, [x0], #0x40+        stur q17, [x0, #-0x30]+        stur q18, [x0, #-0x20]+        stur q19, [x0, #-0x10]+        add x11, x11, #0x20+        cmp x11, #0x100+        b.lt Lmlk_rej_uniform_final_copy+        mov x0, x9++Lmlk_rej_uniform_return:+        add sp, sp, #0x240+        .cfi_adjust_cfa_offset -0x240+        ret+        .cfi_endproc++MLK_ASM_FN_SIZE(rej_uniform_aarch64_asm)++#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/aarch64/src/rej_uniform_table.c view
@@ -0,0 +1,543 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mlkem-native repository.+ *          Do not modify it directly.+ */++#include "../../../common.h"++#if defined(MLK_ARITH_BACKEND_AARCH64) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++#include "arith_native_aarch64.h"++/*+ * Lookup table used by rejection sampling of the public matrix.+ * See autogen for details.+ */+MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t+    mlk_rej_uniform_table[4096] = {+        255, 255, 255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 0 */,+        0,   1,   255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 1 */,+        2,   3,   255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 2 */,+        0,   1,   2,   3,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 3 */,+        4,   5,   255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 4 */,+        0,   1,   4,   5,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 5 */,+        2,   3,   4,   5,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 6 */,+        0,   1,   2,   3,   4,   5,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 7 */,+        6,   7,   255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 8 */,+        0,   1,   6,   7,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 9 */,+        2,   3,   6,   7,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 10 */,+        0,   1,   2,   3,   6,   7,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 11 */,+        4,   5,   6,   7,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 12 */,+        0,   1,   4,   5,   6,   7,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 13 */,+        2,   3,   4,   5,   6,   7,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 14 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        255, 255, 255, 255, 255, 255, 255, 255 /* 15 */,+        8,   9,   255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 16 */,+        0,   1,   8,   9,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 17 */,+        2,   3,   8,   9,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 18 */,+        0,   1,   2,   3,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 19 */,+        4,   5,   8,   9,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 20 */,+        0,   1,   4,   5,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 21 */,+        2,   3,   4,   5,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 22 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        255, 255, 255, 255, 255, 255, 255, 255 /* 23 */,+        6,   7,   8,   9,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 24 */,+        0,   1,   6,   7,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 25 */,+        2,   3,   6,   7,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 26 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        255, 255, 255, 255, 255, 255, 255, 255 /* 27 */,+        4,   5,   6,   7,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 28 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        255, 255, 255, 255, 255, 255, 255, 255 /* 29 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        255, 255, 255, 255, 255, 255, 255, 255 /* 30 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   255, 255, 255, 255, 255, 255 /* 31 */,+        10,  11,  255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 32 */,+        0,   1,   10,  11,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 33 */,+        2,   3,   10,  11,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 34 */,+        0,   1,   2,   3,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 35 */,+        4,   5,   10,  11,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 36 */,+        0,   1,   4,   5,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 37 */,+        2,   3,   4,   5,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 38 */,+        0,   1,   2,   3,   4,   5,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 39 */,+        6,   7,   10,  11,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 40 */,+        0,   1,   6,   7,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 41 */,+        2,   3,   6,   7,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 42 */,+        0,   1,   2,   3,   6,   7,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 43 */,+        4,   5,   6,   7,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 44 */,+        0,   1,   4,   5,   6,   7,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 45 */,+        2,   3,   4,   5,   6,   7,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 46 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        10,  11,  255, 255, 255, 255, 255, 255 /* 47 */,+        8,   9,   10,  11,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 48 */,+        0,   1,   8,   9,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 49 */,+        2,   3,   8,   9,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 50 */,+        0,   1,   2,   3,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 51 */,+        4,   5,   8,   9,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 52 */,+        0,   1,   4,   5,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 53 */,+        2,   3,   4,   5,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 54 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        10,  11,  255, 255, 255, 255, 255, 255 /* 55 */,+        6,   7,   8,   9,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 56 */,+        0,   1,   6,   7,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 57 */,+        2,   3,   6,   7,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 58 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        10,  11,  255, 255, 255, 255, 255, 255 /* 59 */,+        4,   5,   6,   7,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 60 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        10,  11,  255, 255, 255, 255, 255, 255 /* 61 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        10,  11,  255, 255, 255, 255, 255, 255 /* 62 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   10,  11,  255, 255, 255, 255 /* 63 */,+        12,  13,  255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 64 */,+        0,   1,   12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 65 */,+        2,   3,   12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 66 */,+        0,   1,   2,   3,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 67 */,+        4,   5,   12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 68 */,+        0,   1,   4,   5,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 69 */,+        2,   3,   4,   5,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 70 */,+        0,   1,   2,   3,   4,   5,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 71 */,+        6,   7,   12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 72 */,+        0,   1,   6,   7,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 73 */,+        2,   3,   6,   7,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 74 */,+        0,   1,   2,   3,   6,   7,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 75 */,+        4,   5,   6,   7,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 76 */,+        0,   1,   4,   5,   6,   7,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 77 */,+        2,   3,   4,   5,   6,   7,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 78 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        12,  13,  255, 255, 255, 255, 255, 255 /* 79 */,+        8,   9,   12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 80 */,+        0,   1,   8,   9,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 81 */,+        2,   3,   8,   9,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 82 */,+        0,   1,   2,   3,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 83 */,+        4,   5,   8,   9,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 84 */,+        0,   1,   4,   5,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 85 */,+        2,   3,   4,   5,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 86 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        12,  13,  255, 255, 255, 255, 255, 255 /* 87 */,+        6,   7,   8,   9,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 88 */,+        0,   1,   6,   7,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 89 */,+        2,   3,   6,   7,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 90 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        12,  13,  255, 255, 255, 255, 255, 255 /* 91 */,+        4,   5,   6,   7,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 92 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        12,  13,  255, 255, 255, 255, 255, 255 /* 93 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        12,  13,  255, 255, 255, 255, 255, 255 /* 94 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   12,  13,  255, 255, 255, 255 /* 95 */,+        10,  11,  12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 96 */,+        0,   1,   10,  11,  12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 97 */,+        2,   3,   10,  11,  12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 98 */,+        0,   1,   2,   3,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 99 */,+        4,   5,   10,  11,  12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 100 */,+        0,   1,   4,   5,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 101 */,+        2,   3,   4,   5,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 102 */,+        0,   1,   2,   3,   4,   5,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 103 */,+        6,   7,   10,  11,  12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 104 */,+        0,   1,   6,   7,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 105 */,+        2,   3,   6,   7,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 106 */,+        0,   1,   2,   3,   6,   7,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 107 */,+        4,   5,   6,   7,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 108 */,+        0,   1,   4,   5,   6,   7,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 109 */,+        2,   3,   4,   5,   6,   7,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 110 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        10,  11,  12,  13,  255, 255, 255, 255 /* 111 */,+        8,   9,   10,  11,  12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 112 */,+        0,   1,   8,   9,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 113 */,+        2,   3,   8,   9,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 114 */,+        0,   1,   2,   3,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 115 */,+        4,   5,   8,   9,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 116 */,+        0,   1,   4,   5,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 117 */,+        2,   3,   4,   5,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 118 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        10,  11,  12,  13,  255, 255, 255, 255 /* 119 */,+        6,   7,   8,   9,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 120 */,+        0,   1,   6,   7,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 121 */,+        2,   3,   6,   7,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 122 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        10,  11,  12,  13,  255, 255, 255, 255 /* 123 */,+        4,   5,   6,   7,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 124 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        10,  11,  12,  13,  255, 255, 255, 255 /* 125 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        10,  11,  12,  13,  255, 255, 255, 255 /* 126 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   10,  11,  12,  13,  255, 255 /* 127 */,+        14,  15,  255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 128 */,+        0,   1,   14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 129 */,+        2,   3,   14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 130 */,+        0,   1,   2,   3,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 131 */,+        4,   5,   14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 132 */,+        0,   1,   4,   5,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 133 */,+        2,   3,   4,   5,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 134 */,+        0,   1,   2,   3,   4,   5,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 135 */,+        6,   7,   14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 136 */,+        0,   1,   6,   7,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 137 */,+        2,   3,   6,   7,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 138 */,+        0,   1,   2,   3,   6,   7,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 139 */,+        4,   5,   6,   7,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 140 */,+        0,   1,   4,   5,   6,   7,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 141 */,+        2,   3,   4,   5,   6,   7,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 142 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        14,  15,  255, 255, 255, 255, 255, 255 /* 143 */,+        8,   9,   14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 144 */,+        0,   1,   8,   9,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 145 */,+        2,   3,   8,   9,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 146 */,+        0,   1,   2,   3,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 147 */,+        4,   5,   8,   9,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 148 */,+        0,   1,   4,   5,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 149 */,+        2,   3,   4,   5,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 150 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        14,  15,  255, 255, 255, 255, 255, 255 /* 151 */,+        6,   7,   8,   9,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 152 */,+        0,   1,   6,   7,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 153 */,+        2,   3,   6,   7,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 154 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        14,  15,  255, 255, 255, 255, 255, 255 /* 155 */,+        4,   5,   6,   7,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 156 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        14,  15,  255, 255, 255, 255, 255, 255 /* 157 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        14,  15,  255, 255, 255, 255, 255, 255 /* 158 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   14,  15,  255, 255, 255, 255 /* 159 */,+        10,  11,  14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 160 */,+        0,   1,   10,  11,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 161 */,+        2,   3,   10,  11,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 162 */,+        0,   1,   2,   3,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 163 */,+        4,   5,   10,  11,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 164 */,+        0,   1,   4,   5,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 165 */,+        2,   3,   4,   5,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 166 */,+        0,   1,   2,   3,   4,   5,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 167 */,+        6,   7,   10,  11,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 168 */,+        0,   1,   6,   7,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 169 */,+        2,   3,   6,   7,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 170 */,+        0,   1,   2,   3,   6,   7,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 171 */,+        4,   5,   6,   7,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 172 */,+        0,   1,   4,   5,   6,   7,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 173 */,+        2,   3,   4,   5,   6,   7,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 174 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        10,  11,  14,  15,  255, 255, 255, 255 /* 175 */,+        8,   9,   10,  11,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 176 */,+        0,   1,   8,   9,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 177 */,+        2,   3,   8,   9,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 178 */,+        0,   1,   2,   3,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 179 */,+        4,   5,   8,   9,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 180 */,+        0,   1,   4,   5,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 181 */,+        2,   3,   4,   5,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 182 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        10,  11,  14,  15,  255, 255, 255, 255 /* 183 */,+        6,   7,   8,   9,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 184 */,+        0,   1,   6,   7,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 185 */,+        2,   3,   6,   7,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 186 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        10,  11,  14,  15,  255, 255, 255, 255 /* 187 */,+        4,   5,   6,   7,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 188 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        10,  11,  14,  15,  255, 255, 255, 255 /* 189 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        10,  11,  14,  15,  255, 255, 255, 255 /* 190 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   10,  11,  14,  15,  255, 255 /* 191 */,+        12,  13,  14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 192 */,+        0,   1,   12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 193 */,+        2,   3,   12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 194 */,+        0,   1,   2,   3,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 195 */,+        4,   5,   12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 196 */,+        0,   1,   4,   5,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 197 */,+        2,   3,   4,   5,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 198 */,+        0,   1,   2,   3,   4,   5,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 199 */,+        6,   7,   12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 200 */,+        0,   1,   6,   7,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 201 */,+        2,   3,   6,   7,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 202 */,+        0,   1,   2,   3,   6,   7,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 203 */,+        4,   5,   6,   7,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 204 */,+        0,   1,   4,   5,   6,   7,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 205 */,+        2,   3,   4,   5,   6,   7,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 206 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        12,  13,  14,  15,  255, 255, 255, 255 /* 207 */,+        8,   9,   12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 208 */,+        0,   1,   8,   9,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 209 */,+        2,   3,   8,   9,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 210 */,+        0,   1,   2,   3,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 211 */,+        4,   5,   8,   9,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 212 */,+        0,   1,   4,   5,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 213 */,+        2,   3,   4,   5,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 214 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        12,  13,  14,  15,  255, 255, 255, 255 /* 215 */,+        6,   7,   8,   9,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 216 */,+        0,   1,   6,   7,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 217 */,+        2,   3,   6,   7,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 218 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        12,  13,  14,  15,  255, 255, 255, 255 /* 219 */,+        4,   5,   6,   7,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 220 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        12,  13,  14,  15,  255, 255, 255, 255 /* 221 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        12,  13,  14,  15,  255, 255, 255, 255 /* 222 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   12,  13,  14,  15,  255, 255 /* 223 */,+        10,  11,  12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 224 */,+        0,   1,   10,  11,  12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 225 */,+        2,   3,   10,  11,  12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 226 */,+        0,   1,   2,   3,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 227 */,+        4,   5,   10,  11,  12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 228 */,+        0,   1,   4,   5,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 229 */,+        2,   3,   4,   5,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 230 */,+        0,   1,   2,   3,   4,   5,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 231 */,+        6,   7,   10,  11,  12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 232 */,+        0,   1,   6,   7,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 233 */,+        2,   3,   6,   7,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 234 */,+        0,   1,   2,   3,   6,   7,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 235 */,+        4,   5,   6,   7,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 236 */,+        0,   1,   4,   5,   6,   7,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 237 */,+        2,   3,   4,   5,   6,   7,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 238 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        10,  11,  12,  13,  14,  15,  255, 255 /* 239 */,+        8,   9,   10,  11,  12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 240 */,+        0,   1,   8,   9,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 241 */,+        2,   3,   8,   9,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 242 */,+        0,   1,   2,   3,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 243 */,+        4,   5,   8,   9,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 244 */,+        0,   1,   4,   5,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 245 */,+        2,   3,   4,   5,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 246 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        10,  11,  12,  13,  14,  15,  255, 255 /* 247 */,+        6,   7,   8,   9,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 248 */,+        0,   1,   6,   7,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 249 */,+        2,   3,   6,   7,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 250 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        10,  11,  12,  13,  14,  15,  255, 255 /* 251 */,+        4,   5,   6,   7,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 252 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        10,  11,  12,  13,  14,  15,  255, 255 /* 253 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        10,  11,  12,  13,  14,  15,  255, 255 /* 254 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   10,  11,  12,  13,  14,  15 /* 255 */,+};++#else /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED */++MLK_EMPTY_CU(aarch64_rej_uniform_table)++#endif /* !(MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED) */
+ cbits/mlkem/src/native/api.h view
@@ -0,0 +1,651 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ */++#ifndef MLK_NATIVE_API_H+#define MLK_NATIVE_API_H+/*+ * Native arithmetic interface+ *+ * This header is primarily for documentation purposes.+ * It should not be included by backend implementations.+ *+ * To ensure consistency with backends, the header will be+ * included automatically after inclusion of the active+ * backend, to ensure consistency of function signatures,+ * and run sanity checks.+ */++#include "../cbmc.h"+#include "../common.h"++/* Backends must return MLK_NATIVE_FUNC_SUCCESS upon success. */+#define MLK_NATIVE_FUNC_SUCCESS (0)+/* Backends may return MLK_NATIVE_FUNC_FALLBACK to signal to the frontend that+ * the target/parameters are unsupported; typically, this would be because of+ * dependencies on CPU features not detected on the host CPU. In this case,+ * the frontend falls back to the default C implementation. */+#define MLK_NATIVE_FUNC_FALLBACK (-1)+++/* Absolute exclusive upper bound for the output of the inverse NTT+ *+ * NOTE: This is the same bound as in poly.h and has to be kept+ * in sync. */+#define MLK_INVNTT_BOUND (8 * MLKEM_Q)++/* Absolute exclusive upper bound for the output of the forward NTT+ *+ * NOTE: This is the same bound as in poly.h and has to be kept+ * in sync. */+#define MLK_NTT_BOUND (8 * MLKEM_Q)++/*+ * This is the C<->native interface allowing for the drop-in of+ * native code for performance critical arithmetic components of ML-KEM.+ *+ * A _backend_ is a specific implementation of (part of) this interface.+ *+ * To add a function to a backend, define MLK_USE_NATIVE_XXX and+ * implement `static inline xxx(...)` in the profile header.+ *+ * The only exception is MLK_USE_NATIVE_NTT_CUSTOM_ORDER. This option can+ * be set if there are native implementations for all of NTT, invNTT, and+ * base multiplication, and allows the native implementation to use a+ * custom order of polynomial coefficients in NTT domain -- the use of such+ * custom order is not an implementation-detail since the public matrix+ * is generated in NTT domain. In this case, a permutation function+ * mlk_poly_permute_bitrev_to_custom() needs to be provided that permutes+ * polynomials in NTT domain from bitreversed to the custom order.+ */++/*+ * Those functions are meant to be trivial wrappers around the chosen native+ * implementation. The are static inline to avoid unnecessary calls.+ * The macro before each declaration controls whether a native+ * implementation is present.+ */++#if defined(MLK_USE_NATIVE_NTT)+/**+ * Compute the negacyclic number-theoretic transform (NTT) of a polynomial+ * in place.+ *+ * The input polynomial is assumed to be in normal order. The output+ * polynomial is in bitreversed order, or of a custom order if+ * MLK_USE_NATIVE_NTT_CUSTOM_ORDER is set. See the documentation of+ * MLK_USE_NATIVE_NTT_CUSTOM_ORDER for more information.+ *+ * @param[in,out] p Input/output polynomial.+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_ntt_native(int16_t p[MLKEM_N])+__contract__(+  requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N))+  requires(array_abs_bound(p, 0, MLKEM_N, MLKEM_Q))+  assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N))+  ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(p, 0, MLKEM_N, MLK_NTT_BOUND))+  ensures((return_value == MLK_NATIVE_FUNC_FALLBACK) ==> array_abs_bound(p, 0, MLKEM_N, MLKEM_Q))+  ensures((return_value == MLK_NATIVE_FUNC_FALLBACK) ==> array_unchanged(p, MLKEM_N))+);+#endif /* MLK_USE_NATIVE_NTT */++#if defined(MLK_USE_NATIVE_NTT_CUSTOM_ORDER)+/*+ * This must only be set if NTT, invNTT, basemul, mulcache, and+ * to/from byte stream conversions all have native implementations+ * that are adapted to the custom order.+ */+#if !defined(MLK_USE_NATIVE_NTT) || !defined(MLK_USE_NATIVE_INTT) ||  \+    !defined(MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE) ||                 \+    !defined(MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED) || \+    !defined(MLK_USE_NATIVE_POLY_TOBYTES) ||                          \+    !defined(MLK_USE_NATIVE_POLY_FROMBYTES)+#error \+    "Invalid native profile: MLK_USE_NATIVE_NTT_CUSTOM_ORDER can only be \+set if there are native implementations for NTT, invNTT, mulcache, basemul, \+and to/from bytes conversions."+#endif /* !MLK_USE_NATIVE_NTT || !MLK_USE_NATIVE_INTT ||           \+          !MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE ||                 \+          !MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED || \+          !MLK_USE_NATIVE_POLY_TOBYTES || !MLK_USE_NATIVE_POLY_FROMBYTES */++/**+ * When MLK_USE_NATIVE_NTT_CUSTOM_ORDER is defined, convert a polynomial in+ * NTT domain from bitreversed order to the custom order output by the+ * native NTT.+ *+ * This must only be defined if there is native code for all of (a) NTT,+ * (b) invNTT, (c) basemul, (d) mulcache.+ *+ * @param[in,out] p Input/output polynomial.+ */+static MLK_INLINE void mlk_poly_permute_bitrev_to_custom(int16_t p[MLKEM_N])+__contract__(+  /* We don't specify that this should be a permutation, but only+   * that it does not change the bound established at the end of mlk_gen_matrix. */+  requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N))+  requires(array_bound(p, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N))+  ensures(array_bound(p, 0, MLKEM_N, 0, MLKEM_Q)));+#endif /* MLK_USE_NATIVE_NTT_CUSTOM_ORDER */++#if defined(MLK_USE_NATIVE_INTT) && \+    (!defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API))+/**+ * Compute the inverse negacyclic number-theoretic transform (NTT) of a+ * polynomial in place.+ *+ * The input polynomial is in bitreversed order, or of a custom order if+ * MLK_USE_NATIVE_NTT_CUSTOM_ORDER is set. See the documentation of+ * MLK_USE_NATIVE_NTT_CUSTOM_ORDER for more information. The output+ * polynomial is assumed to be in normal order.+ *+ * @param[in,out] p Input/output polynomial.+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_intt_native(int16_t p[MLKEM_N])+__contract__(+  requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N))+  assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N))+  ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(p, 0, MLKEM_N, MLK_INVNTT_BOUND))+  ensures((return_value == MLK_NATIVE_FUNC_FALLBACK) ==> array_unchanged(p, MLKEM_N))+);+#endif /* MLK_USE_NATIVE_INTT && (!MLK_CONFIG_NO_ENCAPS_API || \+          !MLK_CONFIG_NO_DECAPS_API) */++#if defined(MLK_USE_NATIVE_POLY_REDUCE)+/**+ * Apply modular reduction to all coefficients of a polynomial, mapping them+ * to unsigned canonical representatives in [0,..,MLKEM_Q-1].+ *+ * @param[in,out] p Input/output polynomial.+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_reduce_native(int16_t p[MLKEM_N])+__contract__(+  requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N))+  assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N))+  ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_bound(p, 0, MLKEM_N, 0, MLKEM_Q))+  ensures((return_value == MLK_NATIVE_FUNC_FALLBACK) ==> array_unchanged(p, MLKEM_N))+);+#endif /* MLK_USE_NATIVE_POLY_REDUCE */++#if defined(MLK_USE_NATIVE_POLY_TOMONT) && !defined(MLK_CONFIG_NO_KEYPAIR_API)+/**+ * In-place conversion of all coefficients of a polynomial from the normal+ * domain to the Montgomery domain.+ *+ * @param[in,out] p Input/output polynomial.+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_tomont_native(int16_t p[MLKEM_N])+__contract__(+  requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N))+  assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N))+  ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(p, 0, MLKEM_N, MLKEM_Q))+  ensures((return_value == MLK_NATIVE_FUNC_FALLBACK) ==> array_unchanged(p, MLKEM_N))+);+#endif /* MLK_USE_NATIVE_POLY_TOMONT && !MLK_CONFIG_NO_KEYPAIR_API */++#if defined(MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE)+/**+ * Compute multiplication cache for a polynomial in NTT domain.+ *+ * The purpose of the multiplication cache is to cache repeated computations+ * required during a base multiplication of polynomials in NTT domain. The+ * structure of the multiplication-cache is implementation defined.+ *+ * @param[out] cache    Multiplication cache.+ * @param[in]  mlk_poly Input polynomial. Must be in NTT domain and in+ *                      bitreversed order, or of a custom order if+ *                      MLK_USE_NATIVE_NTT_CUSTOM_ORDER is set. See the+ *                      documentation of MLK_USE_NATIVE_NTT_CUSTOM_ORDER for+ *                      more information.+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_mulcache_compute_native(+    int16_t cache[MLKEM_N / 2], const int16_t mlk_poly[MLKEM_N])+__contract__(+  requires(memory_no_alias(cache, sizeof(int16_t) * (MLKEM_N / 2)))+  requires(memory_no_alias(mlk_poly, sizeof(int16_t) * MLKEM_N))+  assigns(memory_slice(cache, sizeof(int16_t) * (MLKEM_N / 2)))+  ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS)+  ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(cache, 0, MLKEM_N/2, MLKEM_Q))+);+#endif /* MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE */++#if defined(MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED)+#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2+/**+ * Compute scalar product of length-2 polynomial vectors in NTT domain.+ *+ * @param[out] r       Result of the scalar product. Again in NTT domain, of+ *                     the same ordering as @p a and @p b.+ * @param[in]  a       First polynomial vector operand. Must be in NTT+ *                     domain and in bitreversed order, or of a custom order+ *                     if MLK_USE_NATIVE_NTT_CUSTOM_ORDER is set. See the+ *                     documentation of MLK_USE_NATIVE_NTT_CUSTOM_ORDER for+ *                     more information.+ * @param[in]  b       Second polynomial vector operand. As for @p a.+ * @param[in]  b_cache Multiplication-cache for @p b.+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k2_native(+    int16_t r[MLKEM_N], const int16_t a[2 * MLKEM_N],+    const int16_t b[2 * MLKEM_N], const int16_t b_cache[2 * (MLKEM_N / 2)])+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  requires(memory_no_alias(a, sizeof(int16_t) * 2 * MLKEM_N))+  requires(memory_no_alias(b, sizeof(int16_t) * 2 * MLKEM_N))+  requires(memory_no_alias(b_cache, sizeof(int16_t) * 2 * (MLKEM_N / 2)))+  requires(array_bound(a, 0, 2 * MLKEM_N, 0, MLKEM_UINT12_LIMIT))+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS)+);+#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 */++#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 3+/**+ * Compute scalar product of length-3 polynomial vectors in NTT domain.+ *+ * @param[out] r       Result of the scalar product. Again in NTT domain, of+ *                     the same ordering as @p a and @p b.+ * @param[in]  a       First polynomial vector operand. Must be in NTT+ *                     domain and in bitreversed order, or of a custom order+ *                     if MLK_USE_NATIVE_NTT_CUSTOM_ORDER is set. See the+ *                     documentation of MLK_USE_NATIVE_NTT_CUSTOM_ORDER for+ *                     more information.+ * @param[in]  b       Second polynomial vector operand. As for @p a.+ * @param[in]  b_cache Multiplication-cache for @p b.+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k3_native(+    int16_t r[MLKEM_N], const int16_t a[3 * MLKEM_N],+    const int16_t b[3 * MLKEM_N], const int16_t b_cache[3 * (MLKEM_N / 2)])+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  requires(memory_no_alias(a, sizeof(int16_t) * 3 * MLKEM_N))+  requires(memory_no_alias(b, sizeof(int16_t) * 3 * MLKEM_N))+  requires(memory_no_alias(b_cache, sizeof(int16_t) * 3 * (MLKEM_N / 2)))+  requires(array_bound(a, 0, 3 * MLKEM_N, 0, MLKEM_UINT12_LIMIT))+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS)+);+#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 3 */++#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4+/**+ * Compute scalar product of length-4 polynomial vectors in NTT domain.+ *+ * @param[out] r       Result of the scalar product. Again in NTT domain, of+ *                     the same ordering as @p a and @p b.+ * @param[in]  a       First polynomial vector operand. Must be in NTT+ *                     domain and in bitreversed order, or of a custom order+ *                     if MLK_USE_NATIVE_NTT_CUSTOM_ORDER is set. See the+ *                     documentation of MLK_USE_NATIVE_NTT_CUSTOM_ORDER for+ *                     more information.+ * @param[in]  b       Second polynomial vector operand. As for @p a.+ * @param[in]  b_cache Multiplication-cache for @p b.+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k4_native(+    int16_t r[MLKEM_N], const int16_t a[4 * MLKEM_N],+    const int16_t b[4 * MLKEM_N], const int16_t b_cache[4 * (MLKEM_N / 2)])+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  requires(memory_no_alias(a, sizeof(int16_t) * 4 * MLKEM_N))+  requires(memory_no_alias(b, sizeof(int16_t) * 4 * MLKEM_N))+  requires(memory_no_alias(b_cache, sizeof(int16_t) * 4 * (MLKEM_N / 2)))+  requires(array_bound(a, 0, 4 * MLKEM_N, 0, MLKEM_UINT12_LIMIT))+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS)+);+#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4 */+#endif /* MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED */++#if defined(MLK_USE_NATIVE_POLY_TOBYTES) && \+    (!defined(MLK_CONFIG_NO_KEYPAIR_API) || \+     !defined(MLK_CONFIG_NO_ENCAPS_API))+/**+ * Serialization of a polynomial with unsigned canonical coefficients.+ *+ * @spec{Implements ByteEncode_12 from @[FIPS203, Algorithm 5].}+ *+ * @param[out] r Output byte array (of MLKEM_POLYBYTES bytes).+ * @param[in]  a Input polynomial, with each coefficient in [0,..,MLKEM_Q-1].+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_tobytes_native(uint8_t r[MLKEM_POLYBYTES],+                                              const int16_t a[MLKEM_N])+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYBYTES))+  requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N))+  requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYBYTES))+  ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK)+);+#endif /* MLK_USE_NATIVE_POLY_TOBYTES && (!MLK_CONFIG_NO_KEYPAIR_API || \+          !MLK_CONFIG_NO_ENCAPS_API) */++#if defined(MLK_USE_NATIVE_POLY_FROMBYTES) && \+    (!defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API))+/**+ * Deserialization of a polynomial.+ *+ * @spec{Implements ByteDecode_12 from @[FIPS203, Algorithm 6].}+ *+ * @param[out] a Output polynomial in NTT domain.+ * @param[in]  r Input byte array (of MLKEM_POLYBYTES bytes).+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_frombytes_native(+    int16_t a[MLKEM_N], const uint8_t r[MLKEM_POLYBYTES])+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYBYTES))+  requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N))+  assigns(memory_slice(a, sizeof(int16_t) * MLKEM_N))+  ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK)+  ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_bound(a, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT))+);+#endif /* MLK_USE_NATIVE_POLY_FROMBYTES && (!MLK_CONFIG_NO_ENCAPS_API || \+          !MLK_CONFIG_NO_DECAPS_API) */++#if defined(MLK_USE_NATIVE_REJ_UNIFORM)+/**+ * Run rejection sampling on uniformly random bytes to generate uniformly random+ * integers mod MLKEM_Q, represented in [0,..,MLKEM_Q-1].+ *+ * @param[out] r      Output buffer.+ * @param      len    Requested number of 16-bit integers (uniform mod MLKEM_Q).+ * @param[in]  buf    Input buffer (assumed to be uniform random bytes).+ * @param      buflen Length of input buffer in bytes.+ *+ * @retval MLK_NATIVE_FUNC_FALLBACK Native implementation does not support+ *                                  the input lengths.+ * @retval other                    Non-negative number of sampled 16-bit+ *                                  integers (at most @p len).+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_rej_uniform_native(int16_t *r, unsigned len,+                                             const uint8_t *buf,+                                             unsigned buflen)+__contract__(+  requires(len <= 4096 && buflen <= 4096 && buflen % 3 == 0)+  requires(memory_no_alias(r, sizeof(int16_t) * len))+  requires(memory_no_alias(buf, buflen))+  assigns(memory_slice(r, sizeof(int16_t) * len))+  ensures(return_value != MLK_NATIVE_FUNC_FALLBACK+              ==> (0 <= return_value && return_value <= len))+  ensures(return_value != MLK_NATIVE_FUNC_FALLBACK+              ==> array_bound(r, 0, (unsigned) return_value, 0, MLKEM_Q))+);+#endif /* MLK_USE_NATIVE_REJ_UNIFORM */++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || (MLKEM_K == 2 || MLKEM_K == 3)+#if defined(MLK_USE_NATIVE_POLY_COMPRESS_D4)+/**+ * Compression (4 bits) and subsequent serialization of a polynomial.+ *+ * @spec{Compress_4 from @[FIPS203, Eq (4.7)].}+ *+ * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D4+ *               bytes).+ * @param[in]  a Input polynomial. Coefficients must be unsigned canonical,+ *               i.e. in [0,1,..,MLKEM_Q-1].+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_compress_d4_native(+    uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D4], const int16_t a[MLKEM_N])+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D4))+  requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N))+  requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D4))+  ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK));+#endif /* MLK_USE_NATIVE_POLY_COMPRESS_D4 */++#if defined(MLK_USE_NATIVE_POLY_COMPRESS_D10)+/**+ * Compression (10 bits) and subsequent serialization of a polynomial.+ *+ * @spec{Compress_10 from @[FIPS203, Eq (4.7)].}+ *+ * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D10+ *               bytes).+ * @param[in]  a Input polynomial. Coefficients must be unsigned canonical,+ *               i.e. in [0,1,..,MLKEM_Q-1].+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_compress_d10_native(+    uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D10], const int16_t a[MLKEM_N])+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D10))+  requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N))+  requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D10))+  ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK));+#endif /* MLK_USE_NATIVE_POLY_COMPRESS_D10 */++#if !defined(MLK_CONFIG_NO_DECAPS_API)+#if defined(MLK_USE_NATIVE_POLY_DECOMPRESS_D4)+/**+ * De-serialization and subsequent decompression (4 bits) of a polynomial;+ * approximate inverse of mlk_poly_compress_d4.+ *+ * @spec{Decompress_4 from @[FIPS203, Eq (4.8)].}+ *+ * Upon return, the coefficients of the output polynomial are+ * unsigned-canonical (non-negative and smaller than MLKEM_Q).+ *+ * @param[out] r Output polynomial.+ * @param[in]  a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D4+ *               bytes).+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_decompress_d4_native(+    int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D4])+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D4))+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK)+  ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_bound(r, 0, MLKEM_N, 0, MLKEM_Q)));+#endif /* MLK_USE_NATIVE_POLY_DECOMPRESS_D4 */++#if defined(MLK_USE_NATIVE_POLY_DECOMPRESS_D10)+/**+ * De-serialization and subsequent decompression (10 bits) of a polynomial;+ * approximate inverse of mlk_poly_compress_d10.+ *+ * @spec{Decompress_10 from @[FIPS203, Eq (4.8)].}+ *+ * Upon return, the coefficients of the output polynomial are+ * unsigned-canonical (non-negative and smaller than MLKEM_Q).+ *+ * @param[out] r Output polynomial.+ * @param[in]  a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D10+ *               bytes).+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_decompress_d10_native(+    int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D10])+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D10))+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK)+  ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_bound(r, 0, MLKEM_N, 0, MLKEM_Q)));+#endif /* MLK_USE_NATIVE_POLY_DECOMPRESS_D10 */+#endif /* !MLK_CONFIG_NO_DECAPS_API */+#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 || MLKEM_K == 3 */++#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4+#if defined(MLK_USE_NATIVE_POLY_COMPRESS_D5)+/**+ * Compression (5 bits) and subsequent serialization of a polynomial.+ *+ * @spec{Compress_5 from @[FIPS203, Eq (4.7)].}+ *+ * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D5+ *               bytes).+ * @param[in]  a Input polynomial. Coefficients must be unsigned canonical,+ *               i.e. in [0,1,..,MLKEM_Q-1].+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_compress_d5_native(+    uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D5], const int16_t a[MLKEM_N])+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D5))+  requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N))+  requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D5))+  ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK));+#endif /* MLK_USE_NATIVE_POLY_COMPRESS_D5 */++#if defined(MLK_USE_NATIVE_POLY_COMPRESS_D11)+/**+ * Compression (11 bits) and subsequent serialization of a polynomial.+ *+ * @spec{Compress_11 from @[FIPS203, Eq (4.7)].}+ *+ * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D11+ *               bytes).+ * @param[in]  a Input polynomial. Coefficients must be unsigned canonical,+ *               i.e. in [0,1,..,MLKEM_Q-1].+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_compress_d11_native(+    uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D11], const int16_t a[MLKEM_N])+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D11))+  requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N))+  requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D11))+  ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK));+#endif /* MLK_USE_NATIVE_POLY_COMPRESS_D11 */++#if !defined(MLK_CONFIG_NO_DECAPS_API)+#if defined(MLK_USE_NATIVE_POLY_DECOMPRESS_D5)+/**+ * De-serialization and subsequent decompression (5 bits) of a polynomial;+ * approximate inverse of mlk_poly_compress_d5.+ *+ * @spec{Decompress_5 from @[FIPS203, Eq (4.8)].}+ *+ * Upon return, the coefficients of the output polynomial are+ * unsigned-canonical (non-negative and smaller than MLKEM_Q).+ *+ * @param[out] r Output polynomial.+ * @param[in]  a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D5+ *               bytes).+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_decompress_d5_native(+    int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D5])+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D5))+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK)+  ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_bound(r, 0, MLKEM_N, 0, MLKEM_Q)));+#endif /* MLK_USE_NATIVE_POLY_DECOMPRESS_D5 */++#if defined(MLK_USE_NATIVE_POLY_DECOMPRESS_D11)+/**+ * De-serialization and subsequent decompression (11 bits) of a polynomial;+ * approximate inverse of mlk_poly_compress_d11.+ *+ * @spec{Decompress_11 from @[FIPS203, Eq (4.8)].}+ *+ * Upon return, the coefficients of the output polynomial are+ * unsigned-canonical (non-negative and smaller than MLKEM_Q).+ *+ * @param[out] r Output polynomial.+ * @param[in]  a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D11+ *               bytes).+ *+ * @retval MLK_NATIVE_FUNC_SUCCESS  Operation succeeded.+ * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_decompress_d11_native(+    int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D11])+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D11))+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK)+  ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_bound(r, 0, MLKEM_N, 0, MLKEM_Q)));+#endif /* MLK_USE_NATIVE_POLY_DECOMPRESS_D11 */+#endif /* !MLK_CONFIG_NO_DECAPS_API */+#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4 */+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++#endif /* !MLK_NATIVE_API_H */
+ cbits/mlkem/src/native/meta.h view
@@ -0,0 +1,30 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLK_NATIVE_META_H+#define MLK_NATIVE_META_H++/*+ * Default arithmetic backend+ */+#include "../sys.h"++#ifdef MLK_SYS_AARCH64_NEON+#include "aarch64/meta.h"+#endif++/* The x86_64 backend requires toolchain support for the SysV ABI */+#if defined(MLK_SYS_X86_64_AVX2) && defined(MLK_SYSV_ABI_SUPPORTED)+#include "x86_64/meta.h"+#endif++#if defined(MLK_SYS_RISCV64_RVV)+#include "riscv64/meta.h"+#endif++#ifdef MLK_SYS_PPC64LE+#include "ppc64le/meta.h"+#endif++#endif /* !MLK_NATIVE_META_H */
+ cbits/mlkem/src/native/x86_64/README.md view
@@ -0,0 +1,4 @@+[//]: # (SPDX-License-Identifier: CC-BY-4.0)++This directory contains the native x86_64 arithmetic backend for ML-KEM provided by the official [AVX2+implementation](https://github.com/pq-crystals/kyber/tree/main/avx2) of the Kyber team.
+ cbits/mlkem/src/native/x86_64/meta.h view
@@ -0,0 +1,324 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#ifndef MLK_NATIVE_X86_64_META_H+#define MLK_NATIVE_X86_64_META_H++/* Identifier for this backend so that source and assembly files+ * in the build can be appropriately guarded. */+#define MLK_ARITH_BACKEND_X86_64_DEFAULT++#define MLK_USE_NATIVE_NTT_CUSTOM_ORDER+#define MLK_USE_NATIVE_REJ_UNIFORM+#define MLK_USE_NATIVE_NTT+#define MLK_USE_NATIVE_INTT+#define MLK_USE_NATIVE_POLY_REDUCE+#define MLK_USE_NATIVE_POLY_TOMONT+#define MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED+#define MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE+#define MLK_USE_NATIVE_POLY_TOBYTES+#define MLK_USE_NATIVE_POLY_FROMBYTES+#define MLK_USE_NATIVE_POLY_COMPRESS_D4+#define MLK_USE_NATIVE_POLY_COMPRESS_D5+#define MLK_USE_NATIVE_POLY_COMPRESS_D10+#define MLK_USE_NATIVE_POLY_COMPRESS_D11+#define MLK_USE_NATIVE_POLY_DECOMPRESS_D4+#define MLK_USE_NATIVE_POLY_DECOMPRESS_D5+#define MLK_USE_NATIVE_POLY_DECOMPRESS_D10+#define MLK_USE_NATIVE_POLY_DECOMPRESS_D11++#if !defined(__ASSEMBLER__)+#include "../../common.h"+#include "../api.h"+#include "src/arith_native_x86_64.h"+#include "src/compress_consts.h"++static MLK_INLINE void mlk_poly_permute_bitrev_to_custom(int16_t data[MLKEM_N])+{+  if (mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    mlk_nttunpack_avx2_asm(data);+  }+}++MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_rej_uniform_native(int16_t *r, unsigned len,+                                             const uint8_t *buf,+                                             unsigned buflen)+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2) || len != MLKEM_N ||+      buflen % 12 != 0)+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }+  return (int)mlk_rej_uniform_avx2_asm(r, buf, buflen, mlk_rej_uniform_table);+}++MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_ntt_native(int16_t data[MLKEM_N])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_ntt_avx2_asm(data, mlk_qdata);+  return MLK_NATIVE_FUNC_SUCCESS;+}++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_intt_native(int16_t data[MLKEM_N])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_invntt_avx2_asm(data, mlk_qdata);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_reduce_native(int16_t data[MLKEM_N])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_reduce_avx2_asm(data);+  return MLK_NATIVE_FUNC_SUCCESS;+}++#if !defined(MLK_CONFIG_NO_KEYPAIR_API)+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_tomont_native(int16_t data[MLKEM_N])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_tomont_avx2_asm(data);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* !MLK_CONFIG_NO_KEYPAIR_API */++MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_mulcache_compute_native(int16_t x[MLKEM_N / 2],+                                                       const int16_t y[MLKEM_N])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_poly_mulcache_compute_avx2_asm(x, y, mlk_qdata);+  return MLK_NATIVE_FUNC_SUCCESS;+}++#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k2_native(+    int16_t r[MLKEM_N], const int16_t a[2 * MLKEM_N],+    const int16_t b[2 * MLKEM_N], const int16_t b_cache[2 * (MLKEM_N / 2)])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm(r, a, b, b_cache);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 */++#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 3+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k3_native(+    int16_t r[MLKEM_N], const int16_t a[3 * MLKEM_N],+    const int16_t b[3 * MLKEM_N], const int16_t b_cache[3 * (MLKEM_N / 2)])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm(r, a, b, b_cache);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 3 */++#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k4_native(+    int16_t r[MLKEM_N], const int16_t a[4 * MLKEM_N],+    const int16_t b[4 * MLKEM_N], const int16_t b_cache[4 * (MLKEM_N / 2)])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm(r, a, b, b_cache);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4 */++#if !defined(MLK_CONFIG_NO_KEYPAIR_API) || !defined(MLK_CONFIG_NO_ENCAPS_API)+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_tobytes_native(uint8_t r[MLKEM_POLYBYTES],+                                              const int16_t a[MLKEM_N])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_ntttobytes_avx2_asm(r, a);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* !MLK_CONFIG_NO_KEYPAIR_API || !MLK_CONFIG_NO_ENCAPS_API */++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_frombytes_native(+    int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYBYTES])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_nttfrombytes_avx2_asm(r, a);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++#if (!defined(MLK_CONFIG_NO_ENCAPS_API) ||                         \+     !defined(MLK_CONFIG_NO_DECAPS_API)) &&                        \+    (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2 || \+     MLKEM_K == 3)+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_compress_d4_native(+    uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D4], const int16_t a[MLKEM_N])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_poly_compress_d4_avx2_asm(r, a, mlk_compress_d4_data);+  return MLK_NATIVE_FUNC_SUCCESS;+}++MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_compress_d10_native(+    uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D10], const int16_t a[MLKEM_N])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_poly_compress_d10_avx2_asm(r, a, mlk_compress_d10_data);+  return MLK_NATIVE_FUNC_SUCCESS;+}++#if !defined(MLK_CONFIG_NO_DECAPS_API)+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_decompress_d4_native(+    int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D4])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_poly_decompress_d4_avx2_asm(r, a, mlk_decompress_d4_data);+  return MLK_NATIVE_FUNC_SUCCESS;+}++MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_decompress_d10_native(+    int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D10])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_poly_decompress_d10_avx2_asm(r, a, mlk_decompress_d10_data);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* !MLK_CONFIG_NO_DECAPS_API */+#endif /* (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) &&         \+          (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 || MLKEM_K == 3) \+        */++#if (!defined(MLK_CONFIG_NO_ENCAPS_API) ||  \+     !defined(MLK_CONFIG_NO_DECAPS_API)) && \+    (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4)+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_compress_d5_native(+    uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D5], const int16_t a[MLKEM_N])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_poly_compress_d5_avx2_asm(r, a, mlk_compress_d5_data);+  return MLK_NATIVE_FUNC_SUCCESS;+}++MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_compress_d11_native(+    uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D11], const int16_t a[MLKEM_N])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_poly_compress_d11_avx2_asm(r, a, mlk_compress_d11_data);+  return MLK_NATIVE_FUNC_SUCCESS;+}++#if !defined(MLK_CONFIG_NO_DECAPS_API)+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_decompress_d5_native(+    int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D5])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_poly_decompress_d5_avx2_asm(r, a, mlk_decompress_d5_data);+  return MLK_NATIVE_FUNC_SUCCESS;+}++MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_poly_decompress_d11_native(+    int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D11])+{+  if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2))+  {+    return MLK_NATIVE_FUNC_FALLBACK;+  }++  mlk_poly_decompress_d11_avx2_asm(r, a, mlk_decompress_d11_data);+  return MLK_NATIVE_FUNC_SUCCESS;+}+#endif /* !MLK_CONFIG_NO_DECAPS_API */+#endif /* (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) && \+          (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4) */++#endif /* !__ASSEMBLER__ */++#endif /* !MLK_NATIVE_X86_64_META_H */
+ cbits/mlkem/src/native/x86_64/src/arith_native_x86_64.h view
@@ -0,0 +1,327 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLK_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H+#define MLK_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H++#include "../../../common.h"++#include <stdint.h>+#include "compress_consts.h"+#include "consts.h"++#define MLK_AVX2_REJ_UNIFORM_BUFLEN \+  (3 * 168) /* REJ_UNIFORM_NBLOCKS * SHAKE128_RATE */++#define mlk_rej_uniform_table MLK_NAMESPACE(rej_uniform_table)+MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_rej_uniform_table[4096];++#define mlk_rej_uniform_avx2_asm MLK_NAMESPACE(rej_uniform_avx2_asm)+MLK_MUST_CHECK_RETURN_VALUE MLK_SYSV_ABI+uint64_t mlk_rej_uniform_avx2_asm(int16_t *r, const uint8_t *buf,+                                  unsigned buflen, const uint8_t *table)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mlkem_rej_uniform_avx2_asm.ml. */+__contract__(+    requires(buflen % 12 == 0)+    requires(memory_no_alias(buf, buflen))+    requires(table == mlk_rej_uniform_table)+    requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+    assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+    ensures(return_value <= MLKEM_N)+    ensures(array_bound(r, 0, (unsigned) return_value, 0, MLKEM_Q))+);++#define mlk_ntt_avx2_asm MLK_NAMESPACE(ntt_avx2_asm)+MLK_SYSV_ABI+void mlk_ntt_avx2_asm(int16_t *r, const int16_t *qdata)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mlkem_ntt_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  requires(array_abs_bound(r, 0, MLKEM_N, 8192))+  requires(qdata == mlk_qdata)+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  /* check-magic: off */+  ensures(array_abs_bound(r, 0, MLKEM_N, 23595))+  /* check-magic: on */+);++#define mlk_invntt_avx2_asm MLK_NAMESPACE(invntt_avx2_asm)+MLK_SYSV_ABI+void mlk_invntt_avx2_asm(int16_t *r, const int16_t *qdata)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mlkem_intt_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  requires(qdata == mlk_qdata)+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  /* check-magic: off */+  ensures(array_abs_bound(r, 0, MLKEM_N, 26632))+  /* check-magic: on */+);++#define mlk_nttunpack_avx2_asm MLK_NAMESPACE(nttunpack_avx2_asm)+MLK_SYSV_ABI+void mlk_nttunpack_avx2_asm(int16_t *r)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mlkem_nttunpack_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  requires(array_bound(r, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  /* Output is a permutation of input: every output coefficient+   * is some input coefficient */+  ensures(forall(i, 0, MLKEM_N, exists(j, 0, MLKEM_N,+    r[i] == old(*(int16_t (*)[MLKEM_N])r)[j])))+);++#define mlk_reduce_avx2_asm MLK_NAMESPACE(reduce_avx2_asm)+MLK_SYSV_ABI+void mlk_reduce_avx2_asm(int16_t *r)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mlkem_reduce_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  ensures(array_bound(r, 0, MLKEM_N, 0, MLKEM_Q))+);++#define mlk_poly_mulcache_compute_avx2_asm \+  MLK_NAMESPACE(poly_mulcache_compute_avx2_asm)+MLK_SYSV_ABI+void mlk_poly_mulcache_compute_avx2_asm(int16_t *out, const int16_t *in,+                                        const int16_t *qdata)+/* This must be kept in sync with the HOL-Light specification+ * in proofs/hol_light/x86_64/proofs/mlkem_poly_mulcache_compute_avx2_asm.ml */+__contract__(+  requires(memory_no_alias(out, sizeof(int16_t) * (MLKEM_N / 2)))+  requires(memory_no_alias(in, sizeof(int16_t) * MLKEM_N))+  requires(qdata == mlk_qdata)+  assigns(memory_slice(out, sizeof(int16_t) * (MLKEM_N / 2)))+  ensures(array_abs_bound(out, 0, MLKEM_N/2, MLKEM_Q))+);++#define mlk_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm \+  MLK_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k2_avx2_asm)+MLK_SYSV_ABI+void mlk_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm(+    int16_t *r, const int16_t *a, const int16_t *b, const int16_t *b_cache)+/* This must be kept in sync with the HOL-Light specification in+ * proofs/hol_light/x86_64/proofs/mlkem_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm.ml.+ */+__contract__(+    requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+    requires(memory_no_alias(a, sizeof(int16_t) * 2 * MLKEM_N))+    requires(memory_no_alias(b, sizeof(int16_t) * 2 * MLKEM_N))+    requires(memory_no_alias(b_cache, sizeof(int16_t) * 2 * (MLKEM_N / 2)))+    requires(array_abs_bound(a, 0, 2 * MLKEM_N, MLKEM_UINT12_LIMIT + 1))+    assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+);++#define mlk_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm \+  MLK_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k3_avx2_asm)+MLK_SYSV_ABI+void mlk_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm(+    int16_t *r, const int16_t *a, const int16_t *b, const int16_t *b_cache)+/* This must be kept in sync with the HOL-Light specification in+ * proofs/hol_light/x86_64/proofs/mlkem_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm.ml.+ */+__contract__(+    requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+    requires(memory_no_alias(a, sizeof(int16_t) * 3 * MLKEM_N))+    requires(memory_no_alias(b, sizeof(int16_t) * 3 * MLKEM_N))+    requires(memory_no_alias(b_cache, sizeof(int16_t) * 3 * (MLKEM_N / 2)))+    requires(array_abs_bound(a, 0, 3 * MLKEM_N, MLKEM_UINT12_LIMIT + 1))+    assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+);++#define mlk_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm \+  MLK_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k4_avx2_asm)+MLK_SYSV_ABI+void mlk_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm(+    int16_t *r, const int16_t *a, const int16_t *b, const int16_t *b_cache)+/* This must be kept in sync with the HOL-Light specification in+ * proofs/hol_light/x86_64/proofs/mlkem_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm.ml.+ */+__contract__(+    requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+    requires(memory_no_alias(a, sizeof(int16_t) * 4 * MLKEM_N))+    requires(memory_no_alias(b, sizeof(int16_t) * 4 * MLKEM_N))+    requires(memory_no_alias(b_cache, sizeof(int16_t) * 4 * (MLKEM_N / 2)))+    requires(array_abs_bound(a, 0, 4 * MLKEM_N, MLKEM_UINT12_LIMIT + 1))+    assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+);++#define mlk_ntttobytes_avx2_asm MLK_NAMESPACE(ntttobytes_avx2_asm)+MLK_SYSV_ABI+void mlk_ntttobytes_avx2_asm(uint8_t *r, const int16_t *a)+/* This must be kept in sync with the HOL-Light specification in+ * proofs/hol_light/x86_64/proofs/mlkem_ntttobytes_avx2_asm.ml.+ */+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYBYTES))+  requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N))+  requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYBYTES))+);++#define mlk_nttfrombytes_avx2_asm MLK_NAMESPACE(nttfrombytes_avx2_asm)+MLK_SYSV_ABI+void mlk_nttfrombytes_avx2_asm(int16_t *r, const uint8_t *a)+/* This must be kept in sync with the HOL-Light specification in+ * proofs/hol_light/x86_64/proofs/mlkem_nttfrombytes_avx2_asm.ml.+ */+__contract__(+  requires(memory_no_alias(a, MLKEM_POLYBYTES))+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  ensures(array_bound(r, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT))+);++#define mlk_tomont_avx2_asm MLK_NAMESPACE(tomont_avx2_asm)+MLK_SYSV_ABI+void mlk_tomont_avx2_asm(int16_t *r)+/* This must be kept in sync with the HOL-Light specification in+ * proofs/hol_light/x86_64/proofs/mlkem_tomont_avx2_asm.ml.+ */+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  ensures(array_abs_bound(r, 0, MLKEM_N, MLKEM_Q))+);++#define mlk_poly_compress_d4_avx2_asm MLK_NAMESPACE(poly_compress_d4_avx2_asm)+MLK_SYSV_ABI+void mlk_poly_compress_d4_avx2_asm(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D4],+                                   const int16_t *MLK_RESTRICT a,+                                   const uint8_t *data)+/* This must be kept in sync with the HOL-Light specification in+ * proofs/hol_light/x86_64/proofs/mlkem_poly_compress_d4_avx2_asm.ml.+ */+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D4))+  requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N))+  requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q))+  requires(data == mlk_compress_d4_data)+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D4))+);++#define mlk_poly_decompress_d4_avx2_asm \+  MLK_NAMESPACE(poly_decompress_d4_avx2_asm)+MLK_SYSV_ABI+void mlk_poly_decompress_d4_avx2_asm(+    int16_t *MLK_RESTRICT r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D4],+    const uint8_t *data)+/* This must be kept in sync with the HOL-Light specification in+ * proofs/hol_light/x86_64/proofs/mlkem_poly_decompress_d4_avx2_asm.ml.+ */+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D4))+  requires(data == mlk_decompress_d4_data)+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  ensures(array_bound(r, 0, MLKEM_N, 0, MLKEM_Q))+);++#define mlk_poly_compress_d10_avx2_asm MLK_NAMESPACE(poly_compress_d10_avx2_asm)+MLK_SYSV_ABI+void mlk_poly_compress_d10_avx2_asm(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D10],+                                    const int16_t *MLK_RESTRICT a,+                                    const uint8_t *data)+/* This must be kept in sync with the HOL-Light specification in+ * proofs/hol_light/x86_64/proofs/mlkem_poly_compress_d10_avx2_asm.ml.+ */+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D10))+  requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N))+  requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q))+  requires(data == mlk_compress_d10_data)+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D10))+);++#define mlk_poly_decompress_d10_avx2_asm \+  MLK_NAMESPACE(poly_decompress_d10_avx2_asm)+MLK_SYSV_ABI+void mlk_poly_decompress_d10_avx2_asm(+    int16_t *MLK_RESTRICT r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D10],+    const uint8_t *data)+/* This must be kept in sync with the HOL-Light specification in+ * proofs/hol_light/x86_64/proofs/mlkem_poly_decompress_d10_avx2_asm.ml.+ */+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D10))+  requires(data == mlk_decompress_d10_data)+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  ensures(array_bound(r, 0, MLKEM_N, 0, MLKEM_Q))+);++#define mlk_poly_compress_d5_avx2_asm MLK_NAMESPACE(poly_compress_d5_avx2_asm)+MLK_SYSV_ABI+void mlk_poly_compress_d5_avx2_asm(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D5],+                                   const int16_t *MLK_RESTRICT a,+                                   const uint8_t *data)+/* This must be kept in sync with the HOL-Light specification in+ * proofs/hol_light/x86_64/proofs/mlkem_poly_compress_d5_avx2_asm.ml.+ */+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D5))+  requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N))+  requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q))+  requires(data == mlk_compress_d5_data)+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D5))+);++#define mlk_poly_decompress_d5_avx2_asm \+  MLK_NAMESPACE(poly_decompress_d5_avx2_asm)+MLK_SYSV_ABI+void mlk_poly_decompress_d5_avx2_asm(+    int16_t *MLK_RESTRICT r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D5],+    const uint8_t *data)+/* This must be kept in sync with the HOL-Light specification in+ * proofs/hol_light/x86_64/proofs/mlkem_poly_decompress_d5_avx2_asm.ml.+ */+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D5))+  requires(data == mlk_decompress_d5_data)+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  ensures(array_bound(r, 0, MLKEM_N, 0, MLKEM_Q))+);++#define mlk_poly_compress_d11_avx2_asm MLK_NAMESPACE(poly_compress_d11_avx2_asm)+MLK_SYSV_ABI+void mlk_poly_compress_d11_avx2_asm(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D11],+                                    const int16_t *MLK_RESTRICT a,+                                    const uint8_t *data)+/* This must be kept in sync with the HOL-Light specification in+ * proofs/hol_light/x86_64/proofs/mlkem_poly_compress_d11_avx2_asm.ml.+ */+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D11))+  requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N))+  requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q))+  requires(data == mlk_compress_d11_data)+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D11))+);++#define mlk_poly_decompress_d11_avx2_asm \+  MLK_NAMESPACE(poly_decompress_d11_avx2_asm)+MLK_SYSV_ABI+void mlk_poly_decompress_d11_avx2_asm(+    int16_t *MLK_RESTRICT r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D11],+    const uint8_t *data)+/* This must be kept in sync with the HOL-Light specification in+ * proofs/hol_light/x86_64/proofs/mlkem_poly_decompress_d11_avx2_asm.ml.+ */+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D11))+  requires(data == mlk_decompress_d11_data)+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  ensures(array_bound(r, 0, MLKEM_N, 0, MLKEM_Q))+);++#endif /* !MLK_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H */
+ cbits/mlkem/src/native/x86_64/src/compress_consts.c view
@@ -0,0 +1,115 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mlkem-native repository.+ *          Do not modify it directly.+ */++#include "../../../common.h"++#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    (!defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API))++#include "compress_consts.h"++#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) &&                   \+    (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2 || \+     MLKEM_K == 3)++MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t+    mlk_compress_d4_data[32] = {+        0, 0, 0, 0, 4, 0, 0, 0, 1, 0, 0, 0, 5, 0, 0, 0,+        2, 0, 0, 0, 6, 0, 0, 0, 3, 0, 0, 0, 7, 0, 0, 0, /* permdidx */+};++#if !defined(MLK_CONFIG_NO_DECAPS_API)+MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t+    mlk_decompress_d4_data[32] = {+        0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3,+        4, 4, 4, 4, 5, 5, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7, /* shufbidx */+};+#endif /* !MLK_CONFIG_NO_DECAPS_API */++MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t+    mlk_compress_d10_data[32] = {+        0,   1,   2,   3,   4,   8, 9,  10, 11, 12,  255,+        255, 255, 255, 255, 255, 9, 10, 11, 12, 255, 255,+        255, 255, 255, 255, 0,   1, 2,  3,  4,  8, /* shufbidx */+};++#if !defined(MLK_CONFIG_NO_DECAPS_API)+MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t+    mlk_decompress_d10_data[32] = {+        0, 1, 1, 2, 2, 3, 3, 4, 5, 6, 6, 7, 7, 8,  8,  9,+        2, 3, 3, 4, 4, 5, 5, 6, 7, 8, 8, 9, 9, 10, 10, 11, /* shufbidx */+};+#endif /* !MLK_CONFIG_NO_DECAPS_API */++#endif /* !MLK_CONFIG_MULTILEVEL_NO_SHARED &&                                 \+          (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 || MLKEM_K == 3) \+        */++#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4)++MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t+    mlk_compress_d5_data[32] = {+        0, 1,  2,  3,  4,   255, 255, 255, 255, 255, 8,+        9, 10, 11, 12, 255, 9,   10,  11,  12,  255, 0,+        1, 2,  3,  4,  255, 255, 255, 255, 255, 8, /* shufbidx */+};++#if !defined(MLK_CONFIG_NO_DECAPS_API)+/* shufbidx[0:32], mask[32:64], shift[64:96] */+MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t+    mlk_decompress_d5_data[96] = {+        0,   0, 0,   1, 1,   1,  1,   2,  2,   3, 3,   3, 3,   4, 4,   4, 5,  5,+        5,   6, 6,   6, 6,   7,  7,   8,  8,   8, 8,   9, 9,   9, /* shufbidx */+        31,  0, 224, 3, 124, 0,  128, 15, 240, 1, 62,  0, 192, 7, 248, 0, 31, 0,+        224, 3, 124, 0, 128, 15, 240, 1,  62,  0, 192, 7, 248, 0, /* mask */+        0,   4, 32,  0, 0,   1,  8,   0,  64,  0, 0,   2, 16,  0, 128, 0, 0,  4,+        32,  0, 0,   1, 8,   0,  64,  0,  0,   2, 16,  0, 128, 0, /* shift */+};+#endif /* !MLK_CONFIG_NO_DECAPS_API */++/* srlvqidx[0:32], shufbidx[32:64] */+MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t+    mlk_compress_d11_data[64] =+        {+            10,  0,   0,   0,   0,   0,  0, 0, 30, 0, 0,+            0,   0,   0,   0,   0,   10, 0, 0, 0,  0, 0,+            0,   0,   30,  0,   0,   0,  0, 0, 0,  0, /* srlvqidx */+            0,   1,   2,   3,   4,   5,  6, 7, 8,  9, 10,+            255, 255, 255, 255, 255, 5,  6, 7, 8,  9, 10,+            255, 255, 255, 255, 0,   0,  1, 2, 3,  4, /* shufbidx */+};++#if !defined(MLK_CONFIG_NO_DECAPS_API)+/* shufbidx[0:32], srlvdidx[32:64], srlvqidx[64:96], shift[96:128] */+MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t+    mlk_decompress_d11_data[128] = {+        0,  1, 1, 2, 2, 3, 4,  5, 5, 6, 6, 7,  8,  9,  9,  10,+        3,  4, 4, 5, 5, 6, 7,  8, 8, 9, 9, 10, 11, 12, 12, 13, /* shufbidx */+        0,  0, 0, 0, 1, 0, 0,  0, 0, 0, 0, 0,  0,  0,  0,  0,+        0,  0, 0, 0, 1, 0, 0,  0, 0, 0, 0, 0,  0,  0,  0,  0, /* srlvdidx */+        0,  0, 0, 0, 0, 0, 0,  0, 2, 0, 0, 0,  0,  0,  0,  0,+        0,  0, 0, 0, 0, 0, 0,  0, 2, 0, 0, 0,  0,  0,  0,  0, /* srlvqidx */+        32, 0, 4, 0, 1, 0, 32, 0, 8, 0, 1, 0,  32, 0,  4,  0,+        32, 0, 4, 0, 1, 0, 32, 0, 8, 0, 1, 0,  32, 0,  4,  0, /* shift */+};+#endif /* !MLK_CONFIG_NO_DECAPS_API */++#endif /* !MLK_CONFIG_MULTILEVEL_NO_SHARED && \+          (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4) */++#else /* MLK_ARITH_BACKEND_X86_64_DEFAULT && (!MLK_CONFIG_NO_ENCAPS_API || \+         !MLK_CONFIG_NO_DECAPS_API) */++MLK_EMPTY_CU(avx2_compress_consts)++#endif /* !(MLK_ARITH_BACKEND_X86_64_DEFAULT && (!MLK_CONFIG_NO_ENCAPS_API || \+          !MLK_CONFIG_NO_DECAPS_API)) */
+ cbits/mlkem/src/native/x86_64/src/compress_consts.h view
@@ -0,0 +1,53 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mlkem-native repository.+ *          Do not modify it directly.+ */++#ifndef MLK_NATIVE_X86_64_SRC_COMPRESS_CONSTS_H+#define MLK_NATIVE_X86_64_SRC_COMPRESS_CONSTS_H++#include "../../../common.h"++#ifndef __ASSEMBLER__++#define mlk_compress_d4_data MLK_NAMESPACE(compress_d4_data)+MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_compress_d4_data[32];++#if !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_decompress_d4_data MLK_NAMESPACE(decompress_d4_data)+MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_decompress_d4_data[32];+#endif++#define mlk_compress_d10_data MLK_NAMESPACE(compress_d10_data)+MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_compress_d10_data[32];++#if !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_decompress_d10_data MLK_NAMESPACE(decompress_d10_data)+MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_decompress_d10_data[32];+#endif++#define mlk_compress_d5_data MLK_NAMESPACE(compress_d5_data)+MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_compress_d5_data[32];++#if !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_decompress_d5_data MLK_NAMESPACE(decompress_d5_data)+MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_decompress_d5_data[96];+#endif++#define mlk_compress_d11_data MLK_NAMESPACE(compress_d11_data)+MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_compress_d11_data[64];++#if !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_decompress_d11_data MLK_NAMESPACE(decompress_d11_data)+MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_decompress_d11_data[128];+#endif++#endif /* !__ASSEMBLER__ */++#endif /* !MLK_NATIVE_X86_64_SRC_COMPRESS_CONSTS_H */
+ cbits/mlkem/src/native/x86_64/src/consts.c view
@@ -0,0 +1,102 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mlkem-native repository.+ *          Do not modify it directly.+ */++#include "../../../common.h"++#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++#include "consts.h"++/*+ * Table of zeta values used in the AVX2 NTTs+ * See autogen for details.+ */+MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const int16_t mlk_qdata[624] = {+    3854,   3340,   2826,   2312,   1798,   1284,   770,    256,    3854,+    3340,   2826,   2312,   1798,   1284,   770,    256,    7,      0,+    6,      0,      5,      0,      4,      0,      3,      0,      2,+    0,      1,      0,      0,      0,      31498,  31498,  31498,  31498,+    -758,   -758,   -758,   -758,   0,      0,      0,      0,      0,+    0,      0,      0,      14745,  14745,  14745,  14745,  14745,  14745,+    14745,  14745,  14745,  14745,  14745,  14745,  14745,  14745,  14745,+    14745,  -359,   -359,   -359,   -359,   -359,   -359,   -359,   -359,+    -359,   -359,   -359,   -359,   -359,   -359,   -359,   -359,   13525,+    13525,  13525,  13525,  13525,  13525,  13525,  13525,  -12402, -12402,+    -12402, -12402, -12402, -12402, -12402, -12402, 1493,   1493,   1493,+    1493,   1493,   1493,   1493,   1493,   1422,   1422,   1422,   1422,+    1422,   1422,   1422,   1422,   -20907, -20907, -20907, -20907, 27758,+    27758,  27758,  27758,  -3799,  -3799,  -3799,  -3799,  -15690, -15690,+    -15690, -15690, -171,   -171,   -171,   -171,   622,    622,    622,+    622,    1577,   1577,   1577,   1577,   182,    182,    182,    182,+    -5827,  -5827,  17363,  17363,  -26360, -26360, -29057, -29057, 5571,+    5571,   -1102,  -1102,  21438,  21438,  -26242, -26242, 573,    573,+    -1325,  -1325,  264,    264,    383,    383,    -829,   -829,   1458,+    1458,   -1602,  -1602,  -130,   -130,   -5689,  -6516,  1496,   30967,+    -23565, 20179,  20710,  25080,  -12796, 26616,  16064,  -12442, 9134,+    -650,   -25986, 27837,  1223,   652,    -552,   1015,   -1293,  1491,+    -282,   -1544,  516,    -8,     -320,   -666,   -1618,  -1162,  126,+    1469,   -335,   -11477, -32227, 20494,  -27738, 945,    -14883, 6182,+    32010,  10631,  29175,  -28762, -18486, 17560,  -14430, -5276,  -1103,+    555,    -1251,  1550,   422,    177,    -291,   1574,   -246,   1159,+    -777,   -602,   -1590,  -872,   418,    -156,   11182,  13387,  -14233,+    -21655, 13131,  -4587,  23092,  5493,   -32502, 30317,  -18741, 12639,+    20100,  18525,  19529,  -12619, 430,    843,    871,    105,    587,+    -235,   -460,   1653,   778,    -147,   1483,   1119,   644,    349,+    329,    -75,    787,    787,    787,    787,    787,    787,    787,+    787,    787,    787,    787,    787,    787,    787,    787,    787,+    -1517,  -1517,  -1517,  -1517,  -1517,  -1517,  -1517,  -1517,  -1517,+    -1517,  -1517,  -1517,  -1517,  -1517,  -1517,  -1517,  28191,  28191,+    28191,  28191,  28191,  28191,  28191,  28191,  -16694, -16694, -16694,+    -16694, -16694, -16694, -16694, -16694, 287,    287,    287,    287,+    287,    287,    287,    287,    202,    202,    202,    202,    202,+    202,    202,    202,    10690,  10690,  10690,  10690,  1358,   1358,+    1358,   1358,   -11202, -11202, -11202, -11202, 31164,  31164,  31164,+    31164,  962,    962,    962,    962,    -1202,  -1202,  -1202,  -1202,+    -1474,  -1474,  -1474,  -1474,  1468,   1468,   1468,   1468,   -28073,+    -28073, 24313,  24313,  -10532, -10532, 8800,   8800,   18426,  18426,+    8859,   8859,   26675,  26675,  -16163, -16163, -681,   -681,   1017,+    1017,   732,    732,    608,    608,    -1542,  -1542,  411,    411,+    -205,   -205,   -1571,  -1571,  19883,  -28250, -15887, -8898,  -28309,+    9075,   -30199, 18249,  13426,  14017,  -29156, -12757, 16832,  4311,+    -24155, -17915, -853,   -90,    -271,   830,    107,    -1421,  -247,+    -951,   -398,   961,    -1508,  -725,   448,    -1065,  677,    -1275,+    -31183, 25435,  -7382,  24391,  -20927, 10946,  24214,  16989,  10335,+    -7934,  -22502, 10906,  31636,  28644,  23998,  -17422, 817,    603,+    1322,   -1465,  -1215,  1218,   -874,   -1187,  -1185,  -1278,  -1510,+    -870,   -108,   996,    958,    1522,   20297,  2146,   15355,  -32384,+    -6280,  -14903, -11044, 14469,  -21498, -20198, 23210,  -17442, -23860,+    -20257, 7756,   23132,  1097,   610,    -1285,  384,    -136,   -1335,+    220,    -1659,  -1530,  794,    -854,   478,    -308,   991,    -1460,+    1628,   -1103,  555,    -1251,  1550,   422,    177,    -291,   1574,+    -246,   1159,   -777,   -602,   -1590,  -872,   418,    -156,   430,+    843,    871,    105,    587,    -235,   -460,   1653,   778,    -147,+    1483,   1119,   644,    349,    329,    -75,    817,    603,    1322,+    -1465,  -1215,  1218,   -874,   -1187,  -1185,  -1278,  -1510,  -870,+    -108,   996,    958,    1522,   1097,   610,    -1285,  384,    -136,+    -1335,  220,    -1659,  -1530,  794,    -854,   478,    -308,   991,+    -1460,  1628,   -335,   -11477, -32227, 20494,  -27738, 945,    -14883,+    6182,   32010,  10631,  29175,  -28762, -18486, 17560,  -14430, -5276,+    11182,  13387,  -14233, -21655, 13131,  -4587,  23092,  5493,   -32502,+    30317,  -18741, 12639,  20100,  18525,  19529,  -12619, -31183, 25435,+    -7382,  24391,  -20927, 10946,  24214,  16989,  10335,  -7934,  -22502,+    10906,  31636,  28644,  23998,  -17422, 20297,  2146,   15355,  -32384,+    -6280,  -14903, -11044, 14469,  -21498, -20198, 23210,  -17442, -23860,+    -20257, 7756,   23132,+};++#else /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+       */++MLK_EMPTY_CU(avx2_consts)++#endif /* !(MLK_ARITH_BACKEND_X86_64_DEFAULT && \+          !MLK_CONFIG_MULTILEVEL_NO_SHARED) */
+ cbits/mlkem/src/native/x86_64/src/consts.h view
@@ -0,0 +1,25 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mlkem-native repository.+ *          Do not modify it directly.+ */++#ifndef MLK_NATIVE_X86_64_SRC_CONSTS_H+#define MLK_NATIVE_X86_64_SRC_CONSTS_H+#include "../../../common.h"+#define MLK_AVX2_BACKEND_DATA_OFFSET_REVIDXB 0+#define MLK_AVX2_BACKEND_DATA_OFFSET_REVIDXD 16+#define MLK_AVX2_BACKEND_DATA_OFFSET_ZETAS_EXP 32+#define MLK_AVX2_BACKEND_DATA_OFFSET_MULCACHE_TWIDDLES 496++#ifndef __ASSEMBLER__+#define mlk_qdata MLK_NAMESPACE(qdata)+MLK_INTERNAL_DATA_DECLARATION const int16_t mlk_qdata[624];+#endif++#endif /* !MLK_NATIVE_X86_64_SRC_CONSTS_H */
+ cbits/mlkem/src/native/x86_64/src/mlkem_intt_avx2_asm.S view
@@ -0,0 +1,743 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [AVX2_NTT]+ *   Faster AVX2 optimized NTT multiplication for Ring-LWE lattice cryptography.+ *   Gregor Seiler+ *   https://eprint.iacr.org/2018/039+ *+ * - [REF_AVX2]+ *   CRYSTALS-Kyber optimized AVX2 implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Kyber implementation @[REF_AVX2].+ *+ * The core ideas behind the implementation are described in @[AVX2_NTT].+ *+ * Changes:+ * - Different placement of modular reductions to simplify+ *   reasoning of non-overflow+ */++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    (!defined(MLK_CONFIG_NO_ENCAPS_API) || \+     !defined(MLK_CONFIG_NO_DECAPS_API))+/*yaml+  Name: invntt_avx2_asm+  Description: x86_64 AVX2 inverse NTT+  Signature: void mlk_invntt_avx2_asm(int16_t *r, const int16_t *qdata)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 512+      permissions: read/write+      c_parameter: int16_t *r+      description: Input/output polynomial (256 x int16_t)+    rsi:+      type: buffer+      size_bytes: 1248+      permissions: read-only+      c_parameter: const int16_t *qdata+      description: Precomputed constants (624 x int16_t)+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_intt_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(invntt_avx2_asm)+MLK_ASM_FN_SYMBOL(invntt_avx2_asm)++        .cfi_startproc+        movl $0xd010d01, %eax        # imm = 0xD010D01+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        movl $0xd8a1d8a1, %eax       # imm = 0xD8A1D8A1+        vmovd %eax, %xmm2+        vpbroadcastd %xmm2, %ymm2+        movl $0x5a105a1, %eax        # imm = 0x5A105A1+        vmovd %eax, %xmm3+        vpbroadcastd %xmm3, %ymm3+        vmovdqa (%rdi), %ymm4+        vmovdqa 0x40(%rdi), %ymm6+        vmovdqa 0x20(%rdi), %ymm5+        vmovdqa 0x60(%rdi), %ymm7+        vpmullw %ymm2, %ymm4, %ymm12+        vpmulhw %ymm3, %ymm4, %ymm4+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm4, %ymm4+        vpmullw %ymm2, %ymm6, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm6+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm6, %ymm6+        vpmullw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm5, %ymm5+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm5, %ymm5+        vpmullw %ymm2, %ymm7, %ymm12+        vpmulhw %ymm3, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm7, %ymm7+        vmovdqa 0x80(%rdi), %ymm8+        vmovdqa 0xc0(%rdi), %ymm10+        vmovdqa 0xa0(%rdi), %ymm9+        vmovdqa 0xe0(%rdi), %ymm11+        vpmullw %ymm2, %ymm8, %ymm12+        vpmulhw %ymm3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm8, %ymm8+        vpmullw %ymm2, %ymm10, %ymm12+        vpmulhw %ymm3, %ymm10, %ymm10+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm10, %ymm10+        vpmullw %ymm2, %ymm9, %ymm12+        vpmulhw %ymm3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm9, %ymm9+        vpmullw %ymm2, %ymm11, %ymm12+        vpmulhw %ymm3, %ymm11, %ymm11+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm11, %ymm11+        vpermq $0x4e, 0x3a0(%rsi), %ymm15 # ymm15 = mem[2,3,0,1]+        vpermq $0x4e, 0x360(%rsi), %ymm1 # ymm1 = mem[2,3,0,1]+        vpermq $0x4e, 0x3c0(%rsi), %ymm2 # ymm2 = mem[2,3,0,1]+        vpermq $0x4e, 0x380(%rsi), %ymm3 # ymm3 = mem[2,3,0,1]+        vmovdqa (%rsi), %ymm12+        vpshufb %ymm12, %ymm15, %ymm15+        vpshufb %ymm12, %ymm1, %ymm1+        vpshufb %ymm12, %ymm2, %ymm2+        vpshufb %ymm12, %ymm3, %ymm3+        vpsubw %ymm4, %ymm6, %ymm12+        vpaddw %ymm6, %ymm4, %ymm4+        vpsubw %ymm5, %ymm7, %ymm13+        vpmullw %ymm15, %ymm12, %ymm6+        vpaddw %ymm7, %ymm5, %ymm5+        vpsubw %ymm8, %ymm10, %ymm14+        vpmullw %ymm15, %ymm13, %ymm7+        vpaddw %ymm10, %ymm8, %ymm8+        vpsubw %ymm9, %ymm11, %ymm15+        vpmullw %ymm1, %ymm14, %ymm10+        vpaddw %ymm11, %ymm9, %ymm9+        vpmullw %ymm1, %ymm15, %ymm11+        vpmulhw %ymm2, %ymm12, %ymm12+        vpmulhw %ymm2, %ymm13, %ymm13+        vpmulhw %ymm3, %ymm14, %ymm14+        vpmulhw %ymm3, %ymm15, %ymm15+        vpmulhw %ymm0, %ymm6, %ymm6+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm10, %ymm10+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm6, %ymm12, %ymm6+        vpsubw %ymm7, %ymm13, %ymm7+        vpsubw %ymm10, %ymm14, %ymm10+        vpsubw %ymm11, %ymm15, %ymm11+        vpermq $0x4e, 0x320(%rsi), %ymm2 # ymm2 = mem[2,3,0,1]+        vpermq $0x4e, 0x340(%rsi), %ymm3 # ymm3 = mem[2,3,0,1]+        vmovdqa (%rsi), %ymm1+        vpshufb %ymm1, %ymm2, %ymm2+        vpshufb %ymm1, %ymm3, %ymm3+        vpsubw %ymm4, %ymm8, %ymm12+        vpaddw %ymm8, %ymm4, %ymm4+        vpsubw %ymm5, %ymm9, %ymm13+        vpmullw %ymm2, %ymm12, %ymm8+        vpaddw %ymm9, %ymm5, %ymm5+        vpsubw %ymm6, %ymm10, %ymm14+        vpmullw %ymm2, %ymm13, %ymm9+        vpaddw %ymm10, %ymm6, %ymm6+        vpsubw %ymm7, %ymm11, %ymm15+        vpmullw %ymm2, %ymm14, %ymm10+        vpaddw %ymm11, %ymm7, %ymm7+        vpmullw %ymm2, %ymm15, %ymm11+        vpmulhw %ymm3, %ymm12, %ymm12+        vpmulhw %ymm3, %ymm13, %ymm13+        vpmulhw %ymm3, %ymm14, %ymm14+        vpmulhw %ymm3, %ymm15, %ymm15+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm10, %ymm10+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm8, %ymm12, %ymm8+        vpsubw %ymm9, %ymm13, %ymm9+        vpsubw %ymm10, %ymm14, %ymm10+        vpsubw %ymm11, %ymm15, %ymm11+        vpslld $0x10, %ymm5, %ymm3+        vpblendw $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7],ymm4[8],ymm3[9],ymm4[10],ymm3[11],ymm4[12],ymm3[13],ymm4[14],ymm3[15]+        vpsrld $0x10, %ymm4, %ymm4+        vpblendw $0xaa, %ymm5, %ymm4, %ymm5 # ymm5 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7],ymm4[8],ymm5[9],ymm4[10],ymm5[11],ymm4[12],ymm5[13],ymm4[14],ymm5[15]+        vpslld $0x10, %ymm7, %ymm4+        vpblendw $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7],ymm6[8],ymm4[9],ymm6[10],ymm4[11],ymm6[12],ymm4[13],ymm6[14],ymm4[15]+        vpsrld $0x10, %ymm6, %ymm6+        vpblendw $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7],ymm6[8],ymm7[9],ymm6[10],ymm7[11],ymm6[12],ymm7[13],ymm6[14],ymm7[15]+        vpslld $0x10, %ymm9, %ymm6+        vpblendw $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7],ymm8[8],ymm6[9],ymm8[10],ymm6[11],ymm8[12],ymm6[13],ymm8[14],ymm6[15]+        vpsrld $0x10, %ymm8, %ymm8+        vpblendw $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7],ymm8[8],ymm9[9],ymm8[10],ymm9[11],ymm8[12],ymm9[13],ymm8[14],ymm9[15]+        vpslld $0x10, %ymm11, %ymm8+        vpblendw $0xaa, %ymm8, %ymm10, %ymm8 # ymm8 = ymm10[0],ymm8[1],ymm10[2],ymm8[3],ymm10[4],ymm8[5],ymm10[6],ymm8[7],ymm10[8],ymm8[9],ymm10[10],ymm8[11],ymm10[12],ymm8[13],ymm10[14],ymm8[15]+        vpsrld $0x10, %ymm10, %ymm10+        vpblendw $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7],ymm10[8],ymm11[9],ymm10[10],ymm11[11],ymm10[12],ymm11[13],ymm10[14],ymm11[15]+        vmovdqa 0x20(%rsi), %ymm12+        vpermd 0x2e0(%rsi), %ymm12, %ymm2+        vpermd 0x300(%rsi), %ymm12, %ymm10+        vpsubw %ymm3, %ymm5, %ymm12+        vpaddw %ymm5, %ymm3, %ymm3+        vpsubw %ymm4, %ymm7, %ymm13+        vpmullw %ymm2, %ymm12, %ymm5+        vpaddw %ymm7, %ymm4, %ymm4+        vpsubw %ymm6, %ymm9, %ymm14+        vpmullw %ymm2, %ymm13, %ymm7+        vpaddw %ymm9, %ymm6, %ymm6+        vpsubw %ymm8, %ymm11, %ymm15+        vpmullw %ymm2, %ymm14, %ymm9+        vpaddw %ymm11, %ymm8, %ymm8+        vpmullw %ymm2, %ymm15, %ymm11+        vpmulhw %ymm10, %ymm12, %ymm12+        vpmulhw %ymm10, %ymm13, %ymm13+        vpmulhw %ymm10, %ymm14, %ymm14+        vpmulhw %ymm10, %ymm15, %ymm15+        vpmulhw %ymm0, %ymm5, %ymm5+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm5, %ymm12, %ymm5+        vpsubw %ymm7, %ymm13, %ymm7+        vpsubw %ymm9, %ymm14, %ymm9+        vpsubw %ymm11, %ymm15, %ymm11+        movl $0x4ebf4ebf, %eax       # imm = 0x4EBF4EBF+        vmovd %eax, %xmm1+        vpbroadcastd %xmm1, %ymm1+        vpmulhw %ymm1, %ymm3, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm3, %ymm3+        vmovsldup %ymm4, %ymm10   # ymm10 = ymm4[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm10, %ymm3, %ymm10 # ymm10 = ymm3[0],ymm10[1],ymm3[2],ymm10[3],ymm3[4],ymm10[5],ymm3[6],ymm10[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7]+        vmovsldup %ymm8, %ymm3    # ymm3 = ymm8[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm6, %ymm3 # ymm3 = ymm6[0],ymm3[1],ymm6[2],ymm3[3],ymm6[4],ymm3[5],ymm6[6],ymm3[7]+        vpsrlq $0x20, %ymm6, %ymm6+        vpblendd $0xaa, %ymm8, %ymm6, %ymm8 # ymm8 = ymm6[0],ymm8[1],ymm6[2],ymm8[3],ymm6[4],ymm8[5],ymm6[6],ymm8[7]+        vmovsldup %ymm7, %ymm6    # ymm6 = ymm7[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7]+        vpsrlq $0x20, %ymm5, %ymm5+        vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7]+        vmovsldup %ymm11, %ymm5   # ymm5 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm5, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm5[1],ymm9[2],ymm5[3],ymm9[4],ymm5[5],ymm9[6],ymm5[7]+        vpsrlq $0x20, %ymm9, %ymm9+        vpblendd $0xaa, %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[0],ymm11[1],ymm9[2],ymm11[3],ymm9[4],ymm11[5],ymm9[6],ymm11[7]+        vpermq $0x1b, 0x2a0(%rsi), %ymm2 # ymm2 = mem[3,2,1,0]+        vpermq $0x1b, 0x2c0(%rsi), %ymm9 # ymm9 = mem[3,2,1,0]+        vpsubw %ymm10, %ymm4, %ymm12+        vpaddw %ymm4, %ymm10, %ymm10+        vpsubw %ymm3, %ymm8, %ymm13+        vpmullw %ymm2, %ymm12, %ymm4+        vpaddw %ymm8, %ymm3, %ymm3+        vpsubw %ymm6, %ymm7, %ymm14+        vpmullw %ymm2, %ymm13, %ymm8+        vpaddw %ymm7, %ymm6, %ymm6+        vpsubw %ymm5, %ymm11, %ymm15+        vpmullw %ymm2, %ymm14, %ymm7+        vpaddw %ymm11, %ymm5, %ymm5+        vpmullw %ymm2, %ymm15, %ymm11+        vpmulhw %ymm9, %ymm12, %ymm12+        vpmulhw %ymm9, %ymm13, %ymm13+        vpmulhw %ymm9, %ymm14, %ymm14+        vpmulhw %ymm9, %ymm15, %ymm15+        vpmulhw %ymm0, %ymm4, %ymm4+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm4, %ymm12, %ymm4+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm7, %ymm14, %ymm7+        vpsubw %ymm11, %ymm15, %ymm11+        vpmulhw %ymm1, %ymm10, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm10, %ymm10+        vpunpcklqdq %ymm3, %ymm10, %ymm9 # ymm9 = ymm10[0],ymm3[0],ymm10[2],ymm3[2]+        vpunpckhqdq %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[1],ymm3[1],ymm10[3],ymm3[3]+        vpunpcklqdq %ymm5, %ymm6, %ymm10 # ymm10 = ymm6[0],ymm5[0],ymm6[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[1],ymm5[1],ymm6[3],ymm5[3]+        vpunpcklqdq %ymm8, %ymm4, %ymm6 # ymm6 = ymm4[0],ymm8[0],ymm4[2],ymm8[2]+        vpunpckhqdq %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[1],ymm8[1],ymm4[3],ymm8[3]+        vpunpcklqdq %ymm11, %ymm7, %ymm4 # ymm4 = ymm7[0],ymm11[0],ymm7[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[1],ymm11[1],ymm7[3],ymm11[3]+        vpermq $0x4e, 0x260(%rsi), %ymm2 # ymm2 = mem[2,3,0,1]+        vpermq $0x4e, 0x280(%rsi), %ymm7 # ymm7 = mem[2,3,0,1]+        vpsubw %ymm9, %ymm3, %ymm12+        vpaddw %ymm3, %ymm9, %ymm9+        vpsubw %ymm10, %ymm5, %ymm13+        vpmullw %ymm2, %ymm12, %ymm3+        vpaddw %ymm5, %ymm10, %ymm10+        vpsubw %ymm6, %ymm8, %ymm14+        vpmullw %ymm2, %ymm13, %ymm5+        vpaddw %ymm8, %ymm6, %ymm6+        vpsubw %ymm4, %ymm11, %ymm15+        vpmullw %ymm2, %ymm14, %ymm8+        vpaddw %ymm11, %ymm4, %ymm4+        vpmullw %ymm2, %ymm15, %ymm11+        vpmulhw %ymm7, %ymm12, %ymm12+        vpmulhw %ymm7, %ymm13, %ymm13+        vpmulhw %ymm7, %ymm14, %ymm14+        vpmulhw %ymm7, %ymm15, %ymm15+        vpmulhw %ymm0, %ymm3, %ymm3+        vpmulhw %ymm0, %ymm5, %ymm5+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm3, %ymm12, %ymm3+        vpsubw %ymm5, %ymm13, %ymm5+        vpsubw %ymm8, %ymm14, %ymm8+        vpsubw %ymm11, %ymm15, %ymm11+        vpmulhw %ymm1, %ymm9, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm9, %ymm9+        vperm2i128 $0x20, %ymm10, %ymm9, %ymm7 # ymm7 = ymm9[0,1],ymm10[0,1]+        vperm2i128 $0x31, %ymm10, %ymm9, %ymm10 # ymm10 = ymm9[2,3],ymm10[2,3]+        vperm2i128 $0x20, %ymm4, %ymm6, %ymm9 # ymm9 = ymm6[0,1],ymm4[0,1]+        vperm2i128 $0x31, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[2,3],ymm4[2,3]+        vperm2i128 $0x20, %ymm5, %ymm3, %ymm6 # ymm6 = ymm3[0,1],ymm5[0,1]+        vperm2i128 $0x31, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[2,3],ymm5[2,3]+        vperm2i128 $0x20, %ymm11, %ymm8, %ymm3 # ymm3 = ymm8[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm8, %ymm11 # ymm11 = ymm8[2,3],ymm11[2,3]+        vmovdqa 0x220(%rsi), %ymm2+        vmovdqa 0x240(%rsi), %ymm8+        vpsubw %ymm7, %ymm10, %ymm12+        vpaddw %ymm10, %ymm7, %ymm7+        vpsubw %ymm9, %ymm4, %ymm13+        vpmullw %ymm2, %ymm12, %ymm10+        vpaddw %ymm4, %ymm9, %ymm9+        vpsubw %ymm6, %ymm5, %ymm14+        vpmullw %ymm2, %ymm13, %ymm4+        vpaddw %ymm5, %ymm6, %ymm6+        vpsubw %ymm3, %ymm11, %ymm15+        vpmullw %ymm2, %ymm14, %ymm5+        vpaddw %ymm11, %ymm3, %ymm3+        vpmullw %ymm2, %ymm15, %ymm11+        vpmulhw %ymm8, %ymm12, %ymm12+        vpmulhw %ymm8, %ymm13, %ymm13+        vpmulhw %ymm8, %ymm14, %ymm14+        vpmulhw %ymm8, %ymm15, %ymm15+        vpmulhw %ymm0, %ymm10, %ymm10+        vpmulhw %ymm0, %ymm4, %ymm4+        vpmulhw %ymm0, %ymm5, %ymm5+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm10, %ymm12, %ymm10+        vpsubw %ymm4, %ymm13, %ymm4+        vpsubw %ymm5, %ymm14, %ymm5+        vpsubw %ymm11, %ymm15, %ymm11+        vpmulhw %ymm1, %ymm7, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm7, %ymm7+        vmovdqa %ymm7, (%rdi)+        vmovdqa %ymm9, 0x20(%rdi)+        vmovdqa %ymm6, 0x40(%rdi)+        vmovdqa %ymm3, 0x60(%rdi)+        vmovdqa %ymm10, 0x80(%rdi)+        vmovdqa %ymm4, 0xa0(%rdi)+        vmovdqa %ymm5, 0xc0(%rdi)+        vmovdqa %ymm11, 0xe0(%rdi)+        movl $0xd8a1d8a1, %eax       # imm = 0xD8A1D8A1+        vmovd %eax, %xmm2+        vpbroadcastd %xmm2, %ymm2+        movl $0x5a105a1, %eax        # imm = 0x5A105A1+        vmovd %eax, %xmm3+        vpbroadcastd %xmm3, %ymm3+        vmovdqa 0x100(%rdi), %ymm4+        vmovdqa 0x140(%rdi), %ymm6+        vmovdqa 0x120(%rdi), %ymm5+        vmovdqa 0x160(%rdi), %ymm7+        vpmullw %ymm2, %ymm4, %ymm12+        vpmulhw %ymm3, %ymm4, %ymm4+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm4, %ymm4+        vpmullw %ymm2, %ymm6, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm6+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm6, %ymm6+        vpmullw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm5, %ymm5+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm5, %ymm5+        vpmullw %ymm2, %ymm7, %ymm12+        vpmulhw %ymm3, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm7, %ymm7+        vmovdqa 0x180(%rdi), %ymm8+        vmovdqa 0x1c0(%rdi), %ymm10+        vmovdqa 0x1a0(%rdi), %ymm9+        vmovdqa 0x1e0(%rdi), %ymm11+        vpmullw %ymm2, %ymm8, %ymm12+        vpmulhw %ymm3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm8, %ymm8+        vpmullw %ymm2, %ymm10, %ymm12+        vpmulhw %ymm3, %ymm10, %ymm10+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm10, %ymm10+        vpmullw %ymm2, %ymm9, %ymm12+        vpmulhw %ymm3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm9, %ymm9+        vpmullw %ymm2, %ymm11, %ymm12+        vpmulhw %ymm3, %ymm11, %ymm11+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm11, %ymm11+        vpermq $0x4e, 0x1e0(%rsi), %ymm15 # ymm15 = mem[2,3,0,1]+        vpermq $0x4e, 0x1a0(%rsi), %ymm1 # ymm1 = mem[2,3,0,1]+        vpermq $0x4e, 0x200(%rsi), %ymm2 # ymm2 = mem[2,3,0,1]+        vpermq $0x4e, 0x1c0(%rsi), %ymm3 # ymm3 = mem[2,3,0,1]+        vmovdqa (%rsi), %ymm12+        vpshufb %ymm12, %ymm15, %ymm15+        vpshufb %ymm12, %ymm1, %ymm1+        vpshufb %ymm12, %ymm2, %ymm2+        vpshufb %ymm12, %ymm3, %ymm3+        vpsubw %ymm4, %ymm6, %ymm12+        vpaddw %ymm6, %ymm4, %ymm4+        vpsubw %ymm5, %ymm7, %ymm13+        vpmullw %ymm15, %ymm12, %ymm6+        vpaddw %ymm7, %ymm5, %ymm5+        vpsubw %ymm8, %ymm10, %ymm14+        vpmullw %ymm15, %ymm13, %ymm7+        vpaddw %ymm10, %ymm8, %ymm8+        vpsubw %ymm9, %ymm11, %ymm15+        vpmullw %ymm1, %ymm14, %ymm10+        vpaddw %ymm11, %ymm9, %ymm9+        vpmullw %ymm1, %ymm15, %ymm11+        vpmulhw %ymm2, %ymm12, %ymm12+        vpmulhw %ymm2, %ymm13, %ymm13+        vpmulhw %ymm3, %ymm14, %ymm14+        vpmulhw %ymm3, %ymm15, %ymm15+        vpmulhw %ymm0, %ymm6, %ymm6+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm10, %ymm10+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm6, %ymm12, %ymm6+        vpsubw %ymm7, %ymm13, %ymm7+        vpsubw %ymm10, %ymm14, %ymm10+        vpsubw %ymm11, %ymm15, %ymm11+        vpermq $0x4e, 0x160(%rsi), %ymm2 # ymm2 = mem[2,3,0,1]+        vpermq $0x4e, 0x180(%rsi), %ymm3 # ymm3 = mem[2,3,0,1]+        vmovdqa (%rsi), %ymm1+        vpshufb %ymm1, %ymm2, %ymm2+        vpshufb %ymm1, %ymm3, %ymm3+        vpsubw %ymm4, %ymm8, %ymm12+        vpaddw %ymm8, %ymm4, %ymm4+        vpsubw %ymm5, %ymm9, %ymm13+        vpmullw %ymm2, %ymm12, %ymm8+        vpaddw %ymm9, %ymm5, %ymm5+        vpsubw %ymm6, %ymm10, %ymm14+        vpmullw %ymm2, %ymm13, %ymm9+        vpaddw %ymm10, %ymm6, %ymm6+        vpsubw %ymm7, %ymm11, %ymm15+        vpmullw %ymm2, %ymm14, %ymm10+        vpaddw %ymm11, %ymm7, %ymm7+        vpmullw %ymm2, %ymm15, %ymm11+        vpmulhw %ymm3, %ymm12, %ymm12+        vpmulhw %ymm3, %ymm13, %ymm13+        vpmulhw %ymm3, %ymm14, %ymm14+        vpmulhw %ymm3, %ymm15, %ymm15+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm10, %ymm10+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm8, %ymm12, %ymm8+        vpsubw %ymm9, %ymm13, %ymm9+        vpsubw %ymm10, %ymm14, %ymm10+        vpsubw %ymm11, %ymm15, %ymm11+        vpslld $0x10, %ymm5, %ymm3+        vpblendw $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7],ymm4[8],ymm3[9],ymm4[10],ymm3[11],ymm4[12],ymm3[13],ymm4[14],ymm3[15]+        vpsrld $0x10, %ymm4, %ymm4+        vpblendw $0xaa, %ymm5, %ymm4, %ymm5 # ymm5 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7],ymm4[8],ymm5[9],ymm4[10],ymm5[11],ymm4[12],ymm5[13],ymm4[14],ymm5[15]+        vpslld $0x10, %ymm7, %ymm4+        vpblendw $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7],ymm6[8],ymm4[9],ymm6[10],ymm4[11],ymm6[12],ymm4[13],ymm6[14],ymm4[15]+        vpsrld $0x10, %ymm6, %ymm6+        vpblendw $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7],ymm6[8],ymm7[9],ymm6[10],ymm7[11],ymm6[12],ymm7[13],ymm6[14],ymm7[15]+        vpslld $0x10, %ymm9, %ymm6+        vpblendw $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7],ymm8[8],ymm6[9],ymm8[10],ymm6[11],ymm8[12],ymm6[13],ymm8[14],ymm6[15]+        vpsrld $0x10, %ymm8, %ymm8+        vpblendw $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7],ymm8[8],ymm9[9],ymm8[10],ymm9[11],ymm8[12],ymm9[13],ymm8[14],ymm9[15]+        vpslld $0x10, %ymm11, %ymm8+        vpblendw $0xaa, %ymm8, %ymm10, %ymm8 # ymm8 = ymm10[0],ymm8[1],ymm10[2],ymm8[3],ymm10[4],ymm8[5],ymm10[6],ymm8[7],ymm10[8],ymm8[9],ymm10[10],ymm8[11],ymm10[12],ymm8[13],ymm10[14],ymm8[15]+        vpsrld $0x10, %ymm10, %ymm10+        vpblendw $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7],ymm10[8],ymm11[9],ymm10[10],ymm11[11],ymm10[12],ymm11[13],ymm10[14],ymm11[15]+        vmovdqa 0x20(%rsi), %ymm12+        vpermd 0x120(%rsi), %ymm12, %ymm2+        vpermd 0x140(%rsi), %ymm12, %ymm10+        vpsubw %ymm3, %ymm5, %ymm12+        vpaddw %ymm5, %ymm3, %ymm3+        vpsubw %ymm4, %ymm7, %ymm13+        vpmullw %ymm2, %ymm12, %ymm5+        vpaddw %ymm7, %ymm4, %ymm4+        vpsubw %ymm6, %ymm9, %ymm14+        vpmullw %ymm2, %ymm13, %ymm7+        vpaddw %ymm9, %ymm6, %ymm6+        vpsubw %ymm8, %ymm11, %ymm15+        vpmullw %ymm2, %ymm14, %ymm9+        vpaddw %ymm11, %ymm8, %ymm8+        vpmullw %ymm2, %ymm15, %ymm11+        vpmulhw %ymm10, %ymm12, %ymm12+        vpmulhw %ymm10, %ymm13, %ymm13+        vpmulhw %ymm10, %ymm14, %ymm14+        vpmulhw %ymm10, %ymm15, %ymm15+        vpmulhw %ymm0, %ymm5, %ymm5+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm5, %ymm12, %ymm5+        vpsubw %ymm7, %ymm13, %ymm7+        vpsubw %ymm9, %ymm14, %ymm9+        vpsubw %ymm11, %ymm15, %ymm11+        movl $0x4ebf4ebf, %eax       # imm = 0x4EBF4EBF+        vmovd %eax, %xmm1+        vpbroadcastd %xmm1, %ymm1+        vpmulhw %ymm1, %ymm3, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm3, %ymm3+        vmovsldup %ymm4, %ymm10   # ymm10 = ymm4[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm10, %ymm3, %ymm10 # ymm10 = ymm3[0],ymm10[1],ymm3[2],ymm10[3],ymm3[4],ymm10[5],ymm3[6],ymm10[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7]+        vmovsldup %ymm8, %ymm3    # ymm3 = ymm8[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm6, %ymm3 # ymm3 = ymm6[0],ymm3[1],ymm6[2],ymm3[3],ymm6[4],ymm3[5],ymm6[6],ymm3[7]+        vpsrlq $0x20, %ymm6, %ymm6+        vpblendd $0xaa, %ymm8, %ymm6, %ymm8 # ymm8 = ymm6[0],ymm8[1],ymm6[2],ymm8[3],ymm6[4],ymm8[5],ymm6[6],ymm8[7]+        vmovsldup %ymm7, %ymm6    # ymm6 = ymm7[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7]+        vpsrlq $0x20, %ymm5, %ymm5+        vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7]+        vmovsldup %ymm11, %ymm5   # ymm5 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm5, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm5[1],ymm9[2],ymm5[3],ymm9[4],ymm5[5],ymm9[6],ymm5[7]+        vpsrlq $0x20, %ymm9, %ymm9+        vpblendd $0xaa, %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[0],ymm11[1],ymm9[2],ymm11[3],ymm9[4],ymm11[5],ymm9[6],ymm11[7]+        vpermq $0x1b, 0xe0(%rsi), %ymm2 # ymm2 = mem[3,2,1,0]+        vpermq $0x1b, 0x100(%rsi), %ymm9 # ymm9 = mem[3,2,1,0]+        vpsubw %ymm10, %ymm4, %ymm12+        vpaddw %ymm4, %ymm10, %ymm10+        vpsubw %ymm3, %ymm8, %ymm13+        vpmullw %ymm2, %ymm12, %ymm4+        vpaddw %ymm8, %ymm3, %ymm3+        vpsubw %ymm6, %ymm7, %ymm14+        vpmullw %ymm2, %ymm13, %ymm8+        vpaddw %ymm7, %ymm6, %ymm6+        vpsubw %ymm5, %ymm11, %ymm15+        vpmullw %ymm2, %ymm14, %ymm7+        vpaddw %ymm11, %ymm5, %ymm5+        vpmullw %ymm2, %ymm15, %ymm11+        vpmulhw %ymm9, %ymm12, %ymm12+        vpmulhw %ymm9, %ymm13, %ymm13+        vpmulhw %ymm9, %ymm14, %ymm14+        vpmulhw %ymm9, %ymm15, %ymm15+        vpmulhw %ymm0, %ymm4, %ymm4+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm4, %ymm12, %ymm4+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm7, %ymm14, %ymm7+        vpsubw %ymm11, %ymm15, %ymm11+        vpmulhw %ymm1, %ymm10, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm10, %ymm10+        vpunpcklqdq %ymm3, %ymm10, %ymm9 # ymm9 = ymm10[0],ymm3[0],ymm10[2],ymm3[2]+        vpunpckhqdq %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[1],ymm3[1],ymm10[3],ymm3[3]+        vpunpcklqdq %ymm5, %ymm6, %ymm10 # ymm10 = ymm6[0],ymm5[0],ymm6[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[1],ymm5[1],ymm6[3],ymm5[3]+        vpunpcklqdq %ymm8, %ymm4, %ymm6 # ymm6 = ymm4[0],ymm8[0],ymm4[2],ymm8[2]+        vpunpckhqdq %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[1],ymm8[1],ymm4[3],ymm8[3]+        vpunpcklqdq %ymm11, %ymm7, %ymm4 # ymm4 = ymm7[0],ymm11[0],ymm7[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[1],ymm11[1],ymm7[3],ymm11[3]+        vpermq $0x4e, 0xa0(%rsi), %ymm2 # ymm2 = mem[2,3,0,1]+        vpermq $0x4e, 0xc0(%rsi), %ymm7 # ymm7 = mem[2,3,0,1]+        vpsubw %ymm9, %ymm3, %ymm12+        vpaddw %ymm3, %ymm9, %ymm9+        vpsubw %ymm10, %ymm5, %ymm13+        vpmullw %ymm2, %ymm12, %ymm3+        vpaddw %ymm5, %ymm10, %ymm10+        vpsubw %ymm6, %ymm8, %ymm14+        vpmullw %ymm2, %ymm13, %ymm5+        vpaddw %ymm8, %ymm6, %ymm6+        vpsubw %ymm4, %ymm11, %ymm15+        vpmullw %ymm2, %ymm14, %ymm8+        vpaddw %ymm11, %ymm4, %ymm4+        vpmullw %ymm2, %ymm15, %ymm11+        vpmulhw %ymm7, %ymm12, %ymm12+        vpmulhw %ymm7, %ymm13, %ymm13+        vpmulhw %ymm7, %ymm14, %ymm14+        vpmulhw %ymm7, %ymm15, %ymm15+        vpmulhw %ymm0, %ymm3, %ymm3+        vpmulhw %ymm0, %ymm5, %ymm5+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm3, %ymm12, %ymm3+        vpsubw %ymm5, %ymm13, %ymm5+        vpsubw %ymm8, %ymm14, %ymm8+        vpsubw %ymm11, %ymm15, %ymm11+        vpmulhw %ymm1, %ymm9, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm9, %ymm9+        vperm2i128 $0x20, %ymm10, %ymm9, %ymm7 # ymm7 = ymm9[0,1],ymm10[0,1]+        vperm2i128 $0x31, %ymm10, %ymm9, %ymm10 # ymm10 = ymm9[2,3],ymm10[2,3]+        vperm2i128 $0x20, %ymm4, %ymm6, %ymm9 # ymm9 = ymm6[0,1],ymm4[0,1]+        vperm2i128 $0x31, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[2,3],ymm4[2,3]+        vperm2i128 $0x20, %ymm5, %ymm3, %ymm6 # ymm6 = ymm3[0,1],ymm5[0,1]+        vperm2i128 $0x31, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[2,3],ymm5[2,3]+        vperm2i128 $0x20, %ymm11, %ymm8, %ymm3 # ymm3 = ymm8[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm8, %ymm11 # ymm11 = ymm8[2,3],ymm11[2,3]+        vmovdqa 0x60(%rsi), %ymm2+        vmovdqa 0x80(%rsi), %ymm8+        vpsubw %ymm7, %ymm10, %ymm12+        vpaddw %ymm10, %ymm7, %ymm7+        vpsubw %ymm9, %ymm4, %ymm13+        vpmullw %ymm2, %ymm12, %ymm10+        vpaddw %ymm4, %ymm9, %ymm9+        vpsubw %ymm6, %ymm5, %ymm14+        vpmullw %ymm2, %ymm13, %ymm4+        vpaddw %ymm5, %ymm6, %ymm6+        vpsubw %ymm3, %ymm11, %ymm15+        vpmullw %ymm2, %ymm14, %ymm5+        vpaddw %ymm11, %ymm3, %ymm3+        vpmullw %ymm2, %ymm15, %ymm11+        vpmulhw %ymm8, %ymm12, %ymm12+        vpmulhw %ymm8, %ymm13, %ymm13+        vpmulhw %ymm8, %ymm14, %ymm14+        vpmulhw %ymm8, %ymm15, %ymm15+        vpmulhw %ymm0, %ymm10, %ymm10+        vpmulhw %ymm0, %ymm4, %ymm4+        vpmulhw %ymm0, %ymm5, %ymm5+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm10, %ymm12, %ymm10+        vpsubw %ymm4, %ymm13, %ymm4+        vpsubw %ymm5, %ymm14, %ymm5+        vpsubw %ymm11, %ymm15, %ymm11+        vpmulhw %ymm1, %ymm7, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm7, %ymm7+        vmovdqa %ymm7, 0x100(%rdi)+        vmovdqa %ymm9, 0x120(%rdi)+        vmovdqa %ymm6, 0x140(%rdi)+        vmovdqa %ymm3, 0x160(%rdi)+        vmovdqa %ymm10, 0x180(%rdi)+        vmovdqa %ymm4, 0x1a0(%rdi)+        vmovdqa %ymm5, 0x1c0(%rdi)+        vmovdqa %ymm11, 0x1e0(%rdi)+        vmovdqa (%rdi), %ymm4+        vmovdqa 0x100(%rdi), %ymm8+        vmovdqa 0x20(%rdi), %ymm5+        vmovdqa 0x120(%rdi), %ymm9+        vpbroadcastq 0x40(%rsi), %ymm2+        vmovdqa 0x40(%rdi), %ymm6+        vmovdqa 0x140(%rdi), %ymm10+        vmovdqa 0x60(%rdi), %ymm7+        vmovdqa 0x160(%rdi), %ymm11+        vpbroadcastq 0x48(%rsi), %ymm3+        vpsubw %ymm4, %ymm8, %ymm12+        vpaddw %ymm8, %ymm4, %ymm4+        vpsubw %ymm5, %ymm9, %ymm13+        vpmullw %ymm2, %ymm12, %ymm8+        vpaddw %ymm9, %ymm5, %ymm5+        vpsubw %ymm6, %ymm10, %ymm14+        vpmullw %ymm2, %ymm13, %ymm9+        vpaddw %ymm10, %ymm6, %ymm6+        vpsubw %ymm7, %ymm11, %ymm15+        vpmullw %ymm2, %ymm14, %ymm10+        vpaddw %ymm11, %ymm7, %ymm7+        vpmullw %ymm2, %ymm15, %ymm11+        vpmulhw %ymm3, %ymm12, %ymm12+        vpmulhw %ymm3, %ymm13, %ymm13+        vpmulhw %ymm3, %ymm14, %ymm14+        vpmulhw %ymm3, %ymm15, %ymm15+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm10, %ymm10+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm8, %ymm12, %ymm8+        vpsubw %ymm9, %ymm13, %ymm9+        vpsubw %ymm10, %ymm14, %ymm10+        vpsubw %ymm11, %ymm15, %ymm11+        vmovdqa %ymm4, (%rdi)+        vmovdqa %ymm5, 0x20(%rdi)+        vmovdqa %ymm6, 0x40(%rdi)+        vmovdqa %ymm7, 0x60(%rdi)+        vmovdqa %ymm8, 0x100(%rdi)+        vmovdqa %ymm9, 0x120(%rdi)+        vmovdqa %ymm10, 0x140(%rdi)+        vmovdqa %ymm11, 0x160(%rdi)+        vmovdqa 0x80(%rdi), %ymm4+        vmovdqa 0x180(%rdi), %ymm8+        vmovdqa 0xa0(%rdi), %ymm5+        vmovdqa 0x1a0(%rdi), %ymm9+        vpbroadcastq 0x40(%rsi), %ymm2+        vmovdqa 0xc0(%rdi), %ymm6+        vmovdqa 0x1c0(%rdi), %ymm10+        vmovdqa 0xe0(%rdi), %ymm7+        vmovdqa 0x1e0(%rdi), %ymm11+        vpbroadcastq 0x48(%rsi), %ymm3+        vpsubw %ymm4, %ymm8, %ymm12+        vpaddw %ymm8, %ymm4, %ymm4+        vpsubw %ymm5, %ymm9, %ymm13+        vpmullw %ymm2, %ymm12, %ymm8+        vpaddw %ymm9, %ymm5, %ymm5+        vpsubw %ymm6, %ymm10, %ymm14+        vpmullw %ymm2, %ymm13, %ymm9+        vpaddw %ymm10, %ymm6, %ymm6+        vpsubw %ymm7, %ymm11, %ymm15+        vpmullw %ymm2, %ymm14, %ymm10+        vpaddw %ymm11, %ymm7, %ymm7+        vpmullw %ymm2, %ymm15, %ymm11+        vpmulhw %ymm3, %ymm12, %ymm12+        vpmulhw %ymm3, %ymm13, %ymm13+        vpmulhw %ymm3, %ymm14, %ymm14+        vpmulhw %ymm3, %ymm15, %ymm15+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm10, %ymm10+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm8, %ymm12, %ymm8+        vpsubw %ymm9, %ymm13, %ymm9+        vpsubw %ymm10, %ymm14, %ymm10+        vpsubw %ymm11, %ymm15, %ymm11+        vmovdqa %ymm4, 0x80(%rdi)+        vmovdqa %ymm5, 0xa0(%rdi)+        vmovdqa %ymm6, 0xc0(%rdi)+        vmovdqa %ymm7, 0xe0(%rdi)+        vmovdqa %ymm8, 0x180(%rdi)+        vmovdqa %ymm9, 0x1a0(%rdi)+        vmovdqa %ymm10, 0x1c0(%rdi)+        vmovdqa %ymm11, 0x1e0(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(invntt_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+          && (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_ntt_avx2_asm.S view
@@ -0,0 +1,661 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [AVX2_NTT]+ *   Faster AVX2 optimized NTT multiplication for Ring-LWE lattice cryptography.+ *   Gregor Seiler+ *   https://eprint.iacr.org/2018/039+ *+ * - [REF_AVX2]+ *   CRYSTALS-Kyber optimized AVX2 implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Kyber implementation @[REF_AVX2].+ *+ * The core ideas behind the implementation are described in @[AVX2_NTT].+ */++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)+/*yaml+  Name: ntt_avx2_asm+  Description: x86_64 AVX2 forward NTT+  Signature: void mlk_ntt_avx2_asm(int16_t *r, const int16_t *qdata)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 512+      permissions: read/write+      c_parameter: int16_t *r+      description: Input/output polynomial (256 x int16_t)+    rsi:+      type: buffer+      size_bytes: 1248+      permissions: read-only+      c_parameter: const int16_t *qdata+      description: Precomputed constants (624 x int16_t)+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_ntt_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(ntt_avx2_asm)+MLK_ASM_FN_SYMBOL(ntt_avx2_asm)++        .cfi_startproc+        movl $0xd010d01, %eax        # imm = 0xD010D01+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        vpbroadcastq 0x40(%rsi), %ymm15+        vmovdqa 0x100(%rdi), %ymm8+        vmovdqa 0x120(%rdi), %ymm9+        vmovdqa 0x140(%rdi), %ymm10+        vmovdqa 0x160(%rdi), %ymm11+        vpbroadcastq 0x48(%rsi), %ymm2+        vpmullw %ymm15, %ymm8, %ymm12+        vpmullw %ymm15, %ymm9, %ymm13+        vpmullw %ymm15, %ymm10, %ymm14+        vpmullw %ymm15, %ymm11, %ymm15+        vpmulhw %ymm2, %ymm8, %ymm8+        vpmulhw %ymm2, %ymm9, %ymm9+        vpmulhw %ymm2, %ymm10, %ymm10+        vpmulhw %ymm2, %ymm11, %ymm11+        vmovdqa (%rdi), %ymm4+        vmovdqa 0x20(%rdi), %ymm5+        vmovdqa 0x40(%rdi), %ymm6+        vmovdqa 0x60(%rdi), %ymm7+        vpmulhw %ymm0, %ymm12, %ymm12+        vpmulhw %ymm0, %ymm13, %ymm13+        vpmulhw %ymm0, %ymm14, %ymm14+        vpmulhw %ymm0, %ymm15, %ymm15+        vpaddw %ymm8, %ymm4, %ymm3+        vpsubw %ymm8, %ymm4, %ymm8+        vpaddw %ymm9, %ymm5, %ymm4+        vpsubw %ymm9, %ymm5, %ymm9+        vpaddw %ymm10, %ymm6, %ymm5+        vpsubw %ymm10, %ymm6, %ymm10+        vpaddw %ymm11, %ymm7, %ymm6+        vpsubw %ymm11, %ymm7, %ymm11+        vpsubw %ymm12, %ymm3, %ymm3+        vpaddw %ymm12, %ymm8, %ymm8+        vpsubw %ymm13, %ymm4, %ymm4+        vpaddw %ymm13, %ymm9, %ymm9+        vpsubw %ymm14, %ymm5, %ymm5+        vpaddw %ymm14, %ymm10, %ymm10+        vpsubw %ymm15, %ymm6, %ymm6+        vpaddw %ymm15, %ymm11, %ymm11+        vmovdqa %ymm3, (%rdi)+        vmovdqa %ymm4, 0x20(%rdi)+        vmovdqa %ymm5, 0x40(%rdi)+        vmovdqa %ymm6, 0x60(%rdi)+        vmovdqa %ymm8, 0x100(%rdi)+        vmovdqa %ymm9, 0x120(%rdi)+        vmovdqa %ymm10, 0x140(%rdi)+        vmovdqa %ymm11, 0x160(%rdi)+        vpbroadcastq 0x40(%rsi), %ymm15+        vmovdqa 0x180(%rdi), %ymm8+        vmovdqa 0x1a0(%rdi), %ymm9+        vmovdqa 0x1c0(%rdi), %ymm10+        vmovdqa 0x1e0(%rdi), %ymm11+        vpbroadcastq 0x48(%rsi), %ymm2+        vpmullw %ymm15, %ymm8, %ymm12+        vpmullw %ymm15, %ymm9, %ymm13+        vpmullw %ymm15, %ymm10, %ymm14+        vpmullw %ymm15, %ymm11, %ymm15+        vpmulhw %ymm2, %ymm8, %ymm8+        vpmulhw %ymm2, %ymm9, %ymm9+        vpmulhw %ymm2, %ymm10, %ymm10+        vpmulhw %ymm2, %ymm11, %ymm11+        vmovdqa 0x80(%rdi), %ymm4+        vmovdqa 0xa0(%rdi), %ymm5+        vmovdqa 0xc0(%rdi), %ymm6+        vmovdqa 0xe0(%rdi), %ymm7+        vpmulhw %ymm0, %ymm12, %ymm12+        vpmulhw %ymm0, %ymm13, %ymm13+        vpmulhw %ymm0, %ymm14, %ymm14+        vpmulhw %ymm0, %ymm15, %ymm15+        vpaddw %ymm8, %ymm4, %ymm3+        vpsubw %ymm8, %ymm4, %ymm8+        vpaddw %ymm9, %ymm5, %ymm4+        vpsubw %ymm9, %ymm5, %ymm9+        vpaddw %ymm10, %ymm6, %ymm5+        vpsubw %ymm10, %ymm6, %ymm10+        vpaddw %ymm11, %ymm7, %ymm6+        vpsubw %ymm11, %ymm7, %ymm11+        vpsubw %ymm12, %ymm3, %ymm3+        vpaddw %ymm12, %ymm8, %ymm8+        vpsubw %ymm13, %ymm4, %ymm4+        vpaddw %ymm13, %ymm9, %ymm9+        vpsubw %ymm14, %ymm5, %ymm5+        vpaddw %ymm14, %ymm10, %ymm10+        vpsubw %ymm15, %ymm6, %ymm6+        vpaddw %ymm15, %ymm11, %ymm11+        vmovdqa %ymm3, 0x80(%rdi)+        vmovdqa %ymm4, 0xa0(%rdi)+        vmovdqa %ymm5, 0xc0(%rdi)+        vmovdqa %ymm6, 0xe0(%rdi)+        vmovdqa %ymm8, 0x180(%rdi)+        vmovdqa %ymm9, 0x1a0(%rdi)+        vmovdqa %ymm10, 0x1c0(%rdi)+        vmovdqa %ymm11, 0x1e0(%rdi)+        vmovdqa 0x60(%rsi), %ymm15+        vmovdqa 0x80(%rdi), %ymm8+        vmovdqa 0xa0(%rdi), %ymm9+        vmovdqa 0xc0(%rdi), %ymm10+        vmovdqa 0xe0(%rdi), %ymm11+        vmovdqa 0x80(%rsi), %ymm2+        vpmullw %ymm15, %ymm8, %ymm12+        vpmullw %ymm15, %ymm9, %ymm13+        vpmullw %ymm15, %ymm10, %ymm14+        vpmullw %ymm15, %ymm11, %ymm15+        vpmulhw %ymm2, %ymm8, %ymm8+        vpmulhw %ymm2, %ymm9, %ymm9+        vpmulhw %ymm2, %ymm10, %ymm10+        vpmulhw %ymm2, %ymm11, %ymm11+        vmovdqa (%rdi), %ymm4+        vmovdqa 0x20(%rdi), %ymm5+        vmovdqa 0x40(%rdi), %ymm6+        vmovdqa 0x60(%rdi), %ymm7+        vpmulhw %ymm0, %ymm12, %ymm12+        vpmulhw %ymm0, %ymm13, %ymm13+        vpmulhw %ymm0, %ymm14, %ymm14+        vpmulhw %ymm0, %ymm15, %ymm15+        vpaddw %ymm8, %ymm4, %ymm3+        vpsubw %ymm8, %ymm4, %ymm8+        vpaddw %ymm9, %ymm5, %ymm4+        vpsubw %ymm9, %ymm5, %ymm9+        vpaddw %ymm10, %ymm6, %ymm5+        vpsubw %ymm10, %ymm6, %ymm10+        vpaddw %ymm11, %ymm7, %ymm6+        vpsubw %ymm11, %ymm7, %ymm11+        vpsubw %ymm12, %ymm3, %ymm3+        vpaddw %ymm12, %ymm8, %ymm8+        vpsubw %ymm13, %ymm4, %ymm4+        vpaddw %ymm13, %ymm9, %ymm9+        vpsubw %ymm14, %ymm5, %ymm5+        vpaddw %ymm14, %ymm10, %ymm10+        vpsubw %ymm15, %ymm6, %ymm6+        vpaddw %ymm15, %ymm11, %ymm11+        vperm2i128 $0x20, %ymm10, %ymm5, %ymm7 # ymm7 = ymm5[0,1],ymm10[0,1]+        vperm2i128 $0x31, %ymm10, %ymm5, %ymm10 # ymm10 = ymm5[2,3],ymm10[2,3]+        vperm2i128 $0x20, %ymm11, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm6, %ymm11 # ymm11 = ymm6[2,3],ymm11[2,3]+        vmovdqa 0xa0(%rsi), %ymm15+        vmovdqa 0xc0(%rsi), %ymm2+        vpmullw %ymm15, %ymm7, %ymm12+        vpmullw %ymm15, %ymm10, %ymm13+        vpmullw %ymm15, %ymm5, %ymm14+        vpmullw %ymm15, %ymm11, %ymm15+        vpmulhw %ymm2, %ymm7, %ymm7+        vpmulhw %ymm2, %ymm10, %ymm10+        vpmulhw %ymm2, %ymm5, %ymm5+        vpmulhw %ymm2, %ymm11, %ymm11+        vperm2i128 $0x20, %ymm8, %ymm3, %ymm6 # ymm6 = ymm3[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm3, %ymm8 # ymm8 = ymm3[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm9, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm9[0,1]+        vperm2i128 $0x31, %ymm9, %ymm4, %ymm9 # ymm9 = ymm4[2,3],ymm9[2,3]+        vpmulhw %ymm0, %ymm12, %ymm12+        vpmulhw %ymm0, %ymm13, %ymm13+        vpmulhw %ymm0, %ymm14, %ymm14+        vpmulhw %ymm0, %ymm15, %ymm15+        vpaddw %ymm7, %ymm6, %ymm4+        vpsubw %ymm7, %ymm6, %ymm7+        vpaddw %ymm10, %ymm8, %ymm6+        vpsubw %ymm10, %ymm8, %ymm10+        vpaddw %ymm5, %ymm3, %ymm8+        vpsubw %ymm5, %ymm3, %ymm5+        vpaddw %ymm11, %ymm9, %ymm3+        vpsubw %ymm11, %ymm9, %ymm11+        vpsubw %ymm12, %ymm4, %ymm4+        vpaddw %ymm12, %ymm7, %ymm7+        vpsubw %ymm13, %ymm6, %ymm6+        vpaddw %ymm13, %ymm10, %ymm10+        vpsubw %ymm14, %ymm8, %ymm8+        vpaddw %ymm14, %ymm5, %ymm5+        vpsubw %ymm15, %ymm3, %ymm3+        vpaddw %ymm15, %ymm11, %ymm11+        vpunpcklqdq %ymm5, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm5[0],ymm8[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm8, %ymm5 # ymm5 = ymm8[1],ymm5[1],ymm8[3],ymm5[3]+        vpunpcklqdq %ymm11, %ymm3, %ymm8 # ymm8 = ymm3[0],ymm11[0],ymm3[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm3, %ymm11 # ymm11 = ymm3[1],ymm11[1],ymm3[3],ymm11[3]+        vmovdqa 0xe0(%rsi), %ymm15+        vmovdqa 0x100(%rsi), %ymm2+        vpmullw %ymm15, %ymm9, %ymm12+        vpmullw %ymm15, %ymm5, %ymm13+        vpmullw %ymm15, %ymm8, %ymm14+        vpmullw %ymm15, %ymm11, %ymm15+        vpmulhw %ymm2, %ymm9, %ymm9+        vpmulhw %ymm2, %ymm5, %ymm5+        vpmulhw %ymm2, %ymm8, %ymm8+        vpmulhw %ymm2, %ymm11, %ymm11+        vpunpcklqdq %ymm7, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm7[0],ymm4[2],ymm7[2]+        vpunpckhqdq %ymm7, %ymm4, %ymm7 # ymm7 = ymm4[1],ymm7[1],ymm4[3],ymm7[3]+        vpunpcklqdq %ymm10, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm10[0],ymm6[2],ymm10[2]+        vpunpckhqdq %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[1],ymm10[1],ymm6[3],ymm10[3]+        vpmulhw %ymm0, %ymm12, %ymm12+        vpmulhw %ymm0, %ymm13, %ymm13+        vpmulhw %ymm0, %ymm14, %ymm14+        vpmulhw %ymm0, %ymm15, %ymm15+        vpaddw %ymm9, %ymm3, %ymm6+        vpsubw %ymm9, %ymm3, %ymm9+        vpaddw %ymm5, %ymm7, %ymm3+        vpsubw %ymm5, %ymm7, %ymm5+        vpaddw %ymm8, %ymm4, %ymm7+        vpsubw %ymm8, %ymm4, %ymm8+        vpaddw %ymm11, %ymm10, %ymm4+        vpsubw %ymm11, %ymm10, %ymm11+        vpsubw %ymm12, %ymm6, %ymm6+        vpaddw %ymm12, %ymm9, %ymm9+        vpsubw %ymm13, %ymm3, %ymm3+        vpaddw %ymm13, %ymm5, %ymm5+        vpsubw %ymm14, %ymm7, %ymm7+        vpaddw %ymm14, %ymm8, %ymm8+        vpsubw %ymm15, %ymm4, %ymm4+        vpaddw %ymm15, %ymm11, %ymm11+        vmovsldup %ymm8, %ymm10   # ymm10 = ymm8[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm10, %ymm7, %ymm10 # ymm10 = ymm7[0],ymm10[1],ymm7[2],ymm10[3],ymm7[4],ymm10[5],ymm7[6],ymm10[7]+        vpsrlq $0x20, %ymm7, %ymm7+        vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7]+        vmovsldup %ymm11, %ymm7   # ymm7 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm7, %ymm4, %ymm7 # ymm7 = ymm4[0],ymm7[1],ymm4[2],ymm7[3],ymm4[4],ymm7[5],ymm4[6],ymm7[7]+        vpsrlq $0x20, %ymm4, %ymm4+        vpblendd $0xaa, %ymm11, %ymm4, %ymm11 # ymm11 = ymm4[0],ymm11[1],ymm4[2],ymm11[3],ymm4[4],ymm11[5],ymm4[6],ymm11[7]+        vmovdqa 0x120(%rsi), %ymm15+        vmovdqa 0x140(%rsi), %ymm2+        vpmullw %ymm15, %ymm10, %ymm12+        vpmullw %ymm15, %ymm8, %ymm13+        vpmullw %ymm15, %ymm7, %ymm14+        vpmullw %ymm15, %ymm11, %ymm15+        vpmulhw %ymm2, %ymm10, %ymm10+        vpmulhw %ymm2, %ymm8, %ymm8+        vpmulhw %ymm2, %ymm7, %ymm7+        vpmulhw %ymm2, %ymm11, %ymm11+        vmovsldup %ymm9, %ymm4    # ymm4 = ymm9[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7]+        vpsrlq $0x20, %ymm6, %ymm6+        vpblendd $0xaa, %ymm9, %ymm6, %ymm9 # ymm9 = ymm6[0],ymm9[1],ymm6[2],ymm9[3],ymm6[4],ymm9[5],ymm6[6],ymm9[7]+        vmovsldup %ymm5, %ymm6    # ymm6 = ymm5[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm6, %ymm3, %ymm6 # ymm6 = ymm3[0],ymm6[1],ymm3[2],ymm6[3],ymm3[4],ymm6[5],ymm3[6],ymm6[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7]+        vpmulhw %ymm0, %ymm12, %ymm12+        vpmulhw %ymm0, %ymm13, %ymm13+        vpmulhw %ymm0, %ymm14, %ymm14+        vpmulhw %ymm0, %ymm15, %ymm15+        vpaddw %ymm10, %ymm4, %ymm3+        vpsubw %ymm10, %ymm4, %ymm10+        vpaddw %ymm8, %ymm9, %ymm4+        vpsubw %ymm8, %ymm9, %ymm8+        vpaddw %ymm7, %ymm6, %ymm9+        vpsubw %ymm7, %ymm6, %ymm7+        vpaddw %ymm11, %ymm5, %ymm6+        vpsubw %ymm11, %ymm5, %ymm11+        vpsubw %ymm12, %ymm3, %ymm3+        vpaddw %ymm12, %ymm10, %ymm10+        vpsubw %ymm13, %ymm4, %ymm4+        vpaddw %ymm13, %ymm8, %ymm8+        vpsubw %ymm14, %ymm9, %ymm9+        vpaddw %ymm14, %ymm7, %ymm7+        vpsubw %ymm15, %ymm6, %ymm6+        vpaddw %ymm15, %ymm11, %ymm11+        vpslld $0x10, %ymm7, %ymm5+        vpblendw $0xaa, %ymm5, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm5[1],ymm9[2],ymm5[3],ymm9[4],ymm5[5],ymm9[6],ymm5[7],ymm9[8],ymm5[9],ymm9[10],ymm5[11],ymm9[12],ymm5[13],ymm9[14],ymm5[15]+        vpsrld $0x10, %ymm9, %ymm9+        vpblendw $0xaa, %ymm7, %ymm9, %ymm7 # ymm7 = ymm9[0],ymm7[1],ymm9[2],ymm7[3],ymm9[4],ymm7[5],ymm9[6],ymm7[7],ymm9[8],ymm7[9],ymm9[10],ymm7[11],ymm9[12],ymm7[13],ymm9[14],ymm7[15]+        vpslld $0x10, %ymm11, %ymm9+        vpblendw $0xaa, %ymm9, %ymm6, %ymm9 # ymm9 = ymm6[0],ymm9[1],ymm6[2],ymm9[3],ymm6[4],ymm9[5],ymm6[6],ymm9[7],ymm6[8],ymm9[9],ymm6[10],ymm9[11],ymm6[12],ymm9[13],ymm6[14],ymm9[15]+        vpsrld $0x10, %ymm6, %ymm6+        vpblendw $0xaa, %ymm11, %ymm6, %ymm11 # ymm11 = ymm6[0],ymm11[1],ymm6[2],ymm11[3],ymm6[4],ymm11[5],ymm6[6],ymm11[7],ymm6[8],ymm11[9],ymm6[10],ymm11[11],ymm6[12],ymm11[13],ymm6[14],ymm11[15]+        vmovdqa 0x160(%rsi), %ymm15+        vmovdqa 0x180(%rsi), %ymm2+        vpmullw %ymm15, %ymm5, %ymm12+        vpmullw %ymm15, %ymm7, %ymm13+        vpmullw %ymm15, %ymm9, %ymm14+        vpmullw %ymm15, %ymm11, %ymm15+        vpmulhw %ymm2, %ymm5, %ymm5+        vpmulhw %ymm2, %ymm7, %ymm7+        vpmulhw %ymm2, %ymm9, %ymm9+        vpmulhw %ymm2, %ymm11, %ymm11+        vpslld $0x10, %ymm10, %ymm6+        vpblendw $0xaa, %ymm6, %ymm3, %ymm6 # ymm6 = ymm3[0],ymm6[1],ymm3[2],ymm6[3],ymm3[4],ymm6[5],ymm3[6],ymm6[7],ymm3[8],ymm6[9],ymm3[10],ymm6[11],ymm3[12],ymm6[13],ymm3[14],ymm6[15]+        vpsrld $0x10, %ymm3, %ymm3+        vpblendw $0xaa, %ymm10, %ymm3, %ymm10 # ymm10 = ymm3[0],ymm10[1],ymm3[2],ymm10[3],ymm3[4],ymm10[5],ymm3[6],ymm10[7],ymm3[8],ymm10[9],ymm3[10],ymm10[11],ymm3[12],ymm10[13],ymm3[14],ymm10[15]+        vpslld $0x10, %ymm8, %ymm3+        vpblendw $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7],ymm4[8],ymm3[9],ymm4[10],ymm3[11],ymm4[12],ymm3[13],ymm4[14],ymm3[15]+        vpsrld $0x10, %ymm4, %ymm4+        vpblendw $0xaa, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm8[1],ymm4[2],ymm8[3],ymm4[4],ymm8[5],ymm4[6],ymm8[7],ymm4[8],ymm8[9],ymm4[10],ymm8[11],ymm4[12],ymm8[13],ymm4[14],ymm8[15]+        vpmulhw %ymm0, %ymm12, %ymm12+        vpmulhw %ymm0, %ymm13, %ymm13+        vpmulhw %ymm0, %ymm14, %ymm14+        vpmulhw %ymm0, %ymm15, %ymm15+        vpaddw %ymm5, %ymm6, %ymm4+        vpsubw %ymm5, %ymm6, %ymm5+        vpaddw %ymm7, %ymm10, %ymm6+        vpsubw %ymm7, %ymm10, %ymm7+        vpaddw %ymm9, %ymm3, %ymm10+        vpsubw %ymm9, %ymm3, %ymm9+        vpaddw %ymm11, %ymm8, %ymm3+        vpsubw %ymm11, %ymm8, %ymm11+        vpsubw %ymm12, %ymm4, %ymm4+        vpaddw %ymm12, %ymm5, %ymm5+        vpsubw %ymm13, %ymm6, %ymm6+        vpaddw %ymm13, %ymm7, %ymm7+        vpsubw %ymm14, %ymm10, %ymm10+        vpaddw %ymm14, %ymm9, %ymm9+        vpsubw %ymm15, %ymm3, %ymm3+        vpaddw %ymm15, %ymm11, %ymm11+        vmovdqa 0x1a0(%rsi), %ymm14+        vmovdqa 0x1e0(%rsi), %ymm15+        vmovdqa 0x1c0(%rsi), %ymm8+        vmovdqa 0x200(%rsi), %ymm2+        vpmullw %ymm14, %ymm10, %ymm12+        vpmullw %ymm14, %ymm3, %ymm13+        vpmullw %ymm15, %ymm9, %ymm14+        vpmullw %ymm15, %ymm11, %ymm15+        vpmulhw %ymm8, %ymm10, %ymm10+        vpmulhw %ymm8, %ymm3, %ymm3+        vpmulhw %ymm2, %ymm9, %ymm9+        vpmulhw %ymm2, %ymm11, %ymm11+        vpmulhw %ymm0, %ymm12, %ymm12+        vpmulhw %ymm0, %ymm13, %ymm13+        vpmulhw %ymm0, %ymm14, %ymm14+        vpmulhw %ymm0, %ymm15, %ymm15+        vpaddw %ymm10, %ymm4, %ymm8+        vpsubw %ymm10, %ymm4, %ymm10+        vpaddw %ymm3, %ymm6, %ymm4+        vpsubw %ymm3, %ymm6, %ymm3+        vpaddw %ymm9, %ymm5, %ymm6+        vpsubw %ymm9, %ymm5, %ymm9+        vpaddw %ymm11, %ymm7, %ymm5+        vpsubw %ymm11, %ymm7, %ymm11+        vpsubw %ymm12, %ymm8, %ymm8+        vpaddw %ymm12, %ymm10, %ymm10+        vpsubw %ymm13, %ymm4, %ymm4+        vpaddw %ymm13, %ymm3, %ymm3+        vpsubw %ymm14, %ymm6, %ymm6+        vpaddw %ymm14, %ymm9, %ymm9+        vpsubw %ymm15, %ymm5, %ymm5+        vpaddw %ymm15, %ymm11, %ymm11+        vmovdqa %ymm8, (%rdi)+        vmovdqa %ymm4, 0x20(%rdi)+        vmovdqa %ymm10, 0x40(%rdi)+        vmovdqa %ymm3, 0x60(%rdi)+        vmovdqa %ymm6, 0x80(%rdi)+        vmovdqa %ymm5, 0xa0(%rdi)+        vmovdqa %ymm9, 0xc0(%rdi)+        vmovdqa %ymm11, 0xe0(%rdi)+        vmovdqa 0x220(%rsi), %ymm15+        vmovdqa 0x180(%rdi), %ymm8+        vmovdqa 0x1a0(%rdi), %ymm9+        vmovdqa 0x1c0(%rdi), %ymm10+        vmovdqa 0x1e0(%rdi), %ymm11+        vmovdqa 0x240(%rsi), %ymm2+        vpmullw %ymm15, %ymm8, %ymm12+        vpmullw %ymm15, %ymm9, %ymm13+        vpmullw %ymm15, %ymm10, %ymm14+        vpmullw %ymm15, %ymm11, %ymm15+        vpmulhw %ymm2, %ymm8, %ymm8+        vpmulhw %ymm2, %ymm9, %ymm9+        vpmulhw %ymm2, %ymm10, %ymm10+        vpmulhw %ymm2, %ymm11, %ymm11+        vmovdqa 0x100(%rdi), %ymm4+        vmovdqa 0x120(%rdi), %ymm5+        vmovdqa 0x140(%rdi), %ymm6+        vmovdqa 0x160(%rdi), %ymm7+        vpmulhw %ymm0, %ymm12, %ymm12+        vpmulhw %ymm0, %ymm13, %ymm13+        vpmulhw %ymm0, %ymm14, %ymm14+        vpmulhw %ymm0, %ymm15, %ymm15+        vpaddw %ymm8, %ymm4, %ymm3+        vpsubw %ymm8, %ymm4, %ymm8+        vpaddw %ymm9, %ymm5, %ymm4+        vpsubw %ymm9, %ymm5, %ymm9+        vpaddw %ymm10, %ymm6, %ymm5+        vpsubw %ymm10, %ymm6, %ymm10+        vpaddw %ymm11, %ymm7, %ymm6+        vpsubw %ymm11, %ymm7, %ymm11+        vpsubw %ymm12, %ymm3, %ymm3+        vpaddw %ymm12, %ymm8, %ymm8+        vpsubw %ymm13, %ymm4, %ymm4+        vpaddw %ymm13, %ymm9, %ymm9+        vpsubw %ymm14, %ymm5, %ymm5+        vpaddw %ymm14, %ymm10, %ymm10+        vpsubw %ymm15, %ymm6, %ymm6+        vpaddw %ymm15, %ymm11, %ymm11+        vperm2i128 $0x20, %ymm10, %ymm5, %ymm7 # ymm7 = ymm5[0,1],ymm10[0,1]+        vperm2i128 $0x31, %ymm10, %ymm5, %ymm10 # ymm10 = ymm5[2,3],ymm10[2,3]+        vperm2i128 $0x20, %ymm11, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm6, %ymm11 # ymm11 = ymm6[2,3],ymm11[2,3]+        vmovdqa 0x260(%rsi), %ymm15+        vmovdqa 0x280(%rsi), %ymm2+        vpmullw %ymm15, %ymm7, %ymm12+        vpmullw %ymm15, %ymm10, %ymm13+        vpmullw %ymm15, %ymm5, %ymm14+        vpmullw %ymm15, %ymm11, %ymm15+        vpmulhw %ymm2, %ymm7, %ymm7+        vpmulhw %ymm2, %ymm10, %ymm10+        vpmulhw %ymm2, %ymm5, %ymm5+        vpmulhw %ymm2, %ymm11, %ymm11+        vperm2i128 $0x20, %ymm8, %ymm3, %ymm6 # ymm6 = ymm3[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm3, %ymm8 # ymm8 = ymm3[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm9, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm9[0,1]+        vperm2i128 $0x31, %ymm9, %ymm4, %ymm9 # ymm9 = ymm4[2,3],ymm9[2,3]+        vpmulhw %ymm0, %ymm12, %ymm12+        vpmulhw %ymm0, %ymm13, %ymm13+        vpmulhw %ymm0, %ymm14, %ymm14+        vpmulhw %ymm0, %ymm15, %ymm15+        vpaddw %ymm7, %ymm6, %ymm4+        vpsubw %ymm7, %ymm6, %ymm7+        vpaddw %ymm10, %ymm8, %ymm6+        vpsubw %ymm10, %ymm8, %ymm10+        vpaddw %ymm5, %ymm3, %ymm8+        vpsubw %ymm5, %ymm3, %ymm5+        vpaddw %ymm11, %ymm9, %ymm3+        vpsubw %ymm11, %ymm9, %ymm11+        vpsubw %ymm12, %ymm4, %ymm4+        vpaddw %ymm12, %ymm7, %ymm7+        vpsubw %ymm13, %ymm6, %ymm6+        vpaddw %ymm13, %ymm10, %ymm10+        vpsubw %ymm14, %ymm8, %ymm8+        vpaddw %ymm14, %ymm5, %ymm5+        vpsubw %ymm15, %ymm3, %ymm3+        vpaddw %ymm15, %ymm11, %ymm11+        vpunpcklqdq %ymm5, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm5[0],ymm8[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm8, %ymm5 # ymm5 = ymm8[1],ymm5[1],ymm8[3],ymm5[3]+        vpunpcklqdq %ymm11, %ymm3, %ymm8 # ymm8 = ymm3[0],ymm11[0],ymm3[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm3, %ymm11 # ymm11 = ymm3[1],ymm11[1],ymm3[3],ymm11[3]+        vmovdqa 0x2a0(%rsi), %ymm15+        vmovdqa 0x2c0(%rsi), %ymm2+        vpmullw %ymm15, %ymm9, %ymm12+        vpmullw %ymm15, %ymm5, %ymm13+        vpmullw %ymm15, %ymm8, %ymm14+        vpmullw %ymm15, %ymm11, %ymm15+        vpmulhw %ymm2, %ymm9, %ymm9+        vpmulhw %ymm2, %ymm5, %ymm5+        vpmulhw %ymm2, %ymm8, %ymm8+        vpmulhw %ymm2, %ymm11, %ymm11+        vpunpcklqdq %ymm7, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm7[0],ymm4[2],ymm7[2]+        vpunpckhqdq %ymm7, %ymm4, %ymm7 # ymm7 = ymm4[1],ymm7[1],ymm4[3],ymm7[3]+        vpunpcklqdq %ymm10, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm10[0],ymm6[2],ymm10[2]+        vpunpckhqdq %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[1],ymm10[1],ymm6[3],ymm10[3]+        vpmulhw %ymm0, %ymm12, %ymm12+        vpmulhw %ymm0, %ymm13, %ymm13+        vpmulhw %ymm0, %ymm14, %ymm14+        vpmulhw %ymm0, %ymm15, %ymm15+        vpaddw %ymm9, %ymm3, %ymm6+        vpsubw %ymm9, %ymm3, %ymm9+        vpaddw %ymm5, %ymm7, %ymm3+        vpsubw %ymm5, %ymm7, %ymm5+        vpaddw %ymm8, %ymm4, %ymm7+        vpsubw %ymm8, %ymm4, %ymm8+        vpaddw %ymm11, %ymm10, %ymm4+        vpsubw %ymm11, %ymm10, %ymm11+        vpsubw %ymm12, %ymm6, %ymm6+        vpaddw %ymm12, %ymm9, %ymm9+        vpsubw %ymm13, %ymm3, %ymm3+        vpaddw %ymm13, %ymm5, %ymm5+        vpsubw %ymm14, %ymm7, %ymm7+        vpaddw %ymm14, %ymm8, %ymm8+        vpsubw %ymm15, %ymm4, %ymm4+        vpaddw %ymm15, %ymm11, %ymm11+        vmovsldup %ymm8, %ymm10   # ymm10 = ymm8[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm10, %ymm7, %ymm10 # ymm10 = ymm7[0],ymm10[1],ymm7[2],ymm10[3],ymm7[4],ymm10[5],ymm7[6],ymm10[7]+        vpsrlq $0x20, %ymm7, %ymm7+        vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7]+        vmovsldup %ymm11, %ymm7   # ymm7 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm7, %ymm4, %ymm7 # ymm7 = ymm4[0],ymm7[1],ymm4[2],ymm7[3],ymm4[4],ymm7[5],ymm4[6],ymm7[7]+        vpsrlq $0x20, %ymm4, %ymm4+        vpblendd $0xaa, %ymm11, %ymm4, %ymm11 # ymm11 = ymm4[0],ymm11[1],ymm4[2],ymm11[3],ymm4[4],ymm11[5],ymm4[6],ymm11[7]+        vmovdqa 0x2e0(%rsi), %ymm15+        vmovdqa 0x300(%rsi), %ymm2+        vpmullw %ymm15, %ymm10, %ymm12+        vpmullw %ymm15, %ymm8, %ymm13+        vpmullw %ymm15, %ymm7, %ymm14+        vpmullw %ymm15, %ymm11, %ymm15+        vpmulhw %ymm2, %ymm10, %ymm10+        vpmulhw %ymm2, %ymm8, %ymm8+        vpmulhw %ymm2, %ymm7, %ymm7+        vpmulhw %ymm2, %ymm11, %ymm11+        vmovsldup %ymm9, %ymm4    # ymm4 = ymm9[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7]+        vpsrlq $0x20, %ymm6, %ymm6+        vpblendd $0xaa, %ymm9, %ymm6, %ymm9 # ymm9 = ymm6[0],ymm9[1],ymm6[2],ymm9[3],ymm6[4],ymm9[5],ymm6[6],ymm9[7]+        vmovsldup %ymm5, %ymm6    # ymm6 = ymm5[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm6, %ymm3, %ymm6 # ymm6 = ymm3[0],ymm6[1],ymm3[2],ymm6[3],ymm3[4],ymm6[5],ymm3[6],ymm6[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7]+        vpmulhw %ymm0, %ymm12, %ymm12+        vpmulhw %ymm0, %ymm13, %ymm13+        vpmulhw %ymm0, %ymm14, %ymm14+        vpmulhw %ymm0, %ymm15, %ymm15+        vpaddw %ymm10, %ymm4, %ymm3+        vpsubw %ymm10, %ymm4, %ymm10+        vpaddw %ymm8, %ymm9, %ymm4+        vpsubw %ymm8, %ymm9, %ymm8+        vpaddw %ymm7, %ymm6, %ymm9+        vpsubw %ymm7, %ymm6, %ymm7+        vpaddw %ymm11, %ymm5, %ymm6+        vpsubw %ymm11, %ymm5, %ymm11+        vpsubw %ymm12, %ymm3, %ymm3+        vpaddw %ymm12, %ymm10, %ymm10+        vpsubw %ymm13, %ymm4, %ymm4+        vpaddw %ymm13, %ymm8, %ymm8+        vpsubw %ymm14, %ymm9, %ymm9+        vpaddw %ymm14, %ymm7, %ymm7+        vpsubw %ymm15, %ymm6, %ymm6+        vpaddw %ymm15, %ymm11, %ymm11+        vpslld $0x10, %ymm7, %ymm5+        vpblendw $0xaa, %ymm5, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm5[1],ymm9[2],ymm5[3],ymm9[4],ymm5[5],ymm9[6],ymm5[7],ymm9[8],ymm5[9],ymm9[10],ymm5[11],ymm9[12],ymm5[13],ymm9[14],ymm5[15]+        vpsrld $0x10, %ymm9, %ymm9+        vpblendw $0xaa, %ymm7, %ymm9, %ymm7 # ymm7 = ymm9[0],ymm7[1],ymm9[2],ymm7[3],ymm9[4],ymm7[5],ymm9[6],ymm7[7],ymm9[8],ymm7[9],ymm9[10],ymm7[11],ymm9[12],ymm7[13],ymm9[14],ymm7[15]+        vpslld $0x10, %ymm11, %ymm9+        vpblendw $0xaa, %ymm9, %ymm6, %ymm9 # ymm9 = ymm6[0],ymm9[1],ymm6[2],ymm9[3],ymm6[4],ymm9[5],ymm6[6],ymm9[7],ymm6[8],ymm9[9],ymm6[10],ymm9[11],ymm6[12],ymm9[13],ymm6[14],ymm9[15]+        vpsrld $0x10, %ymm6, %ymm6+        vpblendw $0xaa, %ymm11, %ymm6, %ymm11 # ymm11 = ymm6[0],ymm11[1],ymm6[2],ymm11[3],ymm6[4],ymm11[5],ymm6[6],ymm11[7],ymm6[8],ymm11[9],ymm6[10],ymm11[11],ymm6[12],ymm11[13],ymm6[14],ymm11[15]+        vmovdqa 0x320(%rsi), %ymm15+        vmovdqa 0x340(%rsi), %ymm2+        vpmullw %ymm15, %ymm5, %ymm12+        vpmullw %ymm15, %ymm7, %ymm13+        vpmullw %ymm15, %ymm9, %ymm14+        vpmullw %ymm15, %ymm11, %ymm15+        vpmulhw %ymm2, %ymm5, %ymm5+        vpmulhw %ymm2, %ymm7, %ymm7+        vpmulhw %ymm2, %ymm9, %ymm9+        vpmulhw %ymm2, %ymm11, %ymm11+        vpslld $0x10, %ymm10, %ymm6+        vpblendw $0xaa, %ymm6, %ymm3, %ymm6 # ymm6 = ymm3[0],ymm6[1],ymm3[2],ymm6[3],ymm3[4],ymm6[5],ymm3[6],ymm6[7],ymm3[8],ymm6[9],ymm3[10],ymm6[11],ymm3[12],ymm6[13],ymm3[14],ymm6[15]+        vpsrld $0x10, %ymm3, %ymm3+        vpblendw $0xaa, %ymm10, %ymm3, %ymm10 # ymm10 = ymm3[0],ymm10[1],ymm3[2],ymm10[3],ymm3[4],ymm10[5],ymm3[6],ymm10[7],ymm3[8],ymm10[9],ymm3[10],ymm10[11],ymm3[12],ymm10[13],ymm3[14],ymm10[15]+        vpslld $0x10, %ymm8, %ymm3+        vpblendw $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7],ymm4[8],ymm3[9],ymm4[10],ymm3[11],ymm4[12],ymm3[13],ymm4[14],ymm3[15]+        vpsrld $0x10, %ymm4, %ymm4+        vpblendw $0xaa, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm8[1],ymm4[2],ymm8[3],ymm4[4],ymm8[5],ymm4[6],ymm8[7],ymm4[8],ymm8[9],ymm4[10],ymm8[11],ymm4[12],ymm8[13],ymm4[14],ymm8[15]+        vpmulhw %ymm0, %ymm12, %ymm12+        vpmulhw %ymm0, %ymm13, %ymm13+        vpmulhw %ymm0, %ymm14, %ymm14+        vpmulhw %ymm0, %ymm15, %ymm15+        vpaddw %ymm5, %ymm6, %ymm4+        vpsubw %ymm5, %ymm6, %ymm5+        vpaddw %ymm7, %ymm10, %ymm6+        vpsubw %ymm7, %ymm10, %ymm7+        vpaddw %ymm9, %ymm3, %ymm10+        vpsubw %ymm9, %ymm3, %ymm9+        vpaddw %ymm11, %ymm8, %ymm3+        vpsubw %ymm11, %ymm8, %ymm11+        vpsubw %ymm12, %ymm4, %ymm4+        vpaddw %ymm12, %ymm5, %ymm5+        vpsubw %ymm13, %ymm6, %ymm6+        vpaddw %ymm13, %ymm7, %ymm7+        vpsubw %ymm14, %ymm10, %ymm10+        vpaddw %ymm14, %ymm9, %ymm9+        vpsubw %ymm15, %ymm3, %ymm3+        vpaddw %ymm15, %ymm11, %ymm11+        vmovdqa 0x360(%rsi), %ymm14+        vmovdqa 0x3a0(%rsi), %ymm15+        vmovdqa 0x380(%rsi), %ymm8+        vmovdqa 0x3c0(%rsi), %ymm2+        vpmullw %ymm14, %ymm10, %ymm12+        vpmullw %ymm14, %ymm3, %ymm13+        vpmullw %ymm15, %ymm9, %ymm14+        vpmullw %ymm15, %ymm11, %ymm15+        vpmulhw %ymm8, %ymm10, %ymm10+        vpmulhw %ymm8, %ymm3, %ymm3+        vpmulhw %ymm2, %ymm9, %ymm9+        vpmulhw %ymm2, %ymm11, %ymm11+        vpmulhw %ymm0, %ymm12, %ymm12+        vpmulhw %ymm0, %ymm13, %ymm13+        vpmulhw %ymm0, %ymm14, %ymm14+        vpmulhw %ymm0, %ymm15, %ymm15+        vpaddw %ymm10, %ymm4, %ymm8+        vpsubw %ymm10, %ymm4, %ymm10+        vpaddw %ymm3, %ymm6, %ymm4+        vpsubw %ymm3, %ymm6, %ymm3+        vpaddw %ymm9, %ymm5, %ymm6+        vpsubw %ymm9, %ymm5, %ymm9+        vpaddw %ymm11, %ymm7, %ymm5+        vpsubw %ymm11, %ymm7, %ymm11+        vpsubw %ymm12, %ymm8, %ymm8+        vpaddw %ymm12, %ymm10, %ymm10+        vpsubw %ymm13, %ymm4, %ymm4+        vpaddw %ymm13, %ymm3, %ymm3+        vpsubw %ymm14, %ymm6, %ymm6+        vpaddw %ymm14, %ymm9, %ymm9+        vpsubw %ymm15, %ymm5, %ymm5+        vpaddw %ymm15, %ymm11, %ymm11+        vmovdqa %ymm8, 0x100(%rdi)+        vmovdqa %ymm4, 0x120(%rdi)+        vmovdqa %ymm10, 0x140(%rdi)+        vmovdqa %ymm3, 0x160(%rdi)+        vmovdqa %ymm6, 0x180(%rdi)+        vmovdqa %ymm5, 0x1a0(%rdi)+        vmovdqa %ymm9, 0x1c0(%rdi)+        vmovdqa %ymm11, 0x1e0(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(ntt_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_nttfrombytes_avx2_asm.S view
@@ -0,0 +1,217 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Kyber optimized AVX2 implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Kyber implementation @[REF_AVX2].+ */++#include "../../../common.h"++#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    (!defined(MLK_CONFIG_NO_ENCAPS_API) || \+     !defined(MLK_CONFIG_NO_DECAPS_API))+/*yaml+  Name: nttfrombytes_avx2_asm+  Description: x86_64 AVX2 polynomial deserialization in NTT domain+  Signature: void mlk_nttfrombytes_avx2_asm(int16_t *r, const uint8_t *a)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 512+      permissions: write-only+      c_parameter: int16_t *r+      description: Output polynomial (256 x int16_t)+    rsi:+      type: buffer+      size_bytes: 384+      permissions: read-only+      c_parameter: const uint8_t *a+      description: Input byte array (MLKEM_POLYBYTES = 384)+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_nttfrombytes_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(nttfrombytes_avx2_asm)+MLK_ASM_FN_SYMBOL(nttfrombytes_avx2_asm)++        .cfi_startproc+        movl $0xfff0fff, %eax        # imm = 0xFFF0FFF+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        vmovdqu (%rsi), %ymm4+        vmovdqu 0x20(%rsi), %ymm5+        vmovdqu 0x40(%rsi), %ymm6+        vmovdqu 0x60(%rsi), %ymm7+        vmovdqu 0x80(%rsi), %ymm8+        vmovdqu 0xa0(%rsi), %ymm9+        vperm2i128 $0x20, %ymm7, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm7[0,1]+        vperm2i128 $0x31, %ymm7, %ymm4, %ymm7 # ymm7 = ymm4[2,3],ymm7[2,3]+        vperm2i128 $0x20, %ymm8, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm5, %ymm8 # ymm8 = ymm5[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm9, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm9[0,1]+        vperm2i128 $0x31, %ymm9, %ymm6, %ymm9 # ymm9 = ymm6[2,3],ymm9[2,3]+        vpunpcklqdq %ymm8, %ymm3, %ymm6 # ymm6 = ymm3[0],ymm8[0],ymm3[2],ymm8[2]+        vpunpckhqdq %ymm8, %ymm3, %ymm8 # ymm8 = ymm3[1],ymm8[1],ymm3[3],ymm8[3]+        vpunpcklqdq %ymm5, %ymm7, %ymm3 # ymm3 = ymm7[0],ymm5[0],ymm7[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm7, %ymm5 # ymm5 = ymm7[1],ymm5[1],ymm7[3],ymm5[3]+        vpunpcklqdq %ymm9, %ymm4, %ymm7 # ymm7 = ymm4[0],ymm9[0],ymm4[2],ymm9[2]+        vpunpckhqdq %ymm9, %ymm4, %ymm9 # ymm9 = ymm4[1],ymm9[1],ymm4[3],ymm9[3]+        vmovsldup %ymm5, %ymm4    # ymm4 = ymm5[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7]+        vpsrlq $0x20, %ymm6, %ymm6+        vpblendd $0xaa, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[0],ymm5[1],ymm6[2],ymm5[3],ymm6[4],ymm5[5],ymm6[6],ymm5[7]+        vmovsldup %ymm7, %ymm6    # ymm6 = ymm7[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7]+        vpsrlq $0x20, %ymm8, %ymm8+        vpblendd $0xaa, %ymm7, %ymm8, %ymm7 # ymm7 = ymm8[0],ymm7[1],ymm8[2],ymm7[3],ymm8[4],ymm7[5],ymm8[6],ymm7[7]+        vmovsldup %ymm9, %ymm8    # ymm8 = ymm9[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm8, %ymm3, %ymm8 # ymm8 = ymm3[0],ymm8[1],ymm3[2],ymm8[3],ymm3[4],ymm8[5],ymm3[6],ymm8[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm9, %ymm3, %ymm9 # ymm9 = ymm3[0],ymm9[1],ymm3[2],ymm9[3],ymm3[4],ymm9[5],ymm3[6],ymm9[7]+        vpslld $0x10, %ymm7, %ymm10+        vpblendw $0xaa, %ymm10, %ymm4, %ymm10 # ymm10 = ymm4[0],ymm10[1],ymm4[2],ymm10[3],ymm4[4],ymm10[5],ymm4[6],ymm10[7],ymm4[8],ymm10[9],ymm4[10],ymm10[11],ymm4[12],ymm10[13],ymm4[14],ymm10[15]+        vpsrld $0x10, %ymm4, %ymm4+        vpblendw $0xaa, %ymm7, %ymm4, %ymm7 # ymm7 = ymm4[0],ymm7[1],ymm4[2],ymm7[3],ymm4[4],ymm7[5],ymm4[6],ymm7[7],ymm4[8],ymm7[9],ymm4[10],ymm7[11],ymm4[12],ymm7[13],ymm4[14],ymm7[15]+        vpslld $0x10, %ymm8, %ymm4+        vpblendw $0xaa, %ymm4, %ymm5, %ymm4 # ymm4 = ymm5[0],ymm4[1],ymm5[2],ymm4[3],ymm5[4],ymm4[5],ymm5[6],ymm4[7],ymm5[8],ymm4[9],ymm5[10],ymm4[11],ymm5[12],ymm4[13],ymm5[14],ymm4[15]+        vpsrld $0x10, %ymm5, %ymm5+        vpblendw $0xaa, %ymm8, %ymm5, %ymm8 # ymm8 = ymm5[0],ymm8[1],ymm5[2],ymm8[3],ymm5[4],ymm8[5],ymm5[6],ymm8[7],ymm5[8],ymm8[9],ymm5[10],ymm8[11],ymm5[12],ymm8[13],ymm5[14],ymm8[15]+        vpslld $0x10, %ymm9, %ymm5+        vpblendw $0xaa, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[0],ymm5[1],ymm6[2],ymm5[3],ymm6[4],ymm5[5],ymm6[6],ymm5[7],ymm6[8],ymm5[9],ymm6[10],ymm5[11],ymm6[12],ymm5[13],ymm6[14],ymm5[15]+        vpsrld $0x10, %ymm6, %ymm6+        vpblendw $0xaa, %ymm9, %ymm6, %ymm9 # ymm9 = ymm6[0],ymm9[1],ymm6[2],ymm9[3],ymm6[4],ymm9[5],ymm6[6],ymm9[7],ymm6[8],ymm9[9],ymm6[10],ymm9[11],ymm6[12],ymm9[13],ymm6[14],ymm9[15]+        vpsrlw $0xc, %ymm10, %ymm11+        vpsllw $0x4, %ymm7, %ymm12+        vpor %ymm11, %ymm12, %ymm11+        vpand %ymm0, %ymm10, %ymm10+        vpand %ymm0, %ymm11, %ymm11+        vpsrlw $0x8, %ymm7, %ymm12+        vpsllw $0x8, %ymm4, %ymm13+        vpor %ymm12, %ymm13, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpsrlw $0x4, %ymm4, %ymm13+        vpand %ymm0, %ymm13, %ymm13+        vpsrlw $0xc, %ymm8, %ymm14+        vpsllw $0x4, %ymm5, %ymm15+        vpor %ymm14, %ymm15, %ymm14+        vpand %ymm0, %ymm8, %ymm8+        vpand %ymm0, %ymm14, %ymm14+        vpsrlw $0x8, %ymm5, %ymm15+        vpsllw $0x8, %ymm9, %ymm1+        vpor %ymm15, %ymm1, %ymm15+        vpand %ymm0, %ymm15, %ymm15+        vpsrlw $0x4, %ymm9, %ymm1+        vpand %ymm0, %ymm1, %ymm1+        vmovdqa %ymm10, (%rdi)+        vmovdqa %ymm11, 0x20(%rdi)+        vmovdqa %ymm12, 0x40(%rdi)+        vmovdqa %ymm13, 0x60(%rdi)+        vmovdqa %ymm8, 0x80(%rdi)+        vmovdqa %ymm14, 0xa0(%rdi)+        vmovdqa %ymm15, 0xc0(%rdi)+        vmovdqa %ymm1, 0xe0(%rdi)+        vmovdqu 0xc0(%rsi), %ymm4+        vmovdqu 0xe0(%rsi), %ymm5+        vmovdqu 0x100(%rsi), %ymm6+        vmovdqu 0x120(%rsi), %ymm7+        vmovdqu 0x140(%rsi), %ymm8+        vmovdqu 0x160(%rsi), %ymm9+        vperm2i128 $0x20, %ymm7, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm7[0,1]+        vperm2i128 $0x31, %ymm7, %ymm4, %ymm7 # ymm7 = ymm4[2,3],ymm7[2,3]+        vperm2i128 $0x20, %ymm8, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm5, %ymm8 # ymm8 = ymm5[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm9, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm9[0,1]+        vperm2i128 $0x31, %ymm9, %ymm6, %ymm9 # ymm9 = ymm6[2,3],ymm9[2,3]+        vpunpcklqdq %ymm8, %ymm3, %ymm6 # ymm6 = ymm3[0],ymm8[0],ymm3[2],ymm8[2]+        vpunpckhqdq %ymm8, %ymm3, %ymm8 # ymm8 = ymm3[1],ymm8[1],ymm3[3],ymm8[3]+        vpunpcklqdq %ymm5, %ymm7, %ymm3 # ymm3 = ymm7[0],ymm5[0],ymm7[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm7, %ymm5 # ymm5 = ymm7[1],ymm5[1],ymm7[3],ymm5[3]+        vpunpcklqdq %ymm9, %ymm4, %ymm7 # ymm7 = ymm4[0],ymm9[0],ymm4[2],ymm9[2]+        vpunpckhqdq %ymm9, %ymm4, %ymm9 # ymm9 = ymm4[1],ymm9[1],ymm4[3],ymm9[3]+        vmovsldup %ymm5, %ymm4    # ymm4 = ymm5[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7]+        vpsrlq $0x20, %ymm6, %ymm6+        vpblendd $0xaa, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[0],ymm5[1],ymm6[2],ymm5[3],ymm6[4],ymm5[5],ymm6[6],ymm5[7]+        vmovsldup %ymm7, %ymm6    # ymm6 = ymm7[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7]+        vpsrlq $0x20, %ymm8, %ymm8+        vpblendd $0xaa, %ymm7, %ymm8, %ymm7 # ymm7 = ymm8[0],ymm7[1],ymm8[2],ymm7[3],ymm8[4],ymm7[5],ymm8[6],ymm7[7]+        vmovsldup %ymm9, %ymm8    # ymm8 = ymm9[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm8, %ymm3, %ymm8 # ymm8 = ymm3[0],ymm8[1],ymm3[2],ymm8[3],ymm3[4],ymm8[5],ymm3[6],ymm8[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm9, %ymm3, %ymm9 # ymm9 = ymm3[0],ymm9[1],ymm3[2],ymm9[3],ymm3[4],ymm9[5],ymm3[6],ymm9[7]+        vpslld $0x10, %ymm7, %ymm10+        vpblendw $0xaa, %ymm10, %ymm4, %ymm10 # ymm10 = ymm4[0],ymm10[1],ymm4[2],ymm10[3],ymm4[4],ymm10[5],ymm4[6],ymm10[7],ymm4[8],ymm10[9],ymm4[10],ymm10[11],ymm4[12],ymm10[13],ymm4[14],ymm10[15]+        vpsrld $0x10, %ymm4, %ymm4+        vpblendw $0xaa, %ymm7, %ymm4, %ymm7 # ymm7 = ymm4[0],ymm7[1],ymm4[2],ymm7[3],ymm4[4],ymm7[5],ymm4[6],ymm7[7],ymm4[8],ymm7[9],ymm4[10],ymm7[11],ymm4[12],ymm7[13],ymm4[14],ymm7[15]+        vpslld $0x10, %ymm8, %ymm4+        vpblendw $0xaa, %ymm4, %ymm5, %ymm4 # ymm4 = ymm5[0],ymm4[1],ymm5[2],ymm4[3],ymm5[4],ymm4[5],ymm5[6],ymm4[7],ymm5[8],ymm4[9],ymm5[10],ymm4[11],ymm5[12],ymm4[13],ymm5[14],ymm4[15]+        vpsrld $0x10, %ymm5, %ymm5+        vpblendw $0xaa, %ymm8, %ymm5, %ymm8 # ymm8 = ymm5[0],ymm8[1],ymm5[2],ymm8[3],ymm5[4],ymm8[5],ymm5[6],ymm8[7],ymm5[8],ymm8[9],ymm5[10],ymm8[11],ymm5[12],ymm8[13],ymm5[14],ymm8[15]+        vpslld $0x10, %ymm9, %ymm5+        vpblendw $0xaa, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[0],ymm5[1],ymm6[2],ymm5[3],ymm6[4],ymm5[5],ymm6[6],ymm5[7],ymm6[8],ymm5[9],ymm6[10],ymm5[11],ymm6[12],ymm5[13],ymm6[14],ymm5[15]+        vpsrld $0x10, %ymm6, %ymm6+        vpblendw $0xaa, %ymm9, %ymm6, %ymm9 # ymm9 = ymm6[0],ymm9[1],ymm6[2],ymm9[3],ymm6[4],ymm9[5],ymm6[6],ymm9[7],ymm6[8],ymm9[9],ymm6[10],ymm9[11],ymm6[12],ymm9[13],ymm6[14],ymm9[15]+        vpsrlw $0xc, %ymm10, %ymm11+        vpsllw $0x4, %ymm7, %ymm12+        vpor %ymm11, %ymm12, %ymm11+        vpand %ymm0, %ymm10, %ymm10+        vpand %ymm0, %ymm11, %ymm11+        vpsrlw $0x8, %ymm7, %ymm12+        vpsllw $0x8, %ymm4, %ymm13+        vpor %ymm12, %ymm13, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpsrlw $0x4, %ymm4, %ymm13+        vpand %ymm0, %ymm13, %ymm13+        vpsrlw $0xc, %ymm8, %ymm14+        vpsllw $0x4, %ymm5, %ymm15+        vpor %ymm14, %ymm15, %ymm14+        vpand %ymm0, %ymm8, %ymm8+        vpand %ymm0, %ymm14, %ymm14+        vpsrlw $0x8, %ymm5, %ymm15+        vpsllw $0x8, %ymm9, %ymm1+        vpor %ymm15, %ymm1, %ymm15+        vpand %ymm0, %ymm15, %ymm15+        vpsrlw $0x4, %ymm9, %ymm1+        vpand %ymm0, %ymm1, %ymm1+        vmovdqa %ymm10, 0x100(%rdi)+        vmovdqa %ymm11, 0x120(%rdi)+        vmovdqa %ymm12, 0x140(%rdi)+        vmovdqa %ymm13, 0x160(%rdi)+        vmovdqa %ymm8, 0x180(%rdi)+        vmovdqa %ymm14, 0x1a0(%rdi)+        vmovdqa %ymm15, 0x1c0(%rdi)+        vmovdqa %ymm1, 0x1e0(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(nttfrombytes_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+          && (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_ntttobytes_avx2_asm.S view
@@ -0,0 +1,205 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Kyber optimized AVX2 implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Kyber implementation @[REF_AVX2].+ */++#include "../../../common.h"++#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    (!defined(MLK_CONFIG_NO_KEYPAIR_API) || \+     !defined(MLK_CONFIG_NO_ENCAPS_API))+/*yaml+  Name: ntttobytes_avx2_asm+  Description: x86_64 AVX2 polynomial serialization in NTT domain+  Signature: void mlk_ntttobytes_avx2_asm(uint8_t *r, const int16_t *a)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 384+      permissions: write-only+      c_parameter: uint8_t *r+      description: Output byte array (MLKEM_POLYBYTES = 384)+    rsi:+      type: buffer+      size_bytes: 512+      permissions: read-only+      c_parameter: const int16_t *a+      description: Input polynomial (256 x int16_t)+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_ntttobytes_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(ntttobytes_avx2_asm)+MLK_ASM_FN_SYMBOL(ntttobytes_avx2_asm)++        .cfi_startproc+        movl $0xd010d01, %eax        # imm = 0xD010D01+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        vmovdqa (%rsi), %ymm5+        vmovdqa 0x20(%rsi), %ymm6+        vmovdqa 0x40(%rsi), %ymm7+        vmovdqa 0x60(%rsi), %ymm8+        vmovdqa 0x80(%rsi), %ymm9+        vmovdqa 0xa0(%rsi), %ymm10+        vmovdqa 0xc0(%rsi), %ymm11+        vmovdqa 0xe0(%rsi), %ymm12+        vpsllw $0xc, %ymm6, %ymm4+        vpor %ymm4, %ymm5, %ymm4+        vpsrlw $0x4, %ymm6, %ymm5+        vpsllw $0x8, %ymm7, %ymm6+        vpor %ymm5, %ymm6, %ymm5+        vpsrlw $0x8, %ymm7, %ymm6+        vpsllw $0x4, %ymm8, %ymm7+        vpor %ymm6, %ymm7, %ymm6+        vpsllw $0xc, %ymm10, %ymm7+        vpor %ymm7, %ymm9, %ymm7+        vpsrlw $0x4, %ymm10, %ymm8+        vpsllw $0x8, %ymm11, %ymm9+        vpor %ymm8, %ymm9, %ymm8+        vpsrlw $0x8, %ymm11, %ymm9+        vpsllw $0x4, %ymm12, %ymm10+        vpor %ymm9, %ymm10, %ymm9+        vpslld $0x10, %ymm5, %ymm3+        vpblendw $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7],ymm4[8],ymm3[9],ymm4[10],ymm3[11],ymm4[12],ymm3[13],ymm4[14],ymm3[15]+        vpsrld $0x10, %ymm4, %ymm4+        vpblendw $0xaa, %ymm5, %ymm4, %ymm5 # ymm5 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7],ymm4[8],ymm5[9],ymm4[10],ymm5[11],ymm4[12],ymm5[13],ymm4[14],ymm5[15]+        vpslld $0x10, %ymm7, %ymm4+        vpblendw $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7],ymm6[8],ymm4[9],ymm6[10],ymm4[11],ymm6[12],ymm4[13],ymm6[14],ymm4[15]+        vpsrld $0x10, %ymm6, %ymm6+        vpblendw $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7],ymm6[8],ymm7[9],ymm6[10],ymm7[11],ymm6[12],ymm7[13],ymm6[14],ymm7[15]+        vpslld $0x10, %ymm9, %ymm6+        vpblendw $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7],ymm8[8],ymm6[9],ymm8[10],ymm6[11],ymm8[12],ymm6[13],ymm8[14],ymm6[15]+        vpsrld $0x10, %ymm8, %ymm8+        vpblendw $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7],ymm8[8],ymm9[9],ymm8[10],ymm9[11],ymm8[12],ymm9[13],ymm8[14],ymm9[15]+        vmovsldup %ymm4, %ymm8    # ymm8 = ymm4[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm8, %ymm3, %ymm8 # ymm8 = ymm3[0],ymm8[1],ymm3[2],ymm8[3],ymm3[4],ymm8[5],ymm3[6],ymm8[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7]+        vmovsldup %ymm5, %ymm3    # ymm3 = ymm5[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm6, %ymm3 # ymm3 = ymm6[0],ymm3[1],ymm6[2],ymm3[3],ymm6[4],ymm3[5],ymm6[6],ymm3[7]+        vpsrlq $0x20, %ymm6, %ymm6+        vpblendd $0xaa, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[0],ymm5[1],ymm6[2],ymm5[3],ymm6[4],ymm5[5],ymm6[6],ymm5[7]+        vmovsldup %ymm9, %ymm6    # ymm6 = ymm9[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm6, %ymm7, %ymm6 # ymm6 = ymm7[0],ymm6[1],ymm7[2],ymm6[3],ymm7[4],ymm6[5],ymm7[6],ymm6[7]+        vpsrlq $0x20, %ymm7, %ymm7+        vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7]+        vpunpcklqdq %ymm3, %ymm8, %ymm7 # ymm7 = ymm8[0],ymm3[0],ymm8[2],ymm3[2]+        vpunpckhqdq %ymm3, %ymm8, %ymm3 # ymm3 = ymm8[1],ymm3[1],ymm8[3],ymm3[3]+        vpunpcklqdq %ymm4, %ymm6, %ymm8 # ymm8 = ymm6[0],ymm4[0],ymm6[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[1],ymm4[1],ymm6[3],ymm4[3]+        vpunpcklqdq %ymm9, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm9[0],ymm5[2],ymm9[2]+        vpunpckhqdq %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[1],ymm9[1],ymm5[3],ymm9[3]+        vperm2i128 $0x20, %ymm8, %ymm7, %ymm5 # ymm5 = ymm7[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm3, %ymm6, %ymm7 # ymm7 = ymm6[0,1],ymm3[0,1]+        vperm2i128 $0x31, %ymm3, %ymm6, %ymm3 # ymm3 = ymm6[2,3],ymm3[2,3]+        vperm2i128 $0x20, %ymm9, %ymm4, %ymm6 # ymm6 = ymm4[0,1],ymm9[0,1]+        vperm2i128 $0x31, %ymm9, %ymm4, %ymm9 # ymm9 = ymm4[2,3],ymm9[2,3]+        vmovdqu %ymm5, (%rdi)+        vmovdqu %ymm7, 0x20(%rdi)+        vmovdqu %ymm6, 0x40(%rdi)+        vmovdqu %ymm8, 0x60(%rdi)+        vmovdqu %ymm3, 0x80(%rdi)+        vmovdqu %ymm9, 0xa0(%rdi)+        vmovdqa 0x100(%rsi), %ymm5+        vmovdqa 0x120(%rsi), %ymm6+        vmovdqa 0x140(%rsi), %ymm7+        vmovdqa 0x160(%rsi), %ymm8+        vmovdqa 0x180(%rsi), %ymm9+        vmovdqa 0x1a0(%rsi), %ymm10+        vmovdqa 0x1c0(%rsi), %ymm11+        vmovdqa 0x1e0(%rsi), %ymm12+        vpsllw $0xc, %ymm6, %ymm4+        vpor %ymm4, %ymm5, %ymm4+        vpsrlw $0x4, %ymm6, %ymm5+        vpsllw $0x8, %ymm7, %ymm6+        vpor %ymm5, %ymm6, %ymm5+        vpsrlw $0x8, %ymm7, %ymm6+        vpsllw $0x4, %ymm8, %ymm7+        vpor %ymm6, %ymm7, %ymm6+        vpsllw $0xc, %ymm10, %ymm7+        vpor %ymm7, %ymm9, %ymm7+        vpsrlw $0x4, %ymm10, %ymm8+        vpsllw $0x8, %ymm11, %ymm9+        vpor %ymm8, %ymm9, %ymm8+        vpsrlw $0x8, %ymm11, %ymm9+        vpsllw $0x4, %ymm12, %ymm10+        vpor %ymm9, %ymm10, %ymm9+        vpslld $0x10, %ymm5, %ymm3+        vpblendw $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7],ymm4[8],ymm3[9],ymm4[10],ymm3[11],ymm4[12],ymm3[13],ymm4[14],ymm3[15]+        vpsrld $0x10, %ymm4, %ymm4+        vpblendw $0xaa, %ymm5, %ymm4, %ymm5 # ymm5 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7],ymm4[8],ymm5[9],ymm4[10],ymm5[11],ymm4[12],ymm5[13],ymm4[14],ymm5[15]+        vpslld $0x10, %ymm7, %ymm4+        vpblendw $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7],ymm6[8],ymm4[9],ymm6[10],ymm4[11],ymm6[12],ymm4[13],ymm6[14],ymm4[15]+        vpsrld $0x10, %ymm6, %ymm6+        vpblendw $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7],ymm6[8],ymm7[9],ymm6[10],ymm7[11],ymm6[12],ymm7[13],ymm6[14],ymm7[15]+        vpslld $0x10, %ymm9, %ymm6+        vpblendw $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7],ymm8[8],ymm6[9],ymm8[10],ymm6[11],ymm8[12],ymm6[13],ymm8[14],ymm6[15]+        vpsrld $0x10, %ymm8, %ymm8+        vpblendw $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7],ymm8[8],ymm9[9],ymm8[10],ymm9[11],ymm8[12],ymm9[13],ymm8[14],ymm9[15]+        vmovsldup %ymm4, %ymm8    # ymm8 = ymm4[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm8, %ymm3, %ymm8 # ymm8 = ymm3[0],ymm8[1],ymm3[2],ymm8[3],ymm3[4],ymm8[5],ymm3[6],ymm8[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7]+        vmovsldup %ymm5, %ymm3    # ymm3 = ymm5[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm6, %ymm3 # ymm3 = ymm6[0],ymm3[1],ymm6[2],ymm3[3],ymm6[4],ymm3[5],ymm6[6],ymm3[7]+        vpsrlq $0x20, %ymm6, %ymm6+        vpblendd $0xaa, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[0],ymm5[1],ymm6[2],ymm5[3],ymm6[4],ymm5[5],ymm6[6],ymm5[7]+        vmovsldup %ymm9, %ymm6    # ymm6 = ymm9[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm6, %ymm7, %ymm6 # ymm6 = ymm7[0],ymm6[1],ymm7[2],ymm6[3],ymm7[4],ymm6[5],ymm7[6],ymm6[7]+        vpsrlq $0x20, %ymm7, %ymm7+        vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7]+        vpunpcklqdq %ymm3, %ymm8, %ymm7 # ymm7 = ymm8[0],ymm3[0],ymm8[2],ymm3[2]+        vpunpckhqdq %ymm3, %ymm8, %ymm3 # ymm3 = ymm8[1],ymm3[1],ymm8[3],ymm3[3]+        vpunpcklqdq %ymm4, %ymm6, %ymm8 # ymm8 = ymm6[0],ymm4[0],ymm6[2],ymm4[2]+        vpunpckhqdq %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[1],ymm4[1],ymm6[3],ymm4[3]+        vpunpcklqdq %ymm9, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm9[0],ymm5[2],ymm9[2]+        vpunpckhqdq %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[1],ymm9[1],ymm5[3],ymm9[3]+        vperm2i128 $0x20, %ymm8, %ymm7, %ymm5 # ymm5 = ymm7[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm3, %ymm6, %ymm7 # ymm7 = ymm6[0,1],ymm3[0,1]+        vperm2i128 $0x31, %ymm3, %ymm6, %ymm3 # ymm3 = ymm6[2,3],ymm3[2,3]+        vperm2i128 $0x20, %ymm9, %ymm4, %ymm6 # ymm6 = ymm4[0,1],ymm9[0,1]+        vperm2i128 $0x31, %ymm9, %ymm4, %ymm9 # ymm9 = ymm4[2,3],ymm9[2,3]+        vmovdqu %ymm5, 0xc0(%rdi)+        vmovdqu %ymm7, 0xe0(%rdi)+        vmovdqu %ymm6, 0x100(%rdi)+        vmovdqu %ymm8, 0x120(%rdi)+        vmovdqu %ymm3, 0x140(%rdi)+        vmovdqu %ymm9, 0x160(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(ntttobytes_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+          && (!MLK_CONFIG_NO_KEYPAIR_API || !MLK_CONFIG_NO_ENCAPS_API) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_nttunpack_avx2_asm.S view
@@ -0,0 +1,190 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Kyber optimized AVX2 implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Kyber implementation @[REF_AVX2].+ */++#include "../../../common.h"++#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)+/*yaml+  Name: nttunpack_avx2_asm+  Description: x86_64 AVX2 NTT unpack from custom to bitreversed order+  Signature: void mlk_nttunpack_avx2_asm(int16_t *r)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 512+      permissions: read/write+      c_parameter: int16_t *r+      description: Input/output polynomial (256 x int16_t)+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_nttunpack_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(nttunpack_avx2_asm)+MLK_ASM_FN_SYMBOL(nttunpack_avx2_asm)++        .cfi_startproc+        vmovdqa (%rdi), %ymm4+        vmovdqa 0x20(%rdi), %ymm5+        vmovdqa 0x40(%rdi), %ymm6+        vmovdqa 0x60(%rdi), %ymm7+        vmovdqa 0x80(%rdi), %ymm8+        vmovdqa 0xa0(%rdi), %ymm9+        vmovdqa 0xc0(%rdi), %ymm10+        vmovdqa 0xe0(%rdi), %ymm11+        vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1]+        vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3]+        vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1]+        vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3]+        vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3]+        vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3]+        vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2]+        vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3]+        vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2]+        vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3]+        vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3]+        vmovsldup %ymm8, %ymm9    # ymm9 = ymm8[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7]+        vpsrlq $0x20, %ymm7, %ymm7+        vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7]+        vmovsldup %ymm6, %ymm7    # ymm7 = ymm6[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7]+        vpsrlq $0x20, %ymm5, %ymm5+        vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7]+        vmovsldup %ymm4, %ymm5    # ymm5 = ymm4[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7]+        vmovsldup %ymm11, %ymm3   # ymm3 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7]+        vpsrlq $0x20, %ymm10, %ymm10+        vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7]+        vpslld $0x10, %ymm5, %ymm10+        vpblendw $0xaa, %ymm10, %ymm9, %ymm10 # ymm10 = ymm9[0],ymm10[1],ymm9[2],ymm10[3],ymm9[4],ymm10[5],ymm9[6],ymm10[7],ymm9[8],ymm10[9],ymm9[10],ymm10[11],ymm9[12],ymm10[13],ymm9[14],ymm10[15]+        vpsrld $0x10, %ymm9, %ymm9+        vpblendw $0xaa, %ymm5, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm5[1],ymm9[2],ymm5[3],ymm9[4],ymm5[5],ymm9[6],ymm5[7],ymm9[8],ymm5[9],ymm9[10],ymm5[11],ymm9[12],ymm5[13],ymm9[14],ymm5[15]+        vpslld $0x10, %ymm4, %ymm9+        vpblendw $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7],ymm8[8],ymm9[9],ymm8[10],ymm9[11],ymm8[12],ymm9[13],ymm8[14],ymm9[15]+        vpsrld $0x10, %ymm8, %ymm8+        vpblendw $0xaa, %ymm4, %ymm8, %ymm4 # ymm4 = ymm8[0],ymm4[1],ymm8[2],ymm4[3],ymm8[4],ymm4[5],ymm8[6],ymm4[7],ymm8[8],ymm4[9],ymm8[10],ymm4[11],ymm8[12],ymm4[13],ymm8[14],ymm4[15]+        vpslld $0x10, %ymm3, %ymm8+        vpblendw $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7],ymm7[8],ymm8[9],ymm7[10],ymm8[11],ymm7[12],ymm8[13],ymm7[14],ymm8[15]+        vpsrld $0x10, %ymm7, %ymm7+        vpblendw $0xaa, %ymm3, %ymm7, %ymm3 # ymm3 = ymm7[0],ymm3[1],ymm7[2],ymm3[3],ymm7[4],ymm3[5],ymm7[6],ymm3[7],ymm7[8],ymm3[9],ymm7[10],ymm3[11],ymm7[12],ymm3[13],ymm7[14],ymm3[15]+        vpslld $0x10, %ymm11, %ymm7+        vpblendw $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7],ymm6[8],ymm7[9],ymm6[10],ymm7[11],ymm6[12],ymm7[13],ymm6[14],ymm7[15]+        vpsrld $0x10, %ymm6, %ymm6+        vpblendw $0xaa, %ymm11, %ymm6, %ymm11 # ymm11 = ymm6[0],ymm11[1],ymm6[2],ymm11[3],ymm6[4],ymm11[5],ymm6[6],ymm11[7],ymm6[8],ymm11[9],ymm6[10],ymm11[11],ymm6[12],ymm11[13],ymm6[14],ymm11[15]+        vmovdqa %ymm10, (%rdi)+        vmovdqa %ymm5, 0x20(%rdi)+        vmovdqa %ymm9, 0x40(%rdi)+        vmovdqa %ymm4, 0x60(%rdi)+        vmovdqa %ymm8, 0x80(%rdi)+        vmovdqa %ymm3, 0xa0(%rdi)+        vmovdqa %ymm7, 0xc0(%rdi)+        vmovdqa %ymm11, 0xe0(%rdi)+        vmovdqa 0x100(%rdi), %ymm4+        vmovdqa 0x120(%rdi), %ymm5+        vmovdqa 0x140(%rdi), %ymm6+        vmovdqa 0x160(%rdi), %ymm7+        vmovdqa 0x180(%rdi), %ymm8+        vmovdqa 0x1a0(%rdi), %ymm9+        vmovdqa 0x1c0(%rdi), %ymm10+        vmovdqa 0x1e0(%rdi), %ymm11+        vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1]+        vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3]+        vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1]+        vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3]+        vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1]+        vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3]+        vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1]+        vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3]+        vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2]+        vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3]+        vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2]+        vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3]+        vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2]+        vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3]+        vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2]+        vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3]+        vmovsldup %ymm8, %ymm9    # ymm9 = ymm8[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7]+        vpsrlq $0x20, %ymm7, %ymm7+        vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7]+        vmovsldup %ymm6, %ymm7    # ymm7 = ymm6[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7]+        vpsrlq $0x20, %ymm5, %ymm5+        vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7]+        vmovsldup %ymm4, %ymm5    # ymm5 = ymm4[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7]+        vpsrlq $0x20, %ymm3, %ymm3+        vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7]+        vmovsldup %ymm11, %ymm3   # ymm3 = ymm11[0,0,2,2,4,4,6,6]+        vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7]+        vpsrlq $0x20, %ymm10, %ymm10+        vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7]+        vpslld $0x10, %ymm5, %ymm10+        vpblendw $0xaa, %ymm10, %ymm9, %ymm10 # ymm10 = ymm9[0],ymm10[1],ymm9[2],ymm10[3],ymm9[4],ymm10[5],ymm9[6],ymm10[7],ymm9[8],ymm10[9],ymm9[10],ymm10[11],ymm9[12],ymm10[13],ymm9[14],ymm10[15]+        vpsrld $0x10, %ymm9, %ymm9+        vpblendw $0xaa, %ymm5, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm5[1],ymm9[2],ymm5[3],ymm9[4],ymm5[5],ymm9[6],ymm5[7],ymm9[8],ymm5[9],ymm9[10],ymm5[11],ymm9[12],ymm5[13],ymm9[14],ymm5[15]+        vpslld $0x10, %ymm4, %ymm9+        vpblendw $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7],ymm8[8],ymm9[9],ymm8[10],ymm9[11],ymm8[12],ymm9[13],ymm8[14],ymm9[15]+        vpsrld $0x10, %ymm8, %ymm8+        vpblendw $0xaa, %ymm4, %ymm8, %ymm4 # ymm4 = ymm8[0],ymm4[1],ymm8[2],ymm4[3],ymm8[4],ymm4[5],ymm8[6],ymm4[7],ymm8[8],ymm4[9],ymm8[10],ymm4[11],ymm8[12],ymm4[13],ymm8[14],ymm4[15]+        vpslld $0x10, %ymm3, %ymm8+        vpblendw $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7],ymm7[8],ymm8[9],ymm7[10],ymm8[11],ymm7[12],ymm8[13],ymm7[14],ymm8[15]+        vpsrld $0x10, %ymm7, %ymm7+        vpblendw $0xaa, %ymm3, %ymm7, %ymm3 # ymm3 = ymm7[0],ymm3[1],ymm7[2],ymm3[3],ymm7[4],ymm3[5],ymm7[6],ymm3[7],ymm7[8],ymm3[9],ymm7[10],ymm3[11],ymm7[12],ymm3[13],ymm7[14],ymm3[15]+        vpslld $0x10, %ymm11, %ymm7+        vpblendw $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7],ymm6[8],ymm7[9],ymm6[10],ymm7[11],ymm6[12],ymm7[13],ymm6[14],ymm7[15]+        vpsrld $0x10, %ymm6, %ymm6+        vpblendw $0xaa, %ymm11, %ymm6, %ymm11 # ymm11 = ymm6[0],ymm11[1],ymm6[2],ymm11[3],ymm6[4],ymm11[5],ymm6[6],ymm11[7],ymm6[8],ymm11[9],ymm6[10],ymm11[11],ymm6[12],ymm11[13],ymm6[14],ymm11[15]+        vmovdqa %ymm10, 0x100(%rdi)+        vmovdqa %ymm5, 0x120(%rdi)+        vmovdqa %ymm9, 0x140(%rdi)+        vmovdqa %ymm4, 0x160(%rdi)+        vmovdqa %ymm8, 0x180(%rdi)+        vmovdqa %ymm3, 0x1a0(%rdi)+        vmovdqa %ymm7, 0x1c0(%rdi)+        vmovdqa %ymm11, 0x1e0(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(nttunpack_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_poly_compress_d10_avx2_asm.S view
@@ -0,0 +1,413 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Kyber optimized AVX2 implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Kyber implementation @[REF_AVX2].+ */++/*************************************************+ * Name:        mlk_poly_compress_d10_avx2_asm+ *+ * Description: Compression of a polynomial to 10 bits per coefficient.+ *+ * Arguments:   - uint8_t *r:       pointer to output byte array+ *                                  (of length MLKEM_POLYCOMPRESSEDBYTES_D10)+ *              - const int16_t *a: pointer to input polynomial+ *              - const uint8_t *data: pointer to shufbidx constant+ **************************************************/++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    (!defined(MLK_CONFIG_NO_ENCAPS_API) || \+     !defined(MLK_CONFIG_NO_DECAPS_API)) && \+    (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2 || MLKEM_K == 3)+/*yaml+  Name: poly_compress_d10_avx2_asm+  Description: x86_64 AVX2 polynomial compression (d=10)+  Signature: void mlk_poly_compress_d10_avx2_asm(uint8_t *r, const int16_t *a, const uint8_t *data)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 320+      permissions: write-only+      c_parameter: uint8_t *r+      description: Output compressed polynomial+    rsi:+      type: buffer+      size_bytes: 512+      permissions: read-only+      c_parameter: const int16_t *a+      description: Input polynomial (256 x int16_t)+    rdx:+      type: buffer+      size_bytes: 32+      permissions: read-only+      c_parameter: const uint8_t *data+      description: Precomputed compression constants+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_poly_compress_d10_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(poly_compress_d10_avx2_asm)+MLK_ASM_FN_SYMBOL(poly_compress_d10_avx2_asm)++        .cfi_startproc+        movl $0x4ebf4ebf, %eax       # imm = 0x4EBF4EBF+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        vpsllw $0x3, %ymm0, %ymm1+        movl $0xf000f, %eax          # imm = 0xF000F+        vmovd %eax, %xmm2+        vpbroadcastd %xmm2, %ymm2+        movl $0x10001000, %eax       # imm = 0x10001000+        vmovd %eax, %xmm3+        vpbroadcastd %xmm3, %ymm3+        movl $0x3ff03ff, %eax        # imm = 0x3FF03FF+        vmovd %eax, %xmm4+        vpbroadcastd %xmm4, %ymm4+        movabsq $0x400000104000001, %rax # imm = 0x400000104000001+        vmovq %rax, %xmm5+        vpbroadcastq %xmm5, %ymm5+        movl $0xc, %eax+        vmovq %rax, %xmm6+        vpbroadcastq %xmm6, %ymm6+        vmovdqa (%rdx), %ymm7+        vmovdqa (%rsi), %ymm8+        vpmullw %ymm1, %ymm8, %ymm9+        vpaddw %ymm2, %ymm8, %ymm10+        vpsllw $0x3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm8, %ymm8+        vpsubw %ymm10, %ymm9, %ymm10+        vpandn %ymm10, %ymm9, %ymm9+        vpsrlw $0xf, %ymm9, %ymm9+        vpsubw %ymm9, %ymm8, %ymm8+        vpmulhrsw %ymm3, %ymm8, %ymm8+        vpand %ymm4, %ymm8, %ymm8+        vpmaddwd %ymm5, %ymm8, %ymm8+        vpsllvd %ymm6, %ymm8, %ymm8+        vpsrlq $0xc, %ymm8, %ymm8+        vpshufb %ymm7, %ymm8, %ymm8+        vextracti128 $0x1, %ymm8, %xmm9+        vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7]+        vmovdqu %xmm8, (%rdi)+        vmovd %xmm9, 0x10(%rdi)+        vmovdqa 0x20(%rsi), %ymm8+        vpmullw %ymm1, %ymm8, %ymm9+        vpaddw %ymm2, %ymm8, %ymm10+        vpsllw $0x3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm8, %ymm8+        vpsubw %ymm10, %ymm9, %ymm10+        vpandn %ymm10, %ymm9, %ymm9+        vpsrlw $0xf, %ymm9, %ymm9+        vpsubw %ymm9, %ymm8, %ymm8+        vpmulhrsw %ymm3, %ymm8, %ymm8+        vpand %ymm4, %ymm8, %ymm8+        vpmaddwd %ymm5, %ymm8, %ymm8+        vpsllvd %ymm6, %ymm8, %ymm8+        vpsrlq $0xc, %ymm8, %ymm8+        vpshufb %ymm7, %ymm8, %ymm8+        vextracti128 $0x1, %ymm8, %xmm9+        vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7]+        vmovdqu %xmm8, 0x14(%rdi)+        vmovd %xmm9, 0x24(%rdi)+        vmovdqa 0x40(%rsi), %ymm8+        vpmullw %ymm1, %ymm8, %ymm9+        vpaddw %ymm2, %ymm8, %ymm10+        vpsllw $0x3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm8, %ymm8+        vpsubw %ymm10, %ymm9, %ymm10+        vpandn %ymm10, %ymm9, %ymm9+        vpsrlw $0xf, %ymm9, %ymm9+        vpsubw %ymm9, %ymm8, %ymm8+        vpmulhrsw %ymm3, %ymm8, %ymm8+        vpand %ymm4, %ymm8, %ymm8+        vpmaddwd %ymm5, %ymm8, %ymm8+        vpsllvd %ymm6, %ymm8, %ymm8+        vpsrlq $0xc, %ymm8, %ymm8+        vpshufb %ymm7, %ymm8, %ymm8+        vextracti128 $0x1, %ymm8, %xmm9+        vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7]+        vmovdqu %xmm8, 0x28(%rdi)+        vmovd %xmm9, 0x38(%rdi)+        vmovdqa 0x60(%rsi), %ymm8+        vpmullw %ymm1, %ymm8, %ymm9+        vpaddw %ymm2, %ymm8, %ymm10+        vpsllw $0x3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm8, %ymm8+        vpsubw %ymm10, %ymm9, %ymm10+        vpandn %ymm10, %ymm9, %ymm9+        vpsrlw $0xf, %ymm9, %ymm9+        vpsubw %ymm9, %ymm8, %ymm8+        vpmulhrsw %ymm3, %ymm8, %ymm8+        vpand %ymm4, %ymm8, %ymm8+        vpmaddwd %ymm5, %ymm8, %ymm8+        vpsllvd %ymm6, %ymm8, %ymm8+        vpsrlq $0xc, %ymm8, %ymm8+        vpshufb %ymm7, %ymm8, %ymm8+        vextracti128 $0x1, %ymm8, %xmm9+        vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7]+        vmovdqu %xmm8, 0x3c(%rdi)+        vmovd %xmm9, 0x4c(%rdi)+        vmovdqa 0x80(%rsi), %ymm8+        vpmullw %ymm1, %ymm8, %ymm9+        vpaddw %ymm2, %ymm8, %ymm10+        vpsllw $0x3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm8, %ymm8+        vpsubw %ymm10, %ymm9, %ymm10+        vpandn %ymm10, %ymm9, %ymm9+        vpsrlw $0xf, %ymm9, %ymm9+        vpsubw %ymm9, %ymm8, %ymm8+        vpmulhrsw %ymm3, %ymm8, %ymm8+        vpand %ymm4, %ymm8, %ymm8+        vpmaddwd %ymm5, %ymm8, %ymm8+        vpsllvd %ymm6, %ymm8, %ymm8+        vpsrlq $0xc, %ymm8, %ymm8+        vpshufb %ymm7, %ymm8, %ymm8+        vextracti128 $0x1, %ymm8, %xmm9+        vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7]+        vmovdqu %xmm8, 0x50(%rdi)+        vmovd %xmm9, 0x60(%rdi)+        vmovdqa 0xa0(%rsi), %ymm8+        vpmullw %ymm1, %ymm8, %ymm9+        vpaddw %ymm2, %ymm8, %ymm10+        vpsllw $0x3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm8, %ymm8+        vpsubw %ymm10, %ymm9, %ymm10+        vpandn %ymm10, %ymm9, %ymm9+        vpsrlw $0xf, %ymm9, %ymm9+        vpsubw %ymm9, %ymm8, %ymm8+        vpmulhrsw %ymm3, %ymm8, %ymm8+        vpand %ymm4, %ymm8, %ymm8+        vpmaddwd %ymm5, %ymm8, %ymm8+        vpsllvd %ymm6, %ymm8, %ymm8+        vpsrlq $0xc, %ymm8, %ymm8+        vpshufb %ymm7, %ymm8, %ymm8+        vextracti128 $0x1, %ymm8, %xmm9+        vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7]+        vmovdqu %xmm8, 0x64(%rdi)+        vmovd %xmm9, 0x74(%rdi)+        vmovdqa 0xc0(%rsi), %ymm8+        vpmullw %ymm1, %ymm8, %ymm9+        vpaddw %ymm2, %ymm8, %ymm10+        vpsllw $0x3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm8, %ymm8+        vpsubw %ymm10, %ymm9, %ymm10+        vpandn %ymm10, %ymm9, %ymm9+        vpsrlw $0xf, %ymm9, %ymm9+        vpsubw %ymm9, %ymm8, %ymm8+        vpmulhrsw %ymm3, %ymm8, %ymm8+        vpand %ymm4, %ymm8, %ymm8+        vpmaddwd %ymm5, %ymm8, %ymm8+        vpsllvd %ymm6, %ymm8, %ymm8+        vpsrlq $0xc, %ymm8, %ymm8+        vpshufb %ymm7, %ymm8, %ymm8+        vextracti128 $0x1, %ymm8, %xmm9+        vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7]+        vmovdqu %xmm8, 0x78(%rdi)+        vmovd %xmm9, 0x88(%rdi)+        vmovdqa 0xe0(%rsi), %ymm8+        vpmullw %ymm1, %ymm8, %ymm9+        vpaddw %ymm2, %ymm8, %ymm10+        vpsllw $0x3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm8, %ymm8+        vpsubw %ymm10, %ymm9, %ymm10+        vpandn %ymm10, %ymm9, %ymm9+        vpsrlw $0xf, %ymm9, %ymm9+        vpsubw %ymm9, %ymm8, %ymm8+        vpmulhrsw %ymm3, %ymm8, %ymm8+        vpand %ymm4, %ymm8, %ymm8+        vpmaddwd %ymm5, %ymm8, %ymm8+        vpsllvd %ymm6, %ymm8, %ymm8+        vpsrlq $0xc, %ymm8, %ymm8+        vpshufb %ymm7, %ymm8, %ymm8+        vextracti128 $0x1, %ymm8, %xmm9+        vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7]+        vmovdqu %xmm8, 0x8c(%rdi)+        vmovd %xmm9, 0x9c(%rdi)+        vmovdqa 0x100(%rsi), %ymm8+        vpmullw %ymm1, %ymm8, %ymm9+        vpaddw %ymm2, %ymm8, %ymm10+        vpsllw $0x3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm8, %ymm8+        vpsubw %ymm10, %ymm9, %ymm10+        vpandn %ymm10, %ymm9, %ymm9+        vpsrlw $0xf, %ymm9, %ymm9+        vpsubw %ymm9, %ymm8, %ymm8+        vpmulhrsw %ymm3, %ymm8, %ymm8+        vpand %ymm4, %ymm8, %ymm8+        vpmaddwd %ymm5, %ymm8, %ymm8+        vpsllvd %ymm6, %ymm8, %ymm8+        vpsrlq $0xc, %ymm8, %ymm8+        vpshufb %ymm7, %ymm8, %ymm8+        vextracti128 $0x1, %ymm8, %xmm9+        vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7]+        vmovdqu %xmm8, 0xa0(%rdi)+        vmovd %xmm9, 0xb0(%rdi)+        vmovdqa 0x120(%rsi), %ymm8+        vpmullw %ymm1, %ymm8, %ymm9+        vpaddw %ymm2, %ymm8, %ymm10+        vpsllw $0x3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm8, %ymm8+        vpsubw %ymm10, %ymm9, %ymm10+        vpandn %ymm10, %ymm9, %ymm9+        vpsrlw $0xf, %ymm9, %ymm9+        vpsubw %ymm9, %ymm8, %ymm8+        vpmulhrsw %ymm3, %ymm8, %ymm8+        vpand %ymm4, %ymm8, %ymm8+        vpmaddwd %ymm5, %ymm8, %ymm8+        vpsllvd %ymm6, %ymm8, %ymm8+        vpsrlq $0xc, %ymm8, %ymm8+        vpshufb %ymm7, %ymm8, %ymm8+        vextracti128 $0x1, %ymm8, %xmm9+        vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7]+        vmovdqu %xmm8, 0xb4(%rdi)+        vmovd %xmm9, 0xc4(%rdi)+        vmovdqa 0x140(%rsi), %ymm8+        vpmullw %ymm1, %ymm8, %ymm9+        vpaddw %ymm2, %ymm8, %ymm10+        vpsllw $0x3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm8, %ymm8+        vpsubw %ymm10, %ymm9, %ymm10+        vpandn %ymm10, %ymm9, %ymm9+        vpsrlw $0xf, %ymm9, %ymm9+        vpsubw %ymm9, %ymm8, %ymm8+        vpmulhrsw %ymm3, %ymm8, %ymm8+        vpand %ymm4, %ymm8, %ymm8+        vpmaddwd %ymm5, %ymm8, %ymm8+        vpsllvd %ymm6, %ymm8, %ymm8+        vpsrlq $0xc, %ymm8, %ymm8+        vpshufb %ymm7, %ymm8, %ymm8+        vextracti128 $0x1, %ymm8, %xmm9+        vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7]+        vmovdqu %xmm8, 0xc8(%rdi)+        vmovd %xmm9, 0xd8(%rdi)+        vmovdqa 0x160(%rsi), %ymm8+        vpmullw %ymm1, %ymm8, %ymm9+        vpaddw %ymm2, %ymm8, %ymm10+        vpsllw $0x3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm8, %ymm8+        vpsubw %ymm10, %ymm9, %ymm10+        vpandn %ymm10, %ymm9, %ymm9+        vpsrlw $0xf, %ymm9, %ymm9+        vpsubw %ymm9, %ymm8, %ymm8+        vpmulhrsw %ymm3, %ymm8, %ymm8+        vpand %ymm4, %ymm8, %ymm8+        vpmaddwd %ymm5, %ymm8, %ymm8+        vpsllvd %ymm6, %ymm8, %ymm8+        vpsrlq $0xc, %ymm8, %ymm8+        vpshufb %ymm7, %ymm8, %ymm8+        vextracti128 $0x1, %ymm8, %xmm9+        vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7]+        vmovdqu %xmm8, 0xdc(%rdi)+        vmovd %xmm9, 0xec(%rdi)+        vmovdqa 0x180(%rsi), %ymm8+        vpmullw %ymm1, %ymm8, %ymm9+        vpaddw %ymm2, %ymm8, %ymm10+        vpsllw $0x3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm8, %ymm8+        vpsubw %ymm10, %ymm9, %ymm10+        vpandn %ymm10, %ymm9, %ymm9+        vpsrlw $0xf, %ymm9, %ymm9+        vpsubw %ymm9, %ymm8, %ymm8+        vpmulhrsw %ymm3, %ymm8, %ymm8+        vpand %ymm4, %ymm8, %ymm8+        vpmaddwd %ymm5, %ymm8, %ymm8+        vpsllvd %ymm6, %ymm8, %ymm8+        vpsrlq $0xc, %ymm8, %ymm8+        vpshufb %ymm7, %ymm8, %ymm8+        vextracti128 $0x1, %ymm8, %xmm9+        vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7]+        vmovdqu %xmm8, 0xf0(%rdi)+        vmovd %xmm9, 0x100(%rdi)+        vmovdqa 0x1a0(%rsi), %ymm8+        vpmullw %ymm1, %ymm8, %ymm9+        vpaddw %ymm2, %ymm8, %ymm10+        vpsllw $0x3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm8, %ymm8+        vpsubw %ymm10, %ymm9, %ymm10+        vpandn %ymm10, %ymm9, %ymm9+        vpsrlw $0xf, %ymm9, %ymm9+        vpsubw %ymm9, %ymm8, %ymm8+        vpmulhrsw %ymm3, %ymm8, %ymm8+        vpand %ymm4, %ymm8, %ymm8+        vpmaddwd %ymm5, %ymm8, %ymm8+        vpsllvd %ymm6, %ymm8, %ymm8+        vpsrlq $0xc, %ymm8, %ymm8+        vpshufb %ymm7, %ymm8, %ymm8+        vextracti128 $0x1, %ymm8, %xmm9+        vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7]+        vmovdqu %xmm8, 0x104(%rdi)+        vmovd %xmm9, 0x114(%rdi)+        vmovdqa 0x1c0(%rsi), %ymm8+        vpmullw %ymm1, %ymm8, %ymm9+        vpaddw %ymm2, %ymm8, %ymm10+        vpsllw $0x3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm8, %ymm8+        vpsubw %ymm10, %ymm9, %ymm10+        vpandn %ymm10, %ymm9, %ymm9+        vpsrlw $0xf, %ymm9, %ymm9+        vpsubw %ymm9, %ymm8, %ymm8+        vpmulhrsw %ymm3, %ymm8, %ymm8+        vpand %ymm4, %ymm8, %ymm8+        vpmaddwd %ymm5, %ymm8, %ymm8+        vpsllvd %ymm6, %ymm8, %ymm8+        vpsrlq $0xc, %ymm8, %ymm8+        vpshufb %ymm7, %ymm8, %ymm8+        vextracti128 $0x1, %ymm8, %xmm9+        vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7]+        vmovdqu %xmm8, 0x118(%rdi)+        vmovd %xmm9, 0x128(%rdi)+        vmovdqa 0x1e0(%rsi), %ymm8+        vpmullw %ymm1, %ymm8, %ymm9+        vpaddw %ymm2, %ymm8, %ymm10+        vpsllw $0x3, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm8, %ymm8+        vpsubw %ymm10, %ymm9, %ymm10+        vpandn %ymm10, %ymm9, %ymm9+        vpsrlw $0xf, %ymm9, %ymm9+        vpsubw %ymm9, %ymm8, %ymm8+        vpmulhrsw %ymm3, %ymm8, %ymm8+        vpand %ymm4, %ymm8, %ymm8+        vpmaddwd %ymm5, %ymm8, %ymm8+        vpsllvd %ymm6, %ymm8, %ymm8+        vpsrlq $0xc, %ymm8, %ymm8+        vpshufb %ymm7, %ymm8, %ymm8+        vextracti128 $0x1, %ymm8, %xmm9+        vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7]+        vmovdqu %xmm8, 0x12c(%rdi)+        vmovd %xmm9, 0x13c(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(poly_compress_d10_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+          && (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) && \+          (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 || MLKEM_K == 3) \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_poly_compress_d11_avx2_asm.S view
@@ -0,0 +1,479 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Kyber optimized AVX2 implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Kyber implementation @[REF_AVX2].+ */++/*************************************************+ * Name:        mlk_poly_compress_d11_avx2_asm+ *+ * Description: Compression of a polynomial to 11 bits per coefficient.+ *+ * Arguments:   - uint8_t *r:       pointer to output byte array+ *                                  (of length MLKEM_POLYCOMPRESSEDBYTES_D11)+ *              - const int16_t *a: pointer to input polynomial+ *              - const uint8_t *data: pointer to constants+ *                                  (srlvqidx[0:32], shufbidx[32:64])+ **************************************************/++#include "../../../common.h"++#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    (!defined(MLK_CONFIG_NO_ENCAPS_API) || \+     !defined(MLK_CONFIG_NO_DECAPS_API)) && \+    (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4)+/*yaml+  Name: poly_compress_d11_avx2_asm+  Description: x86_64 AVX2 polynomial compression (d=11)+  Signature: void mlk_poly_compress_d11_avx2_asm(uint8_t *r, const int16_t *a, const uint8_t *data)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 352+      permissions: write-only+      c_parameter: uint8_t *r+      description: Output compressed polynomial+    rsi:+      type: buffer+      size_bytes: 512+      permissions: read-only+      c_parameter: const int16_t *a+      description: Input polynomial (256 x int16_t)+    rdx:+      type: buffer+      size_bytes: 64+      permissions: read-only+      c_parameter: const uint8_t *data+      description: Precomputed compression constants+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_poly_compress_d11_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(poly_compress_d11_avx2_asm)+MLK_ASM_FN_SYMBOL(poly_compress_d11_avx2_asm)++        .cfi_startproc+        movl $0x4ebf4ebf, %eax       # imm = 0x4EBF4EBF+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        vpsllw $0x3, %ymm0, %ymm1+        movl $0x240024, %eax         # imm = 0x240024+        vmovd %eax, %xmm2+        vpbroadcastd %xmm2, %ymm2+        movl $0x20002000, %eax       # imm = 0x20002000+        vmovd %eax, %xmm3+        vpbroadcastd %xmm3, %ymm3+        movl $0x7ff07ff, %eax        # imm = 0x7FF07FF+        vmovd %eax, %xmm4+        vpbroadcastd %xmm4, %ymm4+        movabsq $0x800000108000001, %rax # imm = 0x800000108000001+        vmovq %rax, %xmm5+        vpbroadcastq %xmm5, %ymm5+        movl $0xa, %eax+        vmovq %rax, %xmm6+        vpbroadcastq %xmm6, %ymm6+        vmovdqa (%rdx), %ymm7+        vmovdqa 0x20(%rdx), %ymm8+        vmovdqa (%rsi), %ymm9+        vpmullw %ymm1, %ymm9, %ymm10+        vpaddw %ymm2, %ymm9, %ymm11+        vpsllw $0x3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm9, %ymm9+        vpsubw %ymm11, %ymm10, %ymm11+        vpandn %ymm11, %ymm10, %ymm10+        vpsrlw $0xf, %ymm10, %ymm10+        vpsubw %ymm10, %ymm9, %ymm9+        vpmulhrsw %ymm3, %ymm9, %ymm9+        vpand %ymm4, %ymm9, %ymm9+        vpmaddwd %ymm5, %ymm9, %ymm9+        vpsllvd %ymm6, %ymm9, %ymm9+        vpsrldq $0x8, %ymm9, %ymm10     # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero+        vpsrlvq %ymm7, %ymm9, %ymm9+        vpsllq $0x22, %ymm10, %ymm10+        vpaddq %ymm10, %ymm9, %ymm9+        vpshufb %ymm8, %ymm9, %ymm9+        vextracti128 $0x1, %ymm9, %xmm10+        vpblendvb %xmm8, %xmm10, %xmm9, %xmm9+        vmovdqu %xmm9, (%rdi)+        vmovd %xmm10, 0x10(%rdi)+        vpextrw $0x2, %xmm10, 0x14(%rdi)+        vmovdqa 0x20(%rsi), %ymm9+        vpmullw %ymm1, %ymm9, %ymm10+        vpaddw %ymm2, %ymm9, %ymm11+        vpsllw $0x3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm9, %ymm9+        vpsubw %ymm11, %ymm10, %ymm11+        vpandn %ymm11, %ymm10, %ymm10+        vpsrlw $0xf, %ymm10, %ymm10+        vpsubw %ymm10, %ymm9, %ymm9+        vpmulhrsw %ymm3, %ymm9, %ymm9+        vpand %ymm4, %ymm9, %ymm9+        vpmaddwd %ymm5, %ymm9, %ymm9+        vpsllvd %ymm6, %ymm9, %ymm9+        vpsrldq $0x8, %ymm9, %ymm10     # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero+        vpsrlvq %ymm7, %ymm9, %ymm9+        vpsllq $0x22, %ymm10, %ymm10+        vpaddq %ymm10, %ymm9, %ymm9+        vpshufb %ymm8, %ymm9, %ymm9+        vextracti128 $0x1, %ymm9, %xmm10+        vpblendvb %xmm8, %xmm10, %xmm9, %xmm9+        vmovdqu %xmm9, 0x16(%rdi)+        vmovd %xmm10, 0x26(%rdi)+        vpextrw $0x2, %xmm10, 0x2a(%rdi)+        vmovdqa 0x40(%rsi), %ymm9+        vpmullw %ymm1, %ymm9, %ymm10+        vpaddw %ymm2, %ymm9, %ymm11+        vpsllw $0x3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm9, %ymm9+        vpsubw %ymm11, %ymm10, %ymm11+        vpandn %ymm11, %ymm10, %ymm10+        vpsrlw $0xf, %ymm10, %ymm10+        vpsubw %ymm10, %ymm9, %ymm9+        vpmulhrsw %ymm3, %ymm9, %ymm9+        vpand %ymm4, %ymm9, %ymm9+        vpmaddwd %ymm5, %ymm9, %ymm9+        vpsllvd %ymm6, %ymm9, %ymm9+        vpsrldq $0x8, %ymm9, %ymm10     # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero+        vpsrlvq %ymm7, %ymm9, %ymm9+        vpsllq $0x22, %ymm10, %ymm10+        vpaddq %ymm10, %ymm9, %ymm9+        vpshufb %ymm8, %ymm9, %ymm9+        vextracti128 $0x1, %ymm9, %xmm10+        vpblendvb %xmm8, %xmm10, %xmm9, %xmm9+        vmovdqu %xmm9, 0x2c(%rdi)+        vmovd %xmm10, 0x3c(%rdi)+        vpextrw $0x2, %xmm10, 0x40(%rdi)+        vmovdqa 0x60(%rsi), %ymm9+        vpmullw %ymm1, %ymm9, %ymm10+        vpaddw %ymm2, %ymm9, %ymm11+        vpsllw $0x3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm9, %ymm9+        vpsubw %ymm11, %ymm10, %ymm11+        vpandn %ymm11, %ymm10, %ymm10+        vpsrlw $0xf, %ymm10, %ymm10+        vpsubw %ymm10, %ymm9, %ymm9+        vpmulhrsw %ymm3, %ymm9, %ymm9+        vpand %ymm4, %ymm9, %ymm9+        vpmaddwd %ymm5, %ymm9, %ymm9+        vpsllvd %ymm6, %ymm9, %ymm9+        vpsrldq $0x8, %ymm9, %ymm10     # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero+        vpsrlvq %ymm7, %ymm9, %ymm9+        vpsllq $0x22, %ymm10, %ymm10+        vpaddq %ymm10, %ymm9, %ymm9+        vpshufb %ymm8, %ymm9, %ymm9+        vextracti128 $0x1, %ymm9, %xmm10+        vpblendvb %xmm8, %xmm10, %xmm9, %xmm9+        vmovdqu %xmm9, 0x42(%rdi)+        vmovd %xmm10, 0x52(%rdi)+        vpextrw $0x2, %xmm10, 0x56(%rdi)+        vmovdqa 0x80(%rsi), %ymm9+        vpmullw %ymm1, %ymm9, %ymm10+        vpaddw %ymm2, %ymm9, %ymm11+        vpsllw $0x3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm9, %ymm9+        vpsubw %ymm11, %ymm10, %ymm11+        vpandn %ymm11, %ymm10, %ymm10+        vpsrlw $0xf, %ymm10, %ymm10+        vpsubw %ymm10, %ymm9, %ymm9+        vpmulhrsw %ymm3, %ymm9, %ymm9+        vpand %ymm4, %ymm9, %ymm9+        vpmaddwd %ymm5, %ymm9, %ymm9+        vpsllvd %ymm6, %ymm9, %ymm9+        vpsrldq $0x8, %ymm9, %ymm10     # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero+        vpsrlvq %ymm7, %ymm9, %ymm9+        vpsllq $0x22, %ymm10, %ymm10+        vpaddq %ymm10, %ymm9, %ymm9+        vpshufb %ymm8, %ymm9, %ymm9+        vextracti128 $0x1, %ymm9, %xmm10+        vpblendvb %xmm8, %xmm10, %xmm9, %xmm9+        vmovdqu %xmm9, 0x58(%rdi)+        vmovd %xmm10, 0x68(%rdi)+        vpextrw $0x2, %xmm10, 0x6c(%rdi)+        vmovdqa 0xa0(%rsi), %ymm9+        vpmullw %ymm1, %ymm9, %ymm10+        vpaddw %ymm2, %ymm9, %ymm11+        vpsllw $0x3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm9, %ymm9+        vpsubw %ymm11, %ymm10, %ymm11+        vpandn %ymm11, %ymm10, %ymm10+        vpsrlw $0xf, %ymm10, %ymm10+        vpsubw %ymm10, %ymm9, %ymm9+        vpmulhrsw %ymm3, %ymm9, %ymm9+        vpand %ymm4, %ymm9, %ymm9+        vpmaddwd %ymm5, %ymm9, %ymm9+        vpsllvd %ymm6, %ymm9, %ymm9+        vpsrldq $0x8, %ymm9, %ymm10     # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero+        vpsrlvq %ymm7, %ymm9, %ymm9+        vpsllq $0x22, %ymm10, %ymm10+        vpaddq %ymm10, %ymm9, %ymm9+        vpshufb %ymm8, %ymm9, %ymm9+        vextracti128 $0x1, %ymm9, %xmm10+        vpblendvb %xmm8, %xmm10, %xmm9, %xmm9+        vmovdqu %xmm9, 0x6e(%rdi)+        vmovd %xmm10, 0x7e(%rdi)+        vpextrw $0x2, %xmm10, 0x82(%rdi)+        vmovdqa 0xc0(%rsi), %ymm9+        vpmullw %ymm1, %ymm9, %ymm10+        vpaddw %ymm2, %ymm9, %ymm11+        vpsllw $0x3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm9, %ymm9+        vpsubw %ymm11, %ymm10, %ymm11+        vpandn %ymm11, %ymm10, %ymm10+        vpsrlw $0xf, %ymm10, %ymm10+        vpsubw %ymm10, %ymm9, %ymm9+        vpmulhrsw %ymm3, %ymm9, %ymm9+        vpand %ymm4, %ymm9, %ymm9+        vpmaddwd %ymm5, %ymm9, %ymm9+        vpsllvd %ymm6, %ymm9, %ymm9+        vpsrldq $0x8, %ymm9, %ymm10     # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero+        vpsrlvq %ymm7, %ymm9, %ymm9+        vpsllq $0x22, %ymm10, %ymm10+        vpaddq %ymm10, %ymm9, %ymm9+        vpshufb %ymm8, %ymm9, %ymm9+        vextracti128 $0x1, %ymm9, %xmm10+        vpblendvb %xmm8, %xmm10, %xmm9, %xmm9+        vmovdqu %xmm9, 0x84(%rdi)+        vmovd %xmm10, 0x94(%rdi)+        vpextrw $0x2, %xmm10, 0x98(%rdi)+        vmovdqa 0xe0(%rsi), %ymm9+        vpmullw %ymm1, %ymm9, %ymm10+        vpaddw %ymm2, %ymm9, %ymm11+        vpsllw $0x3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm9, %ymm9+        vpsubw %ymm11, %ymm10, %ymm11+        vpandn %ymm11, %ymm10, %ymm10+        vpsrlw $0xf, %ymm10, %ymm10+        vpsubw %ymm10, %ymm9, %ymm9+        vpmulhrsw %ymm3, %ymm9, %ymm9+        vpand %ymm4, %ymm9, %ymm9+        vpmaddwd %ymm5, %ymm9, %ymm9+        vpsllvd %ymm6, %ymm9, %ymm9+        vpsrldq $0x8, %ymm9, %ymm10     # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero+        vpsrlvq %ymm7, %ymm9, %ymm9+        vpsllq $0x22, %ymm10, %ymm10+        vpaddq %ymm10, %ymm9, %ymm9+        vpshufb %ymm8, %ymm9, %ymm9+        vextracti128 $0x1, %ymm9, %xmm10+        vpblendvb %xmm8, %xmm10, %xmm9, %xmm9+        vmovdqu %xmm9, 0x9a(%rdi)+        vmovd %xmm10, 0xaa(%rdi)+        vpextrw $0x2, %xmm10, 0xae(%rdi)+        vmovdqa 0x100(%rsi), %ymm9+        vpmullw %ymm1, %ymm9, %ymm10+        vpaddw %ymm2, %ymm9, %ymm11+        vpsllw $0x3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm9, %ymm9+        vpsubw %ymm11, %ymm10, %ymm11+        vpandn %ymm11, %ymm10, %ymm10+        vpsrlw $0xf, %ymm10, %ymm10+        vpsubw %ymm10, %ymm9, %ymm9+        vpmulhrsw %ymm3, %ymm9, %ymm9+        vpand %ymm4, %ymm9, %ymm9+        vpmaddwd %ymm5, %ymm9, %ymm9+        vpsllvd %ymm6, %ymm9, %ymm9+        vpsrldq $0x8, %ymm9, %ymm10     # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero+        vpsrlvq %ymm7, %ymm9, %ymm9+        vpsllq $0x22, %ymm10, %ymm10+        vpaddq %ymm10, %ymm9, %ymm9+        vpshufb %ymm8, %ymm9, %ymm9+        vextracti128 $0x1, %ymm9, %xmm10+        vpblendvb %xmm8, %xmm10, %xmm9, %xmm9+        vmovdqu %xmm9, 0xb0(%rdi)+        vmovd %xmm10, 0xc0(%rdi)+        vpextrw $0x2, %xmm10, 0xc4(%rdi)+        vmovdqa 0x120(%rsi), %ymm9+        vpmullw %ymm1, %ymm9, %ymm10+        vpaddw %ymm2, %ymm9, %ymm11+        vpsllw $0x3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm9, %ymm9+        vpsubw %ymm11, %ymm10, %ymm11+        vpandn %ymm11, %ymm10, %ymm10+        vpsrlw $0xf, %ymm10, %ymm10+        vpsubw %ymm10, %ymm9, %ymm9+        vpmulhrsw %ymm3, %ymm9, %ymm9+        vpand %ymm4, %ymm9, %ymm9+        vpmaddwd %ymm5, %ymm9, %ymm9+        vpsllvd %ymm6, %ymm9, %ymm9+        vpsrldq $0x8, %ymm9, %ymm10     # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero+        vpsrlvq %ymm7, %ymm9, %ymm9+        vpsllq $0x22, %ymm10, %ymm10+        vpaddq %ymm10, %ymm9, %ymm9+        vpshufb %ymm8, %ymm9, %ymm9+        vextracti128 $0x1, %ymm9, %xmm10+        vpblendvb %xmm8, %xmm10, %xmm9, %xmm9+        vmovdqu %xmm9, 0xc6(%rdi)+        vmovd %xmm10, 0xd6(%rdi)+        vpextrw $0x2, %xmm10, 0xda(%rdi)+        vmovdqa 0x140(%rsi), %ymm9+        vpmullw %ymm1, %ymm9, %ymm10+        vpaddw %ymm2, %ymm9, %ymm11+        vpsllw $0x3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm9, %ymm9+        vpsubw %ymm11, %ymm10, %ymm11+        vpandn %ymm11, %ymm10, %ymm10+        vpsrlw $0xf, %ymm10, %ymm10+        vpsubw %ymm10, %ymm9, %ymm9+        vpmulhrsw %ymm3, %ymm9, %ymm9+        vpand %ymm4, %ymm9, %ymm9+        vpmaddwd %ymm5, %ymm9, %ymm9+        vpsllvd %ymm6, %ymm9, %ymm9+        vpsrldq $0x8, %ymm9, %ymm10     # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero+        vpsrlvq %ymm7, %ymm9, %ymm9+        vpsllq $0x22, %ymm10, %ymm10+        vpaddq %ymm10, %ymm9, %ymm9+        vpshufb %ymm8, %ymm9, %ymm9+        vextracti128 $0x1, %ymm9, %xmm10+        vpblendvb %xmm8, %xmm10, %xmm9, %xmm9+        vmovdqu %xmm9, 0xdc(%rdi)+        vmovd %xmm10, 0xec(%rdi)+        vpextrw $0x2, %xmm10, 0xf0(%rdi)+        vmovdqa 0x160(%rsi), %ymm9+        vpmullw %ymm1, %ymm9, %ymm10+        vpaddw %ymm2, %ymm9, %ymm11+        vpsllw $0x3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm9, %ymm9+        vpsubw %ymm11, %ymm10, %ymm11+        vpandn %ymm11, %ymm10, %ymm10+        vpsrlw $0xf, %ymm10, %ymm10+        vpsubw %ymm10, %ymm9, %ymm9+        vpmulhrsw %ymm3, %ymm9, %ymm9+        vpand %ymm4, %ymm9, %ymm9+        vpmaddwd %ymm5, %ymm9, %ymm9+        vpsllvd %ymm6, %ymm9, %ymm9+        vpsrldq $0x8, %ymm9, %ymm10     # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero+        vpsrlvq %ymm7, %ymm9, %ymm9+        vpsllq $0x22, %ymm10, %ymm10+        vpaddq %ymm10, %ymm9, %ymm9+        vpshufb %ymm8, %ymm9, %ymm9+        vextracti128 $0x1, %ymm9, %xmm10+        vpblendvb %xmm8, %xmm10, %xmm9, %xmm9+        vmovdqu %xmm9, 0xf2(%rdi)+        vmovd %xmm10, 0x102(%rdi)+        vpextrw $0x2, %xmm10, 0x106(%rdi)+        vmovdqa 0x180(%rsi), %ymm9+        vpmullw %ymm1, %ymm9, %ymm10+        vpaddw %ymm2, %ymm9, %ymm11+        vpsllw $0x3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm9, %ymm9+        vpsubw %ymm11, %ymm10, %ymm11+        vpandn %ymm11, %ymm10, %ymm10+        vpsrlw $0xf, %ymm10, %ymm10+        vpsubw %ymm10, %ymm9, %ymm9+        vpmulhrsw %ymm3, %ymm9, %ymm9+        vpand %ymm4, %ymm9, %ymm9+        vpmaddwd %ymm5, %ymm9, %ymm9+        vpsllvd %ymm6, %ymm9, %ymm9+        vpsrldq $0x8, %ymm9, %ymm10     # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero+        vpsrlvq %ymm7, %ymm9, %ymm9+        vpsllq $0x22, %ymm10, %ymm10+        vpaddq %ymm10, %ymm9, %ymm9+        vpshufb %ymm8, %ymm9, %ymm9+        vextracti128 $0x1, %ymm9, %xmm10+        vpblendvb %xmm8, %xmm10, %xmm9, %xmm9+        vmovdqu %xmm9, 0x108(%rdi)+        vmovd %xmm10, 0x118(%rdi)+        vpextrw $0x2, %xmm10, 0x11c(%rdi)+        vmovdqa 0x1a0(%rsi), %ymm9+        vpmullw %ymm1, %ymm9, %ymm10+        vpaddw %ymm2, %ymm9, %ymm11+        vpsllw $0x3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm9, %ymm9+        vpsubw %ymm11, %ymm10, %ymm11+        vpandn %ymm11, %ymm10, %ymm10+        vpsrlw $0xf, %ymm10, %ymm10+        vpsubw %ymm10, %ymm9, %ymm9+        vpmulhrsw %ymm3, %ymm9, %ymm9+        vpand %ymm4, %ymm9, %ymm9+        vpmaddwd %ymm5, %ymm9, %ymm9+        vpsllvd %ymm6, %ymm9, %ymm9+        vpsrldq $0x8, %ymm9, %ymm10     # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero+        vpsrlvq %ymm7, %ymm9, %ymm9+        vpsllq $0x22, %ymm10, %ymm10+        vpaddq %ymm10, %ymm9, %ymm9+        vpshufb %ymm8, %ymm9, %ymm9+        vextracti128 $0x1, %ymm9, %xmm10+        vpblendvb %xmm8, %xmm10, %xmm9, %xmm9+        vmovdqu %xmm9, 0x11e(%rdi)+        vmovd %xmm10, 0x12e(%rdi)+        vpextrw $0x2, %xmm10, 0x132(%rdi)+        vmovdqa 0x1c0(%rsi), %ymm9+        vpmullw %ymm1, %ymm9, %ymm10+        vpaddw %ymm2, %ymm9, %ymm11+        vpsllw $0x3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm9, %ymm9+        vpsubw %ymm11, %ymm10, %ymm11+        vpandn %ymm11, %ymm10, %ymm10+        vpsrlw $0xf, %ymm10, %ymm10+        vpsubw %ymm10, %ymm9, %ymm9+        vpmulhrsw %ymm3, %ymm9, %ymm9+        vpand %ymm4, %ymm9, %ymm9+        vpmaddwd %ymm5, %ymm9, %ymm9+        vpsllvd %ymm6, %ymm9, %ymm9+        vpsrldq $0x8, %ymm9, %ymm10     # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero+        vpsrlvq %ymm7, %ymm9, %ymm9+        vpsllq $0x22, %ymm10, %ymm10+        vpaddq %ymm10, %ymm9, %ymm9+        vpshufb %ymm8, %ymm9, %ymm9+        vextracti128 $0x1, %ymm9, %xmm10+        vpblendvb %xmm8, %xmm10, %xmm9, %xmm9+        vmovdqu %xmm9, 0x134(%rdi)+        vmovd %xmm10, 0x144(%rdi)+        vpextrw $0x2, %xmm10, 0x148(%rdi)+        vmovdqa 0x1e0(%rsi), %ymm9+        vpmullw %ymm1, %ymm9, %ymm10+        vpaddw %ymm2, %ymm9, %ymm11+        vpsllw $0x3, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm9, %ymm9+        vpsubw %ymm11, %ymm10, %ymm11+        vpandn %ymm11, %ymm10, %ymm10+        vpsrlw $0xf, %ymm10, %ymm10+        vpsubw %ymm10, %ymm9, %ymm9+        vpmulhrsw %ymm3, %ymm9, %ymm9+        vpand %ymm4, %ymm9, %ymm9+        vpmaddwd %ymm5, %ymm9, %ymm9+        vpsllvd %ymm6, %ymm9, %ymm9+        vpsrldq $0x8, %ymm9, %ymm10     # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero+        vpsrlvq %ymm7, %ymm9, %ymm9+        vpsllq $0x22, %ymm10, %ymm10+        vpaddq %ymm10, %ymm9, %ymm9+        vpshufb %ymm8, %ymm9, %ymm9+        vextracti128 $0x1, %ymm9, %xmm10+        vpblendvb %xmm8, %xmm10, %xmm9, %xmm9+        vmovdqu %xmm9, 0x14a(%rdi)+        vmovd %xmm10, 0x15a(%rdi)+        vpextrw $0x2, %xmm10, 0x15e(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(poly_compress_d11_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+          && (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) && \+          (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_poly_compress_d4_avx2_asm.S view
@@ -0,0 +1,194 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Kyber optimized AVX2 implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Kyber implementation @[REF_AVX2].+ */++/*************************************************+ * Name:        mlk_poly_compress_d4_avx2_asm+ *+ * Description: Compression of a polynomial to 4 bits per coefficient.+ *+ * Arguments:   - uint8_t *r:       pointer to output byte array+ *                                  (of length MLKEM_POLYCOMPRESSEDBYTES_D4)+ *              - const int16_t *a: pointer to input polynomial+ *              - const uint8_t *data: pointer to permdidx constant+ **************************************************/++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    (!defined(MLK_CONFIG_NO_ENCAPS_API) || \+     !defined(MLK_CONFIG_NO_DECAPS_API)) && \+    (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2 || MLKEM_K == 3)+/*yaml+  Name: poly_compress_d4_avx2_asm+  Description: x86_64 AVX2 polynomial compression (d=4)+  Signature: void mlk_poly_compress_d4_avx2_asm(uint8_t *r, const int16_t *a, const uint8_t *data)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 128+      permissions: write-only+      c_parameter: uint8_t *r+      description: Output compressed polynomial+    rsi:+      type: buffer+      size_bytes: 512+      permissions: read-only+      c_parameter: const int16_t *a+      description: Input polynomial (256 x int16_t)+    rdx:+      type: buffer+      size_bytes: 32+      permissions: read-only+      c_parameter: const uint8_t *data+      description: Precomputed compression constants+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_poly_compress_d4_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(poly_compress_d4_avx2_asm)+MLK_ASM_FN_SYMBOL(poly_compress_d4_avx2_asm)++        .cfi_startproc+        movl $0x4ebf4ebf, %eax       # imm = 0x4EBF4EBF+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        movl $0x2000200, %eax        # imm = 0x2000200+        vmovd %eax, %xmm1+        vpbroadcastd %xmm1, %ymm1+        movl $0xf000f, %eax          # imm = 0xF000F+        vmovd %eax, %xmm2+        vpbroadcastd %xmm2, %ymm2+        movl $0x10011001, %eax       # imm = 0x10011001+        vmovd %eax, %xmm3+        vpbroadcastd %xmm3, %ymm3+        vmovdqa (%rdx), %ymm4+        vmovdqa (%rsi), %ymm5+        vmovdqa 0x20(%rsi), %ymm6+        vmovdqa 0x40(%rsi), %ymm7+        vmovdqa 0x60(%rsi), %ymm8+        vpmulhw %ymm0, %ymm5, %ymm5+        vpmulhw %ymm0, %ymm6, %ymm6+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhrsw %ymm1, %ymm5, %ymm5+        vpmulhrsw %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm1, %ymm7, %ymm7+        vpmulhrsw %ymm1, %ymm8, %ymm8+        vpand %ymm2, %ymm5, %ymm5+        vpand %ymm2, %ymm6, %ymm6+        vpand %ymm2, %ymm7, %ymm7+        vpand %ymm2, %ymm8, %ymm8+        vpackuswb %ymm6, %ymm5, %ymm5+        vpackuswb %ymm8, %ymm7, %ymm7+        vpmaddubsw %ymm3, %ymm5, %ymm5+        vpmaddubsw %ymm3, %ymm7, %ymm7+        vpackuswb %ymm7, %ymm5, %ymm5+        vpermd %ymm5, %ymm4, %ymm5+        vmovdqu %ymm5, (%rdi)+        vmovdqa 0x80(%rsi), %ymm5+        vmovdqa 0xa0(%rsi), %ymm6+        vmovdqa 0xc0(%rsi), %ymm7+        vmovdqa 0xe0(%rsi), %ymm8+        vpmulhw %ymm0, %ymm5, %ymm5+        vpmulhw %ymm0, %ymm6, %ymm6+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhrsw %ymm1, %ymm5, %ymm5+        vpmulhrsw %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm1, %ymm7, %ymm7+        vpmulhrsw %ymm1, %ymm8, %ymm8+        vpand %ymm2, %ymm5, %ymm5+        vpand %ymm2, %ymm6, %ymm6+        vpand %ymm2, %ymm7, %ymm7+        vpand %ymm2, %ymm8, %ymm8+        vpackuswb %ymm6, %ymm5, %ymm5+        vpackuswb %ymm8, %ymm7, %ymm7+        vpmaddubsw %ymm3, %ymm5, %ymm5+        vpmaddubsw %ymm3, %ymm7, %ymm7+        vpackuswb %ymm7, %ymm5, %ymm5+        vpermd %ymm5, %ymm4, %ymm5+        vmovdqu %ymm5, 0x20(%rdi)+        vmovdqa 0x100(%rsi), %ymm5+        vmovdqa 0x120(%rsi), %ymm6+        vmovdqa 0x140(%rsi), %ymm7+        vmovdqa 0x160(%rsi), %ymm8+        vpmulhw %ymm0, %ymm5, %ymm5+        vpmulhw %ymm0, %ymm6, %ymm6+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhrsw %ymm1, %ymm5, %ymm5+        vpmulhrsw %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm1, %ymm7, %ymm7+        vpmulhrsw %ymm1, %ymm8, %ymm8+        vpand %ymm2, %ymm5, %ymm5+        vpand %ymm2, %ymm6, %ymm6+        vpand %ymm2, %ymm7, %ymm7+        vpand %ymm2, %ymm8, %ymm8+        vpackuswb %ymm6, %ymm5, %ymm5+        vpackuswb %ymm8, %ymm7, %ymm7+        vpmaddubsw %ymm3, %ymm5, %ymm5+        vpmaddubsw %ymm3, %ymm7, %ymm7+        vpackuswb %ymm7, %ymm5, %ymm5+        vpermd %ymm5, %ymm4, %ymm5+        vmovdqu %ymm5, 0x40(%rdi)+        vmovdqa 0x180(%rsi), %ymm5+        vmovdqa 0x1a0(%rsi), %ymm6+        vmovdqa 0x1c0(%rsi), %ymm7+        vmovdqa 0x1e0(%rsi), %ymm8+        vpmulhw %ymm0, %ymm5, %ymm5+        vpmulhw %ymm0, %ymm6, %ymm6+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhrsw %ymm1, %ymm5, %ymm5+        vpmulhrsw %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm1, %ymm7, %ymm7+        vpmulhrsw %ymm1, %ymm8, %ymm8+        vpand %ymm2, %ymm5, %ymm5+        vpand %ymm2, %ymm6, %ymm6+        vpand %ymm2, %ymm7, %ymm7+        vpand %ymm2, %ymm8, %ymm8+        vpackuswb %ymm6, %ymm5, %ymm5+        vpackuswb %ymm8, %ymm7, %ymm7+        vpmaddubsw %ymm3, %ymm5, %ymm5+        vpmaddubsw %ymm3, %ymm7, %ymm7+        vpackuswb %ymm7, %ymm5, %ymm5+        vpermd %ymm5, %ymm4, %ymm5+        vmovdqu %ymm5, 0x60(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(poly_compress_d4_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+          && (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) && \+          (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 || MLKEM_K == 3) \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_poly_compress_d5_avx2_asm.S view
@@ -0,0 +1,251 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Kyber optimized AVX2 implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Kyber implementation @[REF_AVX2].+ */++/*************************************************+ * Name:        mlk_poly_compress_d5_avx2_asm+ *+ * Description: Compression of a polynomial to 5 bits per coefficient.+ *+ * Arguments:   - uint8_t *r:       pointer to output byte array+ *                                  (of length MLKEM_POLYCOMPRESSEDBYTES_D5)+ *              - const int16_t *a: pointer to input polynomial+ *              - const uint8_t *data: pointer to shufbidx constant+ **************************************************/++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    (!defined(MLK_CONFIG_NO_ENCAPS_API) || \+     !defined(MLK_CONFIG_NO_DECAPS_API)) && \+    (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4)+/*yaml+  Name: poly_compress_d5_avx2_asm+  Description: x86_64 AVX2 polynomial compression (d=5)+  Signature: void mlk_poly_compress_d5_avx2_asm(uint8_t *r, const int16_t *a, const uint8_t *data)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 160+      permissions: write-only+      c_parameter: uint8_t *r+      description: Output compressed polynomial+    rsi:+      type: buffer+      size_bytes: 512+      permissions: read-only+      c_parameter: const int16_t *a+      description: Input polynomial (256 x int16_t)+    rdx:+      type: buffer+      size_bytes: 32+      permissions: read-only+      c_parameter: const uint8_t *data+      description: Precomputed compression constants+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_poly_compress_d5_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(poly_compress_d5_avx2_asm)+MLK_ASM_FN_SYMBOL(poly_compress_d5_avx2_asm)++        .cfi_startproc+        movl $0x4ebf4ebf, %eax       # imm = 0x4EBF4EBF+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        movl $0x4000400, %eax        # imm = 0x4000400+        vmovd %eax, %xmm1+        vpbroadcastd %xmm1, %ymm1+        movl $0x1f001f, %eax         # imm = 0x1F001F+        vmovd %eax, %xmm2+        vpbroadcastd %xmm2, %ymm2+        movl $0x20012001, %eax       # imm = 0x20012001+        vmovd %eax, %xmm3+        vpbroadcastd %xmm3, %ymm3+        movl $0x4000001, %eax        # imm = 0x4000001+        vmovd %eax, %xmm4+        vpbroadcastd %xmm4, %ymm4+        movl $0xc, %eax+        vmovq %rax, %xmm5+        vpbroadcastq %xmm5, %ymm5+        vmovdqa (%rdx), %ymm6+        vmovdqa (%rsi), %ymm7+        vmovdqa 0x20(%rsi), %ymm8+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhrsw %ymm1, %ymm7, %ymm7+        vpmulhrsw %ymm1, %ymm8, %ymm8+        vpand %ymm2, %ymm7, %ymm7+        vpand %ymm2, %ymm8, %ymm8+        vpackuswb %ymm8, %ymm7, %ymm7+        vpmaddubsw %ymm3, %ymm7, %ymm7+        vpmaddwd %ymm4, %ymm7, %ymm7+        vpsllvd %ymm5, %ymm7, %ymm7+        vpsrlvq %ymm5, %ymm7, %ymm7+        vpshufb %ymm6, %ymm7, %ymm7+        vextracti128 $0x1, %ymm7, %xmm8+        vpblendvb %xmm6, %xmm8, %xmm7, %xmm7+        vmovdqu %xmm7, (%rdi)+        vmovd %xmm8, 0x10(%rdi)+        vmovdqa 0x40(%rsi), %ymm7+        vmovdqa 0x60(%rsi), %ymm8+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhrsw %ymm1, %ymm7, %ymm7+        vpmulhrsw %ymm1, %ymm8, %ymm8+        vpand %ymm2, %ymm7, %ymm7+        vpand %ymm2, %ymm8, %ymm8+        vpackuswb %ymm8, %ymm7, %ymm7+        vpmaddubsw %ymm3, %ymm7, %ymm7+        vpmaddwd %ymm4, %ymm7, %ymm7+        vpsllvd %ymm5, %ymm7, %ymm7+        vpsrlvq %ymm5, %ymm7, %ymm7+        vpshufb %ymm6, %ymm7, %ymm7+        vextracti128 $0x1, %ymm7, %xmm8+        vpblendvb %xmm6, %xmm8, %xmm7, %xmm7+        vmovdqu %xmm7, 0x14(%rdi)+        vmovd %xmm8, 0x24(%rdi)+        vmovdqa 0x80(%rsi), %ymm7+        vmovdqa 0xa0(%rsi), %ymm8+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhrsw %ymm1, %ymm7, %ymm7+        vpmulhrsw %ymm1, %ymm8, %ymm8+        vpand %ymm2, %ymm7, %ymm7+        vpand %ymm2, %ymm8, %ymm8+        vpackuswb %ymm8, %ymm7, %ymm7+        vpmaddubsw %ymm3, %ymm7, %ymm7+        vpmaddwd %ymm4, %ymm7, %ymm7+        vpsllvd %ymm5, %ymm7, %ymm7+        vpsrlvq %ymm5, %ymm7, %ymm7+        vpshufb %ymm6, %ymm7, %ymm7+        vextracti128 $0x1, %ymm7, %xmm8+        vpblendvb %xmm6, %xmm8, %xmm7, %xmm7+        vmovdqu %xmm7, 0x28(%rdi)+        vmovd %xmm8, 0x38(%rdi)+        vmovdqa 0xc0(%rsi), %ymm7+        vmovdqa 0xe0(%rsi), %ymm8+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhrsw %ymm1, %ymm7, %ymm7+        vpmulhrsw %ymm1, %ymm8, %ymm8+        vpand %ymm2, %ymm7, %ymm7+        vpand %ymm2, %ymm8, %ymm8+        vpackuswb %ymm8, %ymm7, %ymm7+        vpmaddubsw %ymm3, %ymm7, %ymm7+        vpmaddwd %ymm4, %ymm7, %ymm7+        vpsllvd %ymm5, %ymm7, %ymm7+        vpsrlvq %ymm5, %ymm7, %ymm7+        vpshufb %ymm6, %ymm7, %ymm7+        vextracti128 $0x1, %ymm7, %xmm8+        vpblendvb %xmm6, %xmm8, %xmm7, %xmm7+        vmovdqu %xmm7, 0x3c(%rdi)+        vmovd %xmm8, 0x4c(%rdi)+        vmovdqa 0x100(%rsi), %ymm7+        vmovdqa 0x120(%rsi), %ymm8+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhrsw %ymm1, %ymm7, %ymm7+        vpmulhrsw %ymm1, %ymm8, %ymm8+        vpand %ymm2, %ymm7, %ymm7+        vpand %ymm2, %ymm8, %ymm8+        vpackuswb %ymm8, %ymm7, %ymm7+        vpmaddubsw %ymm3, %ymm7, %ymm7+        vpmaddwd %ymm4, %ymm7, %ymm7+        vpsllvd %ymm5, %ymm7, %ymm7+        vpsrlvq %ymm5, %ymm7, %ymm7+        vpshufb %ymm6, %ymm7, %ymm7+        vextracti128 $0x1, %ymm7, %xmm8+        vpblendvb %xmm6, %xmm8, %xmm7, %xmm7+        vmovdqu %xmm7, 0x50(%rdi)+        vmovd %xmm8, 0x60(%rdi)+        vmovdqa 0x140(%rsi), %ymm7+        vmovdqa 0x160(%rsi), %ymm8+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhrsw %ymm1, %ymm7, %ymm7+        vpmulhrsw %ymm1, %ymm8, %ymm8+        vpand %ymm2, %ymm7, %ymm7+        vpand %ymm2, %ymm8, %ymm8+        vpackuswb %ymm8, %ymm7, %ymm7+        vpmaddubsw %ymm3, %ymm7, %ymm7+        vpmaddwd %ymm4, %ymm7, %ymm7+        vpsllvd %ymm5, %ymm7, %ymm7+        vpsrlvq %ymm5, %ymm7, %ymm7+        vpshufb %ymm6, %ymm7, %ymm7+        vextracti128 $0x1, %ymm7, %xmm8+        vpblendvb %xmm6, %xmm8, %xmm7, %xmm7+        vmovdqu %xmm7, 0x64(%rdi)+        vmovd %xmm8, 0x74(%rdi)+        vmovdqa 0x180(%rsi), %ymm7+        vmovdqa 0x1a0(%rsi), %ymm8+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhrsw %ymm1, %ymm7, %ymm7+        vpmulhrsw %ymm1, %ymm8, %ymm8+        vpand %ymm2, %ymm7, %ymm7+        vpand %ymm2, %ymm8, %ymm8+        vpackuswb %ymm8, %ymm7, %ymm7+        vpmaddubsw %ymm3, %ymm7, %ymm7+        vpmaddwd %ymm4, %ymm7, %ymm7+        vpsllvd %ymm5, %ymm7, %ymm7+        vpsrlvq %ymm5, %ymm7, %ymm7+        vpshufb %ymm6, %ymm7, %ymm7+        vextracti128 $0x1, %ymm7, %xmm8+        vpblendvb %xmm6, %xmm8, %xmm7, %xmm7+        vmovdqu %xmm7, 0x78(%rdi)+        vmovd %xmm8, 0x88(%rdi)+        vmovdqa 0x1c0(%rsi), %ymm7+        vmovdqa 0x1e0(%rsi), %ymm8+        vpmulhw %ymm0, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm8, %ymm8+        vpmulhrsw %ymm1, %ymm7, %ymm7+        vpmulhrsw %ymm1, %ymm8, %ymm8+        vpand %ymm2, %ymm7, %ymm7+        vpand %ymm2, %ymm8, %ymm8+        vpackuswb %ymm8, %ymm7, %ymm7+        vpmaddubsw %ymm3, %ymm7, %ymm7+        vpmaddwd %ymm4, %ymm7, %ymm7+        vpsllvd %ymm5, %ymm7, %ymm7+        vpsrlvq %ymm5, %ymm7, %ymm7+        vpshufb %ymm6, %ymm7, %ymm7+        vextracti128 $0x1, %ymm7, %xmm8+        vpblendvb %xmm6, %xmm8, %xmm7, %xmm7+        vmovdqu %xmm7, 0x8c(%rdi)+        vmovd %xmm8, 0x9c(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(poly_compress_d5_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+          && (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) && \+          (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_poly_decompress_d10_avx2_asm.S view
@@ -0,0 +1,257 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Kyber optimized AVX2 implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Kyber implementation @[REF_AVX2].+ */++/*************************************************+ * Name:        mlk_poly_decompress_d10_avx2_asm+ *+ * Description: Decompression of a polynomial from 10 bits per coefficient.+ *+ * Arguments:   - int16_t *r:       pointer to output polynomial+ *              - const uint8_t *a: pointer to input byte array+ *                                  (of length MLKEM_POLYCOMPRESSEDBYTES_D10)+ *              - const uint8_t *data: pointer to shufbidx constant+ **************************************************/++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    !defined(MLK_CONFIG_NO_DECAPS_API) && \+    (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2 || MLKEM_K == 3)+/*yaml+  Name: poly_decompress_d10_avx2_asm+  Description: x86_64 AVX2 polynomial decompression (d=10)+  Signature: void mlk_poly_decompress_d10_avx2_asm(int16_t *r, const uint8_t *a, const uint8_t *data)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 512+      permissions: write-only+      c_parameter: int16_t *r+      description: Output polynomial (256 x int16_t)+    rsi:+      type: buffer+      size_bytes: 320+      permissions: read-only+      c_parameter: const uint8_t *a+      description: Input compressed polynomial+    rdx:+      type: buffer+      size_bytes: 32+      permissions: read-only+      c_parameter: const uint8_t *data+      description: Precomputed decompression constants+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_poly_decompress_d10_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(poly_decompress_d10_avx2_asm)+MLK_ASM_FN_SYMBOL(poly_decompress_d10_avx2_asm)++        .cfi_startproc+        movl $0xd013404, %eax        # imm = 0xD013404+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        movl $0x4, %eax+        vmovq %rax, %xmm1+        vpbroadcastq %xmm1, %ymm1+        movl $0x7fe01ff8, %eax       # imm = 0x7FE01FF8+        vmovd %eax, %xmm2+        vpbroadcastd %xmm2, %ymm2+        vmovdqa (%rdx), %ymm3+        vmovdqu (%rsi), %xmm4+        vmovd 0x10(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm4, %ymm4+        vpermq $0x94, %ymm4, %ymm4     # ymm4 = ymm4[0,1,1,2]+        vpshufb %ymm3, %ymm4, %ymm4+        vpsllvd %ymm1, %ymm4, %ymm4+        vpsrlw $0x1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, (%rdi)+        vmovdqu 0x14(%rsi), %xmm4+        vmovd 0x24(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm4, %ymm4+        vpermq $0x94, %ymm4, %ymm4     # ymm4 = ymm4[0,1,1,2]+        vpshufb %ymm3, %ymm4, %ymm4+        vpsllvd %ymm1, %ymm4, %ymm4+        vpsrlw $0x1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x20(%rdi)+        vmovdqu 0x28(%rsi), %xmm4+        vmovd 0x38(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm4, %ymm4+        vpermq $0x94, %ymm4, %ymm4     # ymm4 = ymm4[0,1,1,2]+        vpshufb %ymm3, %ymm4, %ymm4+        vpsllvd %ymm1, %ymm4, %ymm4+        vpsrlw $0x1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x40(%rdi)+        vmovdqu 0x3c(%rsi), %xmm4+        vmovd 0x4c(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm4, %ymm4+        vpermq $0x94, %ymm4, %ymm4     # ymm4 = ymm4[0,1,1,2]+        vpshufb %ymm3, %ymm4, %ymm4+        vpsllvd %ymm1, %ymm4, %ymm4+        vpsrlw $0x1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x60(%rdi)+        vmovdqu 0x50(%rsi), %xmm4+        vmovd 0x60(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm4, %ymm4+        vpermq $0x94, %ymm4, %ymm4     # ymm4 = ymm4[0,1,1,2]+        vpshufb %ymm3, %ymm4, %ymm4+        vpsllvd %ymm1, %ymm4, %ymm4+        vpsrlw $0x1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x80(%rdi)+        vmovdqu 0x64(%rsi), %xmm4+        vmovd 0x74(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm4, %ymm4+        vpermq $0x94, %ymm4, %ymm4     # ymm4 = ymm4[0,1,1,2]+        vpshufb %ymm3, %ymm4, %ymm4+        vpsllvd %ymm1, %ymm4, %ymm4+        vpsrlw $0x1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0xa0(%rdi)+        vmovdqu 0x78(%rsi), %xmm4+        vmovd 0x88(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm4, %ymm4+        vpermq $0x94, %ymm4, %ymm4     # ymm4 = ymm4[0,1,1,2]+        vpshufb %ymm3, %ymm4, %ymm4+        vpsllvd %ymm1, %ymm4, %ymm4+        vpsrlw $0x1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0xc0(%rdi)+        vmovdqu 0x8c(%rsi), %xmm4+        vmovd 0x9c(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm4, %ymm4+        vpermq $0x94, %ymm4, %ymm4     # ymm4 = ymm4[0,1,1,2]+        vpshufb %ymm3, %ymm4, %ymm4+        vpsllvd %ymm1, %ymm4, %ymm4+        vpsrlw $0x1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0xe0(%rdi)+        vmovdqu 0xa0(%rsi), %xmm4+        vmovd 0xb0(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm4, %ymm4+        vpermq $0x94, %ymm4, %ymm4     # ymm4 = ymm4[0,1,1,2]+        vpshufb %ymm3, %ymm4, %ymm4+        vpsllvd %ymm1, %ymm4, %ymm4+        vpsrlw $0x1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x100(%rdi)+        vmovdqu 0xb4(%rsi), %xmm4+        vmovd 0xc4(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm4, %ymm4+        vpermq $0x94, %ymm4, %ymm4     # ymm4 = ymm4[0,1,1,2]+        vpshufb %ymm3, %ymm4, %ymm4+        vpsllvd %ymm1, %ymm4, %ymm4+        vpsrlw $0x1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x120(%rdi)+        vmovdqu 0xc8(%rsi), %xmm4+        vmovd 0xd8(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm4, %ymm4+        vpermq $0x94, %ymm4, %ymm4     # ymm4 = ymm4[0,1,1,2]+        vpshufb %ymm3, %ymm4, %ymm4+        vpsllvd %ymm1, %ymm4, %ymm4+        vpsrlw $0x1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x140(%rdi)+        vmovdqu 0xdc(%rsi), %xmm4+        vmovd 0xec(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm4, %ymm4+        vpermq $0x94, %ymm4, %ymm4     # ymm4 = ymm4[0,1,1,2]+        vpshufb %ymm3, %ymm4, %ymm4+        vpsllvd %ymm1, %ymm4, %ymm4+        vpsrlw $0x1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x160(%rdi)+        vmovdqu 0xf0(%rsi), %xmm4+        vmovd 0x100(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm4, %ymm4+        vpermq $0x94, %ymm4, %ymm4     # ymm4 = ymm4[0,1,1,2]+        vpshufb %ymm3, %ymm4, %ymm4+        vpsllvd %ymm1, %ymm4, %ymm4+        vpsrlw $0x1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x180(%rdi)+        vmovdqu 0x104(%rsi), %xmm4+        vmovd 0x114(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm4, %ymm4+        vpermq $0x94, %ymm4, %ymm4     # ymm4 = ymm4[0,1,1,2]+        vpshufb %ymm3, %ymm4, %ymm4+        vpsllvd %ymm1, %ymm4, %ymm4+        vpsrlw $0x1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x1a0(%rdi)+        vmovdqu 0x118(%rsi), %xmm4+        vmovd 0x128(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm4, %ymm4+        vpermq $0x94, %ymm4, %ymm4     # ymm4 = ymm4[0,1,1,2]+        vpshufb %ymm3, %ymm4, %ymm4+        vpsllvd %ymm1, %ymm4, %ymm4+        vpsrlw $0x1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x1c0(%rdi)+        vmovdqu 0x12c(%rsi), %xmm4+        vmovd 0x13c(%rsi), %xmm5+        vinserti128 $0x1, %xmm5, %ymm4, %ymm4+        vpermq $0x94, %ymm4, %ymm4     # ymm4 = ymm4[0,1,1,2]+        vpshufb %ymm3, %ymm4, %ymm4+        vpsllvd %ymm1, %ymm4, %ymm4+        vpsrlw $0x1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x1e0(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(poly_decompress_d10_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+          && !MLK_CONFIG_NO_DECAPS_API && (MLK_CONFIG_MULTILEVEL_WITH_SHARED || \+          MLKEM_K == 2 || MLKEM_K == 3) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_poly_decompress_d11_avx2_asm.S view
@@ -0,0 +1,307 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Kyber optimized AVX2 implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Kyber implementation @[REF_AVX2].+ */++/*************************************************+ * Name:        mlk_poly_decompress_d11_avx2_asm+ *+ * Description: Decompression of a polynomial from 11 bits per coefficient.+ *+ * Arguments:   - int16_t *r:       pointer to output polynomial+ *              - const uint8_t *a: pointer to input byte array+ *                                  (of length MLKEM_POLYCOMPRESSEDBYTES_D11)+ *              - const uint8_t *data: pointer to constants+ *                                  (shufbidx[0:32], srlvdidx[32:64],+ *                                   srlvqidx[64:96], shift[96:128])+ **************************************************/++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    !defined(MLK_CONFIG_NO_DECAPS_API) && \+    (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4)+/*yaml+  Name: poly_decompress_d11_avx2_asm+  Description: x86_64 AVX2 polynomial decompression (d=11)+  Signature: void mlk_poly_decompress_d11_avx2_asm(int16_t *r, const uint8_t *a, const uint8_t *data)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 512+      permissions: write-only+      c_parameter: int16_t *r+      description: Output polynomial (256 x int16_t)+    rsi:+      type: buffer+      size_bytes: 352+      permissions: read-only+      c_parameter: const uint8_t *a+      description: Input compressed polynomial+    rdx:+      type: buffer+      size_bytes: 128+      permissions: read-only+      c_parameter: const uint8_t *data+      description: Precomputed decompression constants+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_poly_decompress_d11_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(poly_decompress_d11_avx2_asm)+MLK_ASM_FN_SYMBOL(poly_decompress_d11_avx2_asm)++        .cfi_startproc+        movl $0xd010d01, %eax        # imm = 0xD010D01+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        movl $0x7ff07ff0, %eax       # imm = 0x7FF07FF0+        vmovd %eax, %xmm1+        vpbroadcastd %xmm1, %ymm1+        vmovdqa (%rdx), %ymm2+        vmovdqa 0x20(%rdx), %ymm3+        vmovdqa 0x40(%rdx), %ymm4+        vmovdqa 0x60(%rdx), %ymm5+        vmovdqu (%rsi), %xmm6+        vmovd 0x10(%rsi), %xmm7+        vpinsrw $0x2, 0x14(%rsi), %xmm7, %xmm7+        vinserti128 $0x1, %xmm7, %ymm6, %ymm6+        vpermq $0x94, %ymm6, %ymm6     # ymm6 = ymm6[0,1,1,2]+        vpshufb %ymm2, %ymm6, %ymm6+        vpsrlvd %ymm3, %ymm6, %ymm6+        vpsrlvq %ymm4, %ymm6, %ymm6+        vpmullw %ymm5, %ymm6, %ymm6+        vpsrlw $0x1, %ymm6, %ymm6+        vpand %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm0, %ymm6, %ymm6+        vmovdqu %ymm6, (%rdi)+        vmovdqu 0x16(%rsi), %xmm6+        vmovd 0x26(%rsi), %xmm7+        vpinsrw $0x2, 0x2a(%rsi), %xmm7, %xmm7+        vinserti128 $0x1, %xmm7, %ymm6, %ymm6+        vpermq $0x94, %ymm6, %ymm6     # ymm6 = ymm6[0,1,1,2]+        vpshufb %ymm2, %ymm6, %ymm6+        vpsrlvd %ymm3, %ymm6, %ymm6+        vpsrlvq %ymm4, %ymm6, %ymm6+        vpmullw %ymm5, %ymm6, %ymm6+        vpsrlw $0x1, %ymm6, %ymm6+        vpand %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm0, %ymm6, %ymm6+        vmovdqu %ymm6, 0x20(%rdi)+        vmovdqu 0x2c(%rsi), %xmm6+        vmovd 0x3c(%rsi), %xmm7+        vpinsrw $0x2, 0x40(%rsi), %xmm7, %xmm7+        vinserti128 $0x1, %xmm7, %ymm6, %ymm6+        vpermq $0x94, %ymm6, %ymm6     # ymm6 = ymm6[0,1,1,2]+        vpshufb %ymm2, %ymm6, %ymm6+        vpsrlvd %ymm3, %ymm6, %ymm6+        vpsrlvq %ymm4, %ymm6, %ymm6+        vpmullw %ymm5, %ymm6, %ymm6+        vpsrlw $0x1, %ymm6, %ymm6+        vpand %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm0, %ymm6, %ymm6+        vmovdqu %ymm6, 0x40(%rdi)+        vmovdqu 0x42(%rsi), %xmm6+        vmovd 0x52(%rsi), %xmm7+        vpinsrw $0x2, 0x56(%rsi), %xmm7, %xmm7+        vinserti128 $0x1, %xmm7, %ymm6, %ymm6+        vpermq $0x94, %ymm6, %ymm6     # ymm6 = ymm6[0,1,1,2]+        vpshufb %ymm2, %ymm6, %ymm6+        vpsrlvd %ymm3, %ymm6, %ymm6+        vpsrlvq %ymm4, %ymm6, %ymm6+        vpmullw %ymm5, %ymm6, %ymm6+        vpsrlw $0x1, %ymm6, %ymm6+        vpand %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm0, %ymm6, %ymm6+        vmovdqu %ymm6, 0x60(%rdi)+        vmovdqu 0x58(%rsi), %xmm6+        vmovd 0x68(%rsi), %xmm7+        vpinsrw $0x2, 0x6c(%rsi), %xmm7, %xmm7+        vinserti128 $0x1, %xmm7, %ymm6, %ymm6+        vpermq $0x94, %ymm6, %ymm6     # ymm6 = ymm6[0,1,1,2]+        vpshufb %ymm2, %ymm6, %ymm6+        vpsrlvd %ymm3, %ymm6, %ymm6+        vpsrlvq %ymm4, %ymm6, %ymm6+        vpmullw %ymm5, %ymm6, %ymm6+        vpsrlw $0x1, %ymm6, %ymm6+        vpand %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm0, %ymm6, %ymm6+        vmovdqu %ymm6, 0x80(%rdi)+        vmovdqu 0x6e(%rsi), %xmm6+        vmovd 0x7e(%rsi), %xmm7+        vpinsrw $0x2, 0x82(%rsi), %xmm7, %xmm7+        vinserti128 $0x1, %xmm7, %ymm6, %ymm6+        vpermq $0x94, %ymm6, %ymm6     # ymm6 = ymm6[0,1,1,2]+        vpshufb %ymm2, %ymm6, %ymm6+        vpsrlvd %ymm3, %ymm6, %ymm6+        vpsrlvq %ymm4, %ymm6, %ymm6+        vpmullw %ymm5, %ymm6, %ymm6+        vpsrlw $0x1, %ymm6, %ymm6+        vpand %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm0, %ymm6, %ymm6+        vmovdqu %ymm6, 0xa0(%rdi)+        vmovdqu 0x84(%rsi), %xmm6+        vmovd 0x94(%rsi), %xmm7+        vpinsrw $0x2, 0x98(%rsi), %xmm7, %xmm7+        vinserti128 $0x1, %xmm7, %ymm6, %ymm6+        vpermq $0x94, %ymm6, %ymm6     # ymm6 = ymm6[0,1,1,2]+        vpshufb %ymm2, %ymm6, %ymm6+        vpsrlvd %ymm3, %ymm6, %ymm6+        vpsrlvq %ymm4, %ymm6, %ymm6+        vpmullw %ymm5, %ymm6, %ymm6+        vpsrlw $0x1, %ymm6, %ymm6+        vpand %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm0, %ymm6, %ymm6+        vmovdqu %ymm6, 0xc0(%rdi)+        vmovdqu 0x9a(%rsi), %xmm6+        vmovd 0xaa(%rsi), %xmm7+        vpinsrw $0x2, 0xae(%rsi), %xmm7, %xmm7+        vinserti128 $0x1, %xmm7, %ymm6, %ymm6+        vpermq $0x94, %ymm6, %ymm6     # ymm6 = ymm6[0,1,1,2]+        vpshufb %ymm2, %ymm6, %ymm6+        vpsrlvd %ymm3, %ymm6, %ymm6+        vpsrlvq %ymm4, %ymm6, %ymm6+        vpmullw %ymm5, %ymm6, %ymm6+        vpsrlw $0x1, %ymm6, %ymm6+        vpand %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm0, %ymm6, %ymm6+        vmovdqu %ymm6, 0xe0(%rdi)+        vmovdqu 0xb0(%rsi), %xmm6+        vmovd 0xc0(%rsi), %xmm7+        vpinsrw $0x2, 0xc4(%rsi), %xmm7, %xmm7+        vinserti128 $0x1, %xmm7, %ymm6, %ymm6+        vpermq $0x94, %ymm6, %ymm6     # ymm6 = ymm6[0,1,1,2]+        vpshufb %ymm2, %ymm6, %ymm6+        vpsrlvd %ymm3, %ymm6, %ymm6+        vpsrlvq %ymm4, %ymm6, %ymm6+        vpmullw %ymm5, %ymm6, %ymm6+        vpsrlw $0x1, %ymm6, %ymm6+        vpand %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm0, %ymm6, %ymm6+        vmovdqu %ymm6, 0x100(%rdi)+        vmovdqu 0xc6(%rsi), %xmm6+        vmovd 0xd6(%rsi), %xmm7+        vpinsrw $0x2, 0xda(%rsi), %xmm7, %xmm7+        vinserti128 $0x1, %xmm7, %ymm6, %ymm6+        vpermq $0x94, %ymm6, %ymm6     # ymm6 = ymm6[0,1,1,2]+        vpshufb %ymm2, %ymm6, %ymm6+        vpsrlvd %ymm3, %ymm6, %ymm6+        vpsrlvq %ymm4, %ymm6, %ymm6+        vpmullw %ymm5, %ymm6, %ymm6+        vpsrlw $0x1, %ymm6, %ymm6+        vpand %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm0, %ymm6, %ymm6+        vmovdqu %ymm6, 0x120(%rdi)+        vmovdqu 0xdc(%rsi), %xmm6+        vmovd 0xec(%rsi), %xmm7+        vpinsrw $0x2, 0xf0(%rsi), %xmm7, %xmm7+        vinserti128 $0x1, %xmm7, %ymm6, %ymm6+        vpermq $0x94, %ymm6, %ymm6     # ymm6 = ymm6[0,1,1,2]+        vpshufb %ymm2, %ymm6, %ymm6+        vpsrlvd %ymm3, %ymm6, %ymm6+        vpsrlvq %ymm4, %ymm6, %ymm6+        vpmullw %ymm5, %ymm6, %ymm6+        vpsrlw $0x1, %ymm6, %ymm6+        vpand %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm0, %ymm6, %ymm6+        vmovdqu %ymm6, 0x140(%rdi)+        vmovdqu 0xf2(%rsi), %xmm6+        vmovd 0x102(%rsi), %xmm7+        vpinsrw $0x2, 0x106(%rsi), %xmm7, %xmm7+        vinserti128 $0x1, %xmm7, %ymm6, %ymm6+        vpermq $0x94, %ymm6, %ymm6     # ymm6 = ymm6[0,1,1,2]+        vpshufb %ymm2, %ymm6, %ymm6+        vpsrlvd %ymm3, %ymm6, %ymm6+        vpsrlvq %ymm4, %ymm6, %ymm6+        vpmullw %ymm5, %ymm6, %ymm6+        vpsrlw $0x1, %ymm6, %ymm6+        vpand %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm0, %ymm6, %ymm6+        vmovdqu %ymm6, 0x160(%rdi)+        vmovdqu 0x108(%rsi), %xmm6+        vmovd 0x118(%rsi), %xmm7+        vpinsrw $0x2, 0x11c(%rsi), %xmm7, %xmm7+        vinserti128 $0x1, %xmm7, %ymm6, %ymm6+        vpermq $0x94, %ymm6, %ymm6     # ymm6 = ymm6[0,1,1,2]+        vpshufb %ymm2, %ymm6, %ymm6+        vpsrlvd %ymm3, %ymm6, %ymm6+        vpsrlvq %ymm4, %ymm6, %ymm6+        vpmullw %ymm5, %ymm6, %ymm6+        vpsrlw $0x1, %ymm6, %ymm6+        vpand %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm0, %ymm6, %ymm6+        vmovdqu %ymm6, 0x180(%rdi)+        vmovdqu 0x11e(%rsi), %xmm6+        vmovd 0x12e(%rsi), %xmm7+        vpinsrw $0x2, 0x132(%rsi), %xmm7, %xmm7+        vinserti128 $0x1, %xmm7, %ymm6, %ymm6+        vpermq $0x94, %ymm6, %ymm6     # ymm6 = ymm6[0,1,1,2]+        vpshufb %ymm2, %ymm6, %ymm6+        vpsrlvd %ymm3, %ymm6, %ymm6+        vpsrlvq %ymm4, %ymm6, %ymm6+        vpmullw %ymm5, %ymm6, %ymm6+        vpsrlw $0x1, %ymm6, %ymm6+        vpand %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm0, %ymm6, %ymm6+        vmovdqu %ymm6, 0x1a0(%rdi)+        vmovdqu 0x134(%rsi), %xmm6+        vmovd 0x144(%rsi), %xmm7+        vpinsrw $0x2, 0x148(%rsi), %xmm7, %xmm7+        vinserti128 $0x1, %xmm7, %ymm6, %ymm6+        vpermq $0x94, %ymm6, %ymm6     # ymm6 = ymm6[0,1,1,2]+        vpshufb %ymm2, %ymm6, %ymm6+        vpsrlvd %ymm3, %ymm6, %ymm6+        vpsrlvq %ymm4, %ymm6, %ymm6+        vpmullw %ymm5, %ymm6, %ymm6+        vpsrlw $0x1, %ymm6, %ymm6+        vpand %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm0, %ymm6, %ymm6+        vmovdqu %ymm6, 0x1c0(%rdi)+        vmovdqu 0x14a(%rsi), %xmm6+        vmovd 0x15a(%rsi), %xmm7+        vpinsrw $0x2, 0x15e(%rsi), %xmm7, %xmm7+        vinserti128 $0x1, %xmm7, %ymm6, %ymm6+        vpermq $0x94, %ymm6, %ymm6     # ymm6 = ymm6[0,1,1,2]+        vpshufb %ymm2, %ymm6, %ymm6+        vpsrlvd %ymm3, %ymm6, %ymm6+        vpsrlvq %ymm4, %ymm6, %ymm6+        vpmullw %ymm5, %ymm6, %ymm6+        vpsrlw $0x1, %ymm6, %ymm6+        vpand %ymm1, %ymm6, %ymm6+        vpmulhrsw %ymm0, %ymm6, %ymm6+        vmovdqu %ymm6, 0x1e0(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(poly_decompress_d11_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+          && !MLK_CONFIG_NO_DECAPS_API && (MLK_CONFIG_MULTILEVEL_WITH_SHARED || \+          MLKEM_K == 4) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_poly_decompress_d4_avx2_asm.S view
@@ -0,0 +1,209 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Kyber optimized AVX2 implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Kyber implementation @[REF_AVX2].+ */++/*************************************************+ * Name:        mlk_poly_decompress_d4_avx2_asm+ *+ * Description: Decompression of a polynomial from 4 bits per coefficient.+ *+ * Arguments:   - int16_t *r:       pointer to output polynomial+ *              - const uint8_t *a: pointer to input byte array+ *                                  (of length MLKEM_POLYCOMPRESSEDBYTES_D4)+ *              - const int8_t *data: pointer to shufbidx constant+ **************************************************/++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    !defined(MLK_CONFIG_NO_DECAPS_API) && \+    (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2 || MLKEM_K == 3)+/*yaml+  Name: poly_decompress_d4_avx2_asm+  Description: x86_64 AVX2 polynomial decompression (d=4)+  Signature: void mlk_poly_decompress_d4_avx2_asm(int16_t *r, const uint8_t *a, const uint8_t *data)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 512+      permissions: write-only+      c_parameter: int16_t *r+      description: Output polynomial (256 x int16_t)+    rsi:+      type: buffer+      size_bytes: 128+      permissions: read-only+      c_parameter: const uint8_t *a+      description: Input compressed polynomial+    rdx:+      type: buffer+      size_bytes: 32+      permissions: read-only+      c_parameter: const uint8_t *data+      description: Precomputed decompression constants+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_poly_decompress_d4_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(poly_decompress_d4_avx2_asm)+MLK_ASM_FN_SYMBOL(poly_decompress_d4_avx2_asm)++        .cfi_startproc+        movl $0xd010d01, %eax        # imm = 0xD010D01+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        movl $0xf0000f, %eax         # imm = 0xF0000F+        vmovd %eax, %xmm1+        vpbroadcastd %xmm1, %ymm1+        movl $0x800800, %eax         # imm = 0x800800+        vmovd %eax, %xmm2+        vpbroadcastd %xmm2, %ymm2+        vmovdqa (%rdx), %ymm3+        vmovq (%rsi), %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm3, %ymm4, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpmullw %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, (%rdi)+        vmovq 0x8(%rsi), %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm3, %ymm4, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpmullw %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x20(%rdi)+        vmovq 0x10(%rsi), %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm3, %ymm4, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpmullw %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x40(%rdi)+        vmovq 0x18(%rsi), %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm3, %ymm4, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpmullw %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x60(%rdi)+        vmovq 0x20(%rsi), %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm3, %ymm4, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpmullw %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x80(%rdi)+        vmovq 0x28(%rsi), %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm3, %ymm4, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpmullw %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0xa0(%rdi)+        vmovq 0x30(%rsi), %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm3, %ymm4, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpmullw %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0xc0(%rdi)+        vmovq 0x38(%rsi), %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm3, %ymm4, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpmullw %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0xe0(%rdi)+        vmovq 0x40(%rsi), %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm3, %ymm4, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpmullw %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x100(%rdi)+        vmovq 0x48(%rsi), %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm3, %ymm4, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpmullw %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x120(%rdi)+        vmovq 0x50(%rsi), %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm3, %ymm4, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpmullw %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x140(%rdi)+        vmovq 0x58(%rsi), %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm3, %ymm4, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpmullw %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x160(%rdi)+        vmovq 0x60(%rsi), %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm3, %ymm4, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpmullw %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x180(%rdi)+        vmovq 0x68(%rsi), %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm3, %ymm4, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpmullw %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x1a0(%rdi)+        vmovq 0x70(%rsi), %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm3, %ymm4, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpmullw %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x1c0(%rdi)+        vmovq 0x78(%rsi), %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm3, %ymm4, %ymm4+        vpand %ymm1, %ymm4, %ymm4+        vpmullw %ymm2, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x1e0(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(poly_decompress_d4_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+          && !MLK_CONFIG_NO_DECAPS_API && (MLK_CONFIG_MULTILEVEL_WITH_SHARED || \+          MLKEM_K == 2 || MLKEM_K == 3) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_poly_decompress_d5_avx2_asm.S view
@@ -0,0 +1,222 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Kyber optimized AVX2 implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Kyber implementation @[REF_AVX2].+ */++/*************************************************+ * Name:        mlk_poly_decompress_d5_avx2_asm+ *+ * Description: Decompression of a polynomial from 5 bits per coefficient.+ *+ * Arguments:   - int16_t *r:       pointer to output polynomial+ *              - const uint8_t *a: pointer to input byte array+ *                                  (of length MLKEM_POLYCOMPRESSEDBYTES_D5)+ *              - const uint8_t *data: pointer to constants+ *                                  (shufbidx[0:32], mask[32:64], shift[64:96])+ **************************************************/++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    !defined(MLK_CONFIG_NO_DECAPS_API) && \+    (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4)+/*yaml+  Name: poly_decompress_d5_avx2_asm+  Description: x86_64 AVX2 polynomial decompression (d=5)+  Signature: void mlk_poly_decompress_d5_avx2_asm(int16_t *r, const uint8_t *a, const uint8_t *data)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 512+      permissions: write-only+      c_parameter: int16_t *r+      description: Output polynomial (256 x int16_t)+    rsi:+      type: buffer+      size_bytes: 160+      permissions: read-only+      c_parameter: const uint8_t *a+      description: Input compressed polynomial+    rdx:+      type: buffer+      size_bytes: 96+      permissions: read-only+      c_parameter: const uint8_t *data+      description: Precomputed decompression constants+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_poly_decompress_d5_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(poly_decompress_d5_avx2_asm)+MLK_ASM_FN_SYMBOL(poly_decompress_d5_avx2_asm)++        .cfi_startproc+        movl $0xd010d01, %eax        # imm = 0xD010D01+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        vmovdqa (%rdx), %ymm1+        vmovdqa 0x20(%rdx), %ymm2+        vmovdqa 0x40(%rdx), %ymm3+        vmovq (%rsi), %xmm4+        vpinsrw $0x4, 0x8(%rsi), %xmm4, %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmullw %ymm3, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, (%rdi)+        vmovq 0xa(%rsi), %xmm4+        vpinsrw $0x4, 0x12(%rsi), %xmm4, %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmullw %ymm3, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x20(%rdi)+        vmovq 0x14(%rsi), %xmm4+        vpinsrw $0x4, 0x1c(%rsi), %xmm4, %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmullw %ymm3, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x40(%rdi)+        vmovq 0x1e(%rsi), %xmm4+        vpinsrw $0x4, 0x26(%rsi), %xmm4, %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmullw %ymm3, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x60(%rdi)+        vmovq 0x28(%rsi), %xmm4+        vpinsrw $0x4, 0x30(%rsi), %xmm4, %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmullw %ymm3, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x80(%rdi)+        vmovq 0x32(%rsi), %xmm4+        vpinsrw $0x4, 0x3a(%rsi), %xmm4, %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmullw %ymm3, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0xa0(%rdi)+        vmovq 0x3c(%rsi), %xmm4+        vpinsrw $0x4, 0x44(%rsi), %xmm4, %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmullw %ymm3, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0xc0(%rdi)+        vmovq 0x46(%rsi), %xmm4+        vpinsrw $0x4, 0x4e(%rsi), %xmm4, %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmullw %ymm3, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0xe0(%rdi)+        vmovq 0x50(%rsi), %xmm4+        vpinsrw $0x4, 0x58(%rsi), %xmm4, %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmullw %ymm3, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x100(%rdi)+        vmovq 0x5a(%rsi), %xmm4+        vpinsrw $0x4, 0x62(%rsi), %xmm4, %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmullw %ymm3, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x120(%rdi)+        vmovq 0x64(%rsi), %xmm4+        vpinsrw $0x4, 0x6c(%rsi), %xmm4, %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmullw %ymm3, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x140(%rdi)+        vmovq 0x6e(%rsi), %xmm4+        vpinsrw $0x4, 0x76(%rsi), %xmm4, %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmullw %ymm3, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x160(%rdi)+        vmovq 0x78(%rsi), %xmm4+        vpinsrw $0x4, 0x80(%rsi), %xmm4, %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmullw %ymm3, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x180(%rdi)+        vmovq 0x82(%rsi), %xmm4+        vpinsrw $0x4, 0x8a(%rsi), %xmm4, %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmullw %ymm3, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x1a0(%rdi)+        vmovq 0x8c(%rsi), %xmm4+        vpinsrw $0x4, 0x94(%rsi), %xmm4, %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmullw %ymm3, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x1c0(%rdi)+        vmovq 0x96(%rsi), %xmm4+        vpinsrw $0x4, 0x9e(%rsi), %xmm4, %xmm4+        vinserti128 $0x1, %xmm4, %ymm4, %ymm4+        vpshufb %ymm1, %ymm4, %ymm4+        vpand %ymm2, %ymm4, %ymm4+        vpmullw %ymm3, %ymm4, %ymm4+        vpmulhrsw %ymm0, %ymm4, %ymm4+        vmovdqu %ymm4, 0x1e0(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(poly_decompress_d5_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+          && !MLK_CONFIG_NO_DECAPS_API && (MLK_CONFIG_MULTILEVEL_WITH_SHARED || \+          MLKEM_K == 4) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_poly_mulcache_compute_avx2_asm.S view
@@ -0,0 +1,118 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)+/*yaml+  Name: poly_mulcache_compute_avx2_asm+  Description: x86_64 AVX2 mulcache computation+  Signature: void mlk_poly_mulcache_compute_avx2_asm(int16_t *out, const int16_t *in, const int16_t *qdata)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 256+      permissions: write-only+      c_parameter: int16_t *out+      description: Output mulcache (128 x int16_t)+    rsi:+      type: buffer+      size_bytes: 512+      permissions: read-only+      c_parameter: const int16_t *in+      description: Input polynomial (256 x int16_t)+    rdx:+      type: buffer+      size_bytes: 1248+      permissions: read-only+      c_parameter: const int16_t *qdata+      description: Precomputed constants (624 x int16_t)+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_poly_mulcache_compute_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(poly_mulcache_compute_avx2_asm)+MLK_ASM_FN_SYMBOL(poly_mulcache_compute_avx2_asm)++        .cfi_startproc+        movl $0xd010d01, %eax        # imm = 0xD010D01+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        vmovdqa 0x20(%rsi), %ymm2+        vmovdqa 0x60(%rsi), %ymm3+        vmovdqa 0x3e0(%rdx), %ymm4+        vmovdqa 0x460(%rdx), %ymm1+        vpmullw %ymm2, %ymm1, %ymm5+        vpmullw %ymm3, %ymm1, %ymm6+        vpmulhw %ymm2, %ymm4, %ymm7+        vpmulhw %ymm3, %ymm4, %ymm8+        vpmulhw %ymm5, %ymm0, %ymm9+        vpmulhw %ymm6, %ymm0, %ymm10+        vpsubw %ymm9, %ymm7, %ymm7+        vpsubw %ymm10, %ymm8, %ymm8+        vmovdqa %ymm7, (%rdi)+        vmovdqa %ymm8, 0x20(%rdi)+        vmovdqa 0xa0(%rsi), %ymm2+        vmovdqa 0xe0(%rsi), %ymm3+        vmovdqa 0x400(%rdx), %ymm4+        vmovdqa 0x480(%rdx), %ymm1+        vpmullw %ymm2, %ymm1, %ymm5+        vpmullw %ymm3, %ymm1, %ymm6+        vpmulhw %ymm2, %ymm4, %ymm7+        vpmulhw %ymm3, %ymm4, %ymm8+        vpmulhw %ymm5, %ymm0, %ymm9+        vpmulhw %ymm6, %ymm0, %ymm10+        vpsubw %ymm9, %ymm7, %ymm7+        vpsubw %ymm10, %ymm8, %ymm8+        vmovdqa %ymm7, 0x40(%rdi)+        vmovdqa %ymm8, 0x60(%rdi)+        vmovdqa 0x120(%rsi), %ymm2+        vmovdqa 0x160(%rsi), %ymm3+        vmovdqa 0x420(%rdx), %ymm4+        vmovdqa 0x4a0(%rdx), %ymm1+        vpmullw %ymm2, %ymm1, %ymm5+        vpmullw %ymm3, %ymm1, %ymm6+        vpmulhw %ymm2, %ymm4, %ymm7+        vpmulhw %ymm3, %ymm4, %ymm8+        vpmulhw %ymm5, %ymm0, %ymm9+        vpmulhw %ymm6, %ymm0, %ymm10+        vpsubw %ymm9, %ymm7, %ymm7+        vpsubw %ymm10, %ymm8, %ymm8+        vmovdqa %ymm7, 0x80(%rdi)+        vmovdqa %ymm8, 0xa0(%rdi)+        vmovdqa 0x1a0(%rsi), %ymm2+        vmovdqa 0x1e0(%rsi), %ymm3+        vmovdqa 0x440(%rdx), %ymm4+        vmovdqa 0x4c0(%rdx), %ymm1+        vpmullw %ymm2, %ymm1, %ymm5+        vpmullw %ymm3, %ymm1, %ymm6+        vpmulhw %ymm2, %ymm4, %ymm7+        vpmulhw %ymm3, %ymm4, %ymm8+        vpmulhw %ymm5, %ymm0, %ymm9+        vpmulhw %ymm6, %ymm0, %ymm10+        vpsubw %ymm9, %ymm7, %ymm7+        vpsubw %ymm10, %ymm8, %ymm8+        vmovdqa %ymm7, 0xc0(%rdi)+        vmovdqa %ymm8, 0xe0(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(poly_mulcache_compute_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm.S view
@@ -0,0 +1,536 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2)+/*yaml+  Name: polyvec_basemul_acc_montgomery_cached_k2_avx2_asm+  Description: x86_64 AVX2 base multiplication with accumulation (k=2)+  Signature: void mlk_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm(int16_t *r, const int16_t *a, const int16_t *b, const int16_t *b_cache)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 512+      permissions: read/write+      c_parameter: int16_t *r+      description: Output polynomial (256 x int16_t)+    rsi:+      type: buffer+      size_bytes: 1024+      permissions: read-only+      c_parameter: const int16_t *a+      description: Input polyvec a (2 x 256 x int16_t)+    rdx:+      type: buffer+      size_bytes: 1024+      permissions: read-only+      c_parameter: const int16_t *b+      description: Input polyvec b (2 x 256 x int16_t)+    rcx:+      type: buffer+      size_bytes: 512+      permissions: read-only+      c_parameter: const int16_t *b_cache+      description: Mulcache for b (2 x 128 x int16_t)+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k2_avx2_asm)+MLK_ASM_FN_SYMBOL(polyvec_basemul_acc_montgomery_cached_k2_avx2_asm)++        .cfi_startproc+        movl $0xd010d01, %eax        # imm = 0xD010D01+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        movl $0xf301f301, %eax       # imm = 0xF301F301+        vmovd %eax, %xmm1+        vpbroadcastd %xmm1, %ymm1+        vmovdqa (%rsi), %ymm2+        vmovdqa 0x20(%rsi), %ymm3+        vmovdqa (%rdx), %ymm4+        vmovdqa 0x20(%rdx), %ymm5+        vmovdqa (%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, (%rdi)+        vmovdqa %ymm9, 0x20(%rdi)+        vmovdqa 0x40(%rsi), %ymm2+        vmovdqa 0x60(%rsi), %ymm3+        vmovdqa 0x40(%rdx), %ymm4+        vmovdqa 0x60(%rdx), %ymm5+        vmovdqa 0x20(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x40(%rdi)+        vmovdqa %ymm9, 0x60(%rdi)+        vmovdqa 0x80(%rsi), %ymm2+        vmovdqa 0xa0(%rsi), %ymm3+        vmovdqa 0x80(%rdx), %ymm4+        vmovdqa 0xa0(%rdx), %ymm5+        vmovdqa 0x40(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x80(%rdi)+        vmovdqa %ymm9, 0xa0(%rdi)+        vmovdqa 0xc0(%rsi), %ymm2+        vmovdqa 0xe0(%rsi), %ymm3+        vmovdqa 0xc0(%rdx), %ymm4+        vmovdqa 0xe0(%rdx), %ymm5+        vmovdqa 0x60(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0xc0(%rdi)+        vmovdqa %ymm9, 0xe0(%rdi)+        vmovdqa 0x100(%rsi), %ymm2+        vmovdqa 0x120(%rsi), %ymm3+        vmovdqa 0x100(%rdx), %ymm4+        vmovdqa 0x120(%rdx), %ymm5+        vmovdqa 0x80(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x100(%rdi)+        vmovdqa %ymm9, 0x120(%rdi)+        vmovdqa 0x140(%rsi), %ymm2+        vmovdqa 0x160(%rsi), %ymm3+        vmovdqa 0x140(%rdx), %ymm4+        vmovdqa 0x160(%rdx), %ymm5+        vmovdqa 0xa0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x140(%rdi)+        vmovdqa %ymm9, 0x160(%rdi)+        vmovdqa 0x180(%rsi), %ymm2+        vmovdqa 0x1a0(%rsi), %ymm3+        vmovdqa 0x180(%rdx), %ymm4+        vmovdqa 0x1a0(%rdx), %ymm5+        vmovdqa 0xc0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x180(%rdi)+        vmovdqa %ymm9, 0x1a0(%rdi)+        vmovdqa 0x1c0(%rsi), %ymm2+        vmovdqa 0x1e0(%rsi), %ymm3+        vmovdqa 0x1c0(%rdx), %ymm4+        vmovdqa 0x1e0(%rdx), %ymm5+        vmovdqa 0xe0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x1c0(%rdi)+        vmovdqa %ymm9, 0x1e0(%rdi)+        vmovdqa 0x200(%rsi), %ymm2+        vmovdqa 0x220(%rsi), %ymm3+        vmovdqa 0x200(%rdx), %ymm4+        vmovdqa 0x220(%rdx), %ymm5+        vmovdqa 0x100(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa (%rdi), %ymm8+        vmovdqa 0x20(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, (%rdi)+        vmovdqa %ymm9, 0x20(%rdi)+        vmovdqa 0x240(%rsi), %ymm2+        vmovdqa 0x260(%rsi), %ymm3+        vmovdqa 0x240(%rdx), %ymm4+        vmovdqa 0x260(%rdx), %ymm5+        vmovdqa 0x120(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x40(%rdi), %ymm8+        vmovdqa 0x60(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x40(%rdi)+        vmovdqa %ymm9, 0x60(%rdi)+        vmovdqa 0x280(%rsi), %ymm2+        vmovdqa 0x2a0(%rsi), %ymm3+        vmovdqa 0x280(%rdx), %ymm4+        vmovdqa 0x2a0(%rdx), %ymm5+        vmovdqa 0x140(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x80(%rdi), %ymm8+        vmovdqa 0xa0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x80(%rdi)+        vmovdqa %ymm9, 0xa0(%rdi)+        vmovdqa 0x2c0(%rsi), %ymm2+        vmovdqa 0x2e0(%rsi), %ymm3+        vmovdqa 0x2c0(%rdx), %ymm4+        vmovdqa 0x2e0(%rdx), %ymm5+        vmovdqa 0x160(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0xc0(%rdi), %ymm8+        vmovdqa 0xe0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0xc0(%rdi)+        vmovdqa %ymm9, 0xe0(%rdi)+        vmovdqa 0x300(%rsi), %ymm2+        vmovdqa 0x320(%rsi), %ymm3+        vmovdqa 0x300(%rdx), %ymm4+        vmovdqa 0x320(%rdx), %ymm5+        vmovdqa 0x180(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x100(%rdi), %ymm8+        vmovdqa 0x120(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x100(%rdi)+        vmovdqa %ymm9, 0x120(%rdi)+        vmovdqa 0x340(%rsi), %ymm2+        vmovdqa 0x360(%rsi), %ymm3+        vmovdqa 0x340(%rdx), %ymm4+        vmovdqa 0x360(%rdx), %ymm5+        vmovdqa 0x1a0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x140(%rdi), %ymm8+        vmovdqa 0x160(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x140(%rdi)+        vmovdqa %ymm9, 0x160(%rdi)+        vmovdqa 0x380(%rsi), %ymm2+        vmovdqa 0x3a0(%rsi), %ymm3+        vmovdqa 0x380(%rdx), %ymm4+        vmovdqa 0x3a0(%rdx), %ymm5+        vmovdqa 0x1c0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x180(%rdi), %ymm8+        vmovdqa 0x1a0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x180(%rdi)+        vmovdqa %ymm9, 0x1a0(%rdi)+        vmovdqa 0x3c0(%rsi), %ymm2+        vmovdqa 0x3e0(%rsi), %ymm3+        vmovdqa 0x3c0(%rdx), %ymm4+        vmovdqa 0x3e0(%rdx), %ymm5+        vmovdqa 0x1e0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x1c0(%rdi), %ymm8+        vmovdqa 0x1e0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x1c0(%rdi)+        vmovdqa %ymm9, 0x1e0(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(polyvec_basemul_acc_montgomery_cached_k2_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+          && (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm.S view
@@ -0,0 +1,784 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 3)+/*yaml+  Name: polyvec_basemul_acc_montgomery_cached_k3_avx2_asm+  Description: x86_64 AVX2 base multiplication with accumulation (k=3)+  Signature: void mlk_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm(int16_t *r, const int16_t *a, const int16_t *b, const int16_t *b_cache)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 512+      permissions: read/write+      c_parameter: int16_t *r+      description: Output polynomial (256 x int16_t)+    rsi:+      type: buffer+      size_bytes: 1536+      permissions: read-only+      c_parameter: const int16_t *a+      description: Input polyvec a (3 x 256 x int16_t)+    rdx:+      type: buffer+      size_bytes: 1536+      permissions: read-only+      c_parameter: const int16_t *b+      description: Input polyvec b (3 x 256 x int16_t)+    rcx:+      type: buffer+      size_bytes: 768+      permissions: read-only+      c_parameter: const int16_t *b_cache+      description: Mulcache for b (3 x 128 x int16_t)+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k3_avx2_asm)+MLK_ASM_FN_SYMBOL(polyvec_basemul_acc_montgomery_cached_k3_avx2_asm)++        .cfi_startproc+        movl $0xd010d01, %eax        # imm = 0xD010D01+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        movl $0xf301f301, %eax       # imm = 0xF301F301+        vmovd %eax, %xmm1+        vpbroadcastd %xmm1, %ymm1+        vmovdqa (%rsi), %ymm2+        vmovdqa 0x20(%rsi), %ymm3+        vmovdqa (%rdx), %ymm4+        vmovdqa 0x20(%rdx), %ymm5+        vmovdqa (%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, (%rdi)+        vmovdqa %ymm9, 0x20(%rdi)+        vmovdqa 0x40(%rsi), %ymm2+        vmovdqa 0x60(%rsi), %ymm3+        vmovdqa 0x40(%rdx), %ymm4+        vmovdqa 0x60(%rdx), %ymm5+        vmovdqa 0x20(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x40(%rdi)+        vmovdqa %ymm9, 0x60(%rdi)+        vmovdqa 0x80(%rsi), %ymm2+        vmovdqa 0xa0(%rsi), %ymm3+        vmovdqa 0x80(%rdx), %ymm4+        vmovdqa 0xa0(%rdx), %ymm5+        vmovdqa 0x40(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x80(%rdi)+        vmovdqa %ymm9, 0xa0(%rdi)+        vmovdqa 0xc0(%rsi), %ymm2+        vmovdqa 0xe0(%rsi), %ymm3+        vmovdqa 0xc0(%rdx), %ymm4+        vmovdqa 0xe0(%rdx), %ymm5+        vmovdqa 0x60(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0xc0(%rdi)+        vmovdqa %ymm9, 0xe0(%rdi)+        vmovdqa 0x100(%rsi), %ymm2+        vmovdqa 0x120(%rsi), %ymm3+        vmovdqa 0x100(%rdx), %ymm4+        vmovdqa 0x120(%rdx), %ymm5+        vmovdqa 0x80(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x100(%rdi)+        vmovdqa %ymm9, 0x120(%rdi)+        vmovdqa 0x140(%rsi), %ymm2+        vmovdqa 0x160(%rsi), %ymm3+        vmovdqa 0x140(%rdx), %ymm4+        vmovdqa 0x160(%rdx), %ymm5+        vmovdqa 0xa0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x140(%rdi)+        vmovdqa %ymm9, 0x160(%rdi)+        vmovdqa 0x180(%rsi), %ymm2+        vmovdqa 0x1a0(%rsi), %ymm3+        vmovdqa 0x180(%rdx), %ymm4+        vmovdqa 0x1a0(%rdx), %ymm5+        vmovdqa 0xc0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x180(%rdi)+        vmovdqa %ymm9, 0x1a0(%rdi)+        vmovdqa 0x1c0(%rsi), %ymm2+        vmovdqa 0x1e0(%rsi), %ymm3+        vmovdqa 0x1c0(%rdx), %ymm4+        vmovdqa 0x1e0(%rdx), %ymm5+        vmovdqa 0xe0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x1c0(%rdi)+        vmovdqa %ymm9, 0x1e0(%rdi)+        vmovdqa 0x200(%rsi), %ymm2+        vmovdqa 0x220(%rsi), %ymm3+        vmovdqa 0x200(%rdx), %ymm4+        vmovdqa 0x220(%rdx), %ymm5+        vmovdqa 0x100(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa (%rdi), %ymm8+        vmovdqa 0x20(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, (%rdi)+        vmovdqa %ymm9, 0x20(%rdi)+        vmovdqa 0x240(%rsi), %ymm2+        vmovdqa 0x260(%rsi), %ymm3+        vmovdqa 0x240(%rdx), %ymm4+        vmovdqa 0x260(%rdx), %ymm5+        vmovdqa 0x120(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x40(%rdi), %ymm8+        vmovdqa 0x60(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x40(%rdi)+        vmovdqa %ymm9, 0x60(%rdi)+        vmovdqa 0x280(%rsi), %ymm2+        vmovdqa 0x2a0(%rsi), %ymm3+        vmovdqa 0x280(%rdx), %ymm4+        vmovdqa 0x2a0(%rdx), %ymm5+        vmovdqa 0x140(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x80(%rdi), %ymm8+        vmovdqa 0xa0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x80(%rdi)+        vmovdqa %ymm9, 0xa0(%rdi)+        vmovdqa 0x2c0(%rsi), %ymm2+        vmovdqa 0x2e0(%rsi), %ymm3+        vmovdqa 0x2c0(%rdx), %ymm4+        vmovdqa 0x2e0(%rdx), %ymm5+        vmovdqa 0x160(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0xc0(%rdi), %ymm8+        vmovdqa 0xe0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0xc0(%rdi)+        vmovdqa %ymm9, 0xe0(%rdi)+        vmovdqa 0x300(%rsi), %ymm2+        vmovdqa 0x320(%rsi), %ymm3+        vmovdqa 0x300(%rdx), %ymm4+        vmovdqa 0x320(%rdx), %ymm5+        vmovdqa 0x180(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x100(%rdi), %ymm8+        vmovdqa 0x120(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x100(%rdi)+        vmovdqa %ymm9, 0x120(%rdi)+        vmovdqa 0x340(%rsi), %ymm2+        vmovdqa 0x360(%rsi), %ymm3+        vmovdqa 0x340(%rdx), %ymm4+        vmovdqa 0x360(%rdx), %ymm5+        vmovdqa 0x1a0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x140(%rdi), %ymm8+        vmovdqa 0x160(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x140(%rdi)+        vmovdqa %ymm9, 0x160(%rdi)+        vmovdqa 0x380(%rsi), %ymm2+        vmovdqa 0x3a0(%rsi), %ymm3+        vmovdqa 0x380(%rdx), %ymm4+        vmovdqa 0x3a0(%rdx), %ymm5+        vmovdqa 0x1c0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x180(%rdi), %ymm8+        vmovdqa 0x1a0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x180(%rdi)+        vmovdqa %ymm9, 0x1a0(%rdi)+        vmovdqa 0x3c0(%rsi), %ymm2+        vmovdqa 0x3e0(%rsi), %ymm3+        vmovdqa 0x3c0(%rdx), %ymm4+        vmovdqa 0x3e0(%rdx), %ymm5+        vmovdqa 0x1e0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x1c0(%rdi), %ymm8+        vmovdqa 0x1e0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x1c0(%rdi)+        vmovdqa %ymm9, 0x1e0(%rdi)+        vmovdqa 0x400(%rsi), %ymm2+        vmovdqa 0x420(%rsi), %ymm3+        vmovdqa 0x400(%rdx), %ymm4+        vmovdqa 0x420(%rdx), %ymm5+        vmovdqa 0x200(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa (%rdi), %ymm8+        vmovdqa 0x20(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, (%rdi)+        vmovdqa %ymm9, 0x20(%rdi)+        vmovdqa 0x440(%rsi), %ymm2+        vmovdqa 0x460(%rsi), %ymm3+        vmovdqa 0x440(%rdx), %ymm4+        vmovdqa 0x460(%rdx), %ymm5+        vmovdqa 0x220(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x40(%rdi), %ymm8+        vmovdqa 0x60(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x40(%rdi)+        vmovdqa %ymm9, 0x60(%rdi)+        vmovdqa 0x480(%rsi), %ymm2+        vmovdqa 0x4a0(%rsi), %ymm3+        vmovdqa 0x480(%rdx), %ymm4+        vmovdqa 0x4a0(%rdx), %ymm5+        vmovdqa 0x240(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x80(%rdi), %ymm8+        vmovdqa 0xa0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x80(%rdi)+        vmovdqa %ymm9, 0xa0(%rdi)+        vmovdqa 0x4c0(%rsi), %ymm2+        vmovdqa 0x4e0(%rsi), %ymm3+        vmovdqa 0x4c0(%rdx), %ymm4+        vmovdqa 0x4e0(%rdx), %ymm5+        vmovdqa 0x260(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0xc0(%rdi), %ymm8+        vmovdqa 0xe0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0xc0(%rdi)+        vmovdqa %ymm9, 0xe0(%rdi)+        vmovdqa 0x500(%rsi), %ymm2+        vmovdqa 0x520(%rsi), %ymm3+        vmovdqa 0x500(%rdx), %ymm4+        vmovdqa 0x520(%rdx), %ymm5+        vmovdqa 0x280(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x100(%rdi), %ymm8+        vmovdqa 0x120(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x100(%rdi)+        vmovdqa %ymm9, 0x120(%rdi)+        vmovdqa 0x540(%rsi), %ymm2+        vmovdqa 0x560(%rsi), %ymm3+        vmovdqa 0x540(%rdx), %ymm4+        vmovdqa 0x560(%rdx), %ymm5+        vmovdqa 0x2a0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x140(%rdi), %ymm8+        vmovdqa 0x160(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x140(%rdi)+        vmovdqa %ymm9, 0x160(%rdi)+        vmovdqa 0x580(%rsi), %ymm2+        vmovdqa 0x5a0(%rsi), %ymm3+        vmovdqa 0x580(%rdx), %ymm4+        vmovdqa 0x5a0(%rdx), %ymm5+        vmovdqa 0x2c0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x180(%rdi), %ymm8+        vmovdqa 0x1a0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x180(%rdi)+        vmovdqa %ymm9, 0x1a0(%rdi)+        vmovdqa 0x5c0(%rsi), %ymm2+        vmovdqa 0x5e0(%rsi), %ymm3+        vmovdqa 0x5c0(%rdx), %ymm4+        vmovdqa 0x5e0(%rdx), %ymm5+        vmovdqa 0x2e0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x1c0(%rdi), %ymm8+        vmovdqa 0x1e0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x1c0(%rdi)+        vmovdqa %ymm9, 0x1e0(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(polyvec_basemul_acc_montgomery_cached_k3_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+          && (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 3) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm.S view
@@ -0,0 +1,1032 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4)+/*yaml+  Name: polyvec_basemul_acc_montgomery_cached_k4_avx2_asm+  Description: x86_64 AVX2 base multiplication with accumulation (k=4)+  Signature: void mlk_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm(int16_t *r, const int16_t *a, const int16_t *b, const int16_t *b_cache)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 512+      permissions: read/write+      c_parameter: int16_t *r+      description: Output polynomial (256 x int16_t)+    rsi:+      type: buffer+      size_bytes: 2048+      permissions: read-only+      c_parameter: const int16_t *a+      description: Input polyvec a (4 x 256 x int16_t)+    rdx:+      type: buffer+      size_bytes: 2048+      permissions: read-only+      c_parameter: const int16_t *b+      description: Input polyvec b (4 x 256 x int16_t)+    rcx:+      type: buffer+      size_bytes: 1024+      permissions: read-only+      c_parameter: const int16_t *b_cache+      description: Mulcache for b (4 x 128 x int16_t)+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k4_avx2_asm)+MLK_ASM_FN_SYMBOL(polyvec_basemul_acc_montgomery_cached_k4_avx2_asm)++        .cfi_startproc+        movl $0xd010d01, %eax        # imm = 0xD010D01+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        movl $0xf301f301, %eax       # imm = 0xF301F301+        vmovd %eax, %xmm1+        vpbroadcastd %xmm1, %ymm1+        vmovdqa (%rsi), %ymm2+        vmovdqa 0x20(%rsi), %ymm3+        vmovdqa (%rdx), %ymm4+        vmovdqa 0x20(%rdx), %ymm5+        vmovdqa (%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, (%rdi)+        vmovdqa %ymm9, 0x20(%rdi)+        vmovdqa 0x40(%rsi), %ymm2+        vmovdqa 0x60(%rsi), %ymm3+        vmovdqa 0x40(%rdx), %ymm4+        vmovdqa 0x60(%rdx), %ymm5+        vmovdqa 0x20(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x40(%rdi)+        vmovdqa %ymm9, 0x60(%rdi)+        vmovdqa 0x80(%rsi), %ymm2+        vmovdqa 0xa0(%rsi), %ymm3+        vmovdqa 0x80(%rdx), %ymm4+        vmovdqa 0xa0(%rdx), %ymm5+        vmovdqa 0x40(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x80(%rdi)+        vmovdqa %ymm9, 0xa0(%rdi)+        vmovdqa 0xc0(%rsi), %ymm2+        vmovdqa 0xe0(%rsi), %ymm3+        vmovdqa 0xc0(%rdx), %ymm4+        vmovdqa 0xe0(%rdx), %ymm5+        vmovdqa 0x60(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0xc0(%rdi)+        vmovdqa %ymm9, 0xe0(%rdi)+        vmovdqa 0x100(%rsi), %ymm2+        vmovdqa 0x120(%rsi), %ymm3+        vmovdqa 0x100(%rdx), %ymm4+        vmovdqa 0x120(%rdx), %ymm5+        vmovdqa 0x80(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x100(%rdi)+        vmovdqa %ymm9, 0x120(%rdi)+        vmovdqa 0x140(%rsi), %ymm2+        vmovdqa 0x160(%rsi), %ymm3+        vmovdqa 0x140(%rdx), %ymm4+        vmovdqa 0x160(%rdx), %ymm5+        vmovdqa 0xa0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x140(%rdi)+        vmovdqa %ymm9, 0x160(%rdi)+        vmovdqa 0x180(%rsi), %ymm2+        vmovdqa 0x1a0(%rsi), %ymm3+        vmovdqa 0x180(%rdx), %ymm4+        vmovdqa 0x1a0(%rdx), %ymm5+        vmovdqa 0xc0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x180(%rdi)+        vmovdqa %ymm9, 0x1a0(%rdi)+        vmovdqa 0x1c0(%rsi), %ymm2+        vmovdqa 0x1e0(%rsi), %ymm3+        vmovdqa 0x1c0(%rdx), %ymm4+        vmovdqa 0x1e0(%rdx), %ymm5+        vmovdqa 0xe0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x1c0(%rdi)+        vmovdqa %ymm9, 0x1e0(%rdi)+        vmovdqa 0x200(%rsi), %ymm2+        vmovdqa 0x220(%rsi), %ymm3+        vmovdqa 0x200(%rdx), %ymm4+        vmovdqa 0x220(%rdx), %ymm5+        vmovdqa 0x100(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa (%rdi), %ymm8+        vmovdqa 0x20(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, (%rdi)+        vmovdqa %ymm9, 0x20(%rdi)+        vmovdqa 0x240(%rsi), %ymm2+        vmovdqa 0x260(%rsi), %ymm3+        vmovdqa 0x240(%rdx), %ymm4+        vmovdqa 0x260(%rdx), %ymm5+        vmovdqa 0x120(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x40(%rdi), %ymm8+        vmovdqa 0x60(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x40(%rdi)+        vmovdqa %ymm9, 0x60(%rdi)+        vmovdqa 0x280(%rsi), %ymm2+        vmovdqa 0x2a0(%rsi), %ymm3+        vmovdqa 0x280(%rdx), %ymm4+        vmovdqa 0x2a0(%rdx), %ymm5+        vmovdqa 0x140(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x80(%rdi), %ymm8+        vmovdqa 0xa0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x80(%rdi)+        vmovdqa %ymm9, 0xa0(%rdi)+        vmovdqa 0x2c0(%rsi), %ymm2+        vmovdqa 0x2e0(%rsi), %ymm3+        vmovdqa 0x2c0(%rdx), %ymm4+        vmovdqa 0x2e0(%rdx), %ymm5+        vmovdqa 0x160(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0xc0(%rdi), %ymm8+        vmovdqa 0xe0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0xc0(%rdi)+        vmovdqa %ymm9, 0xe0(%rdi)+        vmovdqa 0x300(%rsi), %ymm2+        vmovdqa 0x320(%rsi), %ymm3+        vmovdqa 0x300(%rdx), %ymm4+        vmovdqa 0x320(%rdx), %ymm5+        vmovdqa 0x180(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x100(%rdi), %ymm8+        vmovdqa 0x120(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x100(%rdi)+        vmovdqa %ymm9, 0x120(%rdi)+        vmovdqa 0x340(%rsi), %ymm2+        vmovdqa 0x360(%rsi), %ymm3+        vmovdqa 0x340(%rdx), %ymm4+        vmovdqa 0x360(%rdx), %ymm5+        vmovdqa 0x1a0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x140(%rdi), %ymm8+        vmovdqa 0x160(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x140(%rdi)+        vmovdqa %ymm9, 0x160(%rdi)+        vmovdqa 0x380(%rsi), %ymm2+        vmovdqa 0x3a0(%rsi), %ymm3+        vmovdqa 0x380(%rdx), %ymm4+        vmovdqa 0x3a0(%rdx), %ymm5+        vmovdqa 0x1c0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x180(%rdi), %ymm8+        vmovdqa 0x1a0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x180(%rdi)+        vmovdqa %ymm9, 0x1a0(%rdi)+        vmovdqa 0x3c0(%rsi), %ymm2+        vmovdqa 0x3e0(%rsi), %ymm3+        vmovdqa 0x3c0(%rdx), %ymm4+        vmovdqa 0x3e0(%rdx), %ymm5+        vmovdqa 0x1e0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x1c0(%rdi), %ymm8+        vmovdqa 0x1e0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x1c0(%rdi)+        vmovdqa %ymm9, 0x1e0(%rdi)+        vmovdqa 0x400(%rsi), %ymm2+        vmovdqa 0x420(%rsi), %ymm3+        vmovdqa 0x400(%rdx), %ymm4+        vmovdqa 0x420(%rdx), %ymm5+        vmovdqa 0x200(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa (%rdi), %ymm8+        vmovdqa 0x20(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, (%rdi)+        vmovdqa %ymm9, 0x20(%rdi)+        vmovdqa 0x440(%rsi), %ymm2+        vmovdqa 0x460(%rsi), %ymm3+        vmovdqa 0x440(%rdx), %ymm4+        vmovdqa 0x460(%rdx), %ymm5+        vmovdqa 0x220(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x40(%rdi), %ymm8+        vmovdqa 0x60(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x40(%rdi)+        vmovdqa %ymm9, 0x60(%rdi)+        vmovdqa 0x480(%rsi), %ymm2+        vmovdqa 0x4a0(%rsi), %ymm3+        vmovdqa 0x480(%rdx), %ymm4+        vmovdqa 0x4a0(%rdx), %ymm5+        vmovdqa 0x240(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x80(%rdi), %ymm8+        vmovdqa 0xa0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x80(%rdi)+        vmovdqa %ymm9, 0xa0(%rdi)+        vmovdqa 0x4c0(%rsi), %ymm2+        vmovdqa 0x4e0(%rsi), %ymm3+        vmovdqa 0x4c0(%rdx), %ymm4+        vmovdqa 0x4e0(%rdx), %ymm5+        vmovdqa 0x260(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0xc0(%rdi), %ymm8+        vmovdqa 0xe0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0xc0(%rdi)+        vmovdqa %ymm9, 0xe0(%rdi)+        vmovdqa 0x500(%rsi), %ymm2+        vmovdqa 0x520(%rsi), %ymm3+        vmovdqa 0x500(%rdx), %ymm4+        vmovdqa 0x520(%rdx), %ymm5+        vmovdqa 0x280(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x100(%rdi), %ymm8+        vmovdqa 0x120(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x100(%rdi)+        vmovdqa %ymm9, 0x120(%rdi)+        vmovdqa 0x540(%rsi), %ymm2+        vmovdqa 0x560(%rsi), %ymm3+        vmovdqa 0x540(%rdx), %ymm4+        vmovdqa 0x560(%rdx), %ymm5+        vmovdqa 0x2a0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x140(%rdi), %ymm8+        vmovdqa 0x160(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x140(%rdi)+        vmovdqa %ymm9, 0x160(%rdi)+        vmovdqa 0x580(%rsi), %ymm2+        vmovdqa 0x5a0(%rsi), %ymm3+        vmovdqa 0x580(%rdx), %ymm4+        vmovdqa 0x5a0(%rdx), %ymm5+        vmovdqa 0x2c0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x180(%rdi), %ymm8+        vmovdqa 0x1a0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x180(%rdi)+        vmovdqa %ymm9, 0x1a0(%rdi)+        vmovdqa 0x5c0(%rsi), %ymm2+        vmovdqa 0x5e0(%rsi), %ymm3+        vmovdqa 0x5c0(%rdx), %ymm4+        vmovdqa 0x5e0(%rdx), %ymm5+        vmovdqa 0x2e0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x1c0(%rdi), %ymm8+        vmovdqa 0x1e0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x1c0(%rdi)+        vmovdqa %ymm9, 0x1e0(%rdi)+        vmovdqa 0x600(%rsi), %ymm2+        vmovdqa 0x620(%rsi), %ymm3+        vmovdqa 0x600(%rdx), %ymm4+        vmovdqa 0x620(%rdx), %ymm5+        vmovdqa 0x300(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa (%rdi), %ymm8+        vmovdqa 0x20(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, (%rdi)+        vmovdqa %ymm9, 0x20(%rdi)+        vmovdqa 0x640(%rsi), %ymm2+        vmovdqa 0x660(%rsi), %ymm3+        vmovdqa 0x640(%rdx), %ymm4+        vmovdqa 0x660(%rdx), %ymm5+        vmovdqa 0x320(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x40(%rdi), %ymm8+        vmovdqa 0x60(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x40(%rdi)+        vmovdqa %ymm9, 0x60(%rdi)+        vmovdqa 0x680(%rsi), %ymm2+        vmovdqa 0x6a0(%rsi), %ymm3+        vmovdqa 0x680(%rdx), %ymm4+        vmovdqa 0x6a0(%rdx), %ymm5+        vmovdqa 0x340(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x80(%rdi), %ymm8+        vmovdqa 0xa0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x80(%rdi)+        vmovdqa %ymm9, 0xa0(%rdi)+        vmovdqa 0x6c0(%rsi), %ymm2+        vmovdqa 0x6e0(%rsi), %ymm3+        vmovdqa 0x6c0(%rdx), %ymm4+        vmovdqa 0x6e0(%rdx), %ymm5+        vmovdqa 0x360(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0xc0(%rdi), %ymm8+        vmovdqa 0xe0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0xc0(%rdi)+        vmovdqa %ymm9, 0xe0(%rdi)+        vmovdqa 0x700(%rsi), %ymm2+        vmovdqa 0x720(%rsi), %ymm3+        vmovdqa 0x700(%rdx), %ymm4+        vmovdqa 0x720(%rdx), %ymm5+        vmovdqa 0x380(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x100(%rdi), %ymm8+        vmovdqa 0x120(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x100(%rdi)+        vmovdqa %ymm9, 0x120(%rdi)+        vmovdqa 0x740(%rsi), %ymm2+        vmovdqa 0x760(%rsi), %ymm3+        vmovdqa 0x740(%rdx), %ymm4+        vmovdqa 0x760(%rdx), %ymm5+        vmovdqa 0x3a0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x140(%rdi), %ymm8+        vmovdqa 0x160(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x140(%rdi)+        vmovdqa %ymm9, 0x160(%rdi)+        vmovdqa 0x780(%rsi), %ymm2+        vmovdqa 0x7a0(%rsi), %ymm3+        vmovdqa 0x780(%rdx), %ymm4+        vmovdqa 0x7a0(%rdx), %ymm5+        vmovdqa 0x3c0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm8, %ymm13, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x180(%rdi), %ymm8+        vmovdqa 0x1a0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x180(%rdi)+        vmovdqa %ymm9, 0x1a0(%rdi)+        vmovdqa 0x7c0(%rsi), %ymm2+        vmovdqa 0x7e0(%rsi), %ymm3+        vmovdqa 0x7c0(%rdx), %ymm4+        vmovdqa 0x7e0(%rdx), %ymm5+        vmovdqa 0x3e0(%rcx), %ymm6+        vpmullw %ymm2, %ymm1, %ymm13+        vpmullw %ymm3, %ymm1, %ymm14+        vpmullw %ymm13, %ymm4, %ymm7+        vpmullw %ymm13, %ymm5, %ymm9+        vpmullw %ymm14, %ymm6, %ymm8+        vpmullw %ymm14, %ymm4, %ymm10+        vpmulhw %ymm7, %ymm0, %ymm7+        vpmulhw %ymm9, %ymm0, %ymm9+        vpmulhw %ymm8, %ymm0, %ymm8+        vpmulhw %ymm10, %ymm0, %ymm10+        vpmulhw %ymm2, %ymm4, %ymm11+        vpmulhw %ymm2, %ymm5, %ymm12+        vpmulhw %ymm3, %ymm6, %ymm13+        vpmulhw %ymm3, %ymm4, %ymm14+        vpsubw %ymm7, %ymm11, %ymm7+        vpsubw %ymm9, %ymm12, %ymm9+        vpsubw %ymm13, %ymm8, %ymm8+        vpsubw %ymm10, %ymm14, %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa 0x1c0(%rdi), %ymm8+        vmovdqa 0x1e0(%rdi), %ymm10+        vpaddw %ymm7, %ymm8, %ymm7+        vpaddw %ymm9, %ymm10, %ymm9+        vmovdqa %ymm7, 0x1c0(%rdi)+        vmovdqa %ymm9, 0x1e0(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(polyvec_basemul_acc_montgomery_cached_k4_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+          && (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4) */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_reduce_avx2_asm.S view
@@ -0,0 +1,234 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Kyber optimized AVX2 implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/avx2+ */++/*+ * This file is derived from the public domain+ * AVX2 Kyber implementation @[REF_AVX2].+ *+ * Changes:+ * - Add call to csub in reduce128_avx to produce outputs+ *   in [0,1,...,q-1] rather than [0,1,...,q], matching the+ *   semantics of mlk_poly_reduce(),+ * - Use a macro instead of a local function call.+ */++#include "../../../common.h"++#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)+/*yaml+  Name: reduce_avx2_asm+  Description: x86_64 AVX2 modular reduction+  Signature: void mlk_reduce_avx2_asm(int16_t *r)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 512+      permissions: read/write+      c_parameter: int16_t *r+      description: Input/output polynomial (256 x int16_t)+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_reduce_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(reduce_avx2_asm)+MLK_ASM_FN_SYMBOL(reduce_avx2_asm)++        .cfi_startproc+        movl $0xd010d01, %eax        # imm = 0xD010D01+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        movl $0x4ebf4ebf, %eax       # imm = 0x4EBF4EBF+        vmovd %eax, %xmm1+        vpbroadcastd %xmm1, %ymm1+        vmovdqa (%rdi), %ymm2+        vmovdqa 0x20(%rdi), %ymm3+        vmovdqa 0x40(%rdi), %ymm4+        vmovdqa 0x60(%rdi), %ymm5+        vmovdqa 0x80(%rdi), %ymm6+        vmovdqa 0xa0(%rdi), %ymm7+        vmovdqa 0xc0(%rdi), %ymm8+        vmovdqa 0xe0(%rdi), %ymm9+        vpmulhw %ymm1, %ymm2, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm2, %ymm2+        vpmulhw %ymm1, %ymm3, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm3, %ymm3+        vpmulhw %ymm1, %ymm4, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm4, %ymm4+        vpmulhw %ymm1, %ymm5, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm5, %ymm5+        vpmulhw %ymm1, %ymm6, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm6, %ymm6+        vpmulhw %ymm1, %ymm7, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm7, %ymm7+        vpmulhw %ymm1, %ymm8, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm8, %ymm8+        vpmulhw %ymm1, %ymm9, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm9, %ymm9+        vpsubw %ymm0, %ymm2, %ymm2+        vpsraw $0xf, %ymm2, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpaddw %ymm12, %ymm2, %ymm2+        vpsubw %ymm0, %ymm3, %ymm3+        vpsraw $0xf, %ymm3, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpaddw %ymm12, %ymm3, %ymm3+        vpsubw %ymm0, %ymm4, %ymm4+        vpsraw $0xf, %ymm4, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpaddw %ymm12, %ymm4, %ymm4+        vpsubw %ymm0, %ymm5, %ymm5+        vpsraw $0xf, %ymm5, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpaddw %ymm12, %ymm5, %ymm5+        vpsubw %ymm0, %ymm6, %ymm6+        vpsraw $0xf, %ymm6, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpaddw %ymm12, %ymm6, %ymm6+        vpsubw %ymm0, %ymm7, %ymm7+        vpsraw $0xf, %ymm7, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpaddw %ymm12, %ymm7, %ymm7+        vpsubw %ymm0, %ymm8, %ymm8+        vpsraw $0xf, %ymm8, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpaddw %ymm12, %ymm8, %ymm8+        vpsubw %ymm0, %ymm9, %ymm9+        vpsraw $0xf, %ymm9, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpaddw %ymm12, %ymm9, %ymm9+        vmovdqa %ymm2, (%rdi)+        vmovdqa %ymm3, 0x20(%rdi)+        vmovdqa %ymm4, 0x40(%rdi)+        vmovdqa %ymm5, 0x60(%rdi)+        vmovdqa %ymm6, 0x80(%rdi)+        vmovdqa %ymm7, 0xa0(%rdi)+        vmovdqa %ymm8, 0xc0(%rdi)+        vmovdqa %ymm9, 0xe0(%rdi)+        vmovdqa 0x100(%rdi), %ymm2+        vmovdqa 0x120(%rdi), %ymm3+        vmovdqa 0x140(%rdi), %ymm4+        vmovdqa 0x160(%rdi), %ymm5+        vmovdqa 0x180(%rdi), %ymm6+        vmovdqa 0x1a0(%rdi), %ymm7+        vmovdqa 0x1c0(%rdi), %ymm8+        vmovdqa 0x1e0(%rdi), %ymm9+        vpmulhw %ymm1, %ymm2, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm2, %ymm2+        vpmulhw %ymm1, %ymm3, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm3, %ymm3+        vpmulhw %ymm1, %ymm4, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm4, %ymm4+        vpmulhw %ymm1, %ymm5, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm5, %ymm5+        vpmulhw %ymm1, %ymm6, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm6, %ymm6+        vpmulhw %ymm1, %ymm7, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm7, %ymm7+        vpmulhw %ymm1, %ymm8, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm8, %ymm8+        vpmulhw %ymm1, %ymm9, %ymm12+        vpsraw $0xa, %ymm12, %ymm12+        vpmullw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm9, %ymm9+        vpsubw %ymm0, %ymm2, %ymm2+        vpsraw $0xf, %ymm2, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpaddw %ymm12, %ymm2, %ymm2+        vpsubw %ymm0, %ymm3, %ymm3+        vpsraw $0xf, %ymm3, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpaddw %ymm12, %ymm3, %ymm3+        vpsubw %ymm0, %ymm4, %ymm4+        vpsraw $0xf, %ymm4, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpaddw %ymm12, %ymm4, %ymm4+        vpsubw %ymm0, %ymm5, %ymm5+        vpsraw $0xf, %ymm5, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpaddw %ymm12, %ymm5, %ymm5+        vpsubw %ymm0, %ymm6, %ymm6+        vpsraw $0xf, %ymm6, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpaddw %ymm12, %ymm6, %ymm6+        vpsubw %ymm0, %ymm7, %ymm7+        vpsraw $0xf, %ymm7, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpaddw %ymm12, %ymm7, %ymm7+        vpsubw %ymm0, %ymm8, %ymm8+        vpsraw $0xf, %ymm8, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpaddw %ymm12, %ymm8, %ymm8+        vpsubw %ymm0, %ymm9, %ymm9+        vpsraw $0xf, %ymm9, %ymm12+        vpand %ymm0, %ymm12, %ymm12+        vpaddw %ymm12, %ymm9, %ymm9+        vmovdqa %ymm2, 0x100(%rdi)+        vmovdqa %ymm3, 0x120(%rdi)+        vmovdqa %ymm4, 0x140(%rdi)+        vmovdqa %ymm5, 0x160(%rdi)+        vmovdqa %ymm6, 0x180(%rdi)+        vmovdqa %ymm7, 0x1a0(%rdi)+        vmovdqa %ymm8, 0x1c0(%rdi)+        vmovdqa %ymm9, 0x1e0(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(reduce_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_rej_uniform_avx2_asm.S view
@@ -0,0 +1,136 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*************************************************+ * Name:        mlk_rej_uniform_avx2_asm+ *+ * Description: Run rejection sampling on uniform random bytes to generate+ *              uniform random integers mod q+ *+ * Arguments:   - int16_t *r:          pointer to output buffer of MLKEM_N+ *                                     16-bit coefficients.+ *              - const uint8_t *buf:  pointer to input buffer+ *                                     (assumed to be uniform random bytes)+ *              - unsigned buflen:     length of input buffer in bytes.+ *                                     Must be a multiple of 12.+ *+ * Returns number of sampled 16-bit integers (at most MLKEM_N).+ **************************************************/+#include "../../../common.h"++#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)+/*yaml+  Name: rej_uniform_avx2_asm+  Description: x86_64 AVX2 rejection sampling+  Signature: uint64_t mlk_rej_uniform_avx2_asm(int16_t *r, const uint8_t *buf, unsigned buflen, const uint8_t *table)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 512+      permissions: write-only+      c_parameter: int16_t *r+      description: Output buffer (256 x int16_t)+    rsi:+      type: buffer+      size_bytes: rdx+      permissions: read-only+      c_parameter: const uint8_t *buf+      description: Input buffer+    rdx:+      type: scalar+      c_parameter: unsigned buflen+      description: Length of input buffer (must be multiple of 12)+      test_with: 504  # MLKEM_GEN_MATRIX_NBLOCKS * MLK_XOF_RATE+    rcx:+      type: buffer+      size_bytes: 4096+      permissions: read-only+      c_parameter: const uint8_t *table+      description: Lookup table+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_rej_uniform_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(rej_uniform_avx2_asm)+MLK_ASM_FN_SYMBOL(rej_uniform_avx2_asm)++        .cfi_startproc+        subq $0x210, %rsp            # imm = 0x210+        .cfi_adjust_cfa_offset 0x210+        xorl %eax, %eax+        testq %rdx, %rdx+        je Lmlk_rej_uniform_asm_end+        movabsq $0xd010d010d010d01, %rax # imm = 0xD010D010D010D01+        movq %rax, %xmm0+        pinsrq $0x1, %rax, %xmm0+        movabsq $0xfff0fff0fff0fff, %rax # imm = 0xFFF0FFF0FFF0FFF+        movq %rax, %xmm5+        pinsrq $0x1, %rax, %xmm5+        movabsq $0x504040302010100, %rax # imm = 0x504040302010100+        movq %rax, %xmm4+        movabsq $0xb0a0a0908070706, %rax # imm = 0xB0A0A0908070706+        pinsrq $0x1, %rax, %xmm4+        movq $0x0, %rax+        movq $0x0, %r8+        movq $0x5555, %r9            # imm = 0x5555++Lmlk_rej_uniform_asm_loop_start:+        movq (%rsi,%r8), %xmm2+        pinsrd $0x2, 0x8(%rsi,%r8), %xmm2+        pshufb %xmm4, %xmm2+        movdqa %xmm2, %xmm3+        psrlw $0x4, %xmm3+        pblendw $0xaa, %xmm3, %xmm2     # xmm2 = xmm2[0],xmm3[1],xmm2[2],xmm3[3],xmm2[4],xmm3[5],xmm2[6],xmm3[7]+        pand %xmm5, %xmm2+        movdqa %xmm0, %xmm1+        pcmpgtw %xmm2, %xmm1+        pmovmskb %xmm1, %r11d+        pextq %r9, %r11, %r11+        movq %r11, %r10+        shlq $0x4, %r10+        movdqu (%rcx,%r10), %xmm3+        pshufb %xmm3, %xmm2+        movdqu %xmm2, (%rsp,%rax,2)+        popcntq %r11, %r11+        addq %r11, %rax+        cmpq $0x100, %rax            # imm = 0x100+        jae Lmlk_rej_uniform_asm_final_copy+        addq $0xc, %r8+        cmpq %r8, %rdx+        ja Lmlk_rej_uniform_asm_loop_start++Lmlk_rej_uniform_asm_final_copy:+        movq $0x100, %rcx            # imm = 0x100+        cmpq $0x100, %rax            # imm = 0x100+        cmovaq %rcx, %rax+        movq %rsp, %rsi+        movq %rax, %rcx+        shlq %rcx+        rep  movsb (%rsi), %es:(%rdi)++Lmlk_rej_uniform_asm_end:+        addq $0x210, %rsp            # imm = 0x210+        .cfi_adjust_cfa_offset -0x210+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(rej_uniform_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+        */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/mlkem_tomont_avx2_asm.S view
@@ -0,0 +1,172 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [REF_AVX2]+ *   CRYSTALS-Kyber optimized AVX2 implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/avx2+ */++/*+ * Implementation from Kyber reference repository @[REF_AVX2]+ *+ * Changes:+ * - Add call to csub in reduce128_avx to produce outputs+ *   in [0,1,...,q-1] rather than [0,1,...,q], matching the+ *   semantics of mlk_poly_reduce(),+ * - Use a macro instead of a local function call.+ */++#include "../../../common.h"+#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \+    !defined(MLK_CONFIG_NO_KEYPAIR_API)+/*yaml+  Name: tomont_avx2_asm+  Description: x86_64 AVX2 Montgomery conversion+  Signature: void mlk_tomont_avx2_asm(int16_t *r)+  ABI:+    Architecture: x86_64+    CallingConvention: SysV+    Features: [AVX2]+    rdi:+      type: buffer+      size_bytes: 512+      permissions: read/write+      c_parameter: int16_t *r+      description: Input/output polynomial (256 x int16_t)+*/+++/*+ * WARNING: This file is auto-derived from the mlkem-native source file+ *   dev/x86_64/src/mlkem_tomont_avx2_asm.S using scripts/simpasm. Do not modify it directly.+ */++.text+.balign 4+.global MLK_ASM_NAMESPACE(tomont_avx2_asm)+MLK_ASM_FN_SYMBOL(tomont_avx2_asm)++        .cfi_startproc+        movl $0xd010d01, %eax        # imm = 0xD010D01+        vmovd %eax, %xmm0+        vpbroadcastd %xmm0, %ymm0+        movl $0x50495049, %eax       # imm = 0x50495049+        vmovd %eax, %xmm1+        vpbroadcastd %xmm1, %ymm1+        movl $0x5490549, %eax        # imm = 0x5490549+        vmovd %eax, %xmm2+        vpbroadcastd %xmm2, %ymm2+        vmovdqa (%rdi), %ymm3+        vmovdqa 0x20(%rdi), %ymm4+        vmovdqa 0x40(%rdi), %ymm5+        vmovdqa 0x60(%rdi), %ymm6+        vmovdqa 0x80(%rdi), %ymm7+        vmovdqa 0xa0(%rdi), %ymm8+        vmovdqa 0xc0(%rdi), %ymm9+        vmovdqa 0xe0(%rdi), %ymm10+        vpmullw %ymm1, %ymm3, %ymm11+        vpmulhw %ymm2, %ymm3, %ymm3+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm11, %ymm3, %ymm3+        vpmullw %ymm1, %ymm4, %ymm12+        vpmulhw %ymm2, %ymm4, %ymm4+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm4, %ymm4+        vpmullw %ymm1, %ymm5, %ymm13+        vpmulhw %ymm2, %ymm5, %ymm5+        vpmulhw %ymm0, %ymm13, %ymm13+        vpsubw %ymm13, %ymm5, %ymm5+        vpmullw %ymm1, %ymm6, %ymm14+        vpmulhw %ymm2, %ymm6, %ymm6+        vpmulhw %ymm0, %ymm14, %ymm14+        vpsubw %ymm14, %ymm6, %ymm6+        vpmullw %ymm1, %ymm7, %ymm15+        vpmulhw %ymm2, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm15, %ymm15+        vpsubw %ymm15, %ymm7, %ymm7+        vpmullw %ymm1, %ymm8, %ymm11+        vpmulhw %ymm2, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm11, %ymm8, %ymm8+        vpmullw %ymm1, %ymm9, %ymm12+        vpmulhw %ymm2, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm9, %ymm9+        vpmullw %ymm1, %ymm10, %ymm13+        vpmulhw %ymm2, %ymm10, %ymm10+        vpmulhw %ymm0, %ymm13, %ymm13+        vpsubw %ymm13, %ymm10, %ymm10+        vmovdqa %ymm3, (%rdi)+        vmovdqa %ymm4, 0x20(%rdi)+        vmovdqa %ymm5, 0x40(%rdi)+        vmovdqa %ymm6, 0x60(%rdi)+        vmovdqa %ymm7, 0x80(%rdi)+        vmovdqa %ymm8, 0xa0(%rdi)+        vmovdqa %ymm9, 0xc0(%rdi)+        vmovdqa %ymm10, 0xe0(%rdi)+        vmovdqa 0x100(%rdi), %ymm3+        vmovdqa 0x120(%rdi), %ymm4+        vmovdqa 0x140(%rdi), %ymm5+        vmovdqa 0x160(%rdi), %ymm6+        vmovdqa 0x180(%rdi), %ymm7+        vmovdqa 0x1a0(%rdi), %ymm8+        vmovdqa 0x1c0(%rdi), %ymm9+        vmovdqa 0x1e0(%rdi), %ymm10+        vpmullw %ymm1, %ymm3, %ymm11+        vpmulhw %ymm2, %ymm3, %ymm3+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm11, %ymm3, %ymm3+        vpmullw %ymm1, %ymm4, %ymm12+        vpmulhw %ymm2, %ymm4, %ymm4+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm4, %ymm4+        vpmullw %ymm1, %ymm5, %ymm13+        vpmulhw %ymm2, %ymm5, %ymm5+        vpmulhw %ymm0, %ymm13, %ymm13+        vpsubw %ymm13, %ymm5, %ymm5+        vpmullw %ymm1, %ymm6, %ymm14+        vpmulhw %ymm2, %ymm6, %ymm6+        vpmulhw %ymm0, %ymm14, %ymm14+        vpsubw %ymm14, %ymm6, %ymm6+        vpmullw %ymm1, %ymm7, %ymm15+        vpmulhw %ymm2, %ymm7, %ymm7+        vpmulhw %ymm0, %ymm15, %ymm15+        vpsubw %ymm15, %ymm7, %ymm7+        vpmullw %ymm1, %ymm8, %ymm11+        vpmulhw %ymm2, %ymm8, %ymm8+        vpmulhw %ymm0, %ymm11, %ymm11+        vpsubw %ymm11, %ymm8, %ymm8+        vpmullw %ymm1, %ymm9, %ymm12+        vpmulhw %ymm2, %ymm9, %ymm9+        vpmulhw %ymm0, %ymm12, %ymm12+        vpsubw %ymm12, %ymm9, %ymm9+        vpmullw %ymm1, %ymm10, %ymm13+        vpmulhw %ymm2, %ymm10, %ymm10+        vpmulhw %ymm0, %ymm13, %ymm13+        vpsubw %ymm13, %ymm10, %ymm10+        vmovdqa %ymm3, 0x100(%rdi)+        vmovdqa %ymm4, 0x120(%rdi)+        vmovdqa %ymm5, 0x140(%rdi)+        vmovdqa %ymm6, 0x160(%rdi)+        vmovdqa %ymm7, 0x180(%rdi)+        vmovdqa %ymm8, 0x1a0(%rdi)+        vmovdqa %ymm9, 0x1c0(%rdi)+        vmovdqa %ymm10, 0x1e0(%rdi)+        retq+        .cfi_endproc++MLK_ASM_FN_SIZE(tomont_avx2_asm)++#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+          && !MLK_CONFIG_NO_KEYPAIR_API */++#if defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/mlkem/src/native/x86_64/src/rej_uniform_table.c view
@@ -0,0 +1,545 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mlkem-native repository.+ *          Do not modify it directly.+ */++#include "../../../common.h"++#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++#include "arith_native_x86_64.h"++/*+ * Lookup table used by rejection sampling of the public matrix.+ * See autogen for details.+ */+MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t+    mlk_rej_uniform_table[4096] = {+        255, 255, 255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 0 */,+        0,   1,   255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 1 */,+        2,   3,   255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 2 */,+        0,   1,   2,   3,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 3 */,+        4,   5,   255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 4 */,+        0,   1,   4,   5,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 5 */,+        2,   3,   4,   5,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 6 */,+        0,   1,   2,   3,   4,   5,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 7 */,+        6,   7,   255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 8 */,+        0,   1,   6,   7,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 9 */,+        2,   3,   6,   7,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 10 */,+        0,   1,   2,   3,   6,   7,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 11 */,+        4,   5,   6,   7,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 12 */,+        0,   1,   4,   5,   6,   7,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 13 */,+        2,   3,   4,   5,   6,   7,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 14 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        255, 255, 255, 255, 255, 255, 255, 255 /* 15 */,+        8,   9,   255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 16 */,+        0,   1,   8,   9,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 17 */,+        2,   3,   8,   9,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 18 */,+        0,   1,   2,   3,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 19 */,+        4,   5,   8,   9,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 20 */,+        0,   1,   4,   5,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 21 */,+        2,   3,   4,   5,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 22 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        255, 255, 255, 255, 255, 255, 255, 255 /* 23 */,+        6,   7,   8,   9,   255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 24 */,+        0,   1,   6,   7,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 25 */,+        2,   3,   6,   7,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 26 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        255, 255, 255, 255, 255, 255, 255, 255 /* 27 */,+        4,   5,   6,   7,   8,   9,   255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 28 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        255, 255, 255, 255, 255, 255, 255, 255 /* 29 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        255, 255, 255, 255, 255, 255, 255, 255 /* 30 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   255, 255, 255, 255, 255, 255 /* 31 */,+        10,  11,  255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 32 */,+        0,   1,   10,  11,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 33 */,+        2,   3,   10,  11,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 34 */,+        0,   1,   2,   3,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 35 */,+        4,   5,   10,  11,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 36 */,+        0,   1,   4,   5,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 37 */,+        2,   3,   4,   5,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 38 */,+        0,   1,   2,   3,   4,   5,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 39 */,+        6,   7,   10,  11,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 40 */,+        0,   1,   6,   7,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 41 */,+        2,   3,   6,   7,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 42 */,+        0,   1,   2,   3,   6,   7,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 43 */,+        4,   5,   6,   7,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 44 */,+        0,   1,   4,   5,   6,   7,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 45 */,+        2,   3,   4,   5,   6,   7,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 46 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        10,  11,  255, 255, 255, 255, 255, 255 /* 47 */,+        8,   9,   10,  11,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 48 */,+        0,   1,   8,   9,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 49 */,+        2,   3,   8,   9,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 50 */,+        0,   1,   2,   3,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 51 */,+        4,   5,   8,   9,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 52 */,+        0,   1,   4,   5,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 53 */,+        2,   3,   4,   5,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 54 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        10,  11,  255, 255, 255, 255, 255, 255 /* 55 */,+        6,   7,   8,   9,   10,  11,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 56 */,+        0,   1,   6,   7,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 57 */,+        2,   3,   6,   7,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 58 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        10,  11,  255, 255, 255, 255, 255, 255 /* 59 */,+        4,   5,   6,   7,   8,   9,   10,  11,+        255, 255, 255, 255, 255, 255, 255, 255 /* 60 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        10,  11,  255, 255, 255, 255, 255, 255 /* 61 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        10,  11,  255, 255, 255, 255, 255, 255 /* 62 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   10,  11,  255, 255, 255, 255 /* 63 */,+        12,  13,  255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 64 */,+        0,   1,   12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 65 */,+        2,   3,   12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 66 */,+        0,   1,   2,   3,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 67 */,+        4,   5,   12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 68 */,+        0,   1,   4,   5,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 69 */,+        2,   3,   4,   5,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 70 */,+        0,   1,   2,   3,   4,   5,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 71 */,+        6,   7,   12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 72 */,+        0,   1,   6,   7,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 73 */,+        2,   3,   6,   7,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 74 */,+        0,   1,   2,   3,   6,   7,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 75 */,+        4,   5,   6,   7,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 76 */,+        0,   1,   4,   5,   6,   7,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 77 */,+        2,   3,   4,   5,   6,   7,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 78 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        12,  13,  255, 255, 255, 255, 255, 255 /* 79 */,+        8,   9,   12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 80 */,+        0,   1,   8,   9,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 81 */,+        2,   3,   8,   9,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 82 */,+        0,   1,   2,   3,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 83 */,+        4,   5,   8,   9,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 84 */,+        0,   1,   4,   5,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 85 */,+        2,   3,   4,   5,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 86 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        12,  13,  255, 255, 255, 255, 255, 255 /* 87 */,+        6,   7,   8,   9,   12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 88 */,+        0,   1,   6,   7,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 89 */,+        2,   3,   6,   7,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 90 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        12,  13,  255, 255, 255, 255, 255, 255 /* 91 */,+        4,   5,   6,   7,   8,   9,   12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 92 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        12,  13,  255, 255, 255, 255, 255, 255 /* 93 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        12,  13,  255, 255, 255, 255, 255, 255 /* 94 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   12,  13,  255, 255, 255, 255 /* 95 */,+        10,  11,  12,  13,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 96 */,+        0,   1,   10,  11,  12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 97 */,+        2,   3,   10,  11,  12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 98 */,+        0,   1,   2,   3,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 99 */,+        4,   5,   10,  11,  12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 100 */,+        0,   1,   4,   5,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 101 */,+        2,   3,   4,   5,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 102 */,+        0,   1,   2,   3,   4,   5,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 103 */,+        6,   7,   10,  11,  12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 104 */,+        0,   1,   6,   7,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 105 */,+        2,   3,   6,   7,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 106 */,+        0,   1,   2,   3,   6,   7,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 107 */,+        4,   5,   6,   7,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 108 */,+        0,   1,   4,   5,   6,   7,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 109 */,+        2,   3,   4,   5,   6,   7,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 110 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        10,  11,  12,  13,  255, 255, 255, 255 /* 111 */,+        8,   9,   10,  11,  12,  13,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 112 */,+        0,   1,   8,   9,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 113 */,+        2,   3,   8,   9,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 114 */,+        0,   1,   2,   3,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 115 */,+        4,   5,   8,   9,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 116 */,+        0,   1,   4,   5,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 117 */,+        2,   3,   4,   5,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 118 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        10,  11,  12,  13,  255, 255, 255, 255 /* 119 */,+        6,   7,   8,   9,   10,  11,  12,  13,+        255, 255, 255, 255, 255, 255, 255, 255 /* 120 */,+        0,   1,   6,   7,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 121 */,+        2,   3,   6,   7,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 122 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        10,  11,  12,  13,  255, 255, 255, 255 /* 123 */,+        4,   5,   6,   7,   8,   9,   10,  11,+        12,  13,  255, 255, 255, 255, 255, 255 /* 124 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        10,  11,  12,  13,  255, 255, 255, 255 /* 125 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        10,  11,  12,  13,  255, 255, 255, 255 /* 126 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   10,  11,  12,  13,  255, 255 /* 127 */,+        14,  15,  255, 255, 255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 128 */,+        0,   1,   14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 129 */,+        2,   3,   14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 130 */,+        0,   1,   2,   3,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 131 */,+        4,   5,   14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 132 */,+        0,   1,   4,   5,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 133 */,+        2,   3,   4,   5,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 134 */,+        0,   1,   2,   3,   4,   5,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 135 */,+        6,   7,   14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 136 */,+        0,   1,   6,   7,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 137 */,+        2,   3,   6,   7,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 138 */,+        0,   1,   2,   3,   6,   7,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 139 */,+        4,   5,   6,   7,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 140 */,+        0,   1,   4,   5,   6,   7,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 141 */,+        2,   3,   4,   5,   6,   7,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 142 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        14,  15,  255, 255, 255, 255, 255, 255 /* 143 */,+        8,   9,   14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 144 */,+        0,   1,   8,   9,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 145 */,+        2,   3,   8,   9,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 146 */,+        0,   1,   2,   3,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 147 */,+        4,   5,   8,   9,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 148 */,+        0,   1,   4,   5,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 149 */,+        2,   3,   4,   5,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 150 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        14,  15,  255, 255, 255, 255, 255, 255 /* 151 */,+        6,   7,   8,   9,   14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 152 */,+        0,   1,   6,   7,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 153 */,+        2,   3,   6,   7,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 154 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        14,  15,  255, 255, 255, 255, 255, 255 /* 155 */,+        4,   5,   6,   7,   8,   9,   14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 156 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        14,  15,  255, 255, 255, 255, 255, 255 /* 157 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        14,  15,  255, 255, 255, 255, 255, 255 /* 158 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   14,  15,  255, 255, 255, 255 /* 159 */,+        10,  11,  14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 160 */,+        0,   1,   10,  11,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 161 */,+        2,   3,   10,  11,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 162 */,+        0,   1,   2,   3,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 163 */,+        4,   5,   10,  11,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 164 */,+        0,   1,   4,   5,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 165 */,+        2,   3,   4,   5,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 166 */,+        0,   1,   2,   3,   4,   5,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 167 */,+        6,   7,   10,  11,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 168 */,+        0,   1,   6,   7,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 169 */,+        2,   3,   6,   7,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 170 */,+        0,   1,   2,   3,   6,   7,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 171 */,+        4,   5,   6,   7,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 172 */,+        0,   1,   4,   5,   6,   7,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 173 */,+        2,   3,   4,   5,   6,   7,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 174 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        10,  11,  14,  15,  255, 255, 255, 255 /* 175 */,+        8,   9,   10,  11,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 176 */,+        0,   1,   8,   9,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 177 */,+        2,   3,   8,   9,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 178 */,+        0,   1,   2,   3,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 179 */,+        4,   5,   8,   9,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 180 */,+        0,   1,   4,   5,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 181 */,+        2,   3,   4,   5,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 182 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        10,  11,  14,  15,  255, 255, 255, 255 /* 183 */,+        6,   7,   8,   9,   10,  11,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 184 */,+        0,   1,   6,   7,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 185 */,+        2,   3,   6,   7,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 186 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        10,  11,  14,  15,  255, 255, 255, 255 /* 187 */,+        4,   5,   6,   7,   8,   9,   10,  11,+        14,  15,  255, 255, 255, 255, 255, 255 /* 188 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        10,  11,  14,  15,  255, 255, 255, 255 /* 189 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        10,  11,  14,  15,  255, 255, 255, 255 /* 190 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   10,  11,  14,  15,  255, 255 /* 191 */,+        12,  13,  14,  15,  255, 255, 255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 192 */,+        0,   1,   12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 193 */,+        2,   3,   12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 194 */,+        0,   1,   2,   3,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 195 */,+        4,   5,   12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 196 */,+        0,   1,   4,   5,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 197 */,+        2,   3,   4,   5,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 198 */,+        0,   1,   2,   3,   4,   5,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 199 */,+        6,   7,   12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 200 */,+        0,   1,   6,   7,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 201 */,+        2,   3,   6,   7,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 202 */,+        0,   1,   2,   3,   6,   7,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 203 */,+        4,   5,   6,   7,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 204 */,+        0,   1,   4,   5,   6,   7,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 205 */,+        2,   3,   4,   5,   6,   7,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 206 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        12,  13,  14,  15,  255, 255, 255, 255 /* 207 */,+        8,   9,   12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 208 */,+        0,   1,   8,   9,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 209 */,+        2,   3,   8,   9,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 210 */,+        0,   1,   2,   3,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 211 */,+        4,   5,   8,   9,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 212 */,+        0,   1,   4,   5,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 213 */,+        2,   3,   4,   5,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 214 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        12,  13,  14,  15,  255, 255, 255, 255 /* 215 */,+        6,   7,   8,   9,   12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 216 */,+        0,   1,   6,   7,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 217 */,+        2,   3,   6,   7,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 218 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        12,  13,  14,  15,  255, 255, 255, 255 /* 219 */,+        4,   5,   6,   7,   8,   9,   12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 220 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        12,  13,  14,  15,  255, 255, 255, 255 /* 221 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        12,  13,  14,  15,  255, 255, 255, 255 /* 222 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   12,  13,  14,  15,  255, 255 /* 223 */,+        10,  11,  12,  13,  14,  15,  255, 255,+        255, 255, 255, 255, 255, 255, 255, 255 /* 224 */,+        0,   1,   10,  11,  12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 225 */,+        2,   3,   10,  11,  12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 226 */,+        0,   1,   2,   3,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 227 */,+        4,   5,   10,  11,  12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 228 */,+        0,   1,   4,   5,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 229 */,+        2,   3,   4,   5,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 230 */,+        0,   1,   2,   3,   4,   5,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 231 */,+        6,   7,   10,  11,  12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 232 */,+        0,   1,   6,   7,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 233 */,+        2,   3,   6,   7,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 234 */,+        0,   1,   2,   3,   6,   7,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 235 */,+        4,   5,   6,   7,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 236 */,+        0,   1,   4,   5,   6,   7,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 237 */,+        2,   3,   4,   5,   6,   7,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 238 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        10,  11,  12,  13,  14,  15,  255, 255 /* 239 */,+        8,   9,   10,  11,  12,  13,  14,  15,+        255, 255, 255, 255, 255, 255, 255, 255 /* 240 */,+        0,   1,   8,   9,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 241 */,+        2,   3,   8,   9,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 242 */,+        0,   1,   2,   3,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 243 */,+        4,   5,   8,   9,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 244 */,+        0,   1,   4,   5,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 245 */,+        2,   3,   4,   5,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 246 */,+        0,   1,   2,   3,   4,   5,   8,   9,+        10,  11,  12,  13,  14,  15,  255, 255 /* 247 */,+        6,   7,   8,   9,   10,  11,  12,  13,+        14,  15,  255, 255, 255, 255, 255, 255 /* 248 */,+        0,   1,   6,   7,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 249 */,+        2,   3,   6,   7,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 250 */,+        0,   1,   2,   3,   6,   7,   8,   9,+        10,  11,  12,  13,  14,  15,  255, 255 /* 251 */,+        4,   5,   6,   7,   8,   9,   10,  11,+        12,  13,  14,  15,  255, 255, 255, 255 /* 252 */,+        0,   1,   4,   5,   6,   7,   8,   9,+        10,  11,  12,  13,  14,  15,  255, 255 /* 253 */,+        2,   3,   4,   5,   6,   7,   8,   9,+        10,  11,  12,  13,  14,  15,  255, 255 /* 254 */,+        0,   1,   2,   3,   4,   5,   6,   7,+        8,   9,   10,  11,  12,  13,  14,  15 /* 255 */,+};++#else /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \+       */++MLK_EMPTY_CU(avx2_rej_uniform_table)++#endif /* !(MLK_ARITH_BACKEND_X86_64_DEFAULT && \+          !MLK_CONFIG_MULTILEVEL_NO_SHARED) */
+ cbits/mlkem/src/params.h view
@@ -0,0 +1,76 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLK_PARAMS_H+#define MLK_PARAMS_H++#if !defined(MLK_CONFIG_PARAMETER_SET)+#error MLK_CONFIG_PARAMETER_SET is not defined+#endif++#if MLK_CONFIG_PARAMETER_SET == 512+#define MLKEM_K 2+#elif MLK_CONFIG_PARAMETER_SET == 768+#define MLKEM_K 3+#elif MLK_CONFIG_PARAMETER_SET == 1024+#define MLKEM_K 4+#else+#error Invalid value for MLK_CONFIG_PARAMETER_SET. Must be 512, 768, or 1024.+#endif++#define MLKEM_N 256+#define MLKEM_Q 3329+#define MLKEM_Q_HALF ((MLKEM_Q + 1) / 2) /* 1665 */+#define MLKEM_UINT12_LIMIT 4096++#define MLKEM_SYMBYTES 32 /* size in bytes of hashes, and seeds */+#define MLKEM_SSBYTES 32  /* size in bytes of shared key */++#define MLKEM_POLYBYTES 384+#define MLKEM_POLYVECBYTES (MLKEM_K * MLKEM_POLYBYTES)++#define MLKEM_POLYCOMPRESSEDBYTES_D4 128+#define MLKEM_POLYCOMPRESSEDBYTES_D5 160+#define MLKEM_POLYCOMPRESSEDBYTES_D10 320+#define MLKEM_POLYCOMPRESSEDBYTES_D11 352++#if MLKEM_K == 2+#define MLKEM_ETA1 3+#define MLKEM_DU 10+#define MLKEM_DV 4+#define MLKEM_POLYCOMPRESSEDBYTES_DV MLKEM_POLYCOMPRESSEDBYTES_D4+#define MLKEM_POLYCOMPRESSEDBYTES_DU MLKEM_POLYCOMPRESSEDBYTES_D10+#define MLKEM_POLYVECCOMPRESSEDBYTES_DU (MLKEM_K * MLKEM_POLYCOMPRESSEDBYTES_DU)+#elif MLKEM_K == 3+#define MLKEM_ETA1 2+#define MLKEM_DU 10+#define MLKEM_DV 4+#define MLKEM_POLYCOMPRESSEDBYTES_DV MLKEM_POLYCOMPRESSEDBYTES_D4+#define MLKEM_POLYCOMPRESSEDBYTES_DU MLKEM_POLYCOMPRESSEDBYTES_D10+#define MLKEM_POLYVECCOMPRESSEDBYTES_DU (MLKEM_K * MLKEM_POLYCOMPRESSEDBYTES_DU)+#elif MLKEM_K == 4+#define MLKEM_ETA1 2+#define MLKEM_DU 11+#define MLKEM_DV 5+#define MLKEM_POLYCOMPRESSEDBYTES_DV MLKEM_POLYCOMPRESSEDBYTES_D5+#define MLKEM_POLYCOMPRESSEDBYTES_DU MLKEM_POLYCOMPRESSEDBYTES_D11+#define MLKEM_POLYVECCOMPRESSEDBYTES_DU (MLKEM_K * MLKEM_POLYCOMPRESSEDBYTES_DU)+#endif /* MLKEM_K == 4 */++#define MLKEM_ETA2 2++#define MLKEM_INDCPA_MSGBYTES (MLKEM_SYMBYTES)+#define MLKEM_INDCPA_PUBLICKEYBYTES (MLKEM_POLYVECBYTES + MLKEM_SYMBYTES)+#define MLKEM_INDCPA_SECRETKEYBYTES (MLKEM_POLYVECBYTES)+#define MLKEM_INDCPA_BYTES \+  (MLKEM_POLYVECCOMPRESSEDBYTES_DU + MLKEM_POLYCOMPRESSEDBYTES_DV)++#define MLKEM_INDCCA_PUBLICKEYBYTES (MLKEM_INDCPA_PUBLICKEYBYTES)+/* 32 bytes of additional space to save H(pk) */+#define MLKEM_INDCCA_SECRETKEYBYTES                            \+  (MLKEM_INDCPA_SECRETKEYBYTES + MLKEM_INDCPA_PUBLICKEYBYTES + \+   2 * MLKEM_SYMBYTES)+#define MLKEM_INDCCA_CIPHERTEXTBYTES (MLKEM_INDCPA_BYTES)++#endif /* !MLK_PARAMS_H */
+ cbits/mlkem/src/poly.c view
@@ -0,0 +1,581 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [NeonNTT]+ *   Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1+ *   Becker, Hwang, Kannwischer, Yang, Yang+ *   https://eprint.iacr.org/2021/986+ *+ * - [REF]+ *   CRYSTALS-Kyber C reference implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/ref+ */++#include "common.h"+#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)+++#include "cbmc.h"+#include "debug.h"+#include "poly.h"+#include "sampling.h"+#include "symmetric.h"+#include "verify.h"++/**+ * Montgomery multiplication modulo MLKEM_Q.+ *+ * @reference{`fqmul()` in the reference implementation @[REF].}+ *+ * @param a First factor. Can be any int16_t.+ * @param b Second factor. Must be signed canonical+ *          (abs value < (MLKEM_Q+1)/2).+ *+ * @return 16-bit integer congruent to a*b*R^{-1} mod MLKEM_Q, and+ *         smaller than MLKEM_Q in absolute value.+ */+static MLK_INLINE int16_t mlk_fqmul(int16_t a, int16_t b)+__contract__(+  requires(b > -MLKEM_Q_HALF && b < MLKEM_Q_HALF)+  ensures(return_value > -MLKEM_Q && return_value < MLKEM_Q)+)+{+  int16_t res;+  mlk_assert_abs_bound(&b, 1, MLKEM_Q_HALF);++  res = mlk_montgomery_reduce((int32_t)a * (int32_t)b);+  /* Bounds:+   * |res| <= ceil(|a| * |b| / 2^16) + (MLKEM_Q + 1) / 2+   *       <= ceil(2^15 * ((MLKEM_Q - 1)/2) / 2^16) + (MLKEM_Q + 1) / 2+   *       <= ceil((MLKEM_Q - 1) / 4) + (MLKEM_Q + 1) / 2+   *        < MLKEM_Q+   */++  mlk_assert_abs_bound(&res, 1, MLKEM_Q);+  return res;+}++/**+ * Barrett reduction; given a 16-bit integer a, computes the centered+ * representative congruent to a mod MLKEM_Q in [-(MLKEM_Q-1)/2, (MLKEM_Q-1)/2].+ *+ * @reference{`barrett_reduce()` in the reference implementation @[REF].}+ *+ * @param a Input integer to be reduced.+ *+ * @return Integer in [-(MLKEM_Q-1)/2, (MLKEM_Q-1)/2] congruent to @p a modulo+ *         MLKEM_Q.+ */+static MLK_INLINE int16_t mlk_barrett_reduce(int16_t a)+__contract__(+  ensures(return_value > -MLKEM_Q_HALF && return_value < MLKEM_Q_HALF)+)+{+  /* Barrett reduction approximates+   * ```+   *     round(a/MLKEM_Q)+   *   = round(a*(2^N/MLKEM_Q))/2^N)+   *  ~= round(a*round(2^N/MLKEM_Q)/2^N)+   * ```+   * Here, we pick N=26.+   */+  const int32_t magic = 20159; /* check-magic: 20159 == round(2^26 / MLKEM_Q) */++  /*+   * PORTABILITY: Right-shift on a signed integer is+   * implementation-defined for negative left argument.+   * Here, we assume it's sign-preserving "arithmetic" shift right.+   * See (C99 6.5.7 (5))+   */+  const int32_t t = (magic * a + ((int32_t)1 << 25)) >> 26;++  /*+   * t is in -10 .. +10, so we need 32-bit math to+   * evaluate t * MLKEM_Q and the subsequent subtraction+   */+  int16_t res = (int16_t)(a - t * MLKEM_Q);++  mlk_assert_abs_bound(&res, 1, MLKEM_Q_HALF);+  return res;+}++#if !defined(MLK_CONFIG_NO_KEYPAIR_API)+/* Reference: `poly_tomont()` in the reference implementation @[REF]. */+MLK_STATIC_TESTABLE void mlk_poly_tomont_c(mlk_poly *r)+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_abs_bound(r->coeffs, 0, MLKEM_N, MLKEM_Q))+)+{+  unsigned i;+  const int16_t f = 1353; /* check-magic: 1353 == signed_mod(2^32, MLKEM_Q) */+  for (i = 0; i < MLKEM_N; i++)+  __loop__(+    invariant(i <= MLKEM_N)+    invariant(array_abs_bound(r->coeffs, 0, i, MLKEM_Q))+    decreases(MLKEM_N - i))+  {+    r->coeffs[i] = mlk_fqmul(r->coeffs[i], f);+  }++  mlk_assert_abs_bound(r, MLKEM_N, MLKEM_Q);+}++MLK_INTERNAL_API+void mlk_poly_tomont(mlk_poly *r)+{+#if defined(MLK_USE_NATIVE_POLY_TOMONT)+  int ret;+  ret = mlk_poly_tomont_native(r->coeffs);+  if (ret == MLK_NATIVE_FUNC_SUCCESS)+  {+    mlk_assert_abs_bound(r, MLKEM_N, MLKEM_Q);+    return;+  }+#endif /* MLK_USE_NATIVE_POLY_TOMONT */++  mlk_poly_tomont_c(r);+}+#endif /* !MLK_CONFIG_NO_KEYPAIR_API */++/**+ * Constant-time conversion of signed representatives modulo MLKEM_Q within+ * range [-(MLKEM_Q-1), MLKEM_Q-1] into unsigned representatives within+ * range [0, MLKEM_Q-1].+ *+ * @reference{Not present in the reference implementation @[REF]. Used here+ * to implement different semantics of `poly_reduce()`; see below. In the+ * reference implementation @[REF] this logic is part of all compression+ * functions (see `compress.c`).}+ *+ * @param c Signed coefficient to be converted.+ *+ * @return Unsigned representative in [0, MLKEM_Q).+ */+static MLK_INLINE int16_t mlk_scalar_signed_to_unsigned_q(int16_t c)+__contract__(+  requires(c > -MLKEM_Q && c < MLKEM_Q)+  ensures(return_value >= 0 && return_value < MLKEM_Q)+  ensures(return_value == (int32_t)c + (((int32_t)c < 0) * MLKEM_Q)))+{+  mlk_assert_abs_bound(&c, 1, MLKEM_Q);++  /* Add MLKEM_Q if c is negative, but in constant time.+   *+   * Note that c + MLKEM_Q does not overflow in int16_t,+   * so the cast to uint16_t is safe. */+  c = mlk_ct_sel_int16((int16_t)(c + MLKEM_Q), c, mlk_ct_cmask_neg_i16(c));++  mlk_assert_bound(&c, 1, 0, MLKEM_Q);+  return c;+}++/* Reference: `poly_reduce()` in the reference implementation @[REF]+ *            - We use _unsigned_ canonical outputs, while the reference+ *              implementation uses _signed_ canonical outputs.+ *              Accordingly, we need a conditional addition of MLKEM_Q+ *              here to go from signed to unsigned representatives.+ *              This conditional addition is then dropped from all+ *              polynomial compression functions instead (see `compress.c`). */+MLK_STATIC_TESTABLE void mlk_poly_reduce_c(mlk_poly *r)+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+)+{+  unsigned i;++  for (i = 0; i < MLKEM_N; i++)+  __loop__(+    invariant(i <= MLKEM_N)+    invariant(array_bound(r->coeffs, 0, i, 0, MLKEM_Q))+    decreases(MLKEM_N - i))+  {+    /* Barrett reduction, giving signed canonical representative */+    int16_t t = mlk_barrett_reduce(r->coeffs[i]);+    /* Conditional addition to get unsigned canonical representative */+    r->coeffs[i] = mlk_scalar_signed_to_unsigned_q(t);+  }++  mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q);+}++MLK_INTERNAL_API+void mlk_poly_reduce(mlk_poly *r)+{+#if defined(MLK_USE_NATIVE_POLY_REDUCE)+  int ret;+  ret = mlk_poly_reduce_native(r->coeffs);+  if (ret == MLK_NATIVE_FUNC_SUCCESS)+  {+    mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q);+    return;+  }+#endif /* MLK_USE_NATIVE_POLY_REDUCE */++  mlk_poly_reduce_c(r);+}++/* Reference: `poly_add()` in the reference implementation @[REF].+ *            - We use destructive version (output=first input) to avoid+ *              reasoning about aliasing in the CBMC specification */+MLK_INTERNAL_API+void mlk_poly_add(mlk_poly *r, const mlk_poly *b)+{+  unsigned i;+  for (i = 0; i < MLKEM_N; i++)+  __loop__(+    invariant(i <= MLKEM_N)+    invariant(forall(k0, i, MLKEM_N, r->coeffs[k0] == loop_entry(*r).coeffs[k0]))+    invariant(forall(k1, 0, i, r->coeffs[k1] == loop_entry(*r).coeffs[k1] + b->coeffs[k1]))+    decreases(MLKEM_N - i))+  {+    /* The preconditions imply that the addition stays within int16_t. */+    r->coeffs[i] = (int16_t)(r->coeffs[i] + b->coeffs[i]);+  }+}++#if !defined(MLK_CONFIG_NO_DECAPS_API)+/* Reference: `poly_sub()` in the reference implementation @[REF].+ *            - We use destructive version (output=first input) to avoid+ *              reasoning about aliasing in the CBMC specification */+MLK_INTERNAL_API+void mlk_poly_sub(mlk_poly *r, const mlk_poly *b)+{+  unsigned i;+  for (i = 0; i < MLKEM_N; i++)+  __loop__(+    invariant(i <= MLKEM_N)+    invariant(forall(k0, i, MLKEM_N, r->coeffs[k0] == loop_entry(*r).coeffs[k0]))+    invariant(forall(k1, 0, i, r->coeffs[k1] == loop_entry(*r).coeffs[k1] - b->coeffs[k1]))+    decreases(MLKEM_N - i))+  {+    /* The preconditions imply that the subtraction stays within int16_t. */+    r->coeffs[i] = (int16_t)(r->coeffs[i] - b->coeffs[i]);+  }+}+#endif /* !MLK_CONFIG_NO_DECAPS_API */++#include "zetas.inc"++/* Reference: Does not exist in the reference implementation @[REF].+ *            - The reference implementation does not use a+ *              multiplication cache ('mulcache'). This idea originates+ *              from @[NeonNTT] and is used at the C level here. */+MLK_STATIC_TESTABLE void mlk_poly_mulcache_compute_c(mlk_poly_mulcache *x,+                                                     const mlk_poly *a)+__contract__(+  requires(memory_no_alias(x, sizeof(mlk_poly_mulcache)))+  requires(memory_no_alias(a, sizeof(mlk_poly)))+  assigns(memory_slice(x, sizeof(mlk_poly_mulcache)))+)+{+  unsigned i;+  for (i = 0; i < MLKEM_N / 4; i++)+  __loop__(+    invariant(i <= MLKEM_N / 4)+    invariant(array_abs_bound(x->coeffs, 0, 2 * i, MLKEM_Q))+    decreases(MLKEM_N / 4 - i))+  {+    x->coeffs[2 * i + 0] = mlk_fqmul(a->coeffs[4 * i + 1], mlk_zetas[64 + i]);+    /* The values in zeta table are <= MLKEM_Q in absolute value,+     * so the negation in int16_t is safe. */+    x->coeffs[2 * i + 1] =+        mlk_fqmul(a->coeffs[4 * i + 3], (int16_t)(-mlk_zetas[64 + i]));+  }++  /*+   * This bound is true for the C implementation, but not needed+   * in the higher level bounds reasoning. It is thus omitted+   * from the spec to not unnecessarily constrain native+   * implementations, but checked here nonetheless.+   */+  mlk_assert_abs_bound(x, MLKEM_N / 2, MLKEM_Q);+}++MLK_INTERNAL_API+void mlk_poly_mulcache_compute(mlk_poly_mulcache *x, const mlk_poly *a)+{+#if defined(MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE)+  int ret;+  ret = mlk_poly_mulcache_compute_native(x->coeffs, a->coeffs);+  if (ret == MLK_NATIVE_FUNC_SUCCESS)+  {+    return;+  }+#endif /* MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE */++  mlk_poly_mulcache_compute_c(x, a);+}++/*+ * Computes a block CT butterflies with a fixed twiddle factor,+ * using Montgomery multiplication.+ * Parameters:+ * - r: Pointer to base of polynomial (_not_ the base of butterfly block)+ * - root: Twiddle factor to use for the butterfly. This must be in+ *         Montgomery form and signed canonical.+ * - start: Offset to the beginning of the butterfly block+ * - len: Index difference between coefficients subject to a butterfly+ * - bound: Ghost variable describing coefficient bound: Prior to `start`,+ *          coefficients must be bound by `bound + MLKEM_Q`. Post `start`,+ *          they must be bound by `bound`.+ * When this function returns, output coefficients in the index range+ * [start, start+2*len) have bound bumped to `bound + MLKEM_Q`.+ * Example:+ * - start=8, len=4+ *   This would compute the following four butterflies+ *          8     --    12+ *             9    --     13+ *                10   --     14+ *                   11   --     15+ * - start=4, len=2+ *   This would compute the following two butterflies+ *          4 -- 6+ *             5 -- 7+ */++/* Reference: Embedded in `ntt()` in the reference implementation @[REF]. */+static void mlk_ntt_butterfly_block(int16_t r[MLKEM_N], int16_t zeta,+                                    unsigned start, unsigned len,+                                    unsigned bound)+__contract__(+  requires(start < MLKEM_N)+  requires(1 <= len && len <= MLKEM_N / 2 && start + 2 * len <= MLKEM_N)+  requires(0 <= bound && bound < INT16_MAX - MLKEM_Q)+  requires(-MLKEM_Q_HALF < zeta && zeta < MLKEM_Q_HALF)+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  requires(array_abs_bound(r, 0, start, bound + MLKEM_Q))+  requires(array_abs_bound(r, start, MLKEM_N, bound))+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  ensures(array_abs_bound(r, 0, start + 2*len, bound + MLKEM_Q))+  ensures(array_abs_bound(r, start + 2 * len, MLKEM_N, bound)))+{+  /* `bound` is a ghost variable only needed in the CBMC specification */+  unsigned j;+  ((void)bound);+  for (j = start; j < start + len; j++)+  __loop__(+    invariant(start <= j && j <= start + len)+    /*+     * Coefficients are updated in strided pairs, so the bounds for the+     * intermediate states alternate twice between the old and new bound+     */+    invariant(array_abs_bound(r, 0,           j,           bound + MLKEM_Q))+    invariant(array_abs_bound(r, j,           start + len, bound))+    invariant(array_abs_bound(r, start + len, j + len,     bound + MLKEM_Q))+    invariant(array_abs_bound(r, j + len,     MLKEM_N,     bound))+    decreases(start + len - j))+  {+    int16_t t;+    t = mlk_fqmul(r[j + len], zeta);+    /* The precondition implies that the arithmetic does not overflow. */+    r[j + len] = (int16_t)(r[j] - t);+    r[j] = (int16_t)(r[j] + t);+  }+}++/*+ * Compute one layer of forward NTT+ * Parameters:+ * - r: Pointer to base of polynomial+ * - layer: Variable indicating which layer is being applied.+ */++/* Reference: Embedded in `ntt()` in the reference implementation @[REF]. */+static void mlk_ntt_layer(int16_t r[MLKEM_N], unsigned layer)+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  requires(1 <= layer && layer <= 7)+  requires(array_abs_bound(r, 0, MLKEM_N, layer * MLKEM_Q))+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  ensures(array_abs_bound(r, 0, MLKEM_N, (layer + 1) * MLKEM_Q)))+{+  unsigned start, k, len;+  /* Twiddle factors for layer n are at indices 2^(n-1)..2^n-1. */+  k = 1u << (layer - 1);+  len = (unsigned)MLKEM_N >> layer;+  for (start = 0; start < MLKEM_N; start += 2 * len)+  __loop__(+    invariant(start < MLKEM_N + 2 * len)+    invariant(k <= MLKEM_N / 2 && 2 * len * k == start + MLKEM_N)+    invariant(array_abs_bound(r, 0, start, layer * MLKEM_Q + MLKEM_Q))+    invariant(array_abs_bound(r, start, MLKEM_N, layer * MLKEM_Q))+    decreases(MLKEM_N - start))+  {+    int16_t zeta = mlk_zetas[k++];+    mlk_ntt_butterfly_block(r, zeta, start, len, layer * MLKEM_Q);+  }+}++/*+ * Compute full forward NTT+ * NOTE: This particular implementation satisfies a much tighter+ * bound on the output coefficients (5*q) than the contractual one (8*q),+ * but this is not needed in the calling code. Should we change the+ * base multiplication strategy to require smaller NTT output bounds,+ * the proof may need strengthening.+ */++/* Reference: `ntt()` in the reference implementation @[REF].+ * - Iterate over `layer` instead of `len` in the outer loop+ *   to simplify computation of zeta index. */+MLK_STATIC_TESTABLE void mlk_poly_ntt_c(mlk_poly *p)+__contract__(+  requires(memory_no_alias(p, sizeof(mlk_poly)))+  requires(array_abs_bound(p->coeffs, 0, MLKEM_N, MLKEM_Q))+  assigns(memory_slice(p, sizeof(mlk_poly)))+  ensures(array_abs_bound(p->coeffs, 0, MLKEM_N, MLK_NTT_BOUND))+)+{+  unsigned layer;+  int16_t *r;++  mlk_assert_abs_bound(p, MLKEM_N, MLKEM_Q);++  r = p->coeffs;++  for (layer = 1; layer <= 7; layer++)+  __loop__(+    invariant(1 <= layer && layer <= 8)+    invariant(array_abs_bound(r, 0, MLKEM_N, layer * MLKEM_Q))+    decreases(8 - layer))+  {+    mlk_ntt_layer(r, layer);+  }++  /* Check the stronger bound */+  mlk_assert_abs_bound(p, MLKEM_N, MLK_NTT_BOUND);+}++MLK_INTERNAL_API+void mlk_poly_ntt(mlk_poly *r)+{+#if defined(MLK_USE_NATIVE_NTT)+  int ret;+  mlk_assert_abs_bound(r, MLKEM_N, MLKEM_Q);+  ret = mlk_ntt_native(r->coeffs);+  if (ret == MLK_NATIVE_FUNC_SUCCESS)+  {+    mlk_assert_abs_bound(r, MLKEM_N, MLK_NTT_BOUND);+    return;+  }+#endif /* MLK_USE_NATIVE_NTT */++  mlk_poly_ntt_c(r);+}+++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+/* Compute one layer of inverse NTT */++/* Reference: Embedded into `invntt()` in the reference implementation @[REF] */+static void mlk_invntt_layer(int16_t *r, unsigned layer)+__contract__(+  requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N))+  requires(1 <= layer && layer <= 7)+  requires(array_abs_bound(r, 0, MLKEM_N, MLKEM_Q))+  assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N))+  ensures(array_abs_bound(r, 0, MLKEM_N, MLKEM_Q)))+{+  unsigned start, k, len;+  len = (unsigned)MLKEM_N >> layer;+  k = (1u << layer) - 1;+  for (start = 0; start < MLKEM_N; start += 2 * len)+  __loop__(+    invariant(array_abs_bound(r, 0, MLKEM_N, MLKEM_Q))+    invariant(start <= MLKEM_N && k <= 127)+    /* Normalised form of k == MLKEM_N / len - 1 - start / (2 * len) */+    invariant(2 * len * k + start == 2 * MLKEM_N - 2 * len)+    decreases(MLKEM_N - start))+  {+    unsigned j;+    int16_t zeta = mlk_zetas[k--];+    for (j = start; j < start + len; j++)+    __loop__(+      invariant(start <= j && j <= start + len)+      invariant(start <= MLKEM_N && k <= 127)+      invariant(array_abs_bound(r, 0, MLKEM_N, MLKEM_Q))+      decreases(start + len - j))+    {+      int16_t t = r[j];+      /* The preconditions imply that the arithmetic does not overflow. */+      r[j] = mlk_barrett_reduce((int16_t)(t + r[j + len]));+      r[j + len] = (int16_t)(r[j + len] - t);+      r[j + len] = mlk_fqmul(r[j + len], zeta);+    }+  }+}++/* Reference: `invntt()` in the reference implementation @[REF]+ *            - We normalize at the beginning of the inverse NTT,+ *              while the reference implementation normalizes at+ *              the end. This allows us to drop a call to `poly_reduce()`+ *              from the base multiplication. */+MLK_STATIC_TESTABLE void mlk_poly_invntt_tomont_c(mlk_poly *p)+__contract__(+  requires(memory_no_alias(p, sizeof(mlk_poly)))+  assigns(memory_slice(p, sizeof(mlk_poly)))+  ensures(array_abs_bound(p->coeffs, 0, MLKEM_N, MLK_INVNTT_BOUND))+)+{+  unsigned j, layer;+  const int16_t f = 1441; /* check-magic: 1441 == pow(2,32 - 7,MLKEM_Q) */+  int16_t *r = p->coeffs;++  /*+   * Scale input polynomial to account for Montgomery factor+   * and NTT twist. This also brings coefficients down to+   * absolute value < MLKEM_Q.+   */+  for (j = 0; j < MLKEM_N; j++)+  __loop__(+    invariant(j <= MLKEM_N)+    invariant(array_abs_bound(r, 0, j, MLKEM_Q))+    decreases(MLKEM_N - j))+  {+    r[j] = mlk_fqmul(r[j], f);+  }++  /* Run the invNTT layers */+  for (layer = 7; layer > 0; layer--)+  __loop__(+    invariant(0 <= layer && layer < 8)+    invariant(array_abs_bound(r, 0, MLKEM_N, MLKEM_Q))+    decreases(layer))+  {+    mlk_invntt_layer(r, layer);+  }++  mlk_assert_abs_bound(p, MLKEM_N, MLK_INVNTT_BOUND);+}++MLK_INTERNAL_API+void mlk_poly_invntt_tomont(mlk_poly *r)+{+#if defined(MLK_USE_NATIVE_INTT)+  int ret;+  ret = mlk_intt_native(r->coeffs);+  if (ret == MLK_NATIVE_FUNC_SUCCESS)+  {+    mlk_assert_abs_bound(r, MLKEM_N, MLK_INVNTT_BOUND);+    return;+  }+#endif /* MLK_USE_NATIVE_INTT */++  mlk_poly_invntt_tomont_c(r);+}+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */+#else  /* !MLK_CONFIG_MULTILEVEL_NO_SHARED */++MLK_EMPTY_CU(mlk_poly)++#endif /* MLK_CONFIG_MULTILEVEL_NO_SHARED */
+ cbits/mlkem/src/poly.h view
@@ -0,0 +1,296 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ */++#ifndef MLK_POLY_H+#define MLK_POLY_H+++#include "cbmc.h"+#include "common.h"+#include "debug.h"+#include "verify.h"++/* Absolute exclusive upper bound for the output of the inverse NTT */+#define MLK_INVNTT_BOUND (8 * MLKEM_Q)++/* Absolute exclusive upper bound for the output of the forward NTT */+#define MLK_NTT_BOUND (8 * MLKEM_Q)++/**+ * Element of R_q = Z_q[X]/(X^n + 1). Represents polynomial+ * coeffs[0] + X*coeffs[1] + X^2*coeffs[2] + ... + X^{n-1}*coeffs[n-1].+ */+typedef struct+{+  int16_t coeffs[MLKEM_N]; /**< Polynomial coefficients. */+} MLK_ALIGN mlk_poly;++/**+ * INTERNAL representation of precomputed data speeding up+ * the base multiplication of two polynomials in NTT domain.+ */+typedef struct+{+  int16_t coeffs[MLKEM_N >> 1]; /**< Cached coefficients. */+} MLK_ALIGN mlk_poly_mulcache;++/**+ * Generic Montgomery reduction; given a 32-bit integer a, computes a 16-bit+ * integer congruent to a * R^-1 mod MLKEM_Q, where R=2^16.+ *+ * @param a Input integer to be reduced, of absolute value smaller or equal+ *          to INT32_MAX - 2^15 * MLKEM_Q.+ *+ * @return Integer congruent to a * R^-1 modulo MLKEM_Q, with absolute value+ *         <= ceil(|a| / 2^16) + (MLKEM_Q + 1)/2.+ */+static MLK_ALWAYS_INLINE int16_t mlk_montgomery_reduce(int32_t a)+__contract__(+    requires(a < +(INT32_MAX - (((int32_t)1 << 15) * MLKEM_Q)) &&+             a > -(INT32_MAX - (((int32_t)1 << 15) * MLKEM_Q)))+    /* We don't attempt to express an input-dependent output bound+     * as the post-condition here. There are two call-sites for this+     * function:+     * - The base multiplication: Here, we need no output bound.+     * - mlk_fqmul: Here, we inline this function and prove another spec+     *          for mlk_fqmul which does have a post-condition bound. */+)+{+  /* check-magic: 62209 == unsigned_mod(pow(MLKEM_Q, -1, 2^16), 2^16) */+  const uint32_t QINV = 62209;++  /* Compute a*q^{-1} mod 2^16 in unsigned representatives. */+  const uint16_t a_reduced = mlk_cast_int32_to_uint16(a);+  const uint16_t a_inverted = (a_reduced * QINV) & UINT16_MAX;++  /* Lift to signed canonical representative mod 2^16. */+  const int16_t t = mlk_cast_uint16_to_int16(a_inverted);++  int32_t r;++  mlk_assert(a < +(INT32_MAX - (((int32_t)1 << 15) * MLKEM_Q)) &&+             a > -(INT32_MAX - (((int32_t)1 << 15) * MLKEM_Q)));++  r = a - ((int32_t)t * MLKEM_Q);++  /*+   * PORTABILITY: Right-shift on a signed integer is, strictly-speaking,+   * implementation-defined for negative left argument. Here,+   * we assume it's sign-preserving "arithmetic" shift right. (C99 6.5.7 (5))+   */+  r = r >> 16;+  /* Bounds: |r >> 16| <= ceil(|r| / 2^16)+   *                   <= ceil(|a| / 2^16 + MLKEM_Q / 2)+   *                   <= ceil(|a| / 2^16) + (MLKEM_Q + 1) / 2+   *+   * (Note that |a >> n| = ceil(|a| / 2^16) for negative a)+   */+  return (int16_t)r;+}++#if !defined(MLK_CONFIG_NO_KEYPAIR_API)+#define mlk_poly_tomont MLK_NAMESPACE(poly_tomont)+/**+ * In-place conversion of all coefficients of a polynomial from the normal+ * domain to the Montgomery domain.+ *+ * Bounds: output < MLKEM_Q in absolute value.+ *+ * @spec{Internal normalization required in `mlk_indcpa_keypair_derand` as+ * part of matrix-vector multiplication @[FIPS203, Algorithm 13, K-PKE.KeyGen,+ * L18].}+ *+ * @param[in,out] r Input/output polynomial.+ */+MLK_INTERNAL_API+void mlk_poly_tomont(mlk_poly *r)+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_abs_bound(r->coeffs, 0, MLKEM_N, MLKEM_Q))+);+#endif /* !MLK_CONFIG_NO_KEYPAIR_API */++#define mlk_poly_mulcache_compute MLK_NAMESPACE(poly_mulcache_compute)+/**+ * Compute the mulcache for a polynomial in NTT domain.+ *+ * The mulcache of a degree-2 polynomial b := b0 + b1*X in Fq[X]/(X^2-zeta)+ * is the value b1*zeta, needed when computing products of b in+ * Fq[X]/(X^2-zeta).+ *+ * The mulcache of a polynomial in NTT domain -- which is a 128-tuple of+ * degree-2 polynomials in Fq[X]/(X^2-zeta), for varying zeta, is the+ * 128-tuple of mulcaches of those polynomials.+ *+ * @spec{Caches `b_1 * \gamma` in @[FIPS203, Algorithm 12, BaseCaseMultiply,+ * L1].}+ *+ * @param[out] x Mulcache to be populated.+ * @param[in]  a Input polynomial.+ */+/*+ * NOTE: The default C implementation of this function populates+ * the mulcache with values in (-q,q), but this is not needed for the+ * higher level safety proofs, and thus not part of the spec.+ */+MLK_INTERNAL_API+void mlk_poly_mulcache_compute(mlk_poly_mulcache *x, const mlk_poly *a)+__contract__(+  requires(memory_no_alias(x, sizeof(mlk_poly_mulcache)))+  requires(memory_no_alias(a, sizeof(mlk_poly)))+  assigns(memory_slice(x, sizeof(mlk_poly_mulcache)))+);++#define mlk_poly_reduce MLK_NAMESPACE(poly_reduce)+/**+ * Convert a polynomial to unsigned canonical representatives.+ *+ * The input coefficients can be arbitrary integers in int16_t. The output+ * coefficients are in [0,1,..,MLKEM_Q-1].+ *+ * @spec{Normalizes on unsigned canonical representatives ahead of calling+ * @[FIPS203, Compress_d, Eq (4.7)]. This is not made explicit in FIPS 203.}+ *+ * @param[in,out] r Input/output polynomial.+ */+/*+ * NOTE: The semantics of mlk_poly_reduce() is different in+ * the reference implementation, which requires+ * signed canonical output data. Unsigned canonical+ * outputs are better suited to the only remaining+ * use of mlk_poly_reduce() in the context of (de)serialization.+ */+MLK_INTERNAL_API+void mlk_poly_reduce(mlk_poly *r)+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+);++#define mlk_poly_add MLK_NAMESPACE(poly_add)+/**+ * Add two polynomials in place.+ *+ * The coefficients of @p r and @p b must be such that the addition does not+ * overflow. Otherwise, the behaviour of this function is undefined.+ *+ * @spec{@[FIPS203, 2.4.5, Arithmetic With Polynomials and NTT+ * Representations]. Used in @[FIPS203, Algorithm 14 (K-PKE.Encrypt), L21].}+ *+ * @param[in,out] r Input-output polynomial to be added to.+ * @param[in]     b Input polynomial that should be added to @p r. Must be+ *                  disjoint from @p r.+ */+/*+ * NOTE: The reference implementation uses a 3-argument mlk_poly_add.+ * We specialize to the accumulator form to avoid reasoning about aliasing.+ */+MLK_INTERNAL_API+void mlk_poly_add(mlk_poly *r, const mlk_poly *b)+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  requires(memory_no_alias(b, sizeof(mlk_poly)))+  requires(forall(k0, 0, MLKEM_N, (int32_t) r->coeffs[k0] + b->coeffs[k0] <= INT16_MAX))+  requires(forall(k1, 0, MLKEM_N, (int32_t) r->coeffs[k1] + b->coeffs[k1] >= INT16_MIN))+  ensures(forall(k, 0, MLKEM_N, r->coeffs[k] == old(*r).coeffs[k] + b->coeffs[k]))+  assigns(memory_slice(r, sizeof(mlk_poly)))+);++#if !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_poly_sub MLK_NAMESPACE(poly_sub)+/**+ * Subtract two polynomials; no modular reduction is performed.+ *+ * @spec{@[FIPS203, 2.4.5, Arithmetic With Polynomials and NTT+ * Representations]. Used in @[FIPS203, Algorithm 15, K-PKE.Decrypt, L6].}+ *+ * @param[in,out] r Input-output polynomial to be subtracted from.+ * @param[in]     b Second input polynomial.+ */+/*+ * NOTE: The reference implementation uses a 3-argument mlk_poly_sub.+ * We specialize to the accumulator form to avoid reasoning about aliasing.+ */+MLK_INTERNAL_API+void mlk_poly_sub(mlk_poly *r, const mlk_poly *b)+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  requires(memory_no_alias(b, sizeof(mlk_poly)))+  requires(forall(k0, 0, MLKEM_N, (int32_t) r->coeffs[k0] - b->coeffs[k0] <= INT16_MAX))+  requires(forall(k1, 0, MLKEM_N, (int32_t) r->coeffs[k1] - b->coeffs[k1] >= INT16_MIN))+  ensures(forall(k, 0, MLKEM_N, r->coeffs[k] == old(*r).coeffs[k] - b->coeffs[k]))+  assigns(memory_slice(r, sizeof(mlk_poly)))+);+#endif /* !MLK_CONFIG_NO_DECAPS_API */++#define mlk_poly_ntt MLK_NAMESPACE(poly_ntt)+/**+ * Compute the negacyclic number-theoretic transform (NTT) of a polynomial+ * in place.+ *+ * The input is assumed to be in normal order and coefficient-wise bound by+ * MLKEM_Q in absolute value.+ *+ * The output polynomial is in bitreversed order, or of a custom order if+ * MLK_USE_NATIVE_NTT_CUSTOM_ORDER is set, and coefficient-wise bound+ * by MLK_NTT_BOUND in absolute value.+ *+ * (NOTE: Sometimes the input to the NTT is actually smaller, which gives+ * better bounds.)+ *+ * @spec{Implements @[FIPS203, Algorithm 9, NTT].}+ *+ * @param[in,out] r Input/output polynomial.+ */+MLK_INTERNAL_API+void mlk_poly_ntt(mlk_poly *r)+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  requires(array_abs_bound(r->coeffs, 0, MLKEM_N, MLKEM_Q))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_abs_bound(r->coeffs, 0, MLKEM_N, MLK_NTT_BOUND))+);++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_poly_invntt_tomont MLK_NAMESPACE(poly_invntt_tomont)+/**+ * Compute the inverse negacyclic number-theoretic transform (NTT) of a+ * polynomial in place; input assumed to be in bitreversed order, output in+ * normal order.+ *+ * The input is assumed to be in bitreversed order, or of a custom order if+ * MLK_USE_NATIVE_NTT_CUSTOM_ORDER is set, and can have arbitrary+ * coefficients in int16_t.+ *+ * The output polynomial is in normal order, and coefficient-wise bound by+ * MLK_INVNTT_BOUND in absolute value.+ *+ * @spec{Implements composition of @[FIPS203, Algorithm 10, NTT^{-1}] and+ * elementwise modular multiplication with a suitable Montgomery factor+ * introduced during the base multiplication.}+ *+ * @param[in,out] r Input/output polynomial.+ */+MLK_INTERNAL_API+void mlk_poly_invntt_tomont(mlk_poly *r)+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_abs_bound(r->coeffs, 0, MLKEM_N, MLK_INVNTT_BOUND))+);+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++#endif /* !MLK_POLY_H */
+ cbits/mlkem/src/poly_k.c view
@@ -0,0 +1,522 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ *+ * - [NeonNTT]+ *   Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1+ *   Becker, Hwang, Kannwischer, Yang, Yang+ *   https://eprint.iacr.org/2021/986+ *+ * - [REF]+ *   CRYSTALS-Kyber C reference implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/ref+ */++#include "poly_k.h"++#include "debug.h"+#include "sampling.h"+#include "symmetric.h"+#include "verify.h"++/* Parameter set namespacing+ * This is to facilitate building multiple instances+ * of mlkem-native (e.g. with varying parameter sets)+ * within a single compilation unit. */+#define mlk_poly_cbd_eta1 MLK_ADD_PARAM_SET(mlk_poly_cbd_eta1)+#define mlk_poly_cbd_eta2 MLK_ADD_PARAM_SET(mlk_poly_cbd_eta2)+#define mlk_polyvec_basemul_acc_montgomery_cached_c \+  MLK_ADD_PARAM_SET(mlk_polyvec_basemul_acc_montgomery_cached_c)+/* End of parameter set namespacing */++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+/* Reference: `polyvec_compress()` in the reference implementation @[REF]+ *            - In contrast to the reference implementation, we assume+ *              unsigned canonical coefficients here.+ *              The reference implementation works with coefficients+ *              in the range [-(MLKEM_Q-1), MLKEM_Q-1]. */+MLK_INTERNAL_API+void mlk_polyvec_compress_du(uint8_t r[MLKEM_POLYVECCOMPRESSEDBYTES_DU],+                             const mlk_polyvec *a)+{+  unsigned i;+  mlk_assert_bound_2d(a->vec, MLKEM_K, MLKEM_N, 0, MLKEM_Q);++  for (i = 0; i < MLKEM_K; i++)+  {+    mlk_poly_compress_du(r + i * MLKEM_POLYCOMPRESSEDBYTES_DU, &a->vec[i]);+  }+}+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++#if !defined(MLK_CONFIG_NO_DECAPS_API)+/* Reference: `polyvec_decompress()` in the reference implementation @[REF]. */+MLK_INTERNAL_API+void mlk_polyvec_decompress_du(mlk_polyvec *r,+                               const uint8_t a[MLKEM_POLYVECCOMPRESSEDBYTES_DU])+{+  unsigned i;+  for (i = 0; i < MLKEM_K; i++)+  {+    mlk_poly_decompress_du(&r->vec[i], a + i * MLKEM_POLYCOMPRESSEDBYTES_DU);+  }++  mlk_assert_bound_2d(r->vec, MLKEM_K, MLKEM_N, 0, MLKEM_Q);+}+#endif /* !MLK_CONFIG_NO_DECAPS_API */++#if !defined(MLK_CONFIG_NO_KEYPAIR_API) || !defined(MLK_CONFIG_NO_ENCAPS_API)+/* Reference: `polyvec_tobytes()` in the reference implementation @[REF].+ *            - In contrast to the reference implementation, we assume+ *              unsigned canonical coefficients here.+ *              The reference implementation works with coefficients+ *              in the range [-(MLKEM_Q-1), MLKEM_Q-1]. */+MLK_INTERNAL_API+void mlk_polyvec_tobytes(uint8_t r[MLKEM_POLYVECBYTES], const mlk_polyvec *a)+{+  unsigned i;+  mlk_assert_bound_2d(a->vec, MLKEM_K, MLKEM_N, 0, MLKEM_Q);++  for (i = 0; i < MLKEM_K; i++)+  __loop__(+    assigns(i, memory_slice(r, MLKEM_POLYVECBYTES))+    invariant(i <= MLKEM_K)+    decreases(MLKEM_K - i)+  )+  {+    mlk_poly_tobytes(&r[i * MLKEM_POLYBYTES], &a->vec[i]);+  }+}+#endif /* !MLK_CONFIG_NO_KEYPAIR_API || !MLK_CONFIG_NO_ENCAPS_API */++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+/* Reference: `polyvec_frombytes()` in the reference implementation @[REF]. */+MLK_INTERNAL_API+void mlk_polyvec_frombytes(mlk_polyvec *r, const uint8_t a[MLKEM_POLYVECBYTES])+{+  unsigned i;+  for (i = 0; i < MLKEM_K; i++)+  {+    mlk_poly_frombytes(&r->vec[i], a + i * MLKEM_POLYBYTES);+  }++  mlk_assert_bound_2d(r->vec, MLKEM_K, MLKEM_N, 0, MLKEM_UINT12_LIMIT);+}+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++/* Reference: `polyvec_ntt()` in the reference implementation @[REF]. */+MLK_INTERNAL_API+void mlk_polyvec_ntt(mlk_polyvec *r)+{+  unsigned i;+  for (i = 0; i < MLKEM_K; i++)+  {+    mlk_poly_ntt(&r->vec[i]);+  }++  mlk_assert_abs_bound_2d(r->vec, MLKEM_K, MLKEM_N, MLK_NTT_BOUND);+}++/* Reference: `polyvec_invntt_tomont()` in the reference implementation @[REF].+ *            - We normalize at the beginning of the inverse NTT,+ *              while the reference implementation normalizes at+ *              the end. This allows us to drop a call to `poly_reduce()`+ *              from the base multiplication. */+#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+MLK_INTERNAL_API+void mlk_polyvec_invntt_tomont(mlk_polyvec *r)+{+  unsigned i;+  for (i = 0; i < MLKEM_K; i++)+  {+    mlk_poly_invntt_tomont(&r->vec[i]);+  }++  mlk_assert_abs_bound_2d(r->vec, MLKEM_K, MLKEM_N, MLK_INVNTT_BOUND);+}+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++/* Reference: `polyvec_basemul_acc_montgomery()` in the+ *            reference implementation @[REF].+ *            - We use a multiplication cache ('mulcache') here+ *              which is not present in the reference implementation @[REF].+ *              This idea originates from @[NeonNTT] and is used+ *              at the C level here.+ *            - We compute the coefficients of the scalar product in 32-bit+ *              coefficients and perform only a single modular reduction+ *              at the end. The reference implementation uses 2 * MLKEM_K+ *              more modular reductions since it reduces after every modular+ *              multiplication. */+MLK_STATIC_TESTABLE void mlk_polyvec_basemul_acc_montgomery_cached_c(+    mlk_poly *r, const mlk_polyvec *a, const mlk_polyvec *b,+    const mlk_polyvec_mulcache *b_cache)+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  requires(memory_no_alias(a, sizeof(mlk_polyvec)))+  requires(memory_no_alias(b, sizeof(mlk_polyvec)))+  requires(memory_no_alias(b_cache, sizeof(mlk_polyvec_mulcache)))+  requires(forall(k1, 0, MLKEM_K,+     array_bound(a->vec[k1].coeffs, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT)))+  assigns(memory_slice(r, sizeof(mlk_poly)))+)+{+  unsigned i;+  mlk_assert_bound_2d(a->vec, MLKEM_K, MLKEM_N, 0, MLKEM_UINT12_LIMIT);++  for (i = 0; i < MLKEM_N / 2; i++)+  __loop__(invariant(i <= MLKEM_N / 2)+           decreases(MLKEM_N / 2 - i))+  {+    unsigned k;+    int32_t t[2] = {0};+    for (k = 0; k < MLKEM_K; k++)+    __loop__(+      invariant(k <= MLKEM_K &&+         t[0] <=    (int32_t) k * 2 * MLKEM_UINT12_LIMIT * 32768  &&+         t[0] >= - ((int32_t) k * 2 * MLKEM_UINT12_LIMIT * 32768) &&+         t[1] <=   ((int32_t) k * 2 * MLKEM_UINT12_LIMIT * 32768) &&+         t[1] >= - ((int32_t) k * 2 * MLKEM_UINT12_LIMIT * 32768))+      decreases(MLKEM_K - k))+    {+      t[0] += (int32_t)a->vec[k].coeffs[2 * i + 1] * b_cache->vec[k].coeffs[i];+      t[0] += (int32_t)a->vec[k].coeffs[2 * i] * b->vec[k].coeffs[2 * i];+      t[1] += (int32_t)a->vec[k].coeffs[2 * i] * b->vec[k].coeffs[2 * i + 1];+      t[1] += (int32_t)a->vec[k].coeffs[2 * i + 1] * b->vec[k].coeffs[2 * i];+    }+    r->coeffs[2 * i + 0] = mlk_montgomery_reduce(t[0]);+    r->coeffs[2 * i + 1] = mlk_montgomery_reduce(t[1]);+  }+}++MLK_INTERNAL_API+void mlk_polyvec_basemul_acc_montgomery_cached(+    mlk_poly *r, const mlk_polyvec *a, const mlk_polyvec *b,+    const mlk_polyvec_mulcache *b_cache)+{+#if defined(MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED)+  {+    int ret;+    mlk_assert_bound_2d(a->vec, MLKEM_K, MLKEM_N, 0, MLKEM_UINT12_LIMIT);+#if MLKEM_K == 2+    ret = mlk_polyvec_basemul_acc_montgomery_cached_k2_native(+        r->coeffs, (const int16_t *)a, (const int16_t *)b,+        (const int16_t *)b_cache);+#elif MLKEM_K == 3+    ret = mlk_polyvec_basemul_acc_montgomery_cached_k3_native(+        r->coeffs, (const int16_t *)a, (const int16_t *)b,+        (const int16_t *)b_cache);+#elif MLKEM_K == 4+    ret = mlk_polyvec_basemul_acc_montgomery_cached_k4_native(+        r->coeffs, (const int16_t *)a, (const int16_t *)b,+        (const int16_t *)b_cache);+#endif+    if (ret == MLK_NATIVE_FUNC_SUCCESS)+    {+      return;+    }+  }+#endif /* MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED */++  mlk_polyvec_basemul_acc_montgomery_cached_c(r, a, b, b_cache);+}++/* Reference: Does not exist in the reference implementation @[REF].+ *            - The reference implementation does not use a+ *              multiplication cache ('mulcache'). This idea originates+ *              from @[NeonNTT] and is used at the C level here. */+MLK_INTERNAL_API+void mlk_polyvec_mulcache_compute(mlk_polyvec_mulcache *x, const mlk_polyvec *a)+{+  unsigned i;+  for (i = 0; i < MLKEM_K; i++)+  {+    mlk_poly_mulcache_compute(&x->vec[i], &a->vec[i]);+  }+}++/* Reference: `polyvec_reduce()` in the reference implementation @[REF].+ *            - We use _unsigned_ canonical outputs, while the reference+ *              implementation uses _signed_ canonical outputs.+ *              Accordingly, we need a conditional addition of MLKEM_Q+ *              here to go from signed to unsigned representatives.+ *              This conditional addition is then dropped from all+ *              polynomial compression functions instead (see `compress.c`). */+MLK_INTERNAL_API+void mlk_polyvec_reduce(mlk_polyvec *r)+{+  unsigned i;+  for (i = 0; i < MLKEM_K; i++)+  {+    mlk_poly_reduce(&r->vec[i]);+  }++  mlk_assert_bound_2d(r->vec, MLKEM_K, MLKEM_N, 0, MLKEM_Q);+}++/* Reference: `polyvec_add()` in the reference implementation @[REF].+ *            - We use destructive version (output=first input) to avoid+ *              reasoning about aliasing in the CBMC specification */+MLK_INTERNAL_API+void mlk_polyvec_add(mlk_polyvec *r, const mlk_polyvec *b)+{+  unsigned i;+  for (i = 0; i < MLKEM_K; i++)+  __loop__(+    assigns(i, memory_slice(r, sizeof(mlk_polyvec)))+    invariant(i <= MLKEM_K)+    invariant(forall(j0, i, MLKEM_K,+                forall(k0, 0, MLKEM_N,+                       ((int32_t)r->vec[j0].coeffs[k0] + b->vec[j0].coeffs[k0] <= INT16_MAX) &&+                       ((int32_t)r->vec[j0].coeffs[k0] + b->vec[j0].coeffs[k0] >= INT16_MIN))))+    invariant(forall(j2, 0, i,+                forall(k2, 0, MLKEM_N,+                       (r->vec[j2].coeffs[k2] <= INT16_MAX) &&+                       (r->vec[j2].coeffs[k2] >= INT16_MIN))))+    decreases(MLKEM_K - i)+  )+  {+    mlk_poly_add(&r->vec[i], &b->vec[i]);+  }+}++#if !defined(MLK_CONFIG_NO_KEYPAIR_API)+/* Reference: `polyvec_tomont()` in the reference implementation @[REF]. */+MLK_INTERNAL_API+void mlk_polyvec_tomont(mlk_polyvec *r)+{+  unsigned i;+  for (i = 0; i < MLKEM_K; i++)+  {+    mlk_poly_tomont(&r->vec[i]);+  }++  mlk_assert_abs_bound_2d(r->vec, MLKEM_K, MLKEM_N, MLKEM_Q);+}+#endif /* !MLK_CONFIG_NO_KEYPAIR_API */+++/**+ * Given an array of uniformly random bytes, compute a polynomial with+ * coefficients distributed according to a centered binomial distribution+ * with parameter MLKEM_ETA1.+ *+ * @spec{Implements @[FIPS203, Algorithm 8, SamplePolyCBD_eta1], where eta1+ * is specified per parameter set in @[FIPS203, Table 2] and represented as+ * MLKEM_ETA1 here.}+ *+ * @reference{`poly_cbd_eta1` in the reference implementation @[REF].}+ *+ * @param[out] r   Output polynomial.+ * @param[in]  buf Input byte array.+ */+static MLK_INLINE void mlk_poly_cbd_eta1(+    mlk_poly *r, const uint8_t buf[MLKEM_ETA1 * MLKEM_N / 4])+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  requires(memory_no_alias(buf, MLKEM_ETA1 * MLKEM_N / 4))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_abs_bound(r->coeffs, 0, MLKEM_N, MLKEM_ETA1 + 1))+)+{+#if MLKEM_ETA1 == 2+  mlk_poly_cbd2(r, buf);+#elif MLKEM_ETA1 == 3+  mlk_poly_cbd3(r, buf);+#else+#error "Invalid value of MLKEM_ETA1"+#endif+}++#if !defined(MLK_CONFIG_NO_KEYPAIR_API) ||                              \+    (MLKEM_ETA1 == MLKEM_ETA2 && (!defined(MLK_CONFIG_NO_ENCAPS_API) || \+                                  !defined(MLK_CONFIG_NO_DECAPS_API)))+/* Reference: Does not exist in the reference implementation @[REF].+ *            - This implements a x4-batched version of `poly_getnoise_eta1()`+ *              from the reference implementation, to leverage+ *              batched Keccak-f1600.*/+MLK_INTERNAL_API+void mlk_poly_getnoise_eta1_4x(mlk_poly *r0, mlk_poly *r1, mlk_poly *r2,+                               mlk_poly *r3, const uint8_t seed[MLKEM_SYMBYTES],+                               uint8_t nonce0, uint8_t nonce1, uint8_t nonce2,+                               uint8_t nonce3)+{+  MLK_ALIGN uint8_t buf[4][MLK_ALIGN_UP(MLKEM_ETA1 * MLKEM_N / 4)];+  MLK_ALIGN uint8_t extkey[4][MLK_ALIGN_UP(MLKEM_SYMBYTES + 1)];+  mlk_memcpy(extkey[0], seed, MLKEM_SYMBYTES);+  mlk_memcpy(extkey[1], seed, MLKEM_SYMBYTES);+  mlk_memcpy(extkey[2], seed, MLKEM_SYMBYTES);+  mlk_memcpy(extkey[3], seed, MLKEM_SYMBYTES);+  extkey[0][MLKEM_SYMBYTES] = nonce0;+  extkey[1][MLKEM_SYMBYTES] = nonce1;+  extkey[2][MLKEM_SYMBYTES] = nonce2;+  extkey[3][MLKEM_SYMBYTES] = nonce3;++#if !defined(FIPS202_X4_DEFAULT_IMPLEMENTATION) && \+    !defined(MLK_CONFIG_SERIAL_FIPS202_ONLY)+  mlk_prf_eta1_x4(buf, extkey);+#else+  mlk_prf_eta1(buf[0], extkey[0]);+  mlk_prf_eta1(buf[1], extkey[1]);+  mlk_prf_eta1(buf[2], extkey[2]);+  if (r3 != NULL)+  {+    mlk_prf_eta1(buf[3], extkey[3]);+  }+#endif /* !(!FIPS202_X4_DEFAULT_IMPLEMENTATION && \+          !MLK_CONFIG_SERIAL_FIPS202_ONLY) */++  mlk_poly_cbd_eta1(r0, buf[0]);+  mlk_poly_cbd_eta1(r1, buf[1]);+  mlk_poly_cbd_eta1(r2, buf[2]);+  if (r3 != NULL)+  {+    mlk_poly_cbd_eta1(r3, buf[3]);+    mlk_assert_abs_bound(r3, MLKEM_N, MLKEM_ETA1 + 1);+  }++  mlk_assert_abs_bound(r0, MLKEM_N, MLKEM_ETA1 + 1);+  mlk_assert_abs_bound(r1, MLKEM_N, MLKEM_ETA1 + 1);+  mlk_assert_abs_bound(r2, MLKEM_N, MLKEM_ETA1 + 1);++  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  mlk_zeroize(buf, sizeof(buf));+  mlk_zeroize(extkey, sizeof(extkey));+}+#endif /* !MLK_CONFIG_NO_KEYPAIR_API || (MLKEM_ETA1 == MLKEM_ETA2 && \+          (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API)) */++#if (MLKEM_K == 2 || MLKEM_K == 4) && \+    (!defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API))+/**+ * Given an array of uniformly random bytes, compute a polynomial with+ * coefficients distributed according to a centered binomial distribution+ * with parameter MLKEM_ETA2.+ *+ * @spec{Implements @[FIPS203, Algorithm 8, SamplePolyCBD_eta2], where eta2+ * is specified per parameter set in @[FIPS203, Table 2] and represented as+ * MLKEM_ETA2 here.}+ *+ * @reference{`poly_cbd_eta2` in the reference implementation @[REF].}+ *+ * @param[out] r   Output polynomial.+ * @param[in]  buf Input byte array.+ */+static MLK_INLINE void mlk_poly_cbd_eta2(+    mlk_poly *r, const uint8_t buf[MLKEM_ETA2 * MLKEM_N / 4])+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  requires(memory_no_alias(buf, MLKEM_ETA2 * MLKEM_N / 4))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_abs_bound(r->coeffs, 0, MLKEM_N, MLKEM_ETA2 + 1)))+{+#if MLKEM_ETA2 == 2+  mlk_poly_cbd2(r, buf);+#else+#error "Invalid value of MLKEM_ETA2"+#endif+}++/* Reference: `poly_getnoise_eta2()` in the reference implementation @[REF].+ *            - We include buffer zeroization. */+MLK_INTERNAL_API+void mlk_poly_getnoise_eta2(mlk_poly *r, const uint8_t seed[MLKEM_SYMBYTES],+                            uint8_t nonce)+{+  MLK_ALIGN uint8_t buf[MLKEM_ETA2 * MLKEM_N / 4];+  MLK_ALIGN uint8_t extkey[MLKEM_SYMBYTES + 1];++  mlk_memcpy(extkey, seed, MLKEM_SYMBYTES);+  extkey[MLKEM_SYMBYTES] = nonce;+  mlk_prf_eta2(buf, extkey);++  mlk_poly_cbd_eta2(r, buf);++  mlk_assert_abs_bound(r, MLKEM_N, MLKEM_ETA2 + 1);++  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  mlk_zeroize(buf, sizeof(buf));+  mlk_zeroize(extkey, sizeof(extkey));+}+#endif /* (MLKEM_K == 2 || MLKEM_K == 4) && (!MLK_CONFIG_NO_ENCAPS_API || \+          !MLK_CONFIG_NO_DECAPS_API) */++#if MLKEM_K == 2 && \+    (!defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API))+/* Reference: Does not exist in the reference implementation @[REF].+ *            - This implements a x4-batched version of `poly_getnoise_eta1()`+ *              and `poly_getnoise_eta2()` from the reference implementation,+ *              leveraging batched Keccak-f1600.+ *            - If a x4-batched Keccak-f1600 is available, we squeeze+ *              more random data than needed for the eta2 calls, to be+ *              be able to use a x4-batched Keccak-f1600. */+MLK_INTERNAL_API+void mlk_poly_getnoise_eta1122_4x(mlk_poly *r0, mlk_poly *r1, mlk_poly *r2,+                                  mlk_poly *r3,+                                  const uint8_t seed[MLKEM_SYMBYTES],+                                  uint8_t nonce0, uint8_t nonce1,+                                  uint8_t nonce2, uint8_t nonce3)+{+#if MLKEM_ETA2 >= MLKEM_ETA1+#error mlk_poly_getnoise_eta1122_4x assumes MLKEM_ETA1 > MLKEM_ETA2+#endif+  MLK_ALIGN uint8_t buf[4][MLK_ALIGN_UP(MLKEM_ETA1 * MLKEM_N / 4)];+  MLK_ALIGN uint8_t extkey[4][MLK_ALIGN_UP(MLKEM_SYMBYTES + 1)];++  mlk_memcpy(extkey[0], seed, MLKEM_SYMBYTES);+  mlk_memcpy(extkey[1], seed, MLKEM_SYMBYTES);+  mlk_memcpy(extkey[2], seed, MLKEM_SYMBYTES);+  mlk_memcpy(extkey[3], seed, MLKEM_SYMBYTES);+  extkey[0][MLKEM_SYMBYTES] = nonce0;+  extkey[1][MLKEM_SYMBYTES] = nonce1;+  extkey[2][MLKEM_SYMBYTES] = nonce2;+  extkey[3][MLKEM_SYMBYTES] = nonce3;++  /* On systems with fast batched Keccak, we use 4-fold batched PRF,+   * even though that means generating more random data in buf[2] and buf[3]+   * than necessary. */+#if !defined(FIPS202_X4_DEFAULT_IMPLEMENTATION) && \+    !defined(MLK_CONFIG_SERIAL_FIPS202_ONLY)+  mlk_prf_eta1_x4(buf, extkey);+#else+  mlk_prf_eta1(buf[0], extkey[0]);+  mlk_prf_eta1(buf[1], extkey[1]);+  mlk_prf_eta2(buf[2], extkey[2]);+  mlk_prf_eta2(buf[3], extkey[3]);+#endif /* !(!FIPS202_X4_DEFAULT_IMPLEMENTATION && \+          !MLK_CONFIG_SERIAL_FIPS202_ONLY) */++  mlk_poly_cbd_eta1(r0, buf[0]);+  mlk_poly_cbd_eta1(r1, buf[1]);+  mlk_poly_cbd_eta2(r2, buf[2]);+  mlk_poly_cbd_eta2(r3, buf[3]);++  mlk_assert_abs_bound(r0, MLKEM_N, MLKEM_ETA1 + 1);+  mlk_assert_abs_bound(r1, MLKEM_N, MLKEM_ETA1 + 1);+  mlk_assert_abs_bound(r2, MLKEM_N, MLKEM_ETA2 + 1);+  mlk_assert_abs_bound(r3, MLKEM_N, MLKEM_ETA2 + 1);++  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  mlk_zeroize(buf, sizeof(buf));+  mlk_zeroize(extkey, sizeof(extkey));+}+#endif /* MLKEM_K == 2 && (!MLK_CONFIG_NO_ENCAPS_API || \+          !MLK_CONFIG_NO_DECAPS_API) */++/* To facilitate single-compilation-unit (SCU) builds, undefine all macros.+ * Don't modify by hand -- this is auto-generated by scripts/autogen. */+#undef mlk_poly_cbd_eta1+#undef mlk_poly_cbd_eta2+#undef mlk_polyvec_basemul_acc_montgomery_cached_c
+ cbits/mlkem/src/poly_k.h view
@@ -0,0 +1,622 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ */++#ifndef MLK_POLY_K_H+#define MLK_POLY_K_H++#include "common.h"+#include "compress.h"+#include "poly.h"++/* Parameter set namespacing+ * This is to facilitate building multiple instances+ * of mlkem-native (e.g. with varying parameter sets)+ * within a single compilation unit. */+#define mlk_polyvec MLK_ADD_PARAM_SET(mlk_polyvec)+#define mlk_polymat MLK_ADD_PARAM_SET(mlk_polymat)+#define mlk_polyvec_mulcache MLK_ADD_PARAM_SET(mlk_polyvec_mulcache)+/* End of parameter set namespacing */++/** Vector of MLKEM_K polynomials. */+typedef struct+{+  mlk_poly vec[MLKEM_K]; /**< Component polynomials. */+} MLK_ALIGN mlk_polyvec;++/** MLKEM_K x MLKEM_K matrix of polynomials. */+typedef struct+{+  mlk_polyvec vec[MLKEM_K]; /**< Rows of the matrix. */+} MLK_ALIGN mlk_polymat;++/** Vector of MLKEM_K mlk_poly_mulcache entries. */+typedef struct+{+  mlk_poly_mulcache vec[MLKEM_K]; /**< Per-component caches. */+} MLK_ALIGN mlk_polyvec_mulcache;++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_poly_compress_du MLK_NAMESPACE_K(poly_compress_du)+/**+ * Compression (du bits) and subsequent serialization of a polynomial.+ *+ * @spec{Implements `ByteEncode_{d_u} (Compress_{d_u} (u))` in @[FIPS203,+ * Algorithm 14 (K-PKE.Encrypt), L22], with level-specific d_u defined in+ * @[FIPS203, Table 2], and given by MLKEM_DU here.}+ *+ * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_DU+ *               bytes).+ * @param[in]  a Input polynomial. Coefficients must be unsigned canonical,+ *               i.e. in [0,1,..,MLKEM_Q-1].+ */+static MLK_INLINE void mlk_poly_compress_du(+    uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_DU], const mlk_poly *a)+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_DU))+  requires(memory_no_alias(a, sizeof(mlk_poly)))+  requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_DU)))+{+#if MLKEM_DU == 10+  mlk_poly_compress_d10(r, a);+#elif MLKEM_DU == 11+  mlk_poly_compress_d11(r, a);+#else+#error "Invalid value of MLKEM_DU"+#endif+}+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++#if !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_poly_decompress_du MLK_NAMESPACE_K(poly_decompress_du)+/**+ * De-serialization and subsequent decompression (du bits) of a polynomial;+ * approximate inverse of mlk_poly_compress_du.+ *+ * Upon return, the coefficients of the output polynomial are+ * unsigned-canonical (non-negative and smaller than MLKEM_Q).+ *+ * @spec{Implements `Decompress_{d_u} (ByteDecode_{d_u} (u))` in @[FIPS203,+ * Algorithm 15 (K-PKE.Decrypt), L3], with level-specific d_u defined in+ * @[FIPS203, Table 2], and given by MLKEM_DU here.}+ *+ * @param[out] r Output polynomial.+ * @param[in]  a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_DU+ *               bytes).+ */+static MLK_INLINE void mlk_poly_decompress_du(+    mlk_poly *r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_DU])+__contract__(+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_DU))+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q)))+{+#if MLKEM_DU == 10+  mlk_poly_decompress_d10(r, a);+#elif MLKEM_DU == 11+  mlk_poly_decompress_d11(r, a);+#else+#error "Invalid value of MLKEM_DU"+#endif+}+#endif /* !MLK_CONFIG_NO_DECAPS_API */++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_poly_compress_dv MLK_NAMESPACE_K(poly_compress_dv)+/**+ * Compression (dv bits) and subsequent serialization of a polynomial.+ *+ * @spec{Implements `ByteEncode_{d_v} (Compress_{d_v} (v))` in @[FIPS203,+ * Algorithm 14 (K-PKE.Encrypt), L23], with level-specific d_v defined in+ * @[FIPS203, Table 2], and given by MLKEM_DV here.}+ *+ * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_DV+ *               bytes).+ * @param[in]  a Input polynomial. Coefficients must be unsigned canonical,+ *               i.e. in [0,1,..,MLKEM_Q-1].+ */+static MLK_INLINE void mlk_poly_compress_dv(+    uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_DV], const mlk_poly *a)+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_DV))+  requires(memory_no_alias(a, sizeof(mlk_poly)))+  requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+  assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_DV)))+{+#if MLKEM_DV == 4+  mlk_poly_compress_d4(r, a);+#elif MLKEM_DV == 5+  mlk_poly_compress_d5(r, a);+#else+#error "Invalid value of MLKEM_DV"+#endif+}+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */+++#if !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_poly_decompress_dv MLK_NAMESPACE_K(poly_decompress_dv)+/**+ * De-serialization and subsequent decompression (dv bits) of a polynomial;+ * approximate inverse of mlk_poly_compress_dv.+ *+ * Upon return, the coefficients of the output polynomial are+ * unsigned-canonical (non-negative and smaller than MLKEM_Q).+ *+ * @spec{Implements `Decompress_{d_v} (ByteDecode_{d_v} (v))` in @[FIPS203,+ * Algorithm 15 (K-PKE.Decrypt), L4], with level-specific d_v defined in+ * @[FIPS203, Table 2], and given by MLKEM_DV here.}+ *+ * @param[out] r Output polynomial.+ * @param[in]  a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_DV+ *               bytes).+ */+static MLK_INLINE void mlk_poly_decompress_dv(+    mlk_poly *r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_DV])+__contract__(+  requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_DV))+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q)))+{+#if MLKEM_DV == 4+  mlk_poly_decompress_d4(r, a);+#elif MLKEM_DV == 5+  mlk_poly_decompress_d5(r, a);+#else+#error "Invalid value of MLKEM_DV"+#endif+}+#endif /* !MLK_CONFIG_NO_DECAPS_API */++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_polyvec_compress_du MLK_NAMESPACE_K(polyvec_compress_du)+/**+ * Compress and serialize a vector of polynomials.+ *+ * @spec{Implements `ByteEncode_{d_u} (Compress_{d_u} (u))` in @[FIPS203,+ * Algorithm 14 (K-PKE.Encrypt), L22], with level-specific d_u defined in+ * @[FIPS203, Table 2], and given by MLKEM_DU here.}+ *+ * @param[out] r Output byte array (needs space for+ *               MLKEM_POLYVECCOMPRESSEDBYTES_DU bytes).+ * @param[in]  a Input vector of polynomials. Coefficients must be unsigned+ *               canonical, i.e. in [0,1,..,MLKEM_Q-1].+ */+MLK_INTERNAL_API+void mlk_polyvec_compress_du(uint8_t r[MLKEM_POLYVECCOMPRESSEDBYTES_DU],+                             const mlk_polyvec *a)+__contract__(+  requires(memory_no_alias(r, MLKEM_POLYVECCOMPRESSEDBYTES_DU))+  requires(memory_no_alias(a, sizeof(mlk_polyvec)))+  requires(forall(k0, 0, MLKEM_K,+         array_bound(a->vec[k0].coeffs, 0, MLKEM_N, 0, MLKEM_Q)))+  assigns(memory_slice(r, MLKEM_POLYVECCOMPRESSEDBYTES_DU))+);+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++#if !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_polyvec_decompress_du MLK_NAMESPACE_K(polyvec_decompress_du)+/**+ * De-serialize and decompress a vector of polynomials; approximate inverse+ * of mlk_polyvec_compress_du.+ *+ * @spec{Implements `Decompress_{d_u} (ByteDecode_{d_u} (u))` in @[FIPS203,+ * Algorithm 15 (K-PKE.Decrypt), L3], with level-specific d_u defined in+ * @[FIPS203, Table 2], and given by MLKEM_DU here.}+ *+ * @param[out] r Output vector of polynomials. Coefficients are normalized+ *               to [0,1,..,MLKEM_Q-1].+ * @param[in]  a Input byte array (of length MLKEM_POLYVECCOMPRESSEDBYTES_DU+ *               bytes).+ */+MLK_INTERNAL_API+void mlk_polyvec_decompress_du(mlk_polyvec *r,+                               const uint8_t a[MLKEM_POLYVECCOMPRESSEDBYTES_DU])+__contract__(+  requires(memory_no_alias(a, MLKEM_POLYVECCOMPRESSEDBYTES_DU))+  requires(memory_no_alias(r, sizeof(mlk_polyvec)))+  assigns(memory_slice(r, sizeof(mlk_polyvec)))+  ensures(forall(k0, 0, MLKEM_K,+         array_bound(r->vec[k0].coeffs, 0, MLKEM_N, 0, MLKEM_Q)))+);+#endif /* !MLK_CONFIG_NO_DECAPS_API */++#if !defined(MLK_CONFIG_NO_KEYPAIR_API) || !defined(MLK_CONFIG_NO_ENCAPS_API)+#define mlk_polyvec_tobytes MLK_NAMESPACE_K(polyvec_tobytes)+/**+ * Serialize a vector of polynomials.+ *+ * @spec{Implements ByteEncode_12 @[FIPS203, Algorithm 5]. Extended to+ * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays] and+ * @[FIPS203, 2.4.6, Matrices and Vectors].}+ *+ * @param[out] r Output byte array (needs space for MLKEM_POLYVECBYTES bytes).+ * @param[in]  a Input vector of polynomials. Each polynomial must have+ *               coefficients in [0,1,..,MLKEM_Q-1].+ */+MLK_INTERNAL_API+void mlk_polyvec_tobytes(uint8_t r[MLKEM_POLYVECBYTES], const mlk_polyvec *a)+__contract__(+  requires(memory_no_alias(a, sizeof(mlk_polyvec)))+  requires(memory_no_alias(r, MLKEM_POLYVECBYTES))+  requires(forall(k0, 0, MLKEM_K,+         array_bound(a->vec[k0].coeffs, 0, MLKEM_N, 0, MLKEM_Q)))+  assigns(memory_slice(r, MLKEM_POLYVECBYTES))+);+#endif /* !MLK_CONFIG_NO_KEYPAIR_API || !MLK_CONFIG_NO_ENCAPS_API */++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_polyvec_frombytes MLK_NAMESPACE_K(polyvec_frombytes)+/**+ * De-serialize a vector of polynomials; inverse of mlk_polyvec_tobytes.+ *+ * @spec{Implements ByteDecode_12 @[FIPS203, Algorithm 6]. Extended to+ * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays] and+ * @[FIPS203, 2.4.6, Matrices and Vectors].}+ *+ * @param[out] r Output vector of polynomials. Coefficients will be+ *               normalized in [0,1,..,4095].+ * @param[in]  a Input byte array (of length MLKEM_POLYVECBYTES bytes).+ */+MLK_INTERNAL_API+void mlk_polyvec_frombytes(mlk_polyvec *r, const uint8_t a[MLKEM_POLYVECBYTES])+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_polyvec)))+  requires(memory_no_alias(a, MLKEM_POLYVECBYTES))+  assigns(memory_slice(r, sizeof(mlk_polyvec)))+  ensures(forall(k0, 0, MLKEM_K,+        array_bound(r->vec[k0].coeffs, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT)))+);+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++#define mlk_polyvec_ntt MLK_NAMESPACE_K(polyvec_ntt)+/**+ * Apply forward NTT to all elements of a vector of polynomials.+ *+ * The input is assumed to be in normal order and coefficient-wise bound by+ * MLKEM_Q in absolute value.+ *+ * The output polynomial is in bitreversed order, and coefficient-wise bound+ * by MLK_NTT_BOUND in absolute value.+ *+ * @spec{Implements @[FIPS203, Algorithm 9, NTT]. Extended to vectors as per+ * @[FIPS203, 2.4.6, Matrices and Vectors].}+ *+ * @param[in,out] r Input/output vector of polynomials.+ */+MLK_INTERNAL_API+void mlk_polyvec_ntt(mlk_polyvec *r)+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_polyvec)))+  requires(forall(j, 0, MLKEM_K,+  array_abs_bound(r->vec[j].coeffs, 0, MLKEM_N, MLKEM_Q)))+  assigns(memory_slice(r, sizeof(mlk_polyvec)))+  ensures(forall(j, 0, MLKEM_K,+  array_abs_bound(r->vec[j].coeffs, 0, MLKEM_N, MLK_NTT_BOUND)))+);++#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_polyvec_invntt_tomont MLK_NAMESPACE_K(polyvec_invntt_tomont)+/**+ * Apply inverse NTT to all elements of a vector of polynomials and multiply+ * by Montgomery factor 2^16.+ *+ * The input is assumed to be in bitreversed order, and can have arbitrary+ * coefficients in int16_t.+ *+ * The output polynomial is in normal order, and coefficient-wise bound by+ * MLK_INVNTT_BOUND in absolute value.+ *+ * @spec{Implements @[FIPS203, Algorithm 10, NTT^{-1}]. Extended to vectors+ * as per @[FIPS203, 2.4.6, Matrices and Vectors].}+ *+ * @param[in,out] r Input/output vector of polynomials.+ */+MLK_INTERNAL_API+void mlk_polyvec_invntt_tomont(mlk_polyvec *r)+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_polyvec)))+  assigns(memory_slice(r, sizeof(mlk_polyvec)))+  ensures(forall(j, 0, MLKEM_K,+  array_abs_bound(r->vec[j].coeffs, 0, MLKEM_N, MLK_INVNTT_BOUND)))+);+#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++#define mlk_polyvec_basemul_acc_montgomery_cached \+  MLK_NAMESPACE_K(polyvec_basemul_acc_montgomery_cached)+/**+ * Scalar product of two vectors of polynomials in NTT domain, using+ * mulcache for the second operand.+ *+ * Bounds: every coefficient of @p a is assumed to be in [0,1,..,4095]. No+ * bounds guarantees for the coefficients in the result.+ *+ * @spec{Implements @[FIPS203, Section 2.4.7, Eq (2.14)], @[FIPS203,+ * Algorithm 11, MultiplyNTTs], and @[FIPS203, Algorithm 12,+ * BaseCaseMultiply].}+ *+ * @param[out] r       Output polynomial.+ * @param[in]  a       First input polynomial vector.+ * @param[in]  b       Second input polynomial vector.+ * @param[in]  b_cache Mulcache for the second input polynomial vector. Can+ *                     be computed via mlk_polyvec_mulcache_compute().+ */+MLK_INTERNAL_API+void mlk_polyvec_basemul_acc_montgomery_cached(+    mlk_poly *r, const mlk_polyvec *a, const mlk_polyvec *b,+    const mlk_polyvec_mulcache *b_cache)+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  requires(memory_no_alias(a, sizeof(mlk_polyvec)))+  requires(memory_no_alias(b, sizeof(mlk_polyvec)))+  requires(memory_no_alias(b_cache, sizeof(mlk_polyvec_mulcache)))+  requires(forall(k1, 0, MLKEM_K,+     array_bound(a->vec[k1].coeffs, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT)))+  assigns(memory_slice(r, sizeof(mlk_poly)))+);++#define mlk_polyvec_mulcache_compute MLK_NAMESPACE_K(polyvec_mulcache_compute)+/**+ * Compute the mulcache for a vector of polynomials in NTT domain.+ *+ * The mulcache of a degree-2 polynomial b := b0 + b1*X in Fq[X]/(X^2-zeta)+ * is the value b1*zeta, needed when computing products of b in+ * Fq[X]/(X^2-zeta).+ *+ * The mulcache of a polynomial in NTT domain -- which is a 128-tuple of+ * degree-2 polynomials in Fq[X]/(X^2-zeta), for varying zeta, is the+ * 128-tuple of mulcaches of those polynomials.+ *+ * The mulcache of a vector of polynomials is the vector of mulcaches of+ * its entries.+ *+ * @spec{Caches `b_1 * \gamma` in @[FIPS203, Algorithm 12, BaseCaseMultiply,+ * L1].}+ *+ * @param[out] x Mulcache to be populated.+ * @param[in]  a Input polynomial vector.+ */+/*+ * NOTE: The default C implementation of this function populates+ * the mulcache with values in (-q,q), but this is not needed for the+ * higher level safety proofs, and thus not part of the spec.+ */+MLK_INTERNAL_API+void mlk_polyvec_mulcache_compute(mlk_polyvec_mulcache *x, const mlk_polyvec *a)+__contract__(+  requires(memory_no_alias(x, sizeof(mlk_polyvec_mulcache)))+  requires(memory_no_alias(a, sizeof(mlk_polyvec)))+  assigns(memory_slice(x, sizeof(mlk_polyvec_mulcache)))+);++#define mlk_polyvec_reduce MLK_NAMESPACE_K(polyvec_reduce)+/**+ * Apply Barrett reduction to each coefficient of each element of a vector+ * of polynomials. For details of the Barrett reduction see comments in+ * poly.c.+ *+ * @spec{Normalizes on unsigned canonical representatives ahead of calling+ * @[FIPS203, Compress_d, Eq (4.7)]. This is not made explicit in FIPS 203.}+ *+ * @param[in,out] r Input/output polynomial vector.+ */+/*+ * NOTE: The semantics of mlk_polyvec_reduce() is different in+ *       the reference implementation, which requires+ *       signed canonical output data. Unsigned canonical+ *       outputs are better suited to the only remaining+ *       use of mlk_poly_reduce() in the context of (de)serialization.+ */+MLK_INTERNAL_API+void mlk_polyvec_reduce(mlk_polyvec *r)+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_polyvec)))+  assigns(memory_slice(r, sizeof(mlk_polyvec)))+  ensures(forall(k0, 0, MLKEM_K,+    array_bound(r->vec[k0].coeffs, 0, MLKEM_N, 0, MLKEM_Q)))+);++#define mlk_polyvec_add MLK_NAMESPACE_K(polyvec_add)+/**+ * Add vectors of polynomials.+ *+ * The coefficients of @p r and @p b must be such that the addition does+ * not overflow. Otherwise, the behaviour of this function is undefined.+ *+ * The coefficients returned in @p *r are in int16_t which is sufficient to+ * prove type-safety of calling units. Therefore, no stronger ensures clause+ * is required on this function.+ *+ * @spec{@[FIPS203, 2.4.5, Arithmetic With Polynomials and NTT+ * Representations]. Used in @[FIPS203, Algorithm 14 (K-PKE.Encrypt), L19].}+ *+ * @param[in,out] r Input-output vector of polynomials to be added to.+ * @param[in]     b Second input vector of polynomials.+ */+MLK_INTERNAL_API+void mlk_polyvec_add(mlk_polyvec *r, const mlk_polyvec *b)+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_polyvec)))+  requires(memory_no_alias(b, sizeof(mlk_polyvec)))+  requires(forall(j0, 0, MLKEM_K,+          forall(k0, 0, MLKEM_N,+            (int32_t)r->vec[j0].coeffs[k0] + b->vec[j0].coeffs[k0] <= INT16_MAX)))+  requires(forall(j1, 0, MLKEM_K,+          forall(k1, 0, MLKEM_N,+            (int32_t)r->vec[j1].coeffs[k1] + b->vec[j1].coeffs[k1] >= INT16_MIN)))+  assigns(memory_slice(r, sizeof(mlk_polyvec)))+);++#if !defined(MLK_CONFIG_NO_KEYPAIR_API)+#define mlk_polyvec_tomont MLK_NAMESPACE_K(polyvec_tomont)+/**+ * In-place conversion of all coefficients of a polynomial vector from the+ * normal domain to the Montgomery domain.+ *+ * Bounds: output < MLKEM_Q in absolute value.+ *+ * @spec{Internal normalization required in `mlk_indcpa_keypair_derand` as+ * part of matrix-vector multiplication @[FIPS203, Algorithm 13, K-PKE.KeyGen,+ * L18].}+ *+ * @param[in,out] r Input/output polynomial vector.+ */+MLK_INTERNAL_API+void mlk_polyvec_tomont(mlk_polyvec *r)+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_polyvec)))+  assigns(memory_slice(r, sizeof(mlk_polyvec)))+  ensures(forall(j, 0, MLKEM_K,+    array_abs_bound(r->vec[j].coeffs, 0, MLKEM_N, MLKEM_Q)))+);+#endif /* !MLK_CONFIG_NO_KEYPAIR_API */++#if !defined(MLK_CONFIG_NO_KEYPAIR_API) ||                              \+    (MLKEM_ETA1 == MLKEM_ETA2 && (!defined(MLK_CONFIG_NO_ENCAPS_API) || \+                                  !defined(MLK_CONFIG_NO_DECAPS_API)))+#define mlk_poly_getnoise_eta1_4x MLK_NAMESPACE_K(poly_getnoise_eta1_4x)+/**+ * Batch sample four polynomials deterministically from a seed and nonces,+ * with output polynomials close to centered binomial distribution with+ * parameter MLKEM_ETA1.+ *+ * @spec{Implements 4x `SamplePolyCBD_{eta1} (PRF_{eta1} (sigma, N))`:+ * @[FIPS203, Algorithm 8, SamplePolyCBD_eta] and @[FIPS203, Eq (4.3),+ * PRF_eta]. `SamplePolyCBD_{eta1} (PRF_{eta1} (sigma, N))` appears in+ * @[FIPS203, Algorithm 13, K-PKE.KeyGen, L{9, 13}] and @[FIPS203,+ * Algorithm 14, K-PKE.Encrypt, L10].}+ *+ * @param[out] r0     Output polynomial.+ * @param[out] r1     Output polynomial.+ * @param[out] r2     Output polynomial.+ * @param[out] r3     Output polynomial. May be NULL.+ * @param[in]  seed   Input seed (of length MLKEM_SYMBYTES bytes).+ * @param      nonce0 One-byte input nonce.+ * @param      nonce1 One-byte input nonce.+ * @param      nonce2 One-byte input nonce.+ * @param      nonce3 One-byte input nonce.+ */+MLK_INTERNAL_API+void mlk_poly_getnoise_eta1_4x(mlk_poly *r0, mlk_poly *r1, mlk_poly *r2,+                               mlk_poly *r3, const uint8_t seed[MLKEM_SYMBYTES],+                               uint8_t nonce0, uint8_t nonce1, uint8_t nonce2,+                               uint8_t nonce3)+__contract__(+  requires(memory_no_alias(seed, MLKEM_SYMBYTES))+  requires(memory_no_alias(r0, sizeof(mlk_poly)))+  requires(memory_no_alias(r1, sizeof(mlk_poly)))+  requires(memory_no_alias(r2, sizeof(mlk_poly)))+  requires(r3 == NULL || memory_no_alias(r3, sizeof(mlk_poly)))+  assigns(memory_slice(r0, sizeof(mlk_poly)))+  assigns(memory_slice(r1, sizeof(mlk_poly)))+  assigns(memory_slice(r2, sizeof(mlk_poly)))+  assigns(r3 != NULL: memory_slice(r3, sizeof(mlk_poly)))+  ensures(array_abs_bound(r0->coeffs,0, MLKEM_N, MLKEM_ETA1 + 1))+  ensures(array_abs_bound(r1->coeffs,0, MLKEM_N, MLKEM_ETA1 + 1))+  ensures(array_abs_bound(r2->coeffs,0, MLKEM_N, MLKEM_ETA1 + 1))+  ensures(r3 != NULL ==> array_abs_bound(r3->coeffs,0, MLKEM_N, MLKEM_ETA1 + 1))+);++#if MLKEM_ETA1 == MLKEM_ETA2+/*+ * We only require mlk_poly_getnoise_eta2_4x for ml-kem-768 and ml-kem-1024+ * where MLKEM_ETA2 = MLKEM_ETA1 = 2.+ * For ml-kem-512, mlk_poly_getnoise_eta1122_4x is used instead.+ */+#define mlk_poly_getnoise_eta2_4x mlk_poly_getnoise_eta1_4x+#endif /* MLKEM_ETA1 == MLKEM_ETA2 */+#endif /* !MLK_CONFIG_NO_KEYPAIR_API || (MLKEM_ETA1 == MLKEM_ETA2 && \+          (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API)) */++#if (MLKEM_K == 2 || MLKEM_K == 4) && \+    (!defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API))+#define mlk_poly_getnoise_eta2 MLK_NAMESPACE_K(poly_getnoise_eta2)+/**+ * Sample a polynomial deterministically from a seed and a nonce, with+ * output polynomial close to centered binomial distribution with parameter+ * MLKEM_ETA2.+ *+ * @spec{Implements `SamplePolyCBD_{eta2} (PRF_{eta2} (sigma, N))`:+ * @[FIPS203, Algorithm 8, SamplePolyCBD_eta] and @[FIPS203, Eq (4.3),+ * PRF_eta]. `SamplePolyCBD_{eta2} (PRF_{eta2} (sigma, N))` appears in+ * @[FIPS203, Algorithm 14, K-PKE.Encrypt, L14].}+ *+ * @param[out] r     Output polynomial.+ * @param[in]  seed  Input seed (of length MLKEM_SYMBYTES bytes).+ * @param      nonce One-byte input nonce.+ */+MLK_INTERNAL_API+void mlk_poly_getnoise_eta2(mlk_poly *r, const uint8_t seed[MLKEM_SYMBYTES],+                            uint8_t nonce)+__contract__(+  requires(memory_no_alias(r, sizeof(mlk_poly)))+  requires(memory_no_alias(seed, MLKEM_SYMBYTES))+  assigns(memory_slice(r, sizeof(mlk_poly)))+  ensures(array_abs_bound(r->coeffs, 0, MLKEM_N, MLKEM_ETA2 + 1))+);+#endif /* (MLKEM_K == 2 || MLKEM_K == 4) && (!MLK_CONFIG_NO_ENCAPS_API || \+          !MLK_CONFIG_NO_DECAPS_API) */++#if MLKEM_K == 2 && \+    (!defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API))+#define mlk_poly_getnoise_eta1122_4x MLK_NAMESPACE_K(poly_getnoise_eta1122_4x)+/**+ * Batch sample four polynomials deterministically from a seed and nonces,+ * with output polynomials close to centered binomial distribution with+ * parameter MLKEM_ETA1 and MLKEM_ETA2.+ *+ * @spec{Implements two instances each of+ * `SamplePolyCBD_{eta1} (PRF_{eta1} (sigma, N))` and+ * `SamplePolyCBD_{eta2} (PRF_{eta2} (sigma, N))`:+ * @[FIPS203, Algorithm 8, SamplePolyCBD_eta] and @[FIPS203, Eq (4.3),+ * PRF_eta]. `SamplePolyCBD_{eta2} (PRF_{eta2} (sigma, N))` appears in+ * @[FIPS203, Algorithm 14, K-PKE.Encrypt, L14].}+ *+ * @param[out] r0     Output polynomial.+ * @param[out] r1     Output polynomial.+ * @param[out] r2     Output polynomial.+ * @param[out] r3     Output polynomial.+ * @param[in]  seed   Input seed (of length MLKEM_SYMBYTES bytes).+ * @param      nonce0 One-byte input nonce.+ * @param      nonce1 One-byte input nonce.+ * @param      nonce2 One-byte input nonce.+ * @param      nonce3 One-byte input nonce.+ */+MLK_INTERNAL_API+void mlk_poly_getnoise_eta1122_4x(mlk_poly *r0, mlk_poly *r1, mlk_poly *r2,+                                  mlk_poly *r3,+                                  const uint8_t seed[MLKEM_SYMBYTES],+                                  uint8_t nonce0, uint8_t nonce1,+                                  uint8_t nonce2, uint8_t nonce3)+__contract__(+  requires(memory_no_alias(r0, sizeof(mlk_poly)))+  requires(memory_no_alias(r1, sizeof(mlk_poly)))+  requires(memory_no_alias(r2, sizeof(mlk_poly)))+  requires(memory_no_alias(r3, sizeof(mlk_poly)))+  requires(memory_no_alias(seed, MLKEM_SYMBYTES))+  assigns(memory_slice(r0, sizeof(mlk_poly)))+  assigns(memory_slice(r1, sizeof(mlk_poly)))+  assigns(memory_slice(r2, sizeof(mlk_poly)))+  assigns(memory_slice(r3, sizeof(mlk_poly)))+  ensures(array_abs_bound(r0->coeffs,0, MLKEM_N, MLKEM_ETA1 + 1)+       && array_abs_bound(r1->coeffs,0, MLKEM_N, MLKEM_ETA1 + 1)+       && array_abs_bound(r2->coeffs,0, MLKEM_N, MLKEM_ETA2 + 1)+       && array_abs_bound(r3->coeffs,0, MLKEM_N, MLKEM_ETA2 + 1))+);+#endif /* MLKEM_K == 2 && (!MLK_CONFIG_NO_ENCAPS_API || \+          !MLK_CONFIG_NO_DECAPS_API) */++#endif /* !MLK_POLY_K_H */
+ cbits/mlkem/src/randombytes.h view
@@ -0,0 +1,52 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLK_RANDOMBYTES_H+#define MLK_RANDOMBYTES_H+++#include "cbmc.h"+#include "common.h"++#if !defined(MLK_CONFIG_NO_RANDOMIZED_API)+#if !defined(MLK_CONFIG_CUSTOM_RANDOMBYTES)+/**+ * Fill a buffer with cryptographically secure random bytes.+ *+ * mlkem-native does not provide an implementation of this function.+ * It must be provided by the consumer.+ *+ * To use a custom random byte source with a different name or signature,+ * set MLK_CONFIG_CUSTOM_RANDOMBYTES and define mlk_randombytes directly.+ *+ * @param[out] out    Output buffer.+ * @param      outlen Number of random bytes to write.+ *+ * @retval 0     Success.+ * @retval other Failure; top-level APIs propagate this as MLK_ERR_RNG_FAIL.+ */+int randombytes(uint8_t *out, size_t outlen);++/**+ * Internal wrapper around randombytes().+ *+ * Fills a buffer with cryptographically secure random bytes.+ *+ * This function can be replaced by setting MLK_CONFIG_CUSTOM_RANDOMBYTES+ * and defining mlk_randombytes directly.+ *+ * @param[out] out    Output buffer.+ * @param      outlen Number of random bytes to write.+ *+ * @retval 0     Success.+ * @retval other Failure; top-level APIs propagate this as MLK_ERR_RNG_FAIL.+ */+MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_randombytes(uint8_t *out, size_t outlen)+__contract__(+  requires(memory_no_alias(out, outlen))+  assigns(memory_slice(out, outlen))) { return randombytes(out, outlen); }+#endif /* !MLK_CONFIG_CUSTOM_RANDOMBYTES */+#endif /* !MLK_CONFIG_NO_RANDOMIZED_API */+#endif /* !MLK_RANDOMBYTES_H */
+ cbits/mlkem/src/sampling.c view
@@ -0,0 +1,371 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ *+ * - [REF]+ *   CRYSTALS-Kyber C reference implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/ref+ */++#include "common.h"+#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)++#include "debug.h"+#include "sampling.h"+#include "symmetric.h"+#include "verify.h"++/* Reference: `rej_uniform()` in the reference implementation @[REF].+ *            - Our signature differs from the reference implementation+ *              in that it adds the offset and always expects the base of the+ *              target buffer. This avoids shifting the buffer base in the+ *              caller, which appears tricky to reason about. */+MLK_STATIC_TESTABLE unsigned mlk_rej_uniform_c(int16_t *r, unsigned target,+                                               unsigned offset,+                                               const uint8_t *buf,+                                               unsigned buflen)+__contract__(+  requires(offset <= target && target <= 4096 && buflen <= 4096 && buflen % 3 == 0)+  requires(memory_no_alias(r, sizeof(int16_t) * target))+  requires(memory_no_alias(buf, buflen))+  requires(array_bound(r, 0, offset, 0, MLKEM_Q))+  assigns(memory_slice(r, sizeof(int16_t) * target))+  ensures(offset <= return_value && return_value <= target)+  ensures(array_bound(r, 0, return_value, 0, MLKEM_Q)))+{+  unsigned ctr, pos;+  int16_t val0, val1;++  mlk_assert_bound(r, offset, 0, MLKEM_Q);++  ctr = offset;+  pos = 0;+  /* pos + 3 cannot overflow due to the assumption buflen <= 4096 */+  while (ctr < target && pos + 3 <= buflen)+  __loop__(+    invariant(offset <= ctr && ctr <= target && pos <= buflen)+    invariant(array_bound(r, 0, ctr, 0, MLKEM_Q))+    decreases(buflen - pos))+  {+    /* Safety:+     * - The explicit cast to uint16_t ensures that << 8 does+     *   not signed-overflow even on a 16-bit system.+     * - The conversion to int16_t is safe due to the explicit 0xFFF+     *   truncation.+     */+    val0 = (int16_t)(((buf[pos + 0] >> 0) | ((uint16_t)buf[pos + 1] << 8)) &+                     0xFFF);+    val1 = (int16_t)(((buf[pos + 1] >> 4) | (buf[pos + 2] << 4)) & 0xFFF);+    pos += 3;++    if (val0 < MLKEM_Q)+    {+      r[ctr++] = val0;+    }+    if (ctr < target && val1 < MLKEM_Q)+    {+      r[ctr++] = val1;+    }+  }++  mlk_assert_bound(r, ctr, 0, MLKEM_Q);+  return ctr;+}++/**+ * Run rejection sampling on uniform random bytes to generate uniform random+ * integers mod MLKEM_Q.+ *+ * @reference{`rej_uniform()` in the reference implementation @[REF]. Our+ * signature differs from the reference in that it adds the offset and always+ * expects the base of the target buffer; this avoids shifting the buffer+ * base in the caller, which is tricky to reason about. Has an optional+ * fallback to a native implementation.}+ *+ * @param[out] r      Output buffer.+ * @param      target Requested number of 16-bit integers (uniform mod MLKEM_Q).+ *                    Must be <= 4096.+ * @param      offset Number of 16-bit integers that have already been+ *                    sampled. Must be <= @p target.+ * @param[in]  buf    Input buffer (assumed to be uniform random bytes).+ * @param      buflen Length of input buffer in bytes. Must be <= 4096 and a+ *                    multiple of 3.+ *+ * @note Strictly speaking, only a few values of @p buflen near UINT_MAX need+ *       excluding. The limit of 4096 is somewhat arbitrary but sufficient+ *       for all uses of this function. Similarly, the actual limit for+ *       @p target is UINT_MAX/2.+ *+ * @return New offset of sampled 16-bit integers, at most @p target and at+ *         least the initial @p offset. If the new offset is strictly less+ *         than @p target, the entire input buffer is guaranteed to have been+ *         consumed; otherwise no information is provided on how many bytes+ *         of the input buffer have been consumed.+ */+static unsigned mlk_rej_uniform(int16_t *r, unsigned target, unsigned offset,+                                const uint8_t *buf, unsigned buflen)+__contract__(+  requires(offset <= target && target <= 4096 && buflen <= 4096 && buflen % 3 == 0)+  requires(memory_no_alias(r, sizeof(int16_t) * target))+  requires(memory_no_alias(buf, buflen))+  requires(array_bound(r, 0, offset, 0, MLKEM_Q))+  assigns(memory_slice(r, sizeof(int16_t) * target))+  ensures(offset <= return_value && return_value <= target)+  ensures(array_bound(r, 0, return_value, 0, MLKEM_Q))+)+{+#if defined(MLK_USE_NATIVE_REJ_UNIFORM)+  if (offset == 0)+  {+    int ret;+    ret = mlk_rej_uniform_native(r, target, buf, buflen);+    if (ret != MLK_NATIVE_FUNC_FALLBACK)+    {+      unsigned res = (unsigned)ret;+      mlk_assert_bound(r, res, 0, MLKEM_Q);+      return res;+    }+  }+#endif /* MLK_USE_NATIVE_REJ_UNIFORM */++  return mlk_rej_uniform_c(r, target, offset, buf, buflen);+}++#ifndef MLKEM_GEN_MATRIX_NBLOCKS+#define MLKEM_GEN_MATRIX_NBLOCKS                                       \+  ((12 * MLKEM_N / 8 * ((uint32_t)1 << 12) / MLKEM_Q + MLK_XOF_RATE) / \+   MLK_XOF_RATE)+#endif++#if !defined(MLK_CONFIG_SERIAL_FIPS202_ONLY)+/* Reference: Does not exist in the reference implementation @[REF].+ *            - x4-batched version of `rej_uniform()` from the+ *              reference implementation, leveraging x4-batched Keccak-f1600. */+MLK_INTERNAL_API+void mlk_poly_rej_uniform_x4(mlk_poly *vec0, mlk_poly *vec1, mlk_poly *vec2,+                             mlk_poly *vec3,+                             uint8_t seed[4][MLK_ALIGN_UP(MLKEM_SYMBYTES + 2)])+{+  /* Temporary buffers for XOF output before rejection sampling */+  MLK_ALIGN uint8_t+      buf[4][MLK_ALIGN_UP(MLKEM_GEN_MATRIX_NBLOCKS * MLK_XOF_RATE)];++  /* Tracks the number of coefficients we have already sampled */+  unsigned ctr[4];+  mlk_xof_x4_ctx statex;+  unsigned buflen;++  mlk_xof_x4_init(&statex);+  mlk_xof_x4_absorb(&statex, seed, MLKEM_SYMBYTES + 2);++  /*+   * Initially, squeeze heuristic number of MLKEM_GEN_MATRIX_NBLOCKS.+   * This should generate the matrix entries with high probability.+   */+  mlk_xof_x4_squeezeblocks(buf, MLKEM_GEN_MATRIX_NBLOCKS, &statex);+  buflen = MLKEM_GEN_MATRIX_NBLOCKS * MLK_XOF_RATE;+  ctr[0] = mlk_rej_uniform(vec0->coeffs, MLKEM_N, 0, buf[0], buflen);+  ctr[1] = mlk_rej_uniform(vec1->coeffs, MLKEM_N, 0, buf[1], buflen);+  ctr[2] = mlk_rej_uniform(vec2->coeffs, MLKEM_N, 0, buf[2], buflen);+  ctr[3] = mlk_rej_uniform(vec3->coeffs, MLKEM_N, 0, buf[3], buflen);++  /*+   * So long as not all matrix entries have been generated, squeeze+   * one more block a time until we're done.+   */+  buflen = MLK_XOF_RATE;+  while (ctr[0] < MLKEM_N || ctr[1] < MLKEM_N || ctr[2] < MLKEM_N ||+         ctr[3] < MLKEM_N)+  __loop__(+    assigns(ctr, statex,+            memory_slice(vec0, sizeof(mlk_poly)),+            memory_slice(vec1, sizeof(mlk_poly)),+            memory_slice(vec2, sizeof(mlk_poly)),+            memory_slice(vec3, sizeof(mlk_poly)),+            object_whole(buf))+    invariant(ctr[0] <= MLKEM_N && ctr[1] <= MLKEM_N)+    invariant(ctr[2] <= MLKEM_N && ctr[3] <= MLKEM_N)+    invariant(array_bound(vec0->coeffs, 0, ctr[0], 0, MLKEM_Q))+    invariant(array_bound(vec1->coeffs, 0, ctr[1], 0, MLKEM_Q))+    invariant(array_bound(vec2->coeffs, 0, ctr[2], 0, MLKEM_Q))+    invariant(array_bound(vec3->coeffs, 0, ctr[3], 0, MLKEM_Q)))+  {+    mlk_xof_x4_squeezeblocks(buf, 1, &statex);+    ctr[0] = mlk_rej_uniform(vec0->coeffs, MLKEM_N, ctr[0], buf[0], buflen);+    ctr[1] = mlk_rej_uniform(vec1->coeffs, MLKEM_N, ctr[1], buf[1], buflen);+    ctr[2] = mlk_rej_uniform(vec2->coeffs, MLKEM_N, ctr[2], buf[2], buflen);+    ctr[3] = mlk_rej_uniform(vec3->coeffs, MLKEM_N, ctr[3], buf[3], buflen);+  }++  mlk_xof_x4_release(&statex);++  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  mlk_zeroize(buf, sizeof(buf));+}+#endif /* !MLK_CONFIG_SERIAL_FIPS202_ONLY */++MLK_INTERNAL_API+void mlk_poly_rej_uniform(mlk_poly *entry, uint8_t seed[MLKEM_SYMBYTES + 2])+{+  mlk_xof_ctx state;+  MLK_ALIGN uint8_t buf[MLKEM_GEN_MATRIX_NBLOCKS * MLK_XOF_RATE];+  unsigned ctr, buflen;++  mlk_xof_init(&state);+  mlk_xof_absorb(&state, seed, MLKEM_SYMBYTES + 2);++  /* Initially, squeeze + sample heuristic number of MLKEM_GEN_MATRIX_NBLOCKS.+   */+  /* This should generate the matrix entry with high probability. */+  mlk_xof_squeezeblocks(buf, MLKEM_GEN_MATRIX_NBLOCKS, &state);+  buflen = MLKEM_GEN_MATRIX_NBLOCKS * MLK_XOF_RATE;+  ctr = mlk_rej_uniform(entry->coeffs, MLKEM_N, 0, buf, buflen);++  /* Squeeze + sample one more block a time until we're done */+  buflen = MLK_XOF_RATE;+  while (ctr < MLKEM_N)+  __loop__(+    assigns(ctr, state, memory_slice(entry, sizeof(mlk_poly)), object_whole(buf))+    invariant(ctr <= MLKEM_N)+    invariant(array_bound(entry->coeffs, 0, ctr, 0, MLKEM_Q)))+  {+    mlk_xof_squeezeblocks(buf, 1, &state);+    ctr = mlk_rej_uniform(entry->coeffs, MLKEM_N, ctr, buf, buflen);+  }++  mlk_xof_release(&state);++  /* Specification: Partially implements+   * @[FIPS203, Section 3.3, Destruction of intermediate values] */+  mlk_zeroize(buf, sizeof(buf));+}++#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_ETA1 == 2 || \+    !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+/**+ * Load 4 bytes into a 32-bit integer in little-endian order.+ *+ * @reference{`load32_littleendian()` in the reference implementation @[REF].}+ *+ * @param[in] x Input byte array.+ *+ * @return 32-bit unsigned integer loaded from @p x.+ */+static uint32_t mlk_load32_littleendian(const uint8_t x[4])+{+  uint32_t r;+  r = (uint32_t)x[0];+  r |= (uint32_t)x[1] << 8;+  r |= (uint32_t)x[2] << 16;+  r |= (uint32_t)x[3] << 24;+  return r;+}++/* Reference: `cbd2()` in the reference implementation @[REF]. */+MLK_INTERNAL_API+void mlk_poly_cbd2(mlk_poly *r, const uint8_t buf[2 * MLKEM_N / 4])+{+  unsigned i;+  for (i = 0; i < MLKEM_N / 8; i++)+  __loop__(+    invariant(i <= MLKEM_N / 8)+    invariant(array_abs_bound(r->coeffs, 0, 8 * i, 3))+    decreases(MLKEM_N / 8 - i))+  {+    unsigned j;+    uint32_t t = mlk_load32_littleendian(buf + 4 * i);+    uint32_t d = t & 0x55555555;+    d += (t >> 1) & 0x55555555;++    for (j = 0; j < 8; j++)+    __loop__(+      invariant(i <= MLKEM_N / 8 && j <= 8)+      invariant(array_abs_bound(r->coeffs, 0, 8 * i + j, 3))+      decreases(8 - j))+    {+      /* Safety: The & 0x3 masks each value to 2 bits (range [0, 3]), so the+       * truncation and subsequent subtraction in int16_t is lossless. */+      const int16_t a = (int16_t)((d >> (4 * j + 0)) & 0x3);+      const int16_t b = (int16_t)((d >> (4 * j + 2)) & 0x3);+      r->coeffs[8 * i + j] = (int16_t)(a - b);+    }+  }+}+#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_ETA1 == 2 || \+          !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_ETA1 == 3+/**+ * Load 3 bytes into a 32-bit integer in little-endian order.+ *+ * This function is only needed for ML-KEM-512.+ *+ * @reference{`load24_littleendian()` in the reference implementation @[REF].}+ *+ * @param[in] x Input byte array.+ *+ * @return 32-bit unsigned integer loaded from @p x (most significant byte+ *         is zero).+ */+static uint32_t mlk_load24_littleendian(const uint8_t x[3])+{+  uint32_t r;+  r = (uint32_t)x[0];+  r |= (uint32_t)x[1] << 8;+  r |= (uint32_t)x[2] << 16;+  return r;+}++/* Reference: `cbd3()` in the reference implementation @[REF]. */+MLK_INTERNAL_API+void mlk_poly_cbd3(mlk_poly *r, const uint8_t buf[3 * MLKEM_N / 4])+{+  unsigned i;+  for (i = 0; i < MLKEM_N / 4; i++)+  __loop__(+    invariant(i <= MLKEM_N / 4)+    invariant(array_abs_bound(r->coeffs, 0, 4 * i, 4))+    decreases(MLKEM_N / 4 - i))+  {+    unsigned j;+    const uint32_t t = mlk_load24_littleendian(buf + 3 * i);+    uint32_t d = t & 0x00249249;+    d += (t >> 1) & 0x00249249;+    d += (t >> 2) & 0x00249249;++    for (j = 0; j < 4; j++)+    __loop__(+      invariant(i <= MLKEM_N / 4 && j <= 4)+      invariant(array_abs_bound(r->coeffs, 0, 4 * i + j, 4))+      decreases(4 - j))+    {+      /* Safety: The & 0x7 masks each value to 3 bits (range [0, 7]), so the+       * truncation and subsequent subtraction in int16_t is lossless. */+      const int16_t a = (int16_t)((d >> (6 * j + 0)) & 0x7);+      const int16_t b = (int16_t)((d >> (6 * j + 3)) & 0x7);+      r->coeffs[4 * i + j] = (int16_t)(a - b);+    }+  }+}+#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_ETA1 == 3 */++#else /* !MLK_CONFIG_MULTILEVEL_NO_SHARED */++MLK_EMPTY_CU(sampling)++#endif /* MLK_CONFIG_MULTILEVEL_NO_SHARED */++/* To facilitate single-compilation-unit (SCU) builds, undefine all macros.+ * Don't modify by hand -- this is auto-generated by scripts/autogen. */+#undef MLKEM_GEN_MATRIX_NBLOCKS
+ cbits/mlkem/src/sampling.h view
@@ -0,0 +1,111 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ */++#ifndef MLK_SAMPLING_H+#define MLK_SAMPLING_H++#include "cbmc.h"+#include "common.h"+#include "poly.h"++#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_ETA1 == 2 || \+    !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)+#define mlk_poly_cbd2 MLK_NAMESPACE(poly_cbd2)+/**+ * Given an array of uniformly random bytes, compute a polynomial with+ * coefficients distributed according to a centered binomial distribution+ * with parameter eta=2.+ *+ * @spec{Implements @[FIPS203, Algorithm 8, SamplePolyCBD_2].}+ *+ * @param[out] r   Output polynomial.+ * @param[in]  buf Input byte array.+ */+MLK_INTERNAL_API+void mlk_poly_cbd2(mlk_poly *r, const uint8_t buf[2 * MLKEM_N / 4]);+#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_ETA1 == 2 || \+          !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */++#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_ETA1 == 3+#define mlk_poly_cbd3 MLK_NAMESPACE(poly_cbd3)+/**+ * Given an array of uniformly random bytes, compute a polynomial with+ * coefficients distributed according to a centered binomial distribution+ * with parameter eta=3.+ *+ * This function is only needed for ML-KEM-512.+ *+ * @spec{Implements @[FIPS203, Algorithm 8, SamplePolyCBD_3].}+ *+ * @param[out] r   Output polynomial.+ * @param[in]  buf Input byte array.+ */+MLK_INTERNAL_API+void mlk_poly_cbd3(mlk_poly *r, const uint8_t buf[3 * MLKEM_N / 4]);+#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_ETA1 == 3 */++#if !defined(MLK_CONFIG_SERIAL_FIPS202_ONLY)+#define mlk_poly_rej_uniform_x4 MLK_NAMESPACE(poly_rej_uniform_x4)+/**+ * Generate four polynomials using rejection sampling on (pseudo-)uniformly+ * random bytes sampled from a seed.+ *+ * @spec{Implements @[FIPS203, Algorithm 7, SampleNTT].}+ *+ * @param[out] vec0 Polynomial to be sampled.+ * @param[out] vec1 Polynomial to be sampled.+ * @param[out] vec2 Polynomial to be sampled.+ * @param[out] vec3 Polynomial to be sampled.+ * @param[in]  seed Consecutive array of 4 seed buffers of size+ *                  MLKEM_SYMBYTES + 2 each, plus padding for alignment.+ */+MLK_INTERNAL_API+void mlk_poly_rej_uniform_x4(mlk_poly *vec0, mlk_poly *vec1, mlk_poly *vec2,+                             mlk_poly *vec3,+                             uint8_t seed[4][MLK_ALIGN_UP(MLKEM_SYMBYTES + 2)])+__contract__(+  requires(memory_no_alias(vec0, sizeof(mlk_poly)))+  requires(memory_no_alias(vec1, sizeof(mlk_poly)))+  requires(memory_no_alias(vec2, sizeof(mlk_poly)))+  requires(memory_no_alias(vec3, sizeof(mlk_poly)))+  requires(memory_no_alias(seed, 4 * MLK_ALIGN_UP(MLKEM_SYMBYTES + 2)))+  assigns(memory_slice(vec0, sizeof(mlk_poly)))+  assigns(memory_slice(vec1, sizeof(mlk_poly)))+  assigns(memory_slice(vec2, sizeof(mlk_poly)))+  assigns(memory_slice(vec3, sizeof(mlk_poly)))+  ensures(array_bound(vec0->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+  ensures(array_bound(vec1->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+  ensures(array_bound(vec2->coeffs, 0, MLKEM_N, 0, MLKEM_Q))+  ensures(array_bound(vec3->coeffs, 0, MLKEM_N, 0, MLKEM_Q)));+#endif /* !MLK_CONFIG_SERIAL_FIPS202_ONLY */++#define mlk_poly_rej_uniform MLK_NAMESPACE(poly_rej_uniform)+/**+ * Generate a polynomial using rejection sampling on (pseudo-)uniformly+ * random bytes sampled from a seed.+ *+ * @spec{Implements @[FIPS203, Algorithm 7, SampleNTT].}+ *+ * @param[out] entry Polynomial to be sampled.+ * @param[in]  seed  Seed buffer of size MLKEM_SYMBYTES + 2.+ */+MLK_INTERNAL_API+void mlk_poly_rej_uniform(mlk_poly *entry, uint8_t seed[MLKEM_SYMBYTES + 2])+__contract__(+  requires(memory_no_alias(entry, sizeof(mlk_poly)))+  requires(memory_no_alias(seed, MLKEM_SYMBYTES + 2))+  assigns(memory_slice(entry, sizeof(mlk_poly)))+  ensures(array_bound(entry->coeffs, 0, MLKEM_N, 0, MLKEM_Q)));++#endif /* !MLK_SAMPLING_H */
+ cbits/mlkem/src/symmetric.h view
@@ -0,0 +1,70 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ */++#ifndef MLK_SYMMETRIC_H+#define MLK_SYMMETRIC_H+++#include "cbmc.h"+#include "common.h"+#include MLK_FIPS202_HEADER_FILE+#if !defined(MLK_CONFIG_SERIAL_FIPS202_ONLY)+#include MLK_FIPS202X4_HEADER_FILE+#endif++/* Macros denoting FIPS 203 specific Hash functions */++/* Hash function H, @[FIPS203, Section 4.1, Eq (4.4)] */+#define mlk_hash_h(OUT, IN, INBYTES) mlk_sha3_256(OUT, IN, INBYTES)++/* Hash function G, @[FIPS203, Section 4.1, Eq (4.5)] */+#define mlk_hash_g(OUT, IN, INBYTES) mlk_sha3_512(OUT, IN, INBYTES)++/* Hash function J, @[FIPS203, Section 4.1, Eq (4.4)] */+#define mlk_hash_j(OUT, IN, INBYTES) \+  mlk_shake256(OUT, MLKEM_SYMBYTES, IN, INBYTES)++/* PRF function, @[FIPS203, Section 4.1, Eq (4.3)]+ * Referring to (eq 4.3), `OUT` is assumed to contain `s || b`. */+#define mlk_prf_eta(ETA, OUT, IN) \+  mlk_shake256(OUT, (ETA) * MLKEM_N / 4, IN, MLKEM_SYMBYTES + 1)+#define mlk_prf_eta1(OUT, IN) mlk_prf_eta(MLKEM_ETA1, OUT, IN)+#define mlk_prf_eta2(OUT, IN) mlk_prf_eta(MLKEM_ETA2, OUT, IN)+#define mlk_prf_eta1_x4(OUT, IN)                                        \+  mlk_shake256x4((OUT)[0], (OUT)[1], (OUT)[2], (OUT)[3],                \+                 (MLKEM_ETA1 * MLKEM_N / 4), (IN)[0], (IN)[1], (IN)[2], \+                 (IN)[3], MLKEM_SYMBYTES + 1)++/* XOF function, FIPS 203 4.1 */+#define mlk_xof_ctx mlk_shake128ctx+#define mlk_xof_x4_ctx mlk_shake128x4ctx+#define mlk_xof_init(CTX) mlk_shake128_init((CTX))+#define mlk_xof_absorb(CTX, IN, INBYTES) \+  mlk_shake128_absorb_once((CTX), (IN), (INBYTES))+#define mlk_xof_squeezeblocks(BUF, NBLOCKS, CTX) \+  mlk_shake128_squeezeblocks((BUF), (NBLOCKS), (CTX))+#define mlk_xof_release(CTX) mlk_shake128_release((CTX))++#define mlk_xof_x4_init(CTX) mlk_shake128x4_init((CTX))+#define mlk_xof_x4_absorb(CTX, IN, INBYTES)                             \+  mlk_shake128x4_absorb_once((CTX), (IN)[0], (IN)[1], (IN)[2], (IN)[3], \+                             (INBYTES))+#define mlk_xof_x4_squeezeblocks(BUF, NBLOCKS, CTX)                    \+  mlk_shake128x4_squeezeblocks((BUF)[0], (BUF)[1], (BUF)[2], (BUF)[3], \+                               (NBLOCKS), (CTX))+#define mlk_xof_x4_release(CTX) mlk_shake128x4_release((CTX))++#define MLK_XOF_RATE SHAKE128_RATE++#endif /* !MLK_SYMMETRIC_H */
+ cbits/mlkem/src/sys.h view
@@ -0,0 +1,320 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#ifndef MLK_SYS_H+#define MLK_SYS_H++#if !defined(MLK_CONFIG_NO_ASM) && (defined(__GNUC__) || defined(__clang__))+#define MLK_HAVE_INLINE_ASM+#endif++/* Try to find endianness, if not forced through CFLAGS already */+#if !defined(MLK_SYS_LITTLE_ENDIAN) && !defined(MLK_SYS_BIG_ENDIAN)+#if defined(__BYTE_ORDER__)+#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__+#define MLK_SYS_LITTLE_ENDIAN+#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__+#define MLK_SYS_BIG_ENDIAN+#else+#error "__BYTE_ORDER__ defined, but don't recognize value."+#endif+#endif /* __BYTE_ORDER__ */++/* MSVC does not define __BYTE_ORDER__. However, MSVC only supports+ * little endian x86, x86_64, and AArch64. It is, hence, safe to assume+ * little endian. */+#if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_AMD64) || \+                          defined(_M_IX86) || defined(_M_ARM64))+#define MLK_SYS_LITTLE_ENDIAN+#endif++#endif /* !MLK_SYS_LITTLE_ENDIAN && !MLK_SYS_BIG_ENDIAN */++/* Check if we're running on an AArch64 little endian system. _M_ARM64 is set by+ * MSVC. */+#if defined(__AARCH64EL__) || defined(_M_ARM64)+#define MLK_SYS_AARCH64+#endif++/* Check if the AArch64 compilation target supports NEON (Advanced SIMD).+ *+ * Some compilers also define __ARM_NEON__, but __ARM_NEON is the most reliable+ * signal. Specifically, clang on Apple appears to keep __ARM_NEON__ set even if+ * -march=armv8-a+nosimd is set.+ *+ * gcc 4.8 -- the first gcc version introducing Neon support -- sets neither+ * __ARM_NEON nor __ARM_NEON__; in fact, there is no preprocessor signal that+ * Neon is enabled. If you use gcc 4.8, you should set __ARM_NEON manually.+ * gcc 4.9 onwards do set __ARM_NEON.+ */+#if defined(MLK_SYS_AARCH64) && defined(__ARM_NEON)+#define MLK_SYS_AARCH64_NEON+#endif++/* Check if we're running on an AArch64 big endian system. */+#if defined(__AARCH64EB__)+#define MLK_SYS_AARCH64_EB+#endif++/* Check if we're running on an Armv8.1-M system with MVE */+#if defined(__ARM_ARCH_8_1M_MAIN__) || defined(__ARM_FEATURE_MVE)+#define MLK_SYS_ARMV81M_MVE+#endif++/* Check if we're running on an x86_64 system. */+#if defined(__x86_64__) || defined(_M_X64) || defined(_M_AMD64)+#define MLK_SYS_X86_64+#if defined(__AVX2__)+#define MLK_SYS_X86_64_AVX2+#endif+#endif /* __x86_64__ || _M_X64 || _M_AMD64 */++#if defined(MLK_SYS_LITTLE_ENDIAN) && defined(__powerpc64__)+#define MLK_SYS_PPC64LE+#endif++#if defined(__riscv) && defined(__riscv_xlen) && __riscv_xlen == 64+#define MLK_SYS_RISCV64+#endif++#if defined(MLK_SYS_RISCV64) && defined(__riscv_vector) && \+    defined(__riscv_v_intrinsic)+#define MLK_SYS_RISCV64_RVV+#endif++#if defined(__riscv) && defined(__riscv_xlen) && __riscv_xlen == 32+#define MLK_SYS_RISCV32+#endif++#if defined(_WIN32)+#define MLK_SYS_WINDOWS+#endif++#if defined(__linux__)+#define MLK_SYS_LINUX+#endif++#if defined(__APPLE__)+#define MLK_SYS_APPLE+#endif++#if defined(MLK_FORCE_AARCH64) && !defined(MLK_SYS_AARCH64)+#error "MLK_FORCE_AARCH64 is set, but we don't seem to be on an AArch64 system."+#endif++#if defined(MLK_FORCE_AARCH64_EB) && !defined(MLK_SYS_AARCH64_EB)+#error \+    "MLK_FORCE_AARCH64_EB is set, but we don't seem to be on an AArch64 system."+#endif++#if defined(MLK_FORCE_X86_64) && !defined(MLK_SYS_X86_64)+#error "MLK_FORCE_X86_64 is set, but we don't seem to be on an X86_64 system."+#endif++#if defined(MLK_FORCE_PPC64LE) && !defined(MLK_SYS_PPC64LE)+#error "MLK_FORCE_PPC64LE is set, but we don't seem to be on a PPC64LE system."+#endif++#if defined(MLK_FORCE_RISCV64) && !defined(MLK_SYS_RISCV64)+#error "MLK_FORCE_RISCV64 is set, but we don't seem to be on a RISCV64 system."+#endif++#if defined(MLK_FORCE_RISCV32) && !defined(MLK_SYS_RISCV32)+#error "MLK_FORCE_RISCV32 is set, but we don't seem to be on a RISCV32 system."+#endif++/*+ * MLK_INLINE: Hint for inlining.+ * - MSVC: __inline+ * - C99+: inline+ * - GCC/Clang C90: __attribute__((unused)) to silence warnings+ * - Other C90: empty+ */+#if !defined(MLK_INLINE)+#if defined(_MSC_VER)+#define MLK_INLINE __inline+#elif defined(inline) || \+    (defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L)+#define MLK_INLINE inline+#elif defined(__GNUC__) || defined(__clang__)+#define MLK_INLINE __attribute__((unused))+#else+#define MLK_INLINE+#endif+#endif /* !MLK_INLINE */++/*+ * MLK_ALWAYS_INLINE: Force inlining.+ * - MSVC: __forceinline+ * - GCC/Clang C99+: MLK_INLINE __attribute__((always_inline))+ * - Other: MLK_INLINE (no forced inlining)+ */+#if !defined(MLK_ALWAYS_INLINE)+#if defined(_MSC_VER)+#define MLK_ALWAYS_INLINE __forceinline+#elif (defined(__GNUC__) || defined(__clang__)) && \+    (defined(inline) ||                            \+     (defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L))+#define MLK_ALWAYS_INLINE MLK_INLINE __attribute__((always_inline))+#else+#define MLK_ALWAYS_INLINE MLK_INLINE+#endif+#endif /* !MLK_ALWAYS_INLINE */++/*+ * MLK_NOINLINE: Prevent inlining.+ * - MSVC: __declspec(noinline)+ * - GCC/Clang: __attribute__((noinline))+ * - Other: empty+ */+#if !defined(MLK_NOINLINE)+#if defined(_MSC_VER)+#define MLK_NOINLINE __declspec(noinline)+#elif defined(__GNUC__) || defined(__clang__)+#define MLK_NOINLINE __attribute__((noinline))+#else+#define MLK_NOINLINE+#endif+#endif /* !MLK_NOINLINE */++#ifndef MLK_STATIC_TESTABLE+#define MLK_STATIC_TESTABLE static+#endif++/*+ * C90 does not have the restrict compiler directive yet.+ * We don't use it in C90 builds.+ */+#if !defined(restrict)+#if defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L+#define MLK_RESTRICT restrict+#else+#define MLK_RESTRICT+#endif++#else /* !restrict */++#define MLK_RESTRICT restrict+#endif /* restrict */++#define MLK_DEFAULT_ALIGN 32+#define MLK_ALIGN_UP(N) \+  ((((N) + (MLK_DEFAULT_ALIGN - 1)) / MLK_DEFAULT_ALIGN) * MLK_DEFAULT_ALIGN)+#if defined(__GNUC__)+#define MLK_ALIGN __attribute__((aligned(MLK_DEFAULT_ALIGN)))+#elif defined(_MSC_VER)+#define MLK_ALIGN __declspec(align(MLK_DEFAULT_ALIGN))+#else+#define MLK_ALIGN /* No known support for alignment constraints */+#endif+++/* New X86_64 CPUs support control-flow protection using the CET instructions.+ * When enabled (through -fcf-protection=), all compilation units (including+ * empty ones) need to support CET for this to work.+ * For assembly, this means that source files need to signal support for+ * CET by setting the appropriate note.gnu.property section.+ * This can be achieved by including the <cet.h> header in all assembly file.+ * This file also provides the _CET_ENDBR macro which needs to be placed at+ * every potential target of an indirect branch.+ * If CET is enabled _CET_ENDBR maps to the endbr64 instruction, otherwise+ * it is empty.+ * In case the compiler does not support CET (e.g., <gcc8, <clang11),+ * the __CET__ macro is not set and we default to nothing.+ * Note that we only issue _CET_ENDBR instructions through the MLK_ASM_FN_SYMBOL+ * macro as the global symbols are the only possible targets of indirect+ * branches in our code.+ */+#if defined(MLK_SYS_X86_64)+#if defined(__CET__)+#include <cet.h>+#define MLK_CET_ENDBR _CET_ENDBR+#else+#define MLK_CET_ENDBR+#endif+#endif /* MLK_SYS_X86_64 */++#if defined(MLK_CONFIG_CT_TESTING_ENABLED) && !defined(__ASSEMBLER__)+#include <valgrind/memcheck.h>+#define MLK_CT_TESTING_SECRET(ptr, len) \+  VALGRIND_MAKE_MEM_UNDEFINED((ptr), (len))+#define MLK_CT_TESTING_DECLASSIFY(ptr, len) \+  VALGRIND_MAKE_MEM_DEFINED((ptr), (len))+#else /* MLK_CONFIG_CT_TESTING_ENABLED && !__ASSEMBLER__ */+#define MLK_CT_TESTING_SECRET(ptr, len) \+  do                                    \+  {                                     \+  } while (0)+#define MLK_CT_TESTING_DECLASSIFY(ptr, len) \+  do                                        \+  {                                         \+  } while (0)+#endif /* !(MLK_CONFIG_CT_TESTING_ENABLED && !__ASSEMBLER__) */++#if defined(__GNUC__) || defined(__clang__)+#define MLK_MUST_CHECK_RETURN_VALUE __attribute__((warn_unused_result))+#else+#define MLK_MUST_CHECK_RETURN_VALUE+#endif++/* The x86_64 assembly backend uses the SysV calling convention. On Windows,+ * where the Microsoft x64 calling convention is the default, it can still be+ * used with compilers that allow choosing the calling convention per+ * function: GCC and Clang support __attribute__((sysv_abi)), which makes+ * calls to the annotated function follow the SysV calling convention.+ *+ * MLK_SYSV_ABI_SUPPORTED signals that the toolchain can call SysV assembly+ * routines; the x86_64 assembly backend is only enabled if it is defined.+ * MLK_SYSV_ABI is the attribute carried by declarations of x86_64 assembly+ * routines. Both macros can be set externally for toolchains offering an+ * equivalent mechanism that is not recognized here. */+#if defined(MLK_SYS_X86_64) && !defined(MLK_SYSV_ABI_SUPPORTED)+#if !defined(MLK_SYS_WINDOWS) || defined(__GNUC__) || defined(__clang__)+#define MLK_SYSV_ABI_SUPPORTED+#endif+#endif++#if !defined(MLK_SYSV_ABI)+#if defined(MLK_SYS_WINDOWS) && defined(MLK_SYSV_ABI_SUPPORTED)+#define MLK_SYSV_ABI __attribute__((sysv_abi))+#else+#define MLK_SYSV_ABI+#endif+#endif /* !MLK_SYSV_ABI */++#if !defined(__ASSEMBLER__)+/* System capability enumeration */+typedef enum+{+  /* x86_64 */+  MLK_SYS_CAP_X86_64_AVX2,+  /* AArch64 */+  MLK_SYS_CAP_AARCH64_NEON,+  MLK_SYS_CAP_AARCH64_SHA3,+  /* Armv8.1-M */+  MLK_SYS_CAP_ARMV81M_MVE+} mlk_sys_cap;++#if !defined(MLK_CONFIG_CUSTOM_CAPABILITY_FUNC)+#include "cbmc.h"++MLK_MUST_CHECK_RETURN_VALUE+static MLK_INLINE int mlk_sys_check_capability(mlk_sys_cap cap)+__contract__(+  ensures(return_value == 0 || return_value == 1)+)+{+  /* By default, we rely on compile-time feature detection/specification:+   * If a feature is enabled at compile-time, we assume it is supported by+   * the host that the resulting library/binary will be built on.+   * If this assumption is not true, you MUST overwrite this function.+   * See the documentation of MLK_CONFIG_CUSTOM_CAPABILITY_FUNC in+   * mlkem_native_config.h for more information. */+  (void)cap;+  return 1;+}+#endif /* !MLK_CONFIG_CUSTOM_CAPABILITY_FUNC */+#endif /* !__ASSEMBLER__ */++#endif /* !MLK_SYS_H */
+ cbits/mlkem/src/verify.c view
@@ -0,0 +1,20 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */+#include "verify.h"++#if !defined(MLK_USE_ASM_VALUE_BARRIER) && \+    !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)+/*+ * Masking value used in constant-time functions from+ * verify.h to block the compiler's range analysis and+ * thereby reduce the risk of compiler-introduced branches.+ */+volatile uint64_t mlk_ct_opt_blocker_u64 = 0;++#else /* !MLK_USE_ASM_VALUE_BARRIER && !MLK_CONFIG_MULTILEVEL_NO_SHARED */++MLK_EMPTY_CU(verify)++#endif /* !(!MLK_USE_ASM_VALUE_BARRIER && !MLK_CONFIG_MULTILEVEL_NO_SHARED) */
+ cbits/mlkem/src/verify.h view
@@ -0,0 +1,447 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/* References+ * ==========+ *+ * - [FIPS203]+ *   FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard+ *   National Institute of Standards and Technology+ *   https://csrc.nist.gov/pubs/fips/203/final+ *+ * - [REF]+ *   CRYSTALS-Kyber C reference implementation+ *   Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé+ *   https://github.com/pq-crystals/kyber/tree/main/ref+ *+ * - [libmceliece]+ *   libmceliece implementation of Classic McEliece+ *   Bernstein, Chou+ *   https://lib.mceliece.org/+ *+ * - [optblocker]+ *   PQC forum post on opt-blockers using volatile globals+ *   Daniel J. Bernstein+ *   https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/hqbtIGFKIpU/m/H14H0wOlBgAJ+ */++#ifndef MLK_VERIFY_H+#define MLK_VERIFY_H+++#include "cbmc.h"+#include "common.h"++/* Constant-time comparisons and conditional operations++   We reduce the risk for compilation into variable-time code+   through the use of 'value barriers'.++   Functionally, a value barrier is a no-op. To the compiler, however,+   it constitutes an arbitrary modification of its input, and therefore+   harden's value propagation and range analysis.++   We consider two approaches to implement a value barrier:+   - An empty inline asm block which marks the target value as clobbered.+   - XOR'ing with the value of a volatile global that's set to 0;+     see @[optblocker] for a discussion of this idea, and+     @[libmceliece, inttypes/crypto_intN.h] for an implementation.++   The first approach is cheap because it only prevents the compiler+   from reasoning about the value of the variable past the barrier,+   but does not directly generate additional instructions.++   The second approach generates redundant loads and XOR operations+   and therefore comes at a higher runtime cost. However, it appears+   more robust towards optimization, as compilers should never drop+   a volatile load.++   We use the empty-ASM value barrier for GCC and clang, and fall+   back to the global volatile barrier otherwise.++   The global value barrier can be forced by setting+   MLK_CONFIG_NO_ASM_VALUE_BARRIER.++*/++#if defined(MLK_HAVE_INLINE_ASM) && !defined(MLK_CONFIG_NO_ASM_VALUE_BARRIER)+#define MLK_USE_ASM_VALUE_BARRIER+#endif++#if !defined(MLK_USE_ASM_VALUE_BARRIER)++/*+ * Declaration of global volatile that the global value barrier+ * is loading from and masking with.+ */+#define mlk_ct_opt_blocker_u64 MLK_NAMESPACE(ct_opt_blocker_u64)+extern volatile uint64_t mlk_ct_opt_blocker_u64;++/* Helper functions for obtaining global masks of various sizes */++/* This contract is not proved but treated as an axiom.+ *+ * Its validity relies on the assumption that the global opt-blocker+ * constant mlk_ct_opt_blocker_u64 is not modified.+ */+static MLK_INLINE uint64_t mlk_ct_get_optblocker_u64(void)+__contract__(ensures(return_value == 0)) { return mlk_ct_opt_blocker_u64; }++static MLK_INLINE uint8_t mlk_ct_get_optblocker_u8(void)+__contract__(ensures(return_value == 0)) { return (uint8_t)mlk_ct_get_optblocker_u64(); }++static MLK_INLINE uint32_t mlk_ct_get_optblocker_u32(void)+__contract__(ensures(return_value == 0)) { return (uint32_t)mlk_ct_get_optblocker_u64(); }++static MLK_INLINE int32_t mlk_ct_get_optblocker_i32(void)+__contract__(ensures(return_value == 0)) { return (int32_t)mlk_ct_get_optblocker_u64(); }++/* Opt-blocker based implementation of value barriers */+static MLK_INLINE uint32_t mlk_value_barrier_u32(uint32_t b)+__contract__(ensures(return_value == b)) { return (b ^ mlk_ct_get_optblocker_u32()); }++static MLK_INLINE int32_t mlk_value_barrier_i32(int32_t b)+__contract__(ensures(return_value == b)) { return (b ^ mlk_ct_get_optblocker_i32()); }++static MLK_INLINE uint8_t mlk_value_barrier_u8(uint8_t b)+__contract__(ensures(return_value == b)) { return (b ^ mlk_ct_get_optblocker_u8()); }++#else /* !MLK_USE_ASM_VALUE_BARRIER */++static MLK_INLINE uint32_t mlk_value_barrier_u32(uint32_t b)+__contract__(ensures(return_value == b))+{+  __asm__ volatile("" : "+r"(b));+  return b;+}++static MLK_INLINE int32_t mlk_value_barrier_i32(int32_t b)+__contract__(ensures(return_value == b))+{+  __asm__ volatile("" : "+r"(b));+  return b;+}++static MLK_INLINE uint8_t mlk_value_barrier_u8(uint8_t b)+__contract__(ensures(return_value == b))+{+  __asm__ volatile("" : "+r"(b));+  return b;+}++#endif /* MLK_USE_ASM_VALUE_BARRIER */++#ifdef CBMC+#pragma CPROVER check push+#pragma CPROVER check disable "conversion"+#endif+/**+ * Cast uint16 value to int16.+ *+ * @param x Input value.+ *+ * @return For uint16_t x, the unique y in int16_t so that x == y mod 2^16.+ *         Concretely:+ *         - x <  32768: returns x+ *         - x >= 32768: returns x - 65536+ */+static MLK_ALWAYS_INLINE int16_t mlk_cast_uint16_to_int16(uint16_t x)+{+  /*+   * PORTABILITY: This relies on uint16_t -> int16_t+   * being implemented as the inverse of int16_t -> uint16_t,+   * which is implementation-defined (C99 6.3.1.3 (3))+   * CBMC (correctly) fails to prove this conversion is OK,+   * so we have to suppress that check here+   */+  return (int16_t)x;+}+#ifdef CBMC+#pragma CPROVER check pop+#endif++/**+ * Cast int32 value to uint16 as per C standard.+ *+ * @param x Input value.+ *+ * @return For int32_t x, the unique y in uint16_t so that x == y mod 2^16.+ */+static MLK_ALWAYS_INLINE uint16_t mlk_cast_int32_to_uint16(int32_t x)+{+  return (uint16_t)(x & (int32_t)UINT16_MAX);+}++/**+ * Cast int16 value to uint16 as per C standard.+ *+ * @param x Input value.+ *+ * @return For int16_t x, the unique y in uint16_t so that x == y mod 2^16.+ */+static MLK_ALWAYS_INLINE uint16_t mlk_cast_int16_to_uint16(int32_t x)+{+  return mlk_cast_int32_to_uint16(x);+}++/**+ * Return 0 if input is non-negative, and -1 otherwise.+ *+ * @reference{Embedded in the polynomial compression function in the+ * reference implementation @[REF]. Used as part of signed->unsigned+ * conversion for modular representatives to detect whether the input is+ * negative. This happens in `mlk_poly_reduce()` here, and as part of+ * polynomial compression functions in the reference implementation. See+ * `mlk_poly_reduce()`. We use value barriers to reduce the risk of+ * compiler-introduced branches.}+ *+ * @param x Value to be converted into a mask.+ *+ * @return Mask value (0 or 0xFFFF).+ */+static MLK_INLINE uint16_t mlk_ct_cmask_neg_i16(int16_t x)+__contract__(ensures(return_value == ((x < 0) ? 0xFFFF : 0)))+{+  int32_t tmp = mlk_value_barrier_i32((int32_t)x);+  /*+   * PORTABILITY: Right-shift on a signed integer is+   * implementation-defined for negative left argument.+   * Here, we assume it's sign-preserving "arithmetic" shift right.+   * See (C99 6.5.7 (5))+   */+  tmp >>= 16;+  return mlk_cast_int32_to_uint16(tmp);+}++/**+ * Return 0 if input is zero, and -1 otherwise.+ *+ * @reference{Embedded in `cmov_int16()` in the reference implementation+ * @[REF]. Uses a value barrier and shift instead of `b = -b` to convert+ * condition into mask.}+ *+ * @param x Value to be converted into a mask.+ *+ * @return Mask value (0 or 0xFFFF).+ */+static MLK_INLINE uint16_t mlk_ct_cmask_nonzero_u16(uint16_t x)+__contract__(ensures(return_value == ((x == 0) ? 0 : 0xFFFF)))+{+  int32_t tmp = mlk_value_barrier_i32(-((int32_t)x));+  /*+   * PORTABILITY: Right-shift on a signed integer is+   * implementation-defined for negative left argument.+   * Here, we assume it's sign-preserving "arithmetic" shift right.+   * See (C99 6.5.7 (5))+   */+  tmp >>= 16;+  return mlk_cast_int32_to_uint16(tmp);+}++/**+ * Return 0 if input is zero, and -1 otherwise.+ *+ * @reference{Embedded in `verify()` and `cmov()` in the reference+ * implementation @[REF]. We include a value barrier not present in the+ * reference implementation, to prevent the compiler from realizing that+ * this function returns a mask.}+ *+ * @param x Value to be converted into a mask.+ *+ * @return Mask value (0 or 0xFF).+ */+static MLK_INLINE uint8_t mlk_ct_cmask_nonzero_u8(uint8_t x)+__contract__(ensures(return_value == ((x == 0) ? 0 : 0xFF)))+{+  uint16_t mask = mlk_ct_cmask_nonzero_u16((uint16_t)x);+  return (uint8_t)(mask & 0xFF);+}++/**+ * Functionally equivalent to cond ? a : b, but implemented with guards+ * against compiler-introduced branches.+ *+ * @spec{With `a = MLKEM_Q_HALF` and `b=0`, this essentially implements+ * `Decompress_1` @[FIPS203, Eq (4.8)] in `mlk_poly_frommsg()`. With+ * `a = x + MLKEM_Q`, `b = x`, and `cond` indicating whether `x` is negative,+ * implements signed->unsigned conversion of modular representatives.+ * Questions of representation are not considered in the specification+ * @[FIPS203, Section 2.4.1, "The pseudocode is agnostic regarding how an+ * integer modulo 𝑚 is represented in actual implementations"].}+ *+ * @reference{Embedded in the polynomial compression function in the+ * reference implementation @[REF]. Used as part of signed->unsigned+ * conversion for modular representatives. This happens in `mlk_poly_reduce()`+ * here, and as part of polynomial compression functions in @[REF]. See+ * `mlk_poly_reduce()`. Barrier to reduce the risk of compiler-introduced+ * branches. For `a = MLKEM_Q_HALF` and `b=0`, also embedded in+ * `poly_frommsg()` from the reference implementation, which uses+ * `cmov_int16()` instead.}+ *+ * @param a    First alternative.+ * @param b    Second alternative.+ * @param cond Condition variable.+ *+ * @return @p a if @p cond != 0, else @p b.+ */+static MLK_INLINE int16_t mlk_ct_sel_int16(int16_t a, int16_t b, uint16_t cond)+__contract__(ensures(return_value == (cond ? a : b)))+{+  uint16_t au = mlk_cast_int16_to_uint16(a);+  uint16_t bu = mlk_cast_int16_to_uint16(b);+  uint16_t res = bu ^ (mlk_ct_cmask_nonzero_u16(cond) & (au ^ bu));+  return mlk_cast_uint16_to_int16(res);+}++/**+ * Functionally equivalent to cond ? a : b, but implemented with guards+ * against compiler-introduced branches.+ *+ * @reference{Embedded into `cmov()` in the reference implementation @[REF].+ * Uses a value barrier to get mask from condition value.}+ *+ * @param a    First alternative.+ * @param b    Second alternative.+ * @param cond Condition variable.+ *+ * @return @p a if @p cond != 0, else @p b.+ */+static MLK_INLINE uint8_t mlk_ct_sel_uint8(uint8_t a, uint8_t b, uint8_t cond)+__contract__(ensures(return_value == (cond ? a : b)))+{+  return b ^ (mlk_ct_cmask_nonzero_u8(cond) & (a ^ b));+}++/**+ * Compare two arrays for equality in constant time.+ *+ * @spec{Used to securely compute conditional move in @[FIPS203, Algorithm+ * 18 (ML-KEM.Decaps_Internal, L9-11].}+ *+ * @reference{`cmov()` in the reference implementation @[REF]. We return+ * `uint8_t`, not `int`. We use an additional XOR-accumulator in the+ * comparison loop which prevents early abort if the OR-accumulator is 0xFF.+ * We use a value barrier to convert the OR-accumulator into a mask; the+ * reference implementation uses a shift which the compiler can argue to+ * result in either 0 or 0xFF..FF.}+ *+ * @param[in] a   First byte array.+ * @param[in] b   Second byte array.+ * @param     len Length of the byte arrays, upper-bounded to UINT16_MAX to+ *                control proof complexity only.+ *+ * @retval 0    The byte arrays are equal.+ * @retval 0xFF The byte arrays are not equal.+ */+static MLK_INLINE uint8_t mlk_ct_memcmp(const uint8_t *a, const uint8_t *b,+                                        const size_t len)+__contract__(+  requires(len <= UINT16_MAX)+  requires(memory_no_alias(a, len))+  requires(memory_no_alias(b, len))+  ensures((return_value == 0) || (return_value == 0xFF))+  ensures((return_value == 0) == forall(i, 0, len, (a[i] == b[i]))))+{+  uint8_t r = 0, s = 0;+  unsigned i;++  for (i = 0; i < len; i++)+  __loop__(+    invariant(i <= len)+    invariant((r == 0) == (forall(k, 0, i, (a[k] == b[k]))))+    decreases(len - i))+  {+    r |= a[i] ^ b[i];+    /* s is useless, but prevents the loop from being aborted once r=0xff. */+    s ^= a[i] ^ b[i];+  }++  /*+   * - Convert r into a mask; this may not be necessary, but is an additional+   *   safeguard+   *   towards leaking information about a and b.+   * - XOR twice with s, separated by a value barrier, to prevent the compile+   *   from dropping the s computation in the loop.+   */+  return (mlk_value_barrier_u8(mlk_ct_cmask_nonzero_u8(r) ^ s) ^ s);+}++/**+ * Copy len bytes from x to r if b is zero; don't modify x if b is non-zero.+ * Assumes two's complement representation of negative integers. Runs in+ * constant time.+ *+ * @spec{Used to securely compute conditional move in @[FIPS203, Algorithm+ * 18 (ML-KEM.Decaps_Internal, L9-11].}+ *+ * @reference{`cmov()` in the reference implementation @[REF]. We move if+ * condition value is `0`, not `1`. We use `mlk_ct_sel_uint8` for+ * constant-time selection.}+ *+ * @param[out] r   Output byte array.+ * @param[in]  x   Input byte array.+ * @param      len Number of bytes to be copied.+ * @param      b   Condition value.+ */+static MLK_INLINE void mlk_ct_cmov_zero(uint8_t *r, const uint8_t *x,+                                        size_t len, uint8_t b)+__contract__(+  requires(len <= UINT32_MAX)+  requires(memory_no_alias(r, len))+  requires(memory_no_alias(x, len))+  assigns(memory_slice(r, len))+  ensures(forall(i, 0, len, (r[i] == (b == 0 ? x[i] : old(r)[i])))))+{+  size_t i;+  for (i = 0; i < len; i++)+  __loop__(+    invariant(i <= len)+    invariant(forall(k, 0, i, r[k] == (b == 0 ? x[k] : loop_entry(r)[k])))+    decreases(len - i))+  {+    r[i] = mlk_ct_sel_uint8(r[i], x[i], b);+  }+}++/**+ * Force-zeroize a buffer.+ *+ * @spec{Used to implement @[FIPS203, Section 3.3, Destruction of+ * intermediate values].}+ *+ * @reference{Not present in the reference implementation @[REF].}+ *+ * @param[out] ptr Buffer to be zeroed.+ * @param      len Number of bytes to be zeroed.+ */+#if !defined(MLK_CONFIG_CUSTOM_ZEROIZE)+#if defined(MLK_SYS_WINDOWS)+#include <windows.h>+#elif !defined(MLK_HAVE_INLINE_ASM)+#error No plausibly-secure implementation of mlk_zeroize available. Please provide your own using MLK_CONFIG_CUSTOM_ZEROIZE.+#endif++static MLK_INLINE void mlk_zeroize(void *ptr, size_t len)+__contract__(+  requires(len <= UINT32_MAX)+  requires(memory_no_alias(ptr, len))+  assigns(memory_slice(ptr, len))+  ensures(array_zeroized_u8((uint8_t *)ptr, len)))+{+#if defined(MLK_SYS_WINDOWS)+  SecureZeroMemory(ptr, len);+#else+  mlk_memset(ptr, 0, len);+  /* This follows OpenSSL and seems sufficient to prevent the compiler+   * from optimizing away the memset.+   *+   * If there was a reliable way to detect availability of memset_s(),+   * that would be preferred. */+  __asm__ volatile("" : : "r"(ptr) : "memory");+#endif /* !MLK_SYS_WINDOWS */+}+#endif /* !MLK_CONFIG_CUSTOM_ZEROIZE */++#endif /* !MLK_VERIFY_H */
+ cbits/mlkem/src/zetas.inc view
@@ -0,0 +1,30 @@+/*+ * Copyright (c) The mlkem-native project authors+ * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT+ */++/*+ * WARNING: This file is auto-generated from scripts/autogen+ *          in the mlkem-native repository.+ *          Do not modify it directly.+ */+++/*+ * Table of zeta values used in the reference NTT and inverse NTT.+ * See autogen for details.+ */+static MLK_ALIGN const int16_t mlk_zetas[128] = {+    -1044, -758,  -359,  -1517, 1493,  1422,  287,   202,  -171,  622,   1577,+    182,   962,   -1202, -1474, 1468,  573,   -1325, 264,  383,   -829,  1458,+    -1602, -130,  -681,  1017,  732,   608,   -1542, 411,  -205,  -1571, 1223,+    652,   -552,  1015,  -1293, 1491,  -282,  -1544, 516,  -8,    -320,  -666,+    -1618, -1162, 126,   1469,  -853,  -90,   -271,  830,  107,   -1421, -247,+    -951,  -398,  961,   -1508, -725,  448,   -1065, 677,  -1275, -1103, 430,+    555,   843,   -1251, 871,   1550,  105,   422,   587,  177,   -235,  -291,+    -460,  1574,  1653,  -246,  778,   1159,  -147,  -777, 1483,  -602,  1119,+    -1590, 644,   -872,  349,   418,   329,   -156,  -75,  817,   1097,  603,+    610,   1322,  -1285, -1465, 384,   -1215, -136,  1218, -1335, -874,  220,+    -1187, -1659, -1185, -1530, -1278, 794,   -1510, -854, -870,  478,   -108,+    -308,  996,   991,   958,   -1460, 1522,  1628,+};
+ cbits/p256/gen_base_table.py view
@@ -0,0 +1,126 @@+#!/usr/bin/env python3+"""Build the two tables of multiples of the base point that crypton reads.++    ./gen_base_table.py 5 p256_base_table.c        # for p256_scalarmulbase+    ./gen_base_table.py odd 7 p256_wnaf_table.c    # for p256_verify.c++Its own words for the layout: for each i, j with blocksize*i <= 256 and+1 <= j <= B, where B = 2^(blocksize-1), the multiple 2^(blocksize*i) * j * P+goes at tab + 64*(B*i + (j-1)), as a Montgomery-affine pair, four+little-endian 64-bit words each.++Five is the blocksize crypton ships: on an Apple M4 it is 6.40 microseconds+against 7.15 for four and 6.10 for six, and the table is 52 KiB against 33+and 88.  Nothing outside this file knows the number -- it is written into+the generated file and read from there.++The curve constants come from `openssl ecparam`, not from memory, and the+generated table is checked against crypton's own base-point multiplication+by the test suite, so a mistake here does not pass quietly.+"""+import subprocess, re, sys++def curve():+    out = subprocess.run(["openssl","ecparam","-name","prime256v1",+                          "-param_enc","explicit","-text","-noout"],+                         capture_output=True, text=True).stdout+    f, cur = {}, None+    for line in out.splitlines():+        m = re.match(r'^(Prime|A|B|Generator \(uncompressed\)|Order):?\s*$', line.strip())+        if m:+            cur = m.group(1); f[cur] = ""; continue+        if cur and re.match(r'^\s+[0-9a-f]{2}[:0-9a-f]*:?\s*$', line):+            f[cur] += line.strip()+        elif cur and line and not line.startswith(' '):+            cur = None+    def num(s):+        return int.from_bytes(bytes(int(v,16) for v in s.strip(':').split(':') if v), 'big')+    g = bytes(int(v,16) for v in f['Generator (uncompressed)'].strip(':').split(':') if v)+    assert g[0] == 4 and len(g) == 65+    return (num(f['Prime']), num(f['A']), num(f['B']),+            int.from_bytes(g[1:33],'big'), int.from_bytes(g[33:],'big'), num(f['Order']))++P, A, B, GX, GY, N = curve()+assert (GY*GY - GX**3 - A*GX - B) % P == 0, "the generator is not on the curve"++def add(p, q):+    if p is None: return q+    if q is None: return p+    (x1,y1),(x2,y2) = p,q+    if x1 == x2:+        if (y1 + y2) % P == 0: return None+        l = (3*x1*x1 + A) * pow(2*y1, -1, P) % P+    else:+        l = (y2-y1) * pow(x2-x1, -1, P) % P+    x3 = (l*l - x1 - x2) % P+    return (x3, (l*(x1-x3) - y1) % P)++R = 1 << 256+def mont(v):  return v * R % P+def words(v): return [(v >> (64*k)) & 0xFFFFFFFFFFFFFFFF for k in range(4)]++def table(blocksize):+    Bn = 1 << (blocksize - 1)+    blocks = 256 // blocksize + 1+    out = []+    base = (GX, GY)                      # 2^(blocksize*i) * P+    for i in range(blocks):+        acc = None+        for j in range(1, Bn + 1):+            acc = add(acc, base)         # j * base+            out.append(acc)+        for _ in range(blocksize):+            base = add(base, base)+    return blocks, Bn, out++def odd_table(width):+    """The odd multiples 1P, 3P, ..., (2^(width-1)-1)P, which is what the+    windowed form in cbits/p256/p256_verify.c reads for the base point."""+    n = 1 << (width - 2)+    twice = add((GX, GY), (GX, GY))+    out, acc = [], (GX, GY)+    for _ in range(n):+        out.append(acc)+        acc = add(acc, twice)+    return out++def emit_odd(width, path):+    pts = odd_table(width)+    with open(path, 'w') as fh:+        fh.write("/* Generated by cbits/p256/gen_base_table.py; do not hand-edit.\n"+                 " * The odd multiples of the base point, in Montgomery-affine\n"+                 " * form, which cbits/p256/p256_verify.c reads.  A verification\n"+                 " * is public, so this table is walked in variable time. */\n")+        fh.write("#include <stdint.h>\n\n")+        fh.write(f"const uint64_t crypton_p256_wnaf_width = {width};\n\n")+        fh.write(f"const uint64_t crypton_p256_wnaf_table[{len(pts)*8}] = {{\n")+        for (x, y) in pts:+            ws = words(mont(x)) + words(mont(y))+            fh.write("\t" + ",".join(f"0x{w:016x}ULL" for w in ws) + ",\n")+        fh.write("};\n")+    return len(pts), len(pts) * 64++def emit(blocksize, path):+    blocks, Bn, pts = table(blocksize)+    with open(path, 'w') as fh:+        fh.write("/* Generated by cbits/p256/gen_base_table.py; do not hand-edit.\n"+                 " * The multiples of the base point that s2n-bignum's\n"+                 " * p256_scalarmulbase reads, in Montgomery-affine form. */\n")+        fh.write("#include <stdint.h>\n\n")+        fh.write(f"const uint64_t crypton_p256_s2n_base_blocksize = {blocksize};\n\n")+        fh.write(f"const uint64_t crypton_p256_s2n_base_table[{len(pts)*8}] = {{\n")+        for (x, y) in pts:+            ws = words(mont(x)) + words(mont(y))+            fh.write("\t" + ",".join(f"0x{w:016x}ULL" for w in ws) + ",\n")+        fh.write("};\n")+    return blocks, Bn, len(pts)*64++if __name__ == "__main__":+    if sys.argv[1] == "odd":+        w = int(sys.argv[2]); path = sys.argv[3]+        n, size = emit_odd(w, path)+        print(f"width {w}: {n} odd multiples = {size} bytes")+    else:+        b = int(sys.argv[1]); path = sys.argv[2]+        blocks, Bn, size = emit(b, path)+        print(f"blocksize {b}: {blocks} blocks x {Bn} = {size} bytes")
cbits/p256/p256.c view
@@ -39,6 +39,15 @@  #include "p256/p256.h" +#ifdef CRYPTON_S2N_BIGNUM+extern void bignum_modinv(uint64_t k, uint64_t *z, const uint64_t *a,+                          const uint64_t *b, uint64_t *t);+/* the digits are handed over as they lie */+#if P256_BITSPERDIGIT != 64 || P256_NDIGITS != 4+#error "CRYPTON_S2N_BIGNUM wants the 64-bit crypton_p256_int"+#endif+#endif+ const crypton_p256_int crypton_SECP256r1_n =  // curve order   {{P256_LITERAL(0xfc632551, 0xf3b9cac2), P256_LITERAL(0xa7179e84, 0xbce6faad),     P256_LITERAL(-1, -1), P256_LITERAL(0, -1)}};@@ -214,7 +223,11 @@   n %= P256_BITSPERDIGIT;   for (i = P256_NDIGITS - 1; i > 0; --i) {     crypton_p256_digit accu = (P256_DIGIT(a, i) << n);-    accu |= (P256_DIGIT(a, i - 1) >> (P256_BITSPERDIGIT - n));+    /* n is zero as often as it is anything else, and a digit shifted by its+     * own width is undefined: x86 takes the count modulo the width and hands+     * back the whole digit, ARM hands back nothing.  Two shifts that are each+     * inside the width say the intended nothing on both. */+    accu |= (P256_DIGIT(a, i - 1) >> (P256_BITSPERDIGIT - 1 - n) >> 1);     P256_DIGIT(b, i) = accu;   }   P256_DIGIT(b, i) = (P256_DIGIT(a, i) << n);@@ -230,7 +243,8 @@   n %= P256_BITSPERDIGIT;   for (i = 0; i < P256_NDIGITS - 1; ++i) {     crypton_p256_digit accu = (P256_DIGIT(a, i) >> n);-    accu |= (P256_DIGIT(a, i + 1) << (P256_BITSPERDIGIT - n));+    /* the same full-width shift as in crypton_p256_shl above */+    accu |= (P256_DIGIT(a, i + 1) << (P256_BITSPERDIGIT - 1 - n) << 1);     P256_DIGIT(b, i) = accu;   }   P256_DIGIT(b, i) = (P256_DIGIT(a, i) >> n);@@ -244,8 +258,11 @@     accu |= (P256_DIGIT(a, i + 1) << (P256_BITSPERDIGIT - 1));     P256_DIGIT(b, i) = accu;   }+  /* The shift is unsigned: highbit is a carry, zero or one, and one shifted+   * into the sign bit of the signed digit is an overflow the standard leaves+   * undefined.  The value put into b is the same either way. */   P256_DIGIT(b, i) = (P256_DIGIT(a, i) >> 1) |-      (((crypton_p256_sdigit) highbit) << (P256_BITSPERDIGIT - 1));+      (((crypton_p256_digit) highbit) << (P256_BITSPERDIGIT - 1)); }  // Return -1, 0, 1 for a < b, a == b or a > b respectively.@@ -312,6 +329,22 @@   crypton_p256_int U = *MOD;   crypton_p256_int V = *a; +  /* Zero has no inverse, and the loop below never finds that out: V stays+     even forever, so it is halved forever, and the only break is in the+     branch both U and V have to be odd to reach.  The other input without an+     inverse is MOD itself -- 2*MOD does not fit in 256 bits, so there is no+     third -- and that one already leaves here as zero, which is also what+     Crypto.PubKey.ECC.P256's scalarInvSafe answers for both.  Answer the+     same for zero rather than not answering.++     Reachable: Crypto.PubKey.ECC.P256 exports scalarInv, and scalarFromBinary+     accepts any 256 bits.  A hang inside a foreign call cannot be interrupted+     by System.Timeout either. */+  if (crypton_p256_is_zero(a)) {+    crypton_p256_clear(b);+    return;+  }+   for (;;) {     if (crypton_p256_is_even(&U)) {       crypton_p256_shr1(&U, 0, &U);@@ -480,6 +513,16 @@  // b = 1/a mod n, using Fermat's little theorem. void crypton_p256e_scalar_invert(const crypton_p256_int* a, crypton_p256_int* b) {+#ifdef CRYPTON_S2N_BIGNUM+  /* The assembly, which takes a fixed number of division steps instead: 0.80+   * microseconds against 6.02 on an Apple M4.  It answers zero where a has no+   * inverse, which is what the chain below does as well, and the caller reads+   * a zero as "no inverse". */+  uint64_t t[12];++  bignum_modinv(4, P256_DIGITS(b), P256_DIGITS(a),+                P256_DIGITS(&crypton_SECP256r1_n), t);+#else   crypton_p256_int _1, _10, _11, _101, _111, _1010, _1111;   crypton_p256_int _10101, _101010, _101111, x6, x8, x16, x32;   int i;@@ -534,4 +577,5 @@    // Demontgomerize   crypton_p256e_montmul(b, &crypton_SECP256r1_one, b);+#endif }
+ cbits/p256/p256_base_table.c view
@@ -0,0 +1,841 @@+/* Generated by cbits/p256/gen_base_table.py; do not hand-edit.+ * The multiples of the base point that s2n-bignum's+ * p256_scalarmulbase reads, in Montgomery-affine form. */+#include <stdint.h>++const uint64_t crypton_p256_s2n_base_blocksize = 5;++const uint64_t crypton_p256_s2n_base_table[6656] = {+	0x79e730d418a9143cULL,0x75ba95fc5fedb601ULL,0x79fb732b77622510ULL,0x18905f76a53755c6ULL,0xddf25357ce95560aULL,0x8b4ab8e4ba19e45cULL,0xd2e88688dd21f325ULL,0x8571ff1825885d85ULL,+	0x850046d410ddd64dULL,0xaa6ae3c1a433827dULL,0x732205038d1490d9ULL,0xf6bb32e43dcf3a3bULL,0x2f3648d361bee1a5ULL,0x152cd7cbeb236ff8ULL,0x19a8fb0e92042dbeULL,0x78c577510a5b8a3bULL,+	0xffac3f904eebc127ULL,0xb027f84a087d81fbULL,0x66ad77dd87cbbc98ULL,0x26936a3fb6ff747eULL,0xb04c5c1fc983a7ebULL,0x583e47ad0861fe1aULL,0x788208311a2ee98eULL,0xd5f06a29e587cc07ULL,+	0x74b0b50d46918dccULL,0x4650a6edc623c173ULL,0x0cdaacace8100af2ULL,0x577362f541b0176bULL,0x2d96f24ce4cbaba6ULL,0x17628471fad6f447ULL,0x6b6c36dee5ddd22eULL,0x84b14c394c5ab863ULL,+	0xbe1b8aaec45c61f5ULL,0x90ec649a94b9537dULL,0x941cb5aad076c20cULL,0xc9079605890523c8ULL,0xeb309b4ae7ba4f10ULL,0x73c568efe5eb882bULL,0x3540a9877e7a1f68ULL,0x73a076bb2dd1e916ULL,+	0x403947373e77664aULL,0x55ae744f346cee3eULL,0xd50a961a5b17a3adULL,0x13074b5954213673ULL,0x93d36220d377e44bULL,0x299c2b53adff14b5ULL,0xf424d44cef639f11ULL,0xa4c9916d4a07f75fULL,+	0x0746354ea0173b4fULL,0x2bd20213d23c00f7ULL,0xf43eaab50c23bb08ULL,0x13ba5119c3123e03ULL,0x2847d0303f5b9d4dULL,0x6742f2f25da67bddULL,0xef933bdc77c94195ULL,0xeaedd9156e240867ULL,+	0x27f14cd19499a78fULL,0x462ab5c56f9b3455ULL,0x8f90f02af02cfc6bULL,0xb763891eb265230dULL,0xf59da3a9532d4977ULL,0x21e3327dcf9eba15ULL,0x123c7b84be60bbf0ULL,0x56ec12f27706df76ULL,+	0x75c96e8f264e20e8ULL,0xabe6bfed59a7a841ULL,0x2cc09c0444c8eb00ULL,0xe05b3080f0c4e16bULL,0x1eb7777aa45f3314ULL,0x56af7bedce5d45e3ULL,0x2b6e019a88b12f1aULL,0x086659cdfd835f9bULL,+	0x2c18dbd19dc21ec8ULL,0x98f9868a0fcf8139ULL,0x737d2cd648250b49ULL,0xcc61c94724b3428fULL,0x0c2b407880dd9e76ULL,0xc43a8991383fbe08ULL,0x5f7d2d65779be5d2ULL,0x78719a54eb3b4ab5ULL,+	0xea7d260a6245e404ULL,0x9de407956e7fdfe0ULL,0x1ff3a4158dac1ab5ULL,0x3e7090f1649c9073ULL,0x1a7685612b944e88ULL,0x250f939ee57f61c8ULL,0x0c0daa891ead643dULL,0x68930023e125b88eULL,+	0x04b71aa7d2697768ULL,0xabdedef5ca345a33ULL,0x2409d29dee37385eULL,0x4ee1df77cb83e156ULL,0x0cac12d91cbb5b43ULL,0x170ed2f6ca895637ULL,0x28228cfa8ade6d66ULL,0x7ff57c9553238acaULL,+	0xccc425634b2ed709ULL,0x0e356769856fd30dULL,0xbcbcd43f559e9811ULL,0x738477ac5395b759ULL,0x35752b90c00ee17fULL,0x68748390742ed2e3ULL,0x7cd06422bd1f5bc1ULL,0xfbc08769c9e7b797ULL,+	0xa242a35bb0cf664aULL,0x126e48f77f9707e3ULL,0x1717bf54c6832660ULL,0xfaae7332fd12c72eULL,0x27b52db7995d586bULL,0xbe29569e832237c2ULL,0xe8e4193e2a65e7dbULL,0x152706dc2eaa1bbbULL,+	0x72bcd8b7bc60055bULL,0x03cc23ee56e27e4bULL,0xee337424e4819370ULL,0xe2aa0e430ad3da09ULL,0x40b8524f6383c45dULL,0xd766355442a41b25ULL,0x64efa6de778a4797ULL,0x2042170a7079adf4ULL,+	0x808b0b650bc6fb80ULL,0x5882e0753ffe2e6bULL,0xd5ef2f7c2c83f549ULL,0x54d63c809103b723ULL,0xf2f11bd652a23f9bULL,0x3670c3194b0b6587ULL,0x55c4623bb1580e9eULL,0x64edf7b201efe220ULL,+	0xd8c5fccfc5e3a3d8ULL,0xbefd904c4079dfbfULL,0xbc6d6a58fead0197ULL,0x39227077695532a4ULL,0x09e23e6ddbef42f5ULL,0x7e449b64480a9908ULL,0x7b969c1aad9a2e40ULL,0x6231d7929591c2a4ULL,+	0xdb6d96f305968b80ULL,0x380a0913089f73b9ULL,0x7da70b83c2c61e01ULL,0x95fb8394569b38c7ULL,0x9a3c651280edfe2fULL,0x8f726bb98faeaf82ULL,0x8010a4a078424bf8ULL,0x296720440e844970ULL,+	0x802b8d2333e12b70ULL,0x6d490a4b19dd329bULL,0x14f356cc6abc354dULL,0x11eddf7fd0a0da0dULL,0x1e208328d87fd1d8ULL,0xfd2f4f8cfd025813ULL,0x03b48cc47c29bca2ULL,0x3f2a78b3241a2b71ULL,+	0x63c5cb817a2ad62aULL,0x7ef2b6b9ac62ff54ULL,0x3749bba4b3ad9db5ULL,0xad311f2c46d5a617ULL,0xb77a8087c2ff3b6dULL,0xb46feaf3367834ffULL,0xf8aa266d75d6b138ULL,0xfa38d320ec008188ULL,+	0xc04afa1ae3451a09ULL,0x7cc69103bc117423ULL,0x876be3aa51cf56eeULL,0xe7577d57ad844a25ULL,0x266fed8cdb77f341ULL,0xcfa258dc23ae4a2aULL,0x53a7a98cda782760ULL,0x04b48868ceaf7d4aULL,+	0xc0f2affc4e6916c6ULL,0x6fb94957811842daULL,0x6034bcb624b4d157ULL,0xde2efdc7992efb90ULL,0xd66f7eceac793c87ULL,0x02f026267dc6fdcdULL,0x90d3235c9aa1c501ULL,0xf6e494962b4666f0ULL,+	0x082736d19c0859c3ULL,0x89ea5516b269386aULL,0xf25071871aa87b33ULL,0xe9d82f5f704e8236ULL,0x7834612442e855f5ULL,0x209f50fe395e00d8ULL,0xcd9e03aae6e7e62bULL,0xb4b4959e5e5be37bULL,+	0x486d8ffa696946fcULL,0x50fbc6d8b9cba56dULL,0x7e3d423e90f35a15ULL,0x7c3da195c0dd962cULL,0xe673fdb03cfd5d8bULL,0x0704b7c2889dfca5ULL,0xf6ce581ff52305aaULL,0x399d49eb914d5e53ULL,+	0x7966afbb10e6d950ULL,0x37e4a4c4e2bf970aULL,0x23d0c8559d54ca2aULL,0x13d62865fee39a10ULL,0x15f53c38d3bd15e9ULL,0x014b8bed84a80bccULL,0x10674c77bfd8f608ULL,0x4dfab986c93fbfefULL,+	0x24c97c367b92d453ULL,0xd2c271a2249a26c0ULL,0x60eb4b2b89d14a39ULL,0x1198de20432e8005ULL,0x9eabea75799b80d9ULL,0xab6e0c2b8f826ae5ULL,0xca004eedd10061ecULL,0x31a9f439e99c4fd8ULL,+	0x734c8b75b5498a7cULL,0xaeccad8a29f64c2dULL,0x95dd54fa295b1677ULL,0x383902a0b4c54968ULL,0x78cb276feb9d33a9ULL,0x00aedca1af552869ULL,0xa01d14594c5c1630ULL,0xfeba17067cf7d50dULL,+	0x380a496d6ec293cdULL,0x733dbda78e7051f5ULL,0x037e388db849140aULL,0xee4b32b05946dbf6ULL,0xb1c4fda9cae368d1ULL,0x5001a7b0fdb0b2f3ULL,0x6df593742e3ac46eULL,0x4af675f239b3e656ULL,+	0x1a1fffdcc01b0a46ULL,0x07ad675f83f843c2ULL,0xbcec2d076738e81aULL,0x910aec75aa8b7da8ULL,0x13b4d740a4509ba7ULL,0x057010734c7b8216ULL,0x3d75c8f71591f1e5ULL,0x134c1b6f0dfe1d90ULL,+	0xb20e7c44d67826ebULL,0x1212d3cfac379637ULL,0x614f67877de03a5dULL,0x7538a2fc802baa26ULL,0x133c37a19d252415ULL,0x7db390506eb4b587ULL,0x5d40d7574c49d1d9ULL,0xf1126b99a801c4baULL,+	0x39a7aeb9c56729fcULL,0x21a59448e7a8bd85ULL,0xd7c6da5d049d10c6ULL,0x93a4c4a8c5197afbULL,0xd185539c25933861ULL,0x4994bcba27494d9dULL,0xef033de14ce7bfa9ULL,0x22e9b000321f9236ULL,+	0x44e3811039949296ULL,0x5b63827b361db1b5ULL,0x3e5323ed206eaff5ULL,0x942370d2c21f4290ULL,0xf2caaf2ee0d985a1ULL,0x192cc64b7239846dULL,0x7c0b8f47ae6312f8ULL,0x7dc61f9196620108ULL,+	0x4f7081e144cc3addULL,0xd5ffa1d687be82cfULL,0x89890b6c0edd6472ULL,0xada26e1a3ed17863ULL,0x276f271563483caaULL,0xe6924cd92f6077fdULL,0x05a7fe980a466e3cULL,0xf1c794b0b1902d1fULL,+	0x3d2b24b9eb7926b8ULL,0xbff88cb3cdbe5509ULL,0xd0f399afe4dd640bULL,0x3c5fe1302f76ed45ULL,0x6f3562f43764fb3dULL,0x7b5af3183151b62dULL,0xd5bd0bc7d79ce5f3ULL,0xfdaf6b20ec66890fULL,+	0x32027fe891e5d7d3ULL,0xf14b7d1773a07678ULL,0xf88497b3c0dfdd61ULL,0xf7c2eec02a8c4f48ULL,0xaa5573f43756e621ULL,0xc013a2401825b948ULL,0x1c03b34563878572ULL,0xa0472bea653a4184ULL,+	0x6772b0e5ab4b35a2ULL,0x1d8b6001f5eeaacfULL,0x728f7ce4795b9580ULL,0x4a20ed2a41fb81daULL,0x9f685cd44fec01e6ULL,0x3ed7ddcca7ff50adULL,0x460fd2640c2d97fdULL,0x3a241426eb82f4f9ULL,+	0x29ae2cf983dfedc9ULL,0xf84371348d87631aULL,0xaf5717117429c8d2ULL,0x18d15867146d9272ULL,0x83053ecf69769bb7ULL,0xc55eb856c479ab82ULL,0x5ef7791c21b0f4b2ULL,0xaa5956ba3d491525ULL,+	0x84cfbfa1c5c5ea50ULL,0xd3baf14c67960681ULL,0x263984030dd50942ULL,0xe4b7839c4716a663ULL,0xd5f1f794e7de6dc0ULL,0x5cd0f4d4622aa7ceULL,0x5295f3f159acfeecULL,0x8d933552953e0607ULL,+	0xe652533b3cef0d7dULL,0xd94f7b182bbb4381ULL,0x838752be0e80f500ULL,0x8e6e24889e9c9bfbULL,0xc975169716caca6aULL,0x866c49d838531ad9ULL,0xc917e2397151ade1ULL,0x2d016ec16037c407ULL,+	0x80009862d5d721d5ULL,0x0c3357a35bd3a182ULL,0x27f3a83b7aa2cda4ULL,0xb58ae74ef6f83085ULL,0x2a911a812e6dad6bULL,0xde286051f43d6c5bULL,0x4bdccc41f996c4d8ULL,0xe7312ec00ae1e24eULL,+	0x903f6e3960e913afULL,0xb2b58bee98bf140dULL,0x9deff025354890b8ULL,0x155810068d2e924eULL,0xb5755db493c95e5bULL,0x3fac42f0dae20eb8ULL,0x9377c8c109b6d8e0ULL,0xa43e2b46ab47ceffULL,+	0x5f57b2fbfacfa459ULL,0x874b1498c1b5aa6bULL,0xb9e89acac4db2092ULL,0x1362bf8ddf4381daULL,0x25d76830b76328a0ULL,0x38188b7098572ae4ULL,0xb43e941429132f7dULL,0x7895a29f22dd42c9ULL,+	0xcbde78dd5e22cbb2ULL,0xf449c85b76bb4391ULL,0x4289f357b6a4273bULL,0x9fce23fd48e84a19ULL,0xcfc32730939eb3b4ULL,0x8b3d982c16c32280ULL,0x5ac234bad5f1346cULL,0x781954b470769fc9ULL,+	0x6faf68feaae6ee70ULL,0x78f4cc155602b0c9ULL,0x7e3321a86e94052aULL,0x2fb3a0d6734d5d80ULL,0xf3b98f3bb25a43baULL,0x30bf803119ee2951ULL,0x7ffee43321b0612aULL,0x12f775e42eb821d0ULL,+	0x4fdff805f57209b5ULL,0x9bd65ac3f952ac8dULL,0x02a3abd3c7969a6fULL,0x1359927ef523775fULL,0xe09b463f88d2e861ULL,0x661d2199623287c3ULL,0x821e64495a70eb7aULL,0x0afbbb1dd67dc684ULL,+	0x7418e3d3acff89f9ULL,0x227f16aed852251fULL,0xdd5bc6e4eb84658bULL,0xf066b9c8f90a9f7eULL,0xc2369071800a7f87ULL,0x383ddc0d5a72862aULL,0x5b48465d8a776da5ULL,0x3d82f64f5e2d8318ULL,+	0x5852104b87453b28ULL,0x073e8128b387344dULL,0x300e78e4817cfc08ULL,0x3a82ed4799362088ULL,0xe222304c88de46a4ULL,0x666c94fd57fadf4aULL,0x40b2d08ea0c8e108ULL,0x4b2955b909e050faULL,+	0xf8d112e76e6485b3ULL,0x4d3e24db771c52f8ULL,0x48e3ee41684a2f6dULL,0x7161957d21d95551ULL,0x19631283cdb12a6cULL,0xbf3fa8822e50e164ULL,0xf6254b633166cc73ULL,0x3aefa7aeaee8cc38ULL,+	0x79b0fe623b36f9fdULL,0x26543b23fde19fc0ULL,0x136e64a0958482efULL,0x23f637719b095825ULL,0x14cfd596b6a1142eULL,0x5ea6aac6335aac0bULL,0x86a0e8bdf3081dd5ULL,0x5fb89d79003dc12aULL,+	0x0f0165fce3779ee3ULL,0xe00e7f9dbd495d9eULL,0x1fa4efa220284e7aULL,0x4564bade47ac6219ULL,0x90e6312ac4708e8eULL,0x4f5725fba71e9adfULL,0xe95f55ae3d684b9fULL,0x47f7ccb11e94b415ULL,+	0xda3a77e5522e6b69ULL,0x69c908c3bbcd6c18ULL,0x1f1b9e48d924fd56ULL,0x37c64e36aa4bb3f7ULL,0x5a4fdbdfee478d7dULL,0xba75c8bc0193f7a0ULL,0x84bc1e8456cd16dfULL,0x1fb08f0846fad151ULL,+	0x3617890361a341c1ULL,0x3604dc600cfd6142ULL,0x022295eb8533316cULL,0x3dbde4ac44af2922ULL,0x898afc5d1c7eef69ULL,0x58896805d14f4fa1ULL,0x05002160203c21caULL,0x6f0d1f3040ef730bULL,+	0x48201b4b12cfe297ULL,0x3eee129c292f74e5ULL,0xe1fe114ec9e874e8ULL,0x899b055c92c5fc41ULL,0x4e477a643a39c8cfULL,0x82f09efe78963cc9ULL,0x6fd3fd8fd333f863ULL,0x85132b2adc949c63ULL,+	0xbd9b8b1dbe7a2af3ULL,0xec51caa94fb74a72ULL,0xb9937a4b63879697ULL,0x7c9a9d20ec2687d5ULL,0x1773e44f6ef5f014ULL,0x8abcf412e90c6900ULL,0x387bd0228142161eULL,0x50393755fcb6ff2aULL,+	0x766e072232398baaULL,0x205fee425cfca031ULL,0xa49f53417a029cf2ULL,0xa88c68b84023890dULL,0xbc2750417337aaa8ULL,0x9ed364ad0eb384f4ULL,0xe0816f8529aba92fULL,0x2e9e194104e38a88ULL,+	0xfabf770977f7195aULL,0x8ec86167adeb838fULL,0xea1285a8bb4f012dULL,0xd68835039a3eab3fULL,0xee5d24f8309004c2ULL,0xa96e4b7613ffe95eULL,0x0cdffe12bd223ea4ULL,0x8f5c2ee5b6739a53ULL,+	0xecace1dda1887395ULL,0x40960f36932a65deULL,0x9611ff5c3aa95529ULL,0xc58215b07c1e5a36ULL,0xd48c9b58f0e1a524ULL,0xb406856bf590dfb8ULL,0xc7605e049cd95662ULL,0x0dd036eea33ecf82ULL,+	0x3d61333959145a65ULL,0xcd9bc368fa406337ULL,0x82d11be32d8a52a0ULL,0xf6877b2797a1c590ULL,0x837a819bf5cbdb25ULL,0x2a4fd1d8de090249ULL,0x622a7de774990e5fULL,0x840fa5a07945511bULL,+	0xfe2893277946d3f9ULL,0xe132bd2407472273ULL,0xeeeb510c1eb6ae86ULL,0x777708c5f0595067ULL,0x18e2c8cd1297029eULL,0x2c61095cbbf9305eULL,0xe466c2586b85d6d9ULL,0x8ac06c36da1ea530ULL,+	0xe58e90b36b0cf82eULL,0x6438d2462615b5e7ULL,0x07b1f8fc669c145aULL,0xb0d8b2da36f1e1cbULL,0x54d5dadbd9184c4dULL,0x3dbb18d5f93d9976ULL,0x0a3e0f56d1147d47ULL,0x2afa8c8da0a48609ULL,+	0x871239ad653ae326ULL,0x14bcf72aa74cbb43ULL,0x8737650e20d4c083ULL,0x3df86536110ed4afULL,0xd2d86fe7b53ca555ULL,0x688cb00dabd5d538ULL,0xcf81bda31ad38468ULL,0x7ccfe3ccf01167b6ULL,+	0x26e08c07e3533d77ULL,0xd7222e6a2e341c99ULL,0x9d60ec3d8d2dc4edULL,0xbdfe0d8f7c476cf8ULL,0x1fe59ab61d056605ULL,0xa9ea9df686a8551fULL,0x8489941e47fb8d8cULL,0xfeb874eb4a7f1b10ULL,+	0x61fc181060a71676ULL,0xe852d1a8f66a8ad1ULL,0x172bbd656417231eULL,0x0d6de7bd3babb11fULL,0x6fde6f88c8e347f8ULL,0x1c5875479bd99cc3ULL,0x78e54ed034076950ULL,0x97f0f334796e83baULL,+	0xed406aa9bd763802ULL,0xc21486a065303da1ULL,0x61ae291ec7e62ec4ULL,0x622a0492df99333eULL,0x7fd80c9dbb7a8ee0ULL,0xdc2ed3bc6c01aedbULL,0x35c35a1208be74ecULL,0xd540cb1a469f671fULL,+	0xa7a8746a584c5e20ULL,0x267e4ea1b9dc7035ULL,0x593a15cfb9548c9bULL,0x5e6e21354bd012f3ULL,0xdf31cc6a8c8f936eULL,0x8af84d04b5c241dcULL,0x63990a6f345efb86ULL,0x6fef4e61b9b962cbULL,+	0xf6368f0925722608ULL,0x131260db131cf5c6ULL,0x40eb353bfab4f7acULL,0x85c7888037eee829ULL,0x4c1581ffc3bdf24eULL,0x5bff75cbf5c3c5a8ULL,0x35e8c83fa14e6f40ULL,0xb81d1c0f0295e0caULL,+	0xf2efe23d442a8ad1ULL,0xc3816a7d06b9c164ULL,0xa9df2d8bdc0aa5e5ULL,0x191ae46f120a8e65ULL,0x83667f8700611c5bULL,0x83171ed7ff109948ULL,0x33a2ecf8ca695952ULL,0xfa4a73eef48d1a13ULL,+	0xfcde7cc8f43a730fULL,0xe89b6f3c33ab590eULL,0xc823f529ad03240bULL,0x82b79afe98bea5dbULL,0x568f2856962fe5deULL,0x0c590adb60c591f3ULL,0x1fc74a144a28a858ULL,0x3b662498b3203f4cULL,+	0x48fc4ed082dd1b6aULL,0x5783a13867b703afULL,0x2463cb9a005d6aaaULL,0xd31ec55c706ecd43ULL,0x9f8ed33f8e9a7641ULL,0x625453ed098d9e7aULL,0xa3beade4ec887493ULL,0x442b80505a795566ULL,+	0x91e3cf0d6c39765aULL,0xa2db3acdac3cca0bULL,0x288f2f08cb953b50ULL,0x2414582ccf43cf1aULL,0x8dec8bbc60eee9a8ULL,0x54c79f02729aa042ULL,0xd81cd5ec6532f5d5ULL,0xa672303acf82e15fULL,+	0x46df582d3bfab839ULL,0x92474e042f8adadeULL,0x36a7766a147a1bc3ULL,0xb6940f540dc0f979ULL,0x44738ef2f2759f25ULL,0x9dd95789a719f4c6ULL,0x2859b7f40750c345ULL,0x5e788bf2b22180d5ULL,+	0x376aafa8719c0563ULL,0xcd8ad2dcbc5fc79fULL,0x303fdb9fcb750cd3ULL,0x14ff052f4418b08eULL,0xf75084cf3e2d6520ULL,0x7ebdf0f8144ed509ULL,0xf43bf0f2d3f25b98ULL,0x86ad71cfa354d837ULL,+	0xa839c9fdfd67ca25ULL,0x023e626860f2015cULL,0x2414a7930e7b2a65ULL,0x92dbe372b13edcbbULL,0xf64981ee64c2200fULL,0x94fb9cdf8446f2f3ULL,0x01411a6a3f1367bbULL,0x7985c1915a1e8331ULL,+	0xb827fe9226f43572ULL,0xdfd3ab5b5d824758ULL,0x315dd23a539094c1ULL,0x85c0e37a66623d68ULL,0x575c79727be19ae0ULL,0x616a3396df0d36b5ULL,0xa1ebb3c826b1ff7eULL,0x635b9485140ad453ULL,+	0xc8123c6037e2efeaULL,0x8d49b502034a96f6ULL,0x466a346b973e4a95ULL,0xf176b5bab7de00ffULL,0x1c58fa3b82dfa945ULL,0x2eb27a9609e429aeULL,0x57c67a67a12b187cULL,0xb155ba82e2298bbaULL,+	0x92bf3cdada430c0bULL,0x4702850e3a96dac6ULL,0xc91cf0a515ac326aULL,0x95de4f49ab8c25e4ULL,0xb01bad09e265c17cULL,0x24e45464087b3881ULL,0xd43e583ce1fac5caULL,0xe17cb3186ead97a6ULL,+	0xf1a542073d99bcfaULL,0x59db703ce8becf6dULL,0x2e455142d2459569ULL,0xb0ee5143a901b910ULL,0xfc05d451e26d994fULL,0x7a6062b41360caafULL,0xdf1ded5f4fa639b1ULL,0xaf930348d335b8b0ULL,+	0x6cc3924374dcec46ULL,0x33cfc02d54c2b73fULL,0x82917844f26cd99cULL,0x8819dd95d1773f89ULL,0x09572aa60871f427ULL,0x8e0cf365f6f01c34ULL,0x7fa52988bff1f5afULL,0x4eb357eae75e8e50ULL,+	0x3d8f248a21fd0861ULL,0xade3bd649bd5a4b6ULL,0xcb56c953c2e2a6bfULL,0x699cd2b5287d6c5fULL,0xdebce1be47d05e8fULL,0x1a4fbb13a8f53732ULL,0x97163beaa5852b08ULL,0x92c49e6ceec6987aULL,+	0xd9d0c8c4868af75dULL,0xd7325cff45c8c7eaULL,0xab471996cc81ecb0ULL,0xff5d55f3611824edULL,0xbe3145411977a0eeULL,0x5085c4c5722038c6ULL,0x2d5335bff94bb495ULL,0x894ad8a6c8e2a082ULL,+	0x540234b22c11bb37ULL,0x2d0366dded4c74a3ULL,0xf9a968daeec5f25dULL,0x3660106867b63142ULL,0x07cd6d2c68d7b6d4ULL,0xa8f74f090c842942ULL,0xe27514047768b1eeULL,0x4b5f7e89fe62aee4ULL,+	0xf2369f0b879fbbedULL,0x0ff0ae86da9d1869ULL,0x5251d75956766f45ULL,0x4984d8c02be8d0fcULL,0x7ecc95a6d21008f0ULL,0x29bd54a03a1a1c49ULL,0xab9828c5d26c50f3ULL,0x32c0087c51d0d251ULL,+	0x47feeb6662b5f3afULL,0xcefab5610abb3734ULL,0x449de60e19f35cb1ULL,0x39f8db14157f0eb9ULL,0xffaecc5b3c61bfd6ULL,0xa5a4d41d41216703ULL,0x7f8fabed224e1cc2ULL,0x0d5a8186871ad953ULL,+	0x190d8ea601799a52ULL,0xa20cec41b86d2952ULL,0x3062ffb27fff2a7cULL,0x741b32e579f19d37ULL,0xf80d81814eb57d47ULL,0x7a2d0ed416aef06bULL,0x09735fb01cecb588ULL,0x1641caaac6061f5bULL,+	0xae2ad171656e8c3aULL,0xc0e2a4631acd0705ULL,0x006f6a8aa0b6055cULL,0xaf4513d72b65a26eULL,0x3f549e14d616d5bcULL,0x64ee395571253b1fULL,0xe8b10bc1b8ce243aULL,0xbcbeace5913a4e77ULL,+	0x68d32256f779d6a7ULL,0x20423b4d19e7284eULL,0xde19aa1b38f3b153ULL,0x73d0b6c28444f703ULL,0x666161489c64e6a3ULL,0x99587c3737256224ULL,0x6f5277fe61331563ULL,0x7174ad4aa656502eULL,+	0xec1ffc920133918dULL,0x15d9bf5eb2c9ef97ULL,0x7885b542fc6cbb9aULL,0x8abe64535720cf5dULL,0x4e715dced4ec68abULL,0x57a67614279c24a3ULL,0x788ed52a31bb61cfULL,0xaba82444f437a003ULL,+	0x7f99824f20151427ULL,0x206828b692430206ULL,0xaa9097d7e1112357ULL,0xacf9a2f209e414ecULL,0xdbdac9da27915356ULL,0x7e0734b7001efee3ULL,0x54fab5bbd2b288e2ULL,0x4c630fc4f62dd09cULL,+	0x6d883e66bd70a5e1ULL,0x9bd884c1d05de713ULL,0x48d9d445d4d8487dULL,0x8440cd8b0eeae405ULL,0xa3cd0f293d26f83eULL,0x3fd9453022a3c5e4ULL,0x63a078663ffa2ca7ULL,0xbfeadd2900d4a097ULL,+	0x13aea559d68432baULL,0x940043fbd33915dbULL,0x476c7c94e2e8da08ULL,0x6b1630c4443065bbULL,0xd6d8546d19f06c75ULL,0x7e1e98c22eb35abdULL,0x751c9310f0157d8eULL,0xdef84327f6fa6075ULL,+	0x765b993c3a7a4e0eULL,0xfb8f4aef9d79d314ULL,0x5ea2c50e2a19eb24ULL,0x925016e5b891bc8cULL,0x8f70afdc3ad1ece7ULL,0x0abba84cf08fabe2ULL,0x0a9922c815f6132bULL,0x6c0213735d076d77ULL,+	0x00af2615a4453961ULL,0x1705494b993d112cULL,0x0032e12aa1cdd652ULL,0x0ebfa612046d9cb3ULL,0x9f03a9f31b63728cULL,0x8f3618a2de022b05ULL,0xd5b24d903e77c5c9ULL,0x837838a372acb77cULL,+	0xfe65cbd98fceb047ULL,0xde872ef53f8b11a1ULL,0x9aba0d3d8fec802fULL,0x139f1d329a9f381dULL,0x0721aed9c587958dULL,0x066a015cb9f6a7daULL,0x059ec4e3ed5d9d06ULL,0x144285716cbaca1fULL,+	0x66fe0fffe298cdf5ULL,0x3f61bea47b2e51b6ULL,0x7d372117bad3afa4ULL,0x6521a09cef656e2fULL,0xb3b8c966e8a58fe7ULL,0x25203a115a47ebc7ULL,0xfe81588d5c4be573ULL,0x6132e2f31f49a03cULL,+	0xec1ab0130f8c2c99ULL,0x60e8968ff17725c2ULL,0xe1a4a593940a980bULL,0x15ed15b020e9ccb1ULL,0x77d754cc64a00becULL,0x90d09c3341687382ULL,0x294fe02dc31fb651ULL,0x8372cd1a1c94ae53ULL,+	0x8537107a1ac2703bULL,0xb49258d86bc857b5ULL,0x57df14debcdaccd1ULL,0x24ab68d7c4ae8529ULL,0x7ed8b5d4734e59d0ULL,0x5f8740c8c495cc80ULL,0x84aedd5a291db9b3ULL,0x80b360f84fb995beULL,+	0x55d5c68da61a76faULL,0x598b441dca1554dcULL,0xd39923b9773b279cULL,0x33331d3c36bf9efcULL,0x2d4c848e298de399ULL,0xcfdb8e77a1a27f56ULL,0x94c855ea57b8ab70ULL,0xdcdb9dae6f7879baULL,+	0x4c4f07367f636a38ULL,0x9f943fb70e76d5cbULL,0xb03510baa8b68b8bULL,0xc246780a9ed07a1fULL,0x3c0514156d549fc2ULL,0xc2953f31607781caULL,0x955e2c69d8d95413ULL,0xb300fadc7bd282e3ULL,+	0xa14b1163c27901b4ULL,0xfd9236e0899b8bf3ULL,0x42b091eccbc6da0aULL,0xbb1dac6f5ad1d297ULL,0x80e61d53a91cf76eULL,0x4110a412d31f1ee7ULL,0x2d87c3ba13efcf77ULL,0x1f374bb4df450d76ULL,+	0x202886024147519aULL,0xd0981eac26b372f0ULL,0xa9d4a7caa785ebc8ULL,0xd953c50ddbdf58e9ULL,0x9d6361ccfd590f8fULL,0x72e9626b44e6c917ULL,0x7fd9611022eb64cfULL,0x863ebb7e9eb288f3ULL,+	0x15e10a0a097e4403ULL,0xcb3d0a8619854665ULL,0x88d8e211d67d4826ULL,0xb39af66e0b9d2839ULL,0xa5f94588bd475ca8ULL,0xe06b7966c077b80bULL,0xfedb1485da27c26cULL,0xd290d33afe0fd5e0ULL,+	0x686202f31306583dULL,0x05b10da0437c622eULL,0xbf9aaa0f076a7bc8ULL,0x25e94efb8f8f4e43ULL,0x8a35c9b7fa3dc26dULL,0xe0e5fb9396ff03c5ULL,0xa77e3843ebc394ceULL,0xcede65958361de60ULL,+	0xbf9075090c22b540ULL,0x2cde42aab7c267d4ULL,0xba18f9ed5ab0d693ULL,0x3ba62aa66e4660d9ULL,0xb24bf97bab9ea96aULL,0x5d039642e3b60e32ULL,0x4e6a45067c4d9bd5ULL,0x666c5b9e7ed4a6a4ULL,+	0x877b7cf5678a31b0ULL,0xd50301ae3998b620ULL,0x734257c5c00fb396ULL,0xf9fb18a004e672a6ULL,0xff8bd8ebe8758851ULL,0x1e64e4c65d99ba44ULL,0x4b8eaedf7dfd93b7ULL,0xba2f2a9804e76b8cULL,+	0xae02a2f09b56ddbdULL,0x1339b5ac8a2f1cf3ULL,0xf2b569c7839dff0dULL,0xb0b9e864fee9a43dULL,0x4ff8ca4177bb064eULL,0x145a2812fd249f63ULL,0x3ab7beacf86f689aULL,0x9bafec2701d35f5eULL,+	0xbba23fddae57c7b7ULL,0x345342f21b932522ULL,0xfd9c80fe556d4aa3ULL,0xa03907ba6525bb61ULL,0x38b010e1ff218933ULL,0xc066b654aa52117bULL,0x8e14192094f2e6eaULL,0x66a27dca0d32f2b2ULL,+	0xb2e6b121fbabbe92ULL,0x281850fbe1330076ULL,0x093581ec97890015ULL,0x69b1dded75ff77f5ULL,0x7cf0b18fab105105ULL,0x953ced31a89ccfefULL,0x3151f85feb914009ULL,0x3c9f1b8788ed48adULL,+	0xa18f07e0e90fb21eULL,0x00fd2b80bba7fca1ULL,0x20387f2795cd67b5ULL,0x5b89a4e7d39707f7ULL,0x8f83ad3f894407ceULL,0xa0025b946c226132ULL,0xc79563c7f906c13bULL,0x5f548f314e7bb025ULL,+	0xff7589494fde0c1fULL,0xbf8a1abee5b6ec20ULL,0x702278fb87e1db6cULL,0xc447ad7a35ed658fULL,0x48d4aa3803d0ccf2ULL,0x80acb338819a7c03ULL,0x9bc7c89e6e17ceccULL,0x46736b8b03be1d82ULL,+	0x2126f742d43b5eaaULL,0x054a0766dfa59b85ULL,0x9d0d5e36126bfd45ULL,0xa1f8fbd7384f8a8fULL,0x317680f5d563fcccULL,0x48ca5055f280a928ULL,0xe00b81b227b578cfULL,0x10aad9182994a514ULL,+	0xe63f301f358bcdc0ULL,0x07689e990a9d47f8ULL,0x1f689e2f4f43d43aULL,0x4d542a1690920904ULL,0xaea293d59ca0a707ULL,0xd061fe458ac68065ULL,0x1033bf1b0090008cULL,0x29749558c08a6db6ULL,+	0x0ee6d3a7c35d8794ULL,0x042e65580356bae5ULL,0x9f59698d643322fdULL,0x9379ae1550a61967ULL,0x64b9ae62fcc9981eULL,0xaed3d6316d2934c6ULL,0x2454b3025e4e65ebULL,0xab09f647f9950428ULL,+	0xb2083a1222248accULL,0x1f6ec0ef3264e366ULL,0x5659b7045afdee28ULL,0x7a823a40e6430bb5ULL,0x24592a04e1900a79ULL,0xcde09d4ac9ee6576ULL,0x52b6463f4b5ea54aULL,0x1efe9ed3d3ca65a7ULL,+	0xe27a6dbe305406ddULL,0x8eb7dc7fdd5d1957ULL,0xf54a6876387d4d8fULL,0x9c479409c7762de4ULL,0xbe4d5b5d99b30778ULL,0x25380c566e793682ULL,0x602d37f3dac740e3ULL,0x140deabe1566e4aeULL,+	0x4481d067afd32acfULL,0xd8f0fccae1f71ccfULL,0xd208dd0cb596f2daULL,0xd049d7309aad93f9ULL,0xc79f263d42ab580eULL,0x09411bb123f707b4ULL,0x8cfde1ff835e0edaULL,0x7270749090f03402ULL,+	0xeaee6126c49a861eULL,0x024f3b65e14f0d06ULL,0x51a3f1e8c69bfc17ULL,0xc3c3a8e9a7686381ULL,0x3400752cb103d4c8ULL,0x02bc46139218b36bULL,0xc67f75eb7651504aULL,0xd6848b56d02aebfaULL,+	0xbd9802e6c30fa92bULL,0x5a70d96d9a552784ULL,0x9085c4ea3f83169bULL,0xfa9423bb06908228ULL,0x2ffebe12fe97a5b9ULL,0x85da604971b99118ULL,0x9cbc2f7f63178846ULL,0xfd96bc709153218eULL,+	0x958381db1782269bULL,0xae34bf792597e550ULL,0xbb5c60645f385153ULL,0x6f0e96afe3088048ULL,0xbf6a021577884456ULL,0xb3b5688c69310ea7ULL,0x17c9429504fad2deULL,0xe020f0e517896d4dULL,+	0x730ba0ab0976505fULL,0x567f6813095e2ec5ULL,0x470620106331ab71ULL,0x72cfa97741d22b9fULL,0x33e55ead8a2373daULL,0xa8d0d5f47ba45a68ULL,0xba1d8f9c03029d15ULL,0x8f34f1ccfc55b9f3ULL,+	0xcca4428dbbe5a1a9ULL,0x8187fd5f3126bd67ULL,0x0036973a48105826ULL,0xa39b6663b8bd61a0ULL,0x6d42deef2d65a808ULL,0x4969044f94636b19ULL,0xf611ee47dd5d564cULL,0x7b2f3a49d2873077ULL,+	0x94157d45300eb294ULL,0x2b2a656e169c1494ULL,0xc000dd76d3a47aa9ULL,0xa2864e4fa6243ea4ULL,0x82716c47db89842eULL,0x12dfd7d761479fb7ULL,0x3b9a2c56e0b2f6dcULL,0x46be862ad7f85d67ULL,+	0x03b0d8dd0f82b214ULL,0x460c34f9f103cbc6ULL,0xf32e5c0318d79e19ULL,0x8b8888baa84117f8ULL,0x8f3c37dcc0722677ULL,0x10d21be91c1c0f27ULL,0xd47c8468e0f7a0c6ULL,0x9bf02213adecc0e0ULL,+	0x0baa7d1242b48b99ULL,0x1bcb665d48424096ULL,0x8b847cd6ebfb5cfbULL,0x87c2ae569ad4d10dULL,0xf1cbb1220de36726ULL,0xe7043c683fdfbd21ULL,0x4bd0826a4e79d460ULL,0x11f5e5984bd1a2cbULL,+	0x97554160b7fe7b6eULL,0x7d16189a400a3fb2ULL,0xd73e9beae328ca1eULL,0x0dd04b97e793d8ccULL,0xa9c83c9b506db8ccULL,0x5cd47aaecf38814cULL,0x26fc430db64b45e6ULL,0x079b5499d818ea84ULL,+	0xebb01102c1c24a3bULL,0xca24e5681c161c1aULL,0x103eea6936f00a4aULL,0x9ad76ee876176c7bULL,0x97451fc2538e0ff7ULL,0x94f898096604b3b0ULL,0x6311436e3249cfd7ULL,0x27b4a7bd41224f69ULL,+	0x03b5d21ae0ac2941ULL,0x279b0254c2d31937ULL,0x3307c052cac992d0ULL,0x6aa7cb92efa8b1f3ULL,0x5a1825800d37c7a5ULL,0x13380c37342d5422ULL,0x92ac2d66d5d2ef92ULL,0x035a70c9030c63c6ULL,+	0xc16025dd4ce4f152ULL,0x1f419a71f9df7c06ULL,0x6d5b221491e4bb14ULL,0xfc43c6cc839fb4ceULL,0x49f06591925d6b2dULL,0x4b37d9d362186598ULL,0x8c54a971d01b1629ULL,0xe1a9c29f51d50e05ULL,+	0x5109b78571ba1861ULL,0x48b22d5cd0c8f93dULL,0xe8fa84a78633bb93ULL,0x53fba6ba5aebbd08ULL,0x7ff27df3e5eea7d8ULL,0x521c879668ca7158ULL,0xb9d5133bce6f1a05ULL,0x2d50cd53fd0ebee4ULL,+	0x889f6d65533ef217ULL,0x7158c7e4c3ca2e87ULL,0xfb670dfbdc2b4167ULL,0x75910a01844c257fULL,0xf336bf07cf88577dULL,0x22245250e45e2aceULL,0x2ed92e8d7ca23d85ULL,0x29f8be4c2b812f58ULL,+	0xfbb9b2452133ffd9ULL,0x39a8b2f1830f1a20ULL,0x484bc97dd5a1f52aULL,0xd6aebf56a40eddf8ULL,0x32257acb76ccdac6ULL,0xaf4d36ec1586ff27ULL,0x8eaa8863f8de7dd1ULL,0x0045d5cf88647c16ULL,+	0xc51e414351facc61ULL,0xbaf2647de68a25bcULL,0x8f5271a00ff872edULL,0x8f32ef993d2d9659ULL,0xca12488c7593cbd4ULL,0xed266c5d02b82fabULL,0x0a2f78ad14eb3f16ULL,0xc34049484d47afe3ULL,+	0xa6f3d574c005979dULL,0xc2072b426a40e350ULL,0xfca5c1568de2ecf9ULL,0xa8c8bf5ba515344eULL,0x97aee555114df14aULL,0xd4374a4dfdc5ec6bULL,0x754cc28f2ca85418ULL,0x71cb9e27d3c41f78ULL,+	0x09c1670209470496ULL,0xa489a5edebd23815ULL,0xc4dde4648edd4398ULL,0x3ca7b94a80111696ULL,0x3c385d682ad636a4ULL,0x6702702508dc5f1eULL,0x0c1965deafa21943ULL,0x18666e16610be69eULL,+	0x6792fd350369c8e1ULL,0x9271aa62b9dc843bULL,0x8711a4b14d02e2abULL,0x02b2a3e27ee1a383ULL,0xb226e35f0e2b379bULL,0x3d3de39cd652ab25ULL,0xaca6d4c93b560106ULL,0xeced0cf4c95bd877ULL,+	0x45beb4ca2a604b3bULL,0x56f651843a616762ULL,0xf52f5a70978b806eULL,0x7aa3978711dc4480ULL,0xe13fac2a0e01fabcULL,0x7c6ee8a5237d99f9ULL,0x251384ee05211ffeULL,0x4ff6976d1bc9d3ebULL,+	0x8910507903605c39ULL,0xf0843d9ea142c96cULL,0xf374493416923684ULL,0x732caa2ffa0a2893ULL,0xb2e8c27061160170ULL,0xc32788cc437fbaa3ULL,0x39cd818ea6eda3acULL,0xe2e942399e2b2e07ULL,+	0xdde0492316e043a2ULL,0x98a452611dd3d209ULL,0xeaf9f61bd431ebe8ULL,0x00919f4dbaf56abdULL,0xe42417db6d8774b1ULL,0x5fc5279c58e0e309ULL,0x64aa40613adf81eaULL,0xef419edabc627c7fULL,+	0x3919759239ef620fULL,0x9d47284074fa29c4ULL,0x4e428fa39d416d83ULL,0xd1a7c25129f30269ULL,0x46076e1cd746218fULL,0xf3ad6ee8110d967eULL,0xfbb5f434a00ae61fULL,0x3cd2c01980d4c929ULL,+	0xfa24d0537a4af00fULL,0x3f938926ca294614ULL,0x0d700c183982182eULL,0x801334434cc59947ULL,0xf0397106ec87c925ULL,0x62bd59fc0ed6665cULL,0xe8414348c7cca8b5ULL,0x574c76209f9f0a30ULL,+	0x6967d39b0260e52aULL,0xd42585cc90653325ULL,0x0d9bd60521ca7954ULL,0x4fa2087781ed57b3ULL,0x60c1eff8e34a0bbeULL,0x56b0040c84f6ef64ULL,0x28be2b24b1af8483ULL,0xb2278163f5531614ULL,+	0x95be42e2bb8b6a07ULL,0x64be74eeca23f86aULL,0xa73d74fd154ce470ULL,0x1c2d2857d8dc076aULL,0xb1fa1c575a887868ULL,0x38df8e0b3de64818ULL,0xd88e52f9c34e8967ULL,0x274b4f018b4cc76cULL,+	0x59c81ec2c11c208bULL,0x240207da4e3a7234ULL,0x4957eb80f4a089d8ULL,0xc86960bc2137a072ULL,0xde1154fd7f1a932cULL,0xb517fe74d3f2fa17ULL,0x2d5d94557111c0ebULL,0x2aa3378980929d36ULL,+	0x3f5c05b4f8b7559dULL,0x0be4c7acfae29200ULL,0xdd6d3ef756532accULL,0xf6c3ed87eea7a285ULL,0xe463b0a8f46ec59bULL,0x531d9b14ecea6c83ULL,0x3d6bdbafc2dc836bULL,0x3ee501e92ab27f0bULL,+	0x8df275455922ac1cULL,0xa7b3ef5ca52b3f63ULL,0x8e77b21471de57c4ULL,0x31682c10834c008bULL,0xc76824f04bd55d31ULL,0xb6d1c08617b61c71ULL,0x31db0903c2a5089dULL,0x9c092172184e5d3fULL,+	0x5ace5035ea6c3997ULL,0x54259aaac2610befULL,0xef18bb3f3c80dd39ULL,0x6910b95b5fc3fa39ULL,0xfce2f51043e09aeeULL,0xced56c9fa7675665ULL,0x10e265acd872db61ULL,0x6982812eae9fce69ULL,+	0xb8fa3d931341ed7aULL,0x4223272ca7b59d49ULL,0x3dcb194783b8c4a4ULL,0x4e413c01ed1302e4ULL,0x6d999127e17e44ceULL,0xee86bf7533b3adfbULL,0xf6902fe625aa96caULL,0xb73540e4e5aae47dULL,+	0xcc50ef6c872b4a60ULL,0xab2a34a44613521bULL,0x39c5c190983e15d1ULL,0x61dde5df59905512ULL,0xe417f6219f2275f3ULL,0x0750c8b6451d894bULL,0x75b04ab978b0bdaaULL,0x3bfd9fd4458589bdULL,+	0xaafcbfabaf95894cULL,0x7b9bdc07276b2241ULL,0xeaf983625bdda48bULL,0x5977faf2a3fcb4dfULL,0xbed042ef052c4b5bULL,0x9fe87f71067591f0ULL,0xc89c73ca22f24ec7ULL,0x7d37fa9ee64a9f1bULL,+	0xbd78148045ee2f40ULL,0x75e354af416b60cfULL,0xde0b58a18d49a8c4ULL,0xe40e94e2fa359536ULL,0xbd4fa59f62accd76ULL,0x05cf466a8c762837ULL,0xb5abda99448c277bULL,0x5a9e01bf48b13740ULL,+	0xb51e55e6f2606a82ULL,0xe25f706190f2fb57ULL,0xacef6c2ab1a4e37cULL,0x864e359d5dcf2706ULL,0x479e6b187ce57316ULL,0x2cab25003a96b23dULL,0xed4898628ef16df7ULL,0x2056538cef3758b5ULL,+	0xef69a0c3d41d3bd3ULL,0xb533b8c907a26bdeULL,0xe2801d97db2edf9fULL,0xdc4a8269e1877af0ULL,0x6c1c58513d590dbeULL,0x84632f6bee4e9357ULL,0xd36d36b779b33374ULL,0xb46833e39bbca2e6ULL,+	0xcc7a64880a750c0fULL,0x39bacfe34e548e83ULL,0x3d418c760c110f05ULL,0x3e4daa4cb1f11588ULL,0x2733e7b55ffc69ffULL,0x46f147bc92053127ULL,0x885b2434d722df94ULL,0x6a444f65e6fc6b7cULL,+	0x008be4bb346ac9cbULL,0x2e6cb02429811bb2ULL,0xa6ccc747f41540c1ULL,0xf119334efbf6de47ULL,0x4e6bffe9cc97fe6eULL,0x7f4b578bf3d82883ULL,0x459db722753dd1c7ULL,0x05843cd82066b495ULL,+	0x7cb88cd5dbf2af33ULL,0xded9566eaf6b0eabULL,0xd1caf5488e40d844ULL,0x3ae59bd67ce1f67fULL,0xacf4ad33dd82429eULL,0xcc7ebd2bbc4467c7ULL,0xf4f6e95de783d011ULL,0x5197e39116e92db7ULL,+	0xd4f2031885e7a4afULL,0x291bf9a0f7a6878dULL,0xadd8b6eebd051913ULL,0x174f5124f36be034ULL,0x2ac3364527f60189ULL,0xd8902eed8b37ec73ULL,0x546166a11cfdb42eULL,0x31c4e3b807076421ULL,+	0x7de29dfdbc6ebdfbULL,0xa1ee450589de549bULL,0xfd7181aa3ef26a64ULL,0x003179eedf2f3980ULL,0x1bce4d30fd71bc78ULL,0xeb842b14be86a583ULL,0xe00125dc5eb85711ULL,0x9f586983ff11f405ULL,+	0x5b5a089eb833eccbULL,0xc1b3077a7bc51c79ULL,0xe581157f9fbe0e93ULL,0xab487d695b29b172ULL,0xc72551082222f24bULL,0x338cd22cf6dd35dfULL,0x92010e6f5fc24afeULL,0xf8298f15681d46edULL,+	0x2ce7f723042389edULL,0xdf7de0d5a54b1972ULL,0xb8ea2e142c251d75ULL,0xd22a4d37f4e8e8c8ULL,0xe99958566c29c8bcULL,0x9fefdbfe1d1e201eULL,0xb97d946edac885c9ULL,0xf6ea9767d38eac70ULL,+	0x9aaca8e974e75a2eULL,0xcda6fc1eadac968fULL,0x3f6651bf46306befULL,0xc7445e4ecb2ed7f9ULL,0x584a12d8d1571ac1ULL,0x684846c4daf3a679ULL,0xcfc622a9863fbc43ULL,0x2f9e101dea6814f2ULL,+	0x7a1a465ac3f16ea8ULL,0x115a461db2f1d11cULL,0x4767dd956c68a172ULL,0x3392f2ebd13a4698ULL,0xc7a99ccde526cdc7ULL,0x8e537fdc22292b81ULL,0x76d8cf69a6d39198ULL,0xffc5ff432446852dULL,+	0x97b14f7ea90567e6ULL,0x513257b7b6ae5cb7ULL,0x85454a3c9f10903dULL,0xd8d2c9ad69bc3724ULL,0x38da93246b29cb44ULL,0xb540a21d77c8cbacULL,0x9bbfe43501918e42ULL,0xfffa707a56c3614eULL,+	0x56c2e05b1cb76219ULL,0x0ec0bf9171567e7eULL,0xe7076f8661c4c910ULL,0xd67b085bbabc04d9ULL,0x9fb904595e93a96aULL,0x7526c1eafbdc249aULL,0x0d44d367ecdd0bb7ULL,0x953999179dc0d695ULL,+	0x2ccbc583a4c506ecULL,0x957ed188d1acfe97ULL,0x8baed83312f1aea2ULL,0xef2a6cb48325362dULL,0x130dde428e195c43ULL,0xc842025a0e6050c6ULL,0x2da972a708686a5dULL,0xb52999a1e508b4a8ULL,+	0x83f49167ceca9754ULL,0x426d2cf64b7939a0ULL,0x2555e355723fd0bfULL,0xa96e6d06c4f144e2ULL,0x4768a8dd87880e61ULL,0x15543815e508e4d5ULL,0x09d7e772b1b65e15ULL,0x63439dd6ac302fa0ULL,+	0xb11df8e1698e04ccULL,0x877be203169005c8ULL,0x32749e8c4f3c6179ULL,0x2dbc9d0a7853fc05ULL,0x187d4f939454d937ULL,0xe682ce9db4800e1bULL,0xa9129ad8165e68e8ULL,0x0fe29735be7f785bULL,+	0x31019ccf3a4434b4ULL,0xa34581111a7954dcULL,0xa9dac80de34972a7ULL,0xb043d05474f6b8ddULL,0x021c319e11137b1aULL,0x00a754ceed5cc03fULL,0x0aa2c794cbea5ad4ULL,0x093e67f470c015b6ULL,+	0x20ac0351d598d710ULL,0x272c4166cb3a4da4ULL,0xdb82fe1aca71de1fULL,0x746e79f2d8f54b0fULL,0x6e7fc7364b573e9bULL,0x75d03f46fd4b5040ULL,0x5c1cc36d0b98d87bULL,0x513ba3f11f472da1ULL,+	0x859d3145983c38b5ULL,0xb14f176c637abc8bULL,0x2793fb9dcaff7be6ULL,0xebe5a55f35a66a5aULL,0x7cec1dcd9f87dc59ULL,0x7c595cd3fbdbf560ULL,0x5b543b2226eb3257ULL,0x69080646c4c935fdULL,+	0xdca3b70678a6513bULL,0x92ea4a2a9edb1943ULL,0x02642216db6e2dd8ULL,0x9b45d0b49fd57894ULL,0x114e70dbc69d11aeULL,0x1477dd194c57595fULL,0xbc2208b4ec77c272ULL,0x95c5b4d7db68f59cULL,+	0xfe541fa47ea67c77ULL,0x952bd2afe3ea810cULL,0x791fef568d01d374ULL,0xa3a1c6210f11336eULL,0x5ad0d5a9c7ec6d79ULL,0xff7038af3225c342ULL,0x003c6689bc69601bULL,0x25059bc745e8747dULL,+	0x9a75c80676cb2566ULL,0x8f76acb1b24892d9ULL,0x7ae7b9cc1f08fe45ULL,0x19ef73296a4907d8ULL,0x2db4ab715f228bf0ULL,0xf3cdea39817032d7ULL,0x0b1f482edcabe3c0ULL,0x3baf76b4bb86325cULL,+	0x6aac688eadd70482ULL,0x708de92a7b4a4e8aULL,0x75b6dd73758a6eefULL,0xea4bf352725b3c43ULL,0x10041f2c87912868ULL,0xb1b1be95ef09297aULL,0x19ae23c5a9f3860aULL,0xc4f0f839515dcf4bULL,+	0xc71e27bf8538a5c6ULL,0x195c63dd89abff17ULL,0xfd3152851b71e3daULL,0x9cbdfda7fa680fa0ULL,0x9db876ca849d7eabULL,0xebe2764b3c273271ULL,0x663357e3f208dceaULL,0x8c5bd833565b1b70ULL,+	0x75900d7c2fa4f126ULL,0x08a3b8655c99a232ULL,0x2478b6bfdb25e0c3ULL,0x482cc2c271db2edfULL,0x37df7e645f321bb8ULL,0x8a93821b9a8005b4ULL,0x3fa2f10ccc8c1958ULL,0x0d3322182c269d0aULL,+	0xba5514df3fd165e8ULL,0x499fd6a9061f8811ULL,0x72cd1fe0bfef9f00ULL,0x120a4bb979ad7e8aULL,0xf2ffd0955f4a5ac5ULL,0xcfd174f195a7a2f0ULL,0xd42301ba9d17baf1ULL,0xd2fa487a77f22089ULL,+	0xb93452381d531696ULL,0x57201c0088cdde69ULL,0xdde922519a86afc7ULL,0xe3043895bd35cea8ULL,0x7608c1e18555970dULL,0x8267dfa92535935eULL,0xd4c60a57322ea38bULL,0xe0bf7977804ef8b5ULL,+	0x6233ea68c094dbb5ULL,0xb77d062ed968d410ULL,0x3e719bbc58b3002dULL,0x68e7dd3d3dc49d58ULL,0x8d825740013a5e58ULL,0x213117473c9e3c1bULL,0x0cb0a2a77c99b6abULL,0x5c48a3b3c2f888f2ULL,+	0xc7913e91991724f3ULL,0x5eda799c39cbd686ULL,0xddb595c763d4fc1eULL,0x6b63b80bac4fed54ULL,0x6ea0fc697e5fb516ULL,0x737708bad0f1c964ULL,0x9628745f11a92ca5ULL,0x61f379589a86967aULL,+	0x5320fd610979e6b3ULL,0x1d0bd3e593d40723ULL,0x0ac006fcaa80baccULL,0xb1d9c60ed2002515ULL,0x610e7ed0af780b92ULL,0xf5bf446e80a9ce31ULL,0x441c011d3c20b54bULL,0x77f76da1191596c3ULL,+	0x9af39b2caa665072ULL,0x78322fa4efd324efULL,0x3d153394c327bd31ULL,0x81d5f2713129dab0ULL,0xc72e0c42f48027f5ULL,0xaa40cdbc8536e717ULL,0xf45a657a2d369d0fULL,0xb03bbfc4ea7f74e6ULL,+	0x7a04d5429b6a42eaULL,0xfa597e853daf41b9ULL,0x4c58ec27726b0b89ULL,0xed8eb16a030d43deULL,0x65e1e5e1ec9dcf57ULL,0xb7a770c1697cff81ULL,0x1e6d918f9f6e2b22ULL,0x7c277a9ac64e82b7ULL,+	0x46a8c4180d738dedULL,0x6f1a5bb0e0de5729ULL,0xf10230b98ba81675ULL,0x32c6f30c112b33d4ULL,0x7559129dd8fffb62ULL,0x6a281b47b459bf05ULL,0x77c1bd3afa3b6776ULL,0x0709b3807829973aULL,+	0x0875e0c1da36cb47ULL,0xfdf5b7cb1e0210f0ULL,0x7e0c7e4d3a3787c8ULL,0xf043f5262b1c741fULL,0x76df1b006d74d72dULL,0x0514df7338b45ba9ULL,0xaecf7c3e0a6b797aULL,0x5e30b285ddeaac39ULL,+	0x8c26b232a3326505ULL,0x38d69272ee1d41bfULL,0x0459453effe32afaULL,0xce8143ad7cb3ea87ULL,0x932ec1fa7e6ab666ULL,0x6cd2d23022286264ULL,0x459a46fe6736f8edULL,0x50bf0d009eca85bbULL,+	0x15c78f7d605238a5ULL,0xe9d87842448496abULL,0xcfcf75d0d210acc1ULL,0x2948f2295c8c14e2ULL,0xe81fd76de8daf0cbULL,0xd02d11e4a03be800ULL,0x0c4df3518778d30cULL,0x3482bc96965139cbULL,+	0x0b825852877a21ecULL,0x300414a70f537a94ULL,0x3f1cba4021a9a6a2ULL,0x50824eee76943c00ULL,0xa0dbfcecf83cba5dULL,0xf953814893b4f3c0ULL,0x6174416248f24dd7ULL,0x5322d64de4fb09ddULL,+	0x56b6cf278a448bbcULL,0x0ca898dfc85251daULL,0x9082cad836e79b24ULL,0x2e7b9ed31a8e51a7ULL,0xdc7d318a43e1c802ULL,0x4750e523cbf8689dULL,0x9887a072f0071b1aULL,0x52090f87814bfdc1ULL,+	0x574473843d9325f3ULL,0xa9bef2d0f371cb84ULL,0x77d2188ba61e36c5ULL,0xbbd6a7d7c602df72ULL,0xba3aa9028f61bc0bULL,0xf49085ed6ed0b6a1ULL,0x8bc625d6ae6e8298ULL,0x832b0b1da2e9c01dULL,+	0x5bebf2196196bc6eULL,0x0e66736bef097efdULL,0x1128f3b8ea87293aULL,0x2998e4056addfcdeULL,0x55cc31b85d16c961ULL,0x87a434e12418f056ULL,0x2e94aaccdc9fe819ULL,0x94a486c1a56490c6ULL,+	0xa337c447f1f0ced1ULL,0x800cc7939492dd2bULL,0x4b93151dbea08efaULL,0x820cf3f8de0a741eULL,0xff1982dc1c0f7d13ULL,0xef92196084dde6caULL,0x1ad7d97245f96ee3ULL,0x319c8dbe29dea0c7ULL,+	0x1ff2385bde259ec8ULL,0xf6b0836a30f67b0dULL,0x04cc65b006661cffULL,0x467e6358d4230f5cULL,0xce468c802dbed3d3ULL,0x7b984262f1920da6ULL,0x34d257421537479fULL,0x5c8aa88c87bb8de1ULL,+	0xd3ea38717b82b99bULL,0x75922d4d470eb624ULL,0x8f66ec543b95d466ULL,0x66e673ccbee1e346ULL,0x6afe67c4b5f2b89aULL,0x3de9c1e6290e5cd3ULL,0x8c278bb6310a2adaULL,0x420fa3840bdb323bULL,+	0x646f96796424c49bULL,0xf888dfe867c241c9ULL,0xe12d4b9324f68b49ULL,0x9a6b62d8a571df20ULL,0x81b4b26d179483cbULL,0x666f96329511fae2ULL,0xd281b3e4d53aa51fULL,0x7f96a7657f3dbd16ULL,+	0x8553d37c051af62bULL,0xe9a998eb0bf94496ULL,0xe0844f9fb0d59aa1ULL,0x983fd558e6afb813ULL,0x9670c0ca65d69804ULL,0x732b22de6ea5ff2dULL,0xd7640ba95fd8623bULL,0x9f619163a6351782ULL,+	0xf167b4e0bdefdd4fULL,0x69958465f366e401ULL,0x5aa368aba73bbec0ULL,0x121487097b240c21ULL,0x378c323318969006ULL,0xcb4d73cee1fe53d1ULL,0x5f50a80e130c4361ULL,0xd67f59517ef5212bULL,+	0x332f81088cad38c0ULL,0x471b7e906bd68ae2ULL,0x56ac3fb20d8e27a3ULL,0xb54660db136b4b0dULL,0x123a1e11a6fd8de4ULL,0x44dbffeaa37799efULL,0x4540b977ce6ac17cULL,0x495173a8af60acefULL,+	0xeb4437434573eab0ULL,0x11570dfbd1ac6031ULL,0xf7d9b45b44dd9afdULL,0xb8066add22067231ULL,0x15f92ad8f8a3f0b4ULL,0x9e0e4899e0ace2a2ULL,0xbdcd0aadfab38b80ULL,0x46506ae917020052ULL,+	0x429a69f78fca399dULL,0xfe9e27d20207bb63ULL,0xec655ed68788f582ULL,0xa426d748adb75f6eULL,0x18695c02ca81c66dULL,0x84fb8d27a531d425ULL,0x3a3a8956deff48baULL,0xaf1d0d56766d2247ULL,+	0x5a059565352c4b5cULL,0x49261531590bc3e2ULL,0x809f7521f66f9f5fULL,0x2baef6bfc70a4a9bULL,0xe7e6fa6509ed3561ULL,0x11370233984b230cULL,0x2151659bd04cdc69ULL,0xbdb83c63f007d416ULL,+	0x9ebb284d391c2a82ULL,0xbcdd4863158308e8ULL,0x006f16ec83f1edcaULL,0xa13e2c37695dc6c8ULL,0x2ab756f04a057a87ULL,0xa8765500a6b48f98ULL,0x4252face68651c44ULL,0xa52b540be1765e02ULL,+	0xcb35a1a85ca37ff0ULL,0xe1a04f1ccd2f1c8fULL,0x238816ce15a26112ULL,0xe206a111095b177eULL,0x3c10b6048a424149ULL,0xc6a3f56774752cfbULL,0xbf16a37a47f1dbb8ULL,0x7c372f9ad31a3dfbULL,+	0x122d05b5c20b4d2aULL,0xff659cf50c662a67ULL,0xed57c128e8ffc9e9ULL,0x0fbb15859e987683ULL,0xadd70df247319a2bULL,0x4b98baba374be470ULL,0xf03d747356a7b307ULL,0x342e696e3efebf30ULL,+	0xf84b48f7864ac537ULL,0x04713409a6940d3dULL,0x014db22d6174c7aeULL,0xc73a1c438c213034ULL,0x18ac4ea5ffdd93ecULL,0x724fc7576102783eULL,0x9fe13fcc91c3e83fULL,0x92a8c2c8f08f0bf5ULL,+	0x7f2e22fd8f67f6deULL,0xab018833d8693177ULL,0x266d1db6863eca95ULL,0x6bb7732b31b5ef2bULL,0x4fa927c6915f80ceULL,0x6fa1d6d25f90efd8ULL,0x7bd75de8456b48adULL,0xd2cb507a845b6429ULL,+	0xa72cf82ae255d7ecULL,0x52025c23a460e204ULL,0x10ae542d7d5b0a44ULL,0xa85143109305aedaULL,0x958315f5a14bbfe8ULL,0x3f361826385365feULL,0xc2b3a36b66d95040ULL,0x12c7b3347cf4eda2ULL,+	0xa545b4d1e744119dULL,0x4c93b169829a71e7ULL,0x2dbb908c6117fcbfULL,0x4dc97320b35a3d85ULL,0x94c04f856bf88105ULL,0x452e1bce1b51bc1fULL,0x0c41ff50b3013af3ULL,0xf07af445224d7e24ULL,+	0xbdb9e57ca3d24f6aULL,0x8a8246d7f345a763ULL,0x73bd2a6d98cfbb5fULL,0x1dd8e85e86ed04dbULL,0x76f2da42c01f420bULL,0x7ef0547364407bc7ULL,0x7e98ba7faff548f5ULL,0x6b7afbeefd30b64aULL,+	0x4f922fc516a0d2bbULL,0x0d5cc16c1a623499ULL,0x9241cf3a57c62c8bULL,0x2f5e6961fd1b667fULL,0x5c15c70bf5a01797ULL,0x3d20b44d60956192ULL,0x04911b37071fdb52ULL,0xf648f9168d6f0f7bULL,+	0x027cc8b8fac61d9aULL,0x7d25e062e3c6fe8aULL,0xe08805bfe5bff503ULL,0x13271e6c6ff632f7ULL,0x55dca6c0232f76a5ULL,0x8957c32d701ef426ULL,0xee728bcba10a5178ULL,0x5ea60411b62c5173ULL,+	0x9ad5462bb4d8bc50ULL,0x181c0b16a9195770ULL,0xebd4fe1c78412a68ULL,0xae0341bcc0dff48cULL,0xb6bc45cf7003e866ULL,0xf11a6dea8a24a41bULL,0x5407151ad04c24c2ULL,0x62c9d27dda5b7b68ULL,+	0x32865719a8afd30bULL,0x867983288a826dceULL,0xdf04e891c4a8fbe0ULL,0xbb6b6e1bebf56ad3ULL,0x0a695b11471f1ff0ULL,0xd76c3389be15baf0ULL,0x018edb95be96c43eULL,0xf2beaaf490794158ULL,+	0x0a50b12e523b8bf6ULL,0x8009eb5b8f910c1bULL,0xf535af824a167588ULL,0x0f835f9cfb2a2abdULL,0xf59b29312afceb62ULL,0xc797df2a169d383fULL,0xeb3f5fb066ac02b0ULL,0x029d4c6fdaa2d0caULL,+	0x87a7ebd1e0a1b12aULL,0x1e4ef88d770ba95fULL,0x8c33345cdc2ae9cbULL,0xcecf127601cc8403ULL,0x687c012e1b39b80fULL,0xfd90d0ad35c33ba4ULL,0xa3ef5a675c9661c2ULL,0x368fc88ee017429eULL,+	0xb82226052b7ce542ULL,0xe6d4ce997472bde1ULL,0x53e16ebe09d2f4daULL,0x180ff42e53b92b2eULL,0xc59bcc022c34a1c6ULL,0x3803d6f9422c46c2ULL,0x18aff74f5c14a8a2ULL,0x55aebf8010a08b28ULL,+	0xb956970e2fdd23ccULL,0xb80288bc5682e971ULL,0xe6e6d91e9ae86ebcULL,0x0564c83f8c9f1939ULL,0x551932a239560368ULL,0xe893752b049c28e2ULL,0x0b03cee5a6a158c3ULL,0xe12d656b04964263ULL,+	0x58af2010f5b343bcULL,0x0f2e400af2f142feULL,0x3483bfdea85f4bdfULL,0xf0b1d09303bfeaa9ULL,0x2ea01b95c7081603ULL,0xe943e4c93dba1097ULL,0x47be92adb438f3a6ULL,0x00bb7742e5bf6636ULL,+	0x4ed714576be5f7deULL,0xd93006f8c2263c9eULL,0xe073694ccacacb36ULL,0x2ff7a5b43ae118abULL,0x3cce53f1cd871236ULL,0xf156a39dc2aa6d52ULL,0x9cc5f271b198d76dULL,0xbc615b6f81383d39ULL,+	0x137a4fb486df2a61ULL,0xa1ed9c07ecf7b4a2ULL,0xb2e460e27bd042ffULL,0xb7f5e2fa5f62f5ecULL,0x7aa6ec6bcc2423b7ULL,0x75ce0a7fba63eea7ULL,0x67a45fb1f250a6e1ULL,0x93bc919ce53cdc9fULL,+	0x67930af231f63950ULL,0xa77797c114caa2c9ULL,0x526e80ee27ac7e62ULL,0xe1e6e62658b28aecULL,0x636178b0b3c9fef0ULL,0xaf7752e06d5f90beULL,0x94ecaf18eece51cfULL,0x2864d0edca806e1fULL,+	0xec2fccaaddce3345ULL,0x2a6811b7012a4350ULL,0x96760ff1ac598bdcULL,0x054d652ad1bf4128ULL,0x0a1151d492a21005ULL,0xad7f397133110fdfULL,0x8c95928c1960100fULL,0x6c91c8257bf03362ULL,+	0x17785b7799eb6df0ULL,0x26c3cc517386b779ULL,0x345ed9886417a48eULL,0xe990b4e407d6ef31ULL,0x0f456b7e2586abbaULL,0x239ca6a559c96e9aULL,0xe327459ce2eb4206ULL,0x3a4c3313a002b90aULL,+	0x19e6125dec3f1decULL,0x07b1f040911178daULL,0xd93ededa904a6738ULL,0x55187a5a0bebedcdULL,0xf7d04722eb329d41ULL,0xf449099ef170b391ULL,0xfd317a69ca99f828ULL,0x50c3db2b34a4976dULL,+	0x3806b69b92222f1fULL,0x5a2459ca6cf7ae70ULL,0x6789f69ca85217eeULL,0x5f232b5ee3dc85acULL,0x660e3ec548e9e516ULL,0x124b4e473197eb31ULL,0x10a0cb13aafcca23ULL,0x7bd63ba48213224fULL,+	0xb674481b7bfe7178ULL,0x4e1debae65405868ULL,0x061b2821c48c867dULL,0x69c15b35513b30eaULL,0x3b4a166636871088ULL,0xe5e29f5d1220b1ffULL,0x4b82bb35233d9f4dULL,0x4e07633318cdc675ULL,+	0x0d53f5c7a3e6fcedULL,0xe8cbbdd5f45fbdebULL,0xf85c01df13339a70ULL,0x0ff71880142ceb81ULL,0x4c4e8774bd70437aULL,0x5fb32891ba0bda6aULL,0x1cdbebd2f18bd26eULL,0x2f9526f103a9d522ULL,+	0x40ce305192c4d684ULL,0x8b04d7257612efcdULL,0xb9dcda366f9cae20ULL,0x0edc4d24f058856cULL,0x64f2e6bf85427900ULL,0x3de81295dc09dfeaULL,0xd41b4487379bf26cULL,0x50b62c6d6df135a9ULL,+	0xd4f8e3b4c72dfe67ULL,0xc416b0f690e19fdfULL,0x18b9098d4c13bd35ULL,0xac11118a15b8cb9eULL,0xf598a318f0062841ULL,0xbfe0602f89f356f4ULL,0x7ae3637e30177a0cULL,0x3409774761136537ULL,+	0x0db2fb5ed005832aULL,0x5f5efd3b91042e4fULL,0x8c4ffdc6ed70f8caULL,0xe4645d0bb52da9ccULL,0x9596f58bc9001d1fULL,0x52c8f0bc4e117205ULL,0xfd4aa0d2e398a084ULL,0x815bfe3a104f49deULL,+	0x97e5443f23885e5fULL,0xf72f8f99e8433aabULL,0xbd00b154e4d4e604ULL,0xd0b35e6ae5e173ffULL,0x57b2a0489164722dULL,0x3e3c665b88761ec8ULL,0x6bdd13973da83832ULL,0x3c8b1a1e73dafe3bULL,+	0x4497ace654317cacULL,0xbe600ab9521771b3ULL,0xb42e409eb0dfe8b8ULL,0x386a67d73942310fULL,0x25548d8d4431cc28ULL,0xa7cff142985dc524ULL,0x4d60f5a193c4be32ULL,0x83ebd5c8d071c6e1ULL,+	0xba3a80a7b1fd2b0bULL,0x9b3ad3965bec33e8ULL,0xb3868d6179743fb3ULL,0xcfd169fcfdb462faULL,0xd3b499d79ce0a6afULL,0x55dc1cf1e42d3ff8ULL,0x04fb9e6cc6c3e1b2ULL,0x47e6961d6f69a474ULL,+	0x54eb3acce548b37bULL,0xb38e754284d40549ULL,0x8c3daa517b341b4fULL,0x2f6928ec690bf7faULL,0x0496b32386ce6c41ULL,0x01be1c5510adadcdULL,0xc04e67e74bb5faf9ULL,0x3cbaf678e15c9985ULL,+	0x8cd1214550ca4247ULL,0xba1aa47ae7dd30aaULL,0x2f81ddf1e58fee24ULL,0x03452936eec9b0e8ULL,0x8bdc3b81243aea96ULL,0x9a2919af15c3d0e5ULL,0x9ea640ec10948361ULL,0x5ac86d5b6e0bcccfULL,+	0xf892d918c36cf440ULL,0xaed3e837c939719cULL,0xb07b08d2c0218b64ULL,0x6f1bcbbace9790ddULL,0x4a84d6ed60919b8eULL,0xd89007918ac1f9ebULL,0xf84941aa0dd5daefULL,0xb22fe40a67fd62c5ULL,+	0x97e15ba2157f2db3ULL,0xbda2fc8f8e28ca9cULL,0x5d050da437b9f454ULL,0x3d57eb572379d72eULL,0xe9b5eba2fb5ee997ULL,0x01648ca2e11538caULL,0x32bb76f6f6327974ULL,0x338f14b8ff3f4bb7ULL,+	0x524d226ad7ab9a2dULL,0x9c00090d7dfae958ULL,0x0ba5f5398751d8c2ULL,0x8afcbcdd3ab8262dULL,0x57392729e99d043bULL,0xef51263baebc943aULL,0x9feace9320862935ULL,0x639efc03b06c817bULL,+	0x1fe054b366b4be7aULL,0x3f25a9de84a37a1eULL,0xf39ef1ad78d75cd9ULL,0xd7b58f495062c1b5ULL,0x6f74f9a9ff563436ULL,0xf718ff29e8af51e7ULL,0x5234d31315e97fecULL,0xb6a8e2b1292f1c0aULL,+	0xa7f53aa8327720c1ULL,0x956ca322ba092cc8ULL,0x8f03d64a28746c4dULL,0x51fe178266d0d392ULL,0xd19b34db3c832c80ULL,0x60dccc5c6da2e3b4ULL,0x245dd62e0a104cccULL,0xa7ab1de1620b21fdULL,+	0xb293ae0b3893d123ULL,0xf7b75783b15ee71cULL,0x5aa3c61442a9468bULL,0xd686123cdb15d744ULL,0x8c616891a7ab4116ULL,0x6fcd72c8a4e6a459ULL,0xac21911077e5fad7ULL,0xfb6a20e7704fa46bULL,+	0xe839be7d341d81dcULL,0xcddb688932148379ULL,0xda6211a1f7026eadULL,0xf3b2575ff4d1cc5eULL,0x40cfc8f6a7a73ae6ULL,0x83879a5e61d5b483ULL,0xc5acb1ed41a50ebcULL,0x59a60cc83c07d8faULL,+	0x439530b665c7322dULL,0xcf12cc01b3c1b3fbULL,0xc70b01860172f685ULL,0xb915ee221b58391dULL,0x9afdf03ba317db24ULL,0x87dec65917b8ffc4ULL,0x7f46597be4d3d050ULL,0x80a1c1ed006500e7ULL,+	0x3e22a7b397acf4ecULL,0x0426c4005ea8b640ULL,0x5e3295a64e969285ULL,0x22aabc59a6a45670ULL,0xb929714c5f5942bcULL,0x9a6168bdfa3182edULL,0x2216a665104152baULL,0x46908d03b6926368ULL,+	0x52cb8ac6c2babcc1ULL,0x4748d448fe81ae8dULL,0x5844f03f80f1a711ULL,0x3db784b2f8df4ac4ULL,0xad918f122df1fe36ULL,0xe40f25b9f33cc7c0ULL,0x4700d0e73b5e5555ULL,0x5c28fa08b03326f9ULL,+	0xa9f5d8745a1251fbULL,0x967747a8c72725c7ULL,0x195c33e531ffe89eULL,0x609d210fe964935eULL,0xcafd6ca82fe12227ULL,0xaf9b5b960426469dULL,0x2e9ee04c5693183cULL,0x1084a333c8146fefULL,+	0x56dab1c8321a518cULL,0xfd4439a68bce226fULL,0xe0b30d194facb9faULL,0xb5052f307583571bULL,0x1442641012afd476ULL,0xd02e417203fe624aULL,0xfc394f65531c92e6ULL,0x16d4bf5ad4bc0b52ULL,+	0xce06b88210395755ULL,0x117ce6345ec1df80ULL,0xfefae513eff55e96ULL,0xcf36cba6fd7fed1eULL,0x7340eca9a40ebf88ULL,0xe6ec1bcfb3d37e12ULL,0xca51b64e86bbf9ffULL,0x4e0dbb588b40e05eULL,+	0x120ad0bf5f8b8a84ULL,0xbfa00f36866f3edeULL,0xa8c6064ec30ebc2cULL,0xc39e40b823001e3bULL,0x9614f7cde7b1cbabULL,0xcd4420c704a56284ULL,0x8446a8f316857a19ULL,0x519b93b1e6f706cbULL,+	0x96649933aed1d1f7ULL,0x566eaff350563090ULL,0x345057f0ad2e39cfULL,0x148ff65b1f832124ULL,0x042e89d4cf94cf0dULL,0x319bec84520c58b3ULL,0x2a2676265361aa0dULL,0xc86fa3028fbc87adULL,+	0x3805e5b80863b664ULL,0x8be7ac6b3cabdb53ULL,0x9f7d70505dfeff91ULL,0x7dea8bd095896206ULL,0x28005a3b410e3c4eULL,0x24a4f0e9bc603ebfULL,0xcc4fd5ae4aec15d2ULL,0x4dc253f80f96641dULL,+	0x359d7b9c7ea2ee34ULL,0x3fd0d94c09cc3a71ULL,0xbb53c31c3a1ea37aULL,0x533425facf818c87ULL,0x7cd199c3810156e0ULL,0x0ea020e430c16448ULL,0xe557ba094a642542ULL,0xe657e7e79465f5eaULL,+	0xea723ad1fbc82439ULL,0xc726868bf896e9fdULL,0xd97c913c511c33ffULL,0x8a3fa8e2e2114231ULL,0x2a6c0e5608445b3eULL,0x2c4cd884f8d098d3ULL,0x7bf51faf2fea77d4ULL,0x709f208a7b1c4f71ULL,+	0xfc83d2ab5c8b06d5ULL,0xb1a785a2fe4eac46ULL,0xb99315bc846f7779ULL,0xcf31d816ef9ea505ULL,0x2391fe6a15d7dc85ULL,0x2f132b04b4016b33ULL,0x29547fe3181cb4c7ULL,0xdb66d8a6650155a1ULL,+	0x5b9e4843f45aac50ULL,0xc31e042e91eaaad0ULL,0x6e8c0b345a54eea0ULL,0xf0437b94962c7a57ULL,0xe4531ce8fe1d348bULL,0xe1489378e3786432ULL,0xd5e19c4a3f510d38ULL,0x4df3f016ce348b00ULL,+	0x59cd0e8b593d070fULL,0x437575165255625dULL,0x551fdda75b7a0399ULL,0x7bb6e6b02dec1eebULL,0x729bb662334c0922ULL,0x3df631df0cf41b79ULL,0x01abf3c578f32402ULL,0xfcb4666c9cd33c88ULL,+	0xb805b445735e843cULL,0x2a8e890d97379134ULL,0xebc7c10c52ab9f87ULL,0xcbb5e1ecb80a92b6ULL,0xd6ada2d9dc2c4efeULL,0xfccf504eae8cc7bdULL,0x650115acb2418a74ULL,0x8dd90e06c52bd80cULL,+	0x6b66d7e1adc1696fULL,0x98ebe5930acd72d0ULL,0x65f24550cc1b7435ULL,0xce231393b4b9a5ecULL,0x234a22d4db067df9ULL,0x98dda095caff9b00ULL,0x1bbc75a06100c9c1ULL,0x1560a9c8939cf695ULL,+	0xe4050f1cf1c367caULL,0x9bc85a9bc90fbc7dULL,0xa373c4a2e1a11032ULL,0xb64232b7ad0393a9ULL,0xf5577eb0167dad29ULL,0x1604f30194b78ab2ULL,0x0baa94afe829348bULL,0x77fbd8dd41654342ULL,+	0x31f14802fcf0a7fdULL,0x42fd07895488b01eULL,0x71d78d6d9952b498ULL,0x8eb572d907ac5201ULL,0xe0a2a44c4d194a88ULL,0xd2b63fd9ba017e66ULL,0x78efc6c8f888aefcULL,0xb76f6bda4a881a11ULL,+	0xa2f7932c68af43eeULL,0x5502468e703d00bdULL,0xe5dc978f2fb061f5ULL,0xc9a1904a28c815adULL,0xd3af538d470c56a4ULL,0x159abc5f193d8cedULL,0x2a37245f20108ef3ULL,0xfa17081e223f7178ULL,+	0x1fe2a9b2b4b4b67cULL,0xc1d10df0e8020604ULL,0x9d64abfcbc8058d8ULL,0x8943b9b2712a0fbbULL,0x90eed9143b3def04ULL,0x85ab3aa24ce775ffULL,0x605fd4ca7bbc9040ULL,0x8b34a564e2c75dfbULL,+	0x5c18acf88e2f7d90ULL,0xfdbf33d777be32cdULL,0x0a085cd7d2eb5ee9ULL,0x2d702cfbb3201115ULL,0xb6e0ebdb85c88ce8ULL,0x23a3ce3c1e01d617ULL,0x3041618e567333acULL,0x9dd0fd8f157edb6bULL,+	0xb2b2610798fa7aaaULL,0x41209ee4f073aa4eULL,0xf1570359f2d6b19bULL,0xcbe6868cfc577cafULL,0x186c4bdc32c04dd3ULL,0xa6c35faecfeee397ULL,0xb4a1b312f086c0cfULL,0xe0a5ccc6d9461fe2ULL,+	0x516ff3a36fa6110cULL,0x74fb1eb1fb93561fULL,0x6c0c90478457522bULL,0xcfd321046bb8bdc6ULL,0x2d6884a2cc80ad57ULL,0x7c27fc3586a9b637ULL,0x3461baedadf4e8cdULL,0x1d56251a617242f0ULL,+	0xb84011a9431dd80eULL,0xeb7c7cca73306cd9ULL,0x20fadd29d1b3b730ULL,0x83858b5bfe37b3d3ULL,0xbf4cd193b6251d5cULL,0x1cca1fd31352d952ULL,0xc66157a490fbc051ULL,0x7990a63889b98636ULL,+	0x892c81a321175ec1ULL,0x9159a505ee018109ULL,0xc70130532d8be316ULL,0x76060c21426fa2e5ULL,0x074d2dfc6b6f0f22ULL,0x9725fc64ca01a671ULL,0x3f6679b92770bd8eULL,0x8fe6604fd7c9b3feULL,+	0xce711154b6e00a84ULL,0xd9fe7e4224890e60ULL,0xd10bc6c34560988fULL,0xbdc2ef526859b004ULL,0xdcf0d868d5c890eeULL,0x893115e6119c47dcULL,0xe97966fbee714567ULL,0x117813355c85aa53ULL,+	0x71d530cc73204349ULL,0xc9df473d94a0679cULL,0xc572f0014261e031ULL,0x9786b71f22f135feULL,0xed6505fa6b64e56fULL,0xe2fb48e905219c46ULL,0x0dbec45bedf53d71ULL,0xd7d782f2c589f406ULL,+	0x350fb66b73ed0966ULL,0x968e4b082886032bULL,0x5a16ed6e88390493ULL,0x0a83ce84a121edbeULL,0x7c86fc10a3d9cda0ULL,0x5a40a6d595ce67fbULL,0x6cb8cda7f937dbf6ULL,0x95b44768651f6283ULL,+	0x06513c8a446cd7f4ULL,0x158c423b906d52a6ULL,0x71503261c423866cULL,0x4b96f57093c148eeULL,0x5daf9cc7239a8523ULL,0x611b597695ac4b8bULL,0xde3981db724bf7f6ULL,0x7e7d0f7867afc443ULL,+	0x984f101ed6fc3837ULL,0x340bf99f5e1b3a09ULL,0xbb96036f06942626ULL,0x7bc878ab0c7da618ULL,0xb37416441c6fb035ULL,0xc65bd5aea182fe9fULL,0x1b9c2fb86cc7a67aULL,0x8d1b19af5ce68d7dULL,+	0x3d1ab80c8ce59954ULL,0x742c5a9478222ac0ULL,0x3ddacbf894f878ddULL,0xfc085117e7d54a99ULL,0xfb0f1dfa21e38ec2ULL,0x1c7b59cb16f4ff7fULL,0x988752397ea888feULL,0x705d270cb10dc889ULL,+	0xe5aa692a87dec0e1ULL,0x010ded8df7b39d00ULL,0x7b1b80c854cfa0b5ULL,0x66beb876a0f8ea28ULL,0x50d7f5313476cd0eULL,0xa63d0e65b08d3949ULL,0x1a09eea953479fc6ULL,0x82ae9891f499e742ULL,+	0xab58b9105ca7d866ULL,0x582967e23adb3b34ULL,0x89ae4447cceac0bcULL,0x919c667c7bf56af5ULL,0x9aec17b160f5dcd7ULL,0xec697b9fddcaadbcULL,0x0b98f341463467f5ULL,0xb187f1f7a967132fULL,+	0xeb5ddcb6ec7fae9fULL,0x995f2714efb66e5aULL,0xdee95d8e69445d52ULL,0x1b6c2d4609e27620ULL,0x32621c318129d716ULL,0xb03909f10958c1aaULL,0x8c468ef91af4af63ULL,0x162c429ffba5cdf6ULL,+	0xe8cb5eef9c053df7ULL,0x8de25b37b300ea6fULL,0xdb03fa92c849cffbULL,0x242e43a7e84169bbULL,0xe4fa51f4dd6f958eULL,0x6925a77ff4445a8dULL,0xe6e72a50e90d8949ULL,0xc66648e32b1f6390ULL,+	0x6c3b96f31711ebecULL,0x2da40f1fce98fdc4ULL,0xb99774d357b4411fULL,0x87c8bdf415b65bb6ULL,0xda3a89e3c2eef12dULL,0xde95bb9b3c7471f3ULL,0x600f225bd812c594ULL,0x54907c5d2b75a56bULL,+	0x699e4d2945c1dd53ULL,0xcadc5898231debb5ULL,0xdf49fcc7a77f00e0ULL,0x93057bbfa73e5a0eULL,0x2f8b7ecd027a4cd1ULL,0x114734b3c614011aULL,0xe7a01db767677c68ULL,0x89d9be5e7e273f4fULL,+	0xe0deee5931fba239ULL,0xf47424d398bd91d1ULL,0x0f8886f4071a3c1dULL,0x3f7d41e8a819233bULL,0x708623c2cf6eb998ULL,0x86bb49af609a287fULL,0x942bb24963c90762ULL,0x0ef6eea555a9654bULL,+	0x4add4a2e649d4e57ULL,0xcd53a2b01917526eULL,0xc526233020b44ac4ULL,0x4028746abaa2c31dULL,0x5131839064291d4cULL,0xbf48f151ee5ad909ULL,0xcce57f597b185681ULL,0x7c3ac1b04854d442ULL,+	0xa80d1db6f79588c0ULL,0xfa52fc69b55768ccULL,0x0b4df1ae7f54438aULL,0x0cadd1a7f9b46a4fULL,0xb40ea6b31803dd6fULL,0x488e4fa555eaae35ULL,0x9f047d55382e4e16ULL,0xc9b5b7e02f6e0c98ULL,+	0xc19972d0b611c24bULL,0x1d468e6560a8f351ULL,0xeb7580697bcf6421ULL,0xec9dd0ee88fbc491ULL,0x5b59d2bf956c2e32ULL,0x73dc6864dcddf94eULL,0xfd5e2321bcee7665ULL,0xa7b4f8ef5e9a06c4ULL,+	0x03cc8f17cf41c6e8ULL,0xf1f03c2a037b925cULL,0xc39c19cc66d2427cULL,0x823d24ba7b6c18e4ULL,0x32ef9013901f0b4fULL,0x684360f1f8941c2eULL,0x0ebaff522c28092eULL,0x7891e4e3256c932fULL,+	0x174e8f82d8d38a9bULL,0x2e97c600e7de1391ULL,0xc5709850a1c175ddULL,0x969041a032ae5035ULL,0xcbfd533b76a2086bULL,0xd6bba71bd7c2e8feULL,0xb2d58ee6099dfb67ULL,0x3a8b342d064a85d9ULL,+	0xf83cbf0502f40d9aULL,0x4681c4682c318a4dULL,0x985756180e9c2674ULL,0xbe79d0461847092eULL,0xaf1e480a78bd01e0ULL,0x6dd359e472a51db9ULL,0x62ce3821e3afbab6ULL,0xc5cee5b617733199ULL,+	0x671ed8fc3b922bf8ULL,0xe4d8c0a04c29b133ULL,0x87eb12393b6e99c4ULL,0xaff3974c8793bebaULL,0x037494052c18df9bULL,0xc5c3a29391007139ULL,0x6a77234fe37a0b95ULL,0x02c29a21b661c96bULL,+	0x5a52fe2e34d74e31ULL,0xa352c3103bf79ab6ULL,0x97ff6c5aabfeeb8fULL,0xbfbe8feff5c97305ULL,0xd6081ce6a7904608ULL,0x1f812f3ac4fca249ULL,0x9b24bc9ab9e5e200ULL,0x91022c6738012ee8ULL,+	0x184de7d7cc5f4394ULL,0xb5551b5c4536e142ULL,0x2e89b212d34aa60aULL,0x14a96feaf50051d5ULL,0x4e21ef740d12bb0bULL,0xc522f02060b9677eULL,0x8b12e4672df7731dULL,0x39f803827b326d31ULL,+	0xc12738b67c4a658aULL,0xb3c4763940e72182ULL,0x3b77be468798e44fULL,0xdc047df217a7f85fULL,0x2439d4c55e59d92dULL,0xcedca475e8e64d8dULL,0xa724cd0d87ca9b16ULL,0x35e4fd59a5540dfeULL,+	0x9894344f3a29467aULL,0xde81e949c51eba6dULL,0xdaea066ba5e5c2f2ULL,0x3fc8a61408c8c7b3ULL,0x7adff88f06d0de9fULL,0xbbc11cf53b75ce0aULL,0x9fbb7accfbbc87d5ULL,0xa1458e267badfde2ULL,+	0x1cb43668e039c256ULL,0x5f26fb8b7c17fd5dULL,0xeee426af79aa062bULL,0x072002d0d78fbf04ULL,0x4c9ca237e84fb7e3ULL,0xb401d8a10c82133dULL,0xaaa525926d7e4181ULL,0xe943083373dbb152ULL,+	0x2f5fad1e6ee7c983ULL,0xeb0f9d7cb41328f5ULL,0x9ba68b441d78d5f7ULL,0xa06b3b9e35bc726fULL,0xa2550255593e1ff1ULL,0x552dd43ddfbec115ULL,0x2c48a7abbba8f046ULL,0xd4fc56a3ad0bf133ULL,+	0xf92dda31be24319aULL,0x03f7d28be095a8e7ULL,0xa52fe84098782185ULL,0x276ddafe29c24dbcULL,0x80cd54961d7a64ebULL,0xe43608897f1dbe42ULL,0x2f81a8778438d2d5ULL,0x7e4d52a885169036ULL,+	0x62f21cefba04b5d1ULL,0x9a442707224c7352ULL,0xbf07966c33c6171fULL,0xdb7ba8911e0816b0ULL,0x306fea59033745a9ULL,0x2aacf7e0c0a78f67ULL,0xb5aa3883ad251bf9ULL,0x345aede926bd7086ULL,+	0x19e3d5b11d59715dULL,0xc7eaa762d788983eULL,0xe5a730b0abf1f248ULL,0xfbab8084fae3fd83ULL,0x65e50d2153765b2fULL,0xbdd4e083fa127f3dULL,0x9cf3c074397b1b10ULL,0x59f8090cb1b59fd3ULL,+	0xdc1de17d98119f10ULL,0x74353c5d488c36a6ULL,0x14aaf33a3d8e23dfULL,0x31e075c078baf593ULL,0x0f7ca03a46d1ca3cULL,0x99c5e3ac47b660c7ULL,0x70d0241388fe2e59ULL,0x2e9a6be12a7ec005ULL,+	0x7b15fd9d615faa8fULL,0x8fa1eb40968554edULL,0x7bb4447e7aa44882ULL,0x2bb2d0d1029fff32ULL,0x075e2a646caa6d2fULL,0x8eb879de22e7351bULL,0xbcd5624e9a506c62ULL,0x218eaef0a87e24dcULL,+	0xac449695241fbd6fULL,0x67c9b170081c1223ULL,0x16868f21b56aac6fULL,0x34bd8fa3f8bcb721ULL,0x06b6bd33b6691c76ULL,0x6c924766381a7973ULL,0x6a12444ca54078dbULL,0xd02e91a96d1051ccULL,+	0x37e5684744ddfa35ULL,0x9ccfc5c5dab3f747ULL,0x9ac1df3f1ee96cf4ULL,0x0c0571a13b480b8fULL,0x2fbeb3d54b3a7b3cULL,0x35c036695dcdbb99ULL,0x52a0f5dcb2415b3aULL,0xd57759b44413ed9aULL,+	0x077379c00b33d3f8ULL,0x421883c67064e409ULL,0x2d0873d76c29c8f6ULL,0xbfa433a3d274c0c8ULL,0x56dc778f23a5891eULL,0xd663bf6535e2de04ULL,0x488fdb485db517ceULL,0x00bba55e19b226c2ULL,+	0x1fe647d83d30a2c5ULL,0x0857f77ef78a81dcULL,0x11d5a334131a4a9bULL,0xc0a94af929d393f5ULL,0xbc3a5c0bdaa6ec1aULL,0xba9fe49388d2d7edULL,0xbb4335b4bb614797ULL,0x991c4d6872f83533ULL,+	0xedbbeee78a058fb6ULL,0xb9d19ddcfb09121aULL,0xa41bb45bd34dddceULL,0x2dbc80b900964bc4ULL,0x4ed9137d1d6cb654ULL,0x1b9016db483d01c5ULL,0x5fc501bc6528e22eULL,0xb2d2f8816cad646bULL,+	0x53258c28d2f01cb3ULL,0x93d6eaa3d75db0b1ULL,0x419a2b0de87d0db4ULL,0xa1e48f03d8fe8493ULL,0xf747faf6c508b23aULL,0xf137571a35d53549ULL,0x9f5e58e2fcf9b838ULL,0xc7186ceea7fd3cf5ULL,+	0x5e76fb2f286bad39ULL,0xbad9efe39dcad1e2ULL,0x60e75190edc7e904ULL,0x6a6f063e0fecb5a5ULL,0x5150ed85aed8acc3ULL,0xb56ccfbc6d20af6cULL,0x7e0d1e982c69dbfaULL,0xabf5628a7c7e10a9ULL,+	0x77b868cee978a1d3ULL,0xe3a68b337ab92d04ULL,0x5102979487a5b862ULL,0x5f0606c33a61d41dULL,0x2814be276f9326f1ULL,0x2f521c14c6fe3c2eULL,0x17464d7dacdf7351ULL,0x10f5f9d3777f7e44ULL,+	0xb06b1244c5f95cd8ULL,0xda8c8af0f4ab95f4ULL,0x1bae59c2b9e5836dULL,0x07d51e7e3acffffcULL,0x01e15e6ac2ccbcdaULL,0x3bc1923f8528c3e0ULL,0x43324577a49fead4ULL,0x61a1b8842aa7a711ULL,+	0x4fe7ee31b0e63d34ULL,0xf4600572a9e54fabULL,0xc0493334d5e7b5a4ULL,0x8589fb9206d54831ULL,0xaa70f5cc6583553aULL,0x0879094ae25649e5ULL,0xcc90450710044652ULL,0xebb0696d02541c4fULL,+	0x172a5247d95ea168ULL,0x1758fada2970764aULL,0xac803a511d978169ULL,0x299cfe2ede77e01bULL,0x652a1e17b0a98927ULL,0x2e26e1d120014495ULL,0x7ae0af9f7175b56aULL,0xc2e22a80d64b9f95ULL,+	0x758c1a3ea2dee7a6ULL,0xdcde2f3c734b2284ULL,0xaba445d24eaba6adULL,0x35aaf66876cee0a7ULL,0x7e0b04a9e5aa049aULL,0xe74083ad91103e84ULL,0xbeb183ce40afecc3ULL,0x6b89de9fea043f7aULL,+	0xafbfb1eda4f7e665ULL,0x403cce6aa23df8e7ULL,0xb49cc83f1312c2f4ULL,0xe1cc2366771a9c34ULL,0x7ab6a6c0db92faacULL,0xacd15e0dec3befe1ULL,0x7f8ed988583a0f36ULL,0x1821de7705f9be09ULL,+	0xb99f0e0399375235ULL,0x7614c847b9917970ULL,0xfec93ce9524ec067ULL,0xe40e7bf89b122520ULL,0xb5670631ee4c4774ULL,0x6f03847a3b04914cULL,0xc96e9429dc9dd226ULL,0x43489b6c8c57c1f8ULL,+	0x7b9722a5c5f26464ULL,0xca4c3dfba442809aULL,0x7d10986723644810ULL,0x9e4951723c924f82ULL,0xef4a6968a2c5bc14ULL,0x750eac4f68de6b7aULL,0x4e01884d52a2cbb5ULL,0xac40830af4a5f446ULL,+	0x0e299d23fe67ba66ULL,0x9145076093cf2f34ULL,0xf45b5ea997fcf913ULL,0x5be008438bd7dddaULL,0x358c3e05d53ff04dULL,0xbf7ccdc35de91ef7ULL,0xad684dbfb69ec1a0ULL,0x367e7cf2801fd997ULL,+	0x8c54f1076103adf8ULL,0x2d813d6cbe8e9810ULL,0xb1466fa85fbd3c9bULL,0x68c65d2240e1ca76ULL,0xb81baa40255f9164ULL,0x5b34c3eed1a864b2ULL,0x3602209b122ca141ULL,0xe7d7248e885badebULL,+	0x46ffd227cc2338fbULL,0x89ff6fa990e26153ULL,0xbe570779331a0076ULL,0x43d241c506e1f3afULL,0xfdcdb97dde9b62a3ULL,0x6a06e984a0ae30eaULL,0xc9bf16804fbddf7dULL,0x170471a2d36163c4ULL,+	0xfd23e207a6469d43ULL,0xcb9f5f112f753a85ULL,0xde2625d4fbb5ca72ULL,0x82e4e54ab7b1c78bULL,0x2cd0ca5378b9e814ULL,0xfcd44051125b817aULL,0xb68f719f30cfd965ULL,0x31644719d848a974ULL,+	0xff5ba8ae3113655eULL,0xfa2c6e2b57b83180ULL,0x1c48271977e0eabeULL,0xf9f3c555337fea97ULL,0x340f7022a42581cbULL,0xe1de0bc218f710e3ULL,0xee640adef62e5aa8ULL,0x16b2389149428940ULL,+	0x5045738f25f653f5ULL,0xe42b8cb83764f635ULL,0xb4f89406dc11ffc3ULL,0x99593144b6b3e4aaULL,0x81c849f3c9740052ULL,0x2c9cf4c155ffc48bULL,0x6299e52177f67a49ULL,0x5869f6e3c00c6c62ULL,+	0x361619e455950cc3ULL,0xc71d665c56b66bb8ULL,0xea034b34afac6d84ULL,0xa987f832e5e4c7e3ULL,0xa07427727a79a6a7ULL,0x56e5d017e26d6c23ULL,0x7e50b97638167e10ULL,0xaa6c81efe88aa84eULL,+	0xb84186f75fe01576ULL,0x446e276cdea51395ULL,0xf3c5ef8105f3f8d4ULL,0x3d7f7df674f7f142ULL,0x7c69b565f9ef1656ULL,0x87efa4247c414ef6ULL,0xeb7e620d3d292060ULL,0x50b1de346eec21aeULL,+	0x0ca1f3b7b0dc8595ULL,0x27de46089f1d9f2eULL,0x1af3bf39badd82a7ULL,0x79356a7965862448ULL,0xc0602345f5f9a052ULL,0x1a8b0f89139a42f9ULL,0xb53eee42844d40fcULL,0x93b0bfe54e5b6368ULL,+	0x0f893a5dc8de610bULL,0xe8c515fb67e223ceULL,0x7774bfa64ead6dc5ULL,0x89d20f95925c728fULL,0x7a1e0966098583ceULL,0xa2eedb9493f2a7d7ULL,0x1b2820974c304d4aULL,0x0842e3dac077282dULL,+	0xa1010e9d74cd06ffULL,0x9c17c7dfaca3eeacULL,0x74c86cd38063aa2bULL,0x8595c4b3734614ffULL,0xa3de00ca990f62ccULL,0xd9bed213ca0c3be5ULL,0x7886078adf8ce9f5ULL,0xddb27ce35cd44444ULL,+	0x5a3097befc15aa1eULL,0x40d12548b54b0745ULL,0x5bad4706519a5f12ULL,0xed03f717a439dee6ULL,0x0794bb6c4a02c499ULL,0xf725083dcffe71d2ULL,0x2cad75190f3adcafULL,0x7f68ea1c43729310ULL,+	0x9c7c581d26ee8382ULL,0xcf17dcc5359d638eULL,0xee8273abb728ae3dULL,0x1d112926f821f047ULL,0x1149847750491a74ULL,0x687fa761fde0dfb9ULL,0x2c2580227ea435abULL,0x6b8bdb9491ce7e3fULL,+	0x9c806d8af7f91d0fULL,0x3b61b0f1a82a5728ULL,0x4640032d94d76754ULL,0x273eb5de47d834c6ULL,0x2988abf77b4e4d53ULL,0xb7ce66bfde401777ULL,0x9fba6b32715071b3ULL,0x82413c24ad3a1a98ULL,+	0x75537b7e3cc8ac85ULL,0x8d725f57dd02753bULL,0xfd05ff64b737df2fULL,0x55fe8712f6d2531dULL,0x57ce04a96ab6b01cULL,0x69a02a897cd93724ULL,0x4f82ac35cf86699bULL,0x8242d3ad9cb4b232ULL,+	0x69c435269be47be0ULL,0x323b7dd8cb28fea1ULL,0xfa5538ba3a6c67e5ULL,0xef921d701d378e46ULL,0xf92961fc3c4b880eULL,0x3f6f914e98940a67ULL,0xa990eb0afef0ff39ULL,0xa6c2920ff0eeff9cULL,+	0xb23a03a553fb2b56ULL,0x6ce141e74e057f78ULL,0x796525c389e490d9ULL,0x0bc95725a31a7e75ULL,0x1ec567911220fd06ULL,0x716e3a3c408b0bd6ULL,0x31cd6bf7e8ebeba9ULL,0xa7326ca6bee6b670ULL,+	0x70b63d32343bf1a9ULL,0x8fd3bd2837d1a6b1ULL,0x0454879c316865b4ULL,0xee959ff6c458efa2ULL,0x0461dcf89706dc3fULL,0x737db0e2164e4b2eULL,0x092626802f8843c8ULL,0x54498bbc7745e6f6ULL,+	0x5341352b5acf6e10ULL,0xc50343fdafe652c3ULL,0x4af3792d18577a7fULL,0xe1a4c617af16823dULL,0x9b26d0cd33425d0aULL,0x306399ed9b7bc47fULL,0x2a792f33706bb20bULL,0x3121961498111055ULL,+	0x4c1f428cd5f30851ULL,0x94dfed272a4f6630ULL,0x4df53772fc5d48a4ULL,0xdd2d5a2f933260ceULL,0x574115bdd44cc7a5ULL,0x4ba6b20dbd12533aULL,0x30e93cb8243057c9ULL,0x794c486a14de320eULL,+	0x6095355699f241d7ULL,0xee4adbd7001a349dULL,0x0b35bf6aaa89e491ULL,0x7f0076f4136f7546ULL,0xd19a18ba9264da3dULL,0x6eb2d2cd62a7a28bULL,0xcdba941f8761c971ULL,0x1550518ba3be4a5dULL,+	0xc232d97302f1cd1eULL,0xce87eacb1dd212a4ULL,0x6e4c8c73e69802f7ULL,0x12ef02901fffddbdULL,0x941ec74e1bcea6e2ULL,0xd0b540243cb92cbbULL,0x809fb9d47e8f9d05ULL,0x3bf16159f2992aaeULL,+	0xb2497007eafbb1e1ULL,0xd75c9ce6e75b7a93ULL,0x3558352defb68d78ULL,0xa2f26699223f6396ULL,0xeb911ecfe469b17aULL,0x62545779e72d3ec2ULL,0x8ea47de782cb113fULL,0xebe4b0864e1fa98dULL,+	0xbdb8e675b055cb40ULL,0x898f8e7b977b5167ULL,0xecc65651b82fb863ULL,0x565448146d88f01fULL,0xb0928e95263a75a9ULL,0xcfb6836f1a22fcdaULL,0x651d14db3f3bd37cULL,0x1d3837fbb6ad4664ULL,+	0x20d3c982cf7d62d2ULL,0x1f36e29d23ba8150ULL,0x48ae0bf092763f9eULL,0x7a527e6b1d3a7007ULL,0xb4a89097581a85e3ULL,0x1f1a520fdc158be5ULL,0xf98db37d167d726eULL,0x8802786e1113e862ULL,+	0xefb2149e36f09ab0ULL,0x03f163ca4a10bb5bULL,0xd029704506e20998ULL,0x56f0af001b5a3babULL,0x7af4cfec70880e0dULL,0x7332a66fbe3d913fULL,0x32e6c84a7eceb4bdULL,0xedc4a79a9c228f55ULL,+	0xc37c7dd0c55c4496ULL,0xa6a9635725bbabd2ULL,0x5b7e63f2add7f363ULL,0x9dce37822e73f1dfULL,0xe1e5a16ab2b91f71ULL,0xe44898235ba0163cULL,0xf2759c32f6e515adULL,0xa5e2f1f88615eecfULL,+	0x74519be7abded551ULL,0x03d358b8c8b74410ULL,0x4d00b10b0e10d9a9ULL,0x6392b0b128da52b7ULL,0x6744a2980b75c904ULL,0xc305b0aea8f7f96cULL,0x042e421d182cf932ULL,0xf6fc5d509e4636caULL,+	0x795847c9d64cc78cULL,0x6c50621b9b6cb27bULL,0x07099bf8df8022abULL,0x48f862ebc04eda1dULL,0xd12732ede1603c16ULL,0x19a80e0f5c9a9450ULL,0xe2257f54b429b4fcULL,0x66d3b2c645460515ULL,+	0x6ca4f87e822e37beULL,0x73f237b4253bda4eULL,0xf747f3a241190aebULL,0xf06fa36f804cf284ULL,0x0a6bbb6efc621c12ULL,0x5d624b6440b80ec6ULL,0x4b0724257ba556f3ULL,0x7fa0c3543e2d20a8ULL,+	0xe921fa31e3229d41ULL,0xa929c65294531bd4ULL,0x84156027a6d38209ULL,0xf3d69f736bdb97bdULL,0x8906d19a16833631ULL,0x68a34c2e03d51be3ULL,0xcb59583b0e511cd8ULL,0x99ce6bfdfdc132a8ULL,+	0x3facdaaaffcdb463ULL,0x658bbc1a34a38b08ULL,0x12a801f8f1a9078dULL,0x1567bcf96ab855deULL,0xe08498e03572359bULL,0xcf0353e58659e68bULL,0xbb86e9c87d23807cULL,0xbc08728d2198e8a2ULL,+	0x8de2b7bc453cadd6ULL,0x203900a7bc0bc1f8ULL,0xbcd86e47a6abd3afULL,0x911cac128502effbULL,0x2d550242ec965469ULL,0x0e9f769229e0017eULL,0x633f078f65979885ULL,0xfb87d4494cf751efULL,+	0xe1790e4bfc25419aULL,0x364672034bff3cfdULL,0xc8db638625b6e83fULL,0x6cc69f236cad6fd2ULL,0x0219e45a6bc68bb9ULL,0xe43d79b6297f7334ULL,0x7d445368465dc97cULL,0x4b9eea322a0b949aULL,+	0x1b96c6ba6102d021ULL,0xeaafac782f4461eaULL,0xd4b85c41c49f19a8ULL,0x275c28e4cf538875ULL,0x35451a9ddd2e54e0ULL,0x6991adb50605618bULL,0x5b8b4bcd7b36cd24ULL,0x372a4f8c56f37216ULL,+	0xc890bd73a6a5da60ULL,0x6f083da0dc4c9ff0ULL,0xf4e14d94f0536e57ULL,0xf9ee1edaaaec8243ULL,0x571241ec8bdcf8e7ULL,0xa5db82710b041e26ULL,0x9a0b9a99e3fff040ULL,0xcaaf21dd7c271202ULL,+	0xb4e2b2e14f0dd2e8ULL,0xe77e7c4f0a377ac7ULL,0x69202c3f0d7a2198ULL,0xf759b7ff28200eb8ULL,0xc87526eddcfe314eULL,0xeb84c52453d5cf99ULL,0xb1b52ace515138b6ULL,0x5aa7ff8c23fca3f4ULL,+	0xff0b13c3b9791a26ULL,0x960022dacdd58b16ULL,0xdbd55c9257aad2deULL,0x3baaaaa3f30fe619ULL,0x9a4b23460d881efdULL,0x506416c046325e2aULL,0x91381e76035c18d4ULL,0xb3bb68bef27817b0ULL,+	0x15bfb8bf5116f937ULL,0x7c64a586c1268943ULL,0x71e25cc38419a2c8ULL,0x9fd6b0c48335f463ULL,0x4bf0ba3ce8ee0e0eULL,0x6f6fba60298c21faULL,0x57d57b39ae66bee0ULL,0x292d513022672544ULL,+	0xf451105dbab093b3ULL,0x012f59b902839986ULL,0x8a9158023474a89cULL,0x048c919c2de03e97ULL,0xc476a2b591071cd5ULL,0x791ed89a034970a5ULL,0x89bd9042e1b7994bULL,0x8eaf5179a1057ffdULL,+	0x6066e2a2d551ee10ULL,0x87a8f1d8727e09a6ULL,0x00d08bab2c01148dULL,0x6da8e4f1424f33feULL,0x466d17f0cf9a4e71ULL,0xff5020103bf5cb19ULL,0xdccf97d8d062ecc0ULL,0x80c0d9af81d80ac4ULL,+	0x98857ceb1bf4581cULL,0xe635e186aca7b166ULL,0x278ddd22659722acULL,0xa0903c4c1db68007ULL,0x366e458948f21402ULL,0x31b49c14b96abda2ULL,0x329c4b09e0403190ULL,0x97197ca3d29f43feULL,+	0x03e2de1cf3480d4aULL,0xf0d8edc7bc8acf1aULL,0xf23e330368295a9cULL,0xfadd5f68c546a97dULL,0x895597ad96f8acb1ULL,0xbddd49d5671bdae2ULL,0x16fcd52821dd43f4ULL,0xa5a454126619141aULL,+	0xfa32c79f439f29cfULL,0x7bf321c04dd82a3bULL,0xff127f54eaa2c1b1ULL,0xa8365f2df35e9618ULL,0x852d29024d0ef8ddULL,0x395ce2c159228c4aULL,0xb69f44e8215afed3ULL,0x16c1f898b27458e3ULL,+	0x8ce9b6bfc360e25aULL,0xe6425195075a1a78ULL,0x9dc756a8481732f4ULL,0x83c0440f5432b57aULL,0xc670b3f1d720281fULL,0x2205910ed135e051ULL,0xded14b0edb052be7ULL,0x697b3d27c568ea39ULL,+	0xdef29005dc453233ULL,0xc208c47a2fac4bcfULL,0x6057a3feac3d55acULL,0x1723725d902c1207ULL,0x9c31c62733eb0fecULL,0x4913ccdbbb3c63bbULL,0x113e542b07305838ULL,0x9d48e72a310c2d97ULL,+	0x54424ec4d9bbeb3dULL,0x34ceafe3d7b2921cULL,0x5e68022e296d37c5ULL,0xa28e0a2b359f16b0ULL,0xfdd82dd9bc3f9d73ULL,0x6939a8f9baf3e1ffULL,0x55cff45c31736dc4ULL,0x910f56427892e8e7ULL,+	0xde09b349dfe39260ULL,0x984612f773549093ULL,0x7cede28167853b02ULL,0x7d809bb429d17703ULL,0x450b6bfb2756c4f0ULL,0xc59ff9ba93f02b80ULL,0x3e69545720ad9561ULL,0x0c7cf0be3331e2c7ULL,+	0x2e599b9afb3ff9edULL,0x28c2e0ab17f6515cULL,0x1cbee4fd474da449ULL,0x071279a44f364452ULL,0x97abff6601fbe855ULL,0x3ee394e85fda51c4ULL,0x190385f667597c0bULL,0x6e9fccc6a27ee34bULL,+	0xced2d419362fb228ULL,0x894637c206aa0be4ULL,0x7a4fc55eb294b197ULL,0xd9cbac1cfd4ca1e8ULL,0x068b74800ccdb6ceULL,0xca4c556580dfaa49ULL,0xe835382176033b78ULL,0x35db7525c5ce98a7ULL,+	0x8cf5927274fd1997ULL,0x987d2031ffadeb58ULL,0x5647c6c3d4db260cULL,0xf08bb13b985543beULL,0x566eeb1056fad695ULL,0x39e17dde68334cc7ULL,0x8a97b3bb7ddd1db2ULL,0xf91199d8c1c5a300ULL,+	0x402fa437bdc47fe4ULL,0x35bd25234f3751f0ULL,0x8c2281b7cf57d485ULL,0x50083ea58f607cb6ULL,0x6f41e480df6d730dULL,0x1164e47e91bb06a2ULL,0xd9040c22391fb48aULL,0xe12df251da23dda7ULL,+	0x0b89de9314092ebbULL,0xf17256bd428e240cULL,0xcf89a7f393d2f064ULL,0x4f57841ee1ed3b14ULL,0x4ee14405e708d855ULL,0x856aae7203f1c3d0ULL,0xc8e5424fbdd7eed5ULL,0x3333e4ef73ab4270ULL,+	0xf9cf2b5148f835b1ULL,0x4a70faf32adaaba8ULL,0xa0d2e24fed7beadeULL,0x2b1e4e6715f04032ULL,0xeadbc0b91fb3ceeeULL,0xb817b0863a54b0b4ULL,0x14a787257fd4a188ULL,0xff13304623482e60ULL,+	0x312012ec4efe42dcULL,0x7251b2c2c664b2bcULL,0x996c2e6b798f9cb9ULL,0x3543376a3b8f3465ULL,0x337ae8416fbfb6a9ULL,0xe0893a840cb91a03ULL,0x53744e9d7b02d855ULL,0x7e673186206d473fULL,+	0x9509996f3ef143dbULL,0x9cf1e82e399378bcULL,0xb59cd09b0b8ccc89ULL,0x64b4a2fc52cda6beULL,0xd3bda8b31bcd55dfULL,0xfec4e87d2507cb37ULL,0x3d48a85eb610e49bULL,0x02fbe1bd1f9cc445ULL,+	0x3bc77adedda492f8ULL,0xc11a3aea78297205ULL,0x5e89a3e734931b4cULL,0x17512e2e9f5694bbULL,0x5dc349f3177bf8b6ULL,0x232ea4ba08c7ff3eULL,0x9c4f9d16f511145dULL,0xccf109a333b379c3ULL,+	0x60e2857080eb24a9ULL,0x7bedfb4d488e0cfdULL,0x721ebbd7c259cdb8ULL,0x0b0da855bc6390a9ULL,0x2b4d04dbde314c70ULL,0xcdbf1fbc6c32e846ULL,0x33833eabb162fc9eULL,0x9939b48bb0dd3ab7ULL,+	0x96892c1f711b0eb9ULL,0xb905f2c8780ab954ULL,0xace26309a20792dbULL,0xec8ac9b30684e126ULL,0x486ad8b6b40a2447ULL,0x60121fc19fe3fb24ULL,0x5626fccf1a8e3b3fULL,0x4e5686226ad1f394ULL,+	0x5cfbbb22236f4a98ULL,0x0b0c59e9066800bbULL,0x4ac69a8f5a9a7774ULL,0x2b33f804d6bec948ULL,0xb372929532e6c466ULL,0x68956d0f4e599c73ULL,0xa47a249f155c31ccULL,0x24d80f0de1ce284eULL,+	0x5a4b46c64a8a3d62ULL,0x8469c4d0247743d2ULL,0x2bb3a13d88f7e433ULL,0x62b23a1001be5849ULL,0xe83596b4a63d1a4cULL,0x454e7fea7d183f3eULL,0x643fce6117afb01cULL,0x4e65e5e61c4c3638ULL,+	0xb7e830e3dc09508bULL,0xfaf6d2cf74317655ULL,0x72606cebdf690355ULL,0x48bb92b3d0c3ded6ULL,0x65b754845c7cf892ULL,0xf6cd7ac9d5d5f01fULL,0xc2c30a5996401d69ULL,0x91268650ed921878ULL,+	0xe5db77176add8545ULL,0x1b71cb6672c49b66ULL,0xd856073968421d77ULL,0x03840fe883e3afeaULL,0xb391dad51ec69977ULL,0xae243fb9307f6726ULL,0xc88ac87be8ca160cULL,0x5174cced4ce355f4ULL,+	0x752067f8ff81578eULL,0x786221509045447dULL,0xc0c22fcf0505aa6fULL,0x1030f0a66bed1c77ULL,0x31f29f151f0bd739ULL,0x2d7989c7e6debe85ULL,0x5c070e728e677e98ULL,0x0a817bd306e81fd5ULL,+	0xc1e17eb6cbc613e5ULL,0x33131d55497ea61cULL,0x2f69d39eaf7eded5ULL,0x73c2f434de6af11bULL,0x4ca52493a4a375faULL,0x5f06787cb833c5c2ULL,0x814e091f3e6e71cfULL,0x76451f578b746666ULL,+	0x01c7e082e1539388ULL,0xfd286f30759a9c6bULL,0x94581041176bacfbULL,0xe580f07c3bc3de53ULL,0xdcdb6f75884d772fULL,0xd75c3bb840bb9d4fULL,0xdc6c2e4edb083011ULL,0xe18789a2cd9c298dULL,+	0x5ee6ab8495fe1347ULL,0xab0f6c396f24503cULL,0x807e3ffb4486dd6bULL,0xf00b6c748002fef5ULL,0x48bff9a6a7862999ULL,0x85e5a06cbed89e26ULL,0x86d311af3d8419ebULL,0x24f3ad7834733f16ULL,+	0x5e20551311820d44ULL,0x0c651bcc86c4473aULL,0x1d230c2b9bd80eefULL,0x042c4a1207515be5ULL,0x42517ca0bfe9e284ULL,0xe8f605782369827aULL,0x184f04f01638699dULL,0x174618edf2bc6d05ULL,+	0x5e3e03fc6c68d687ULL,0x3e732c3d1ff052c7ULL,0xf2d0efa66ed16e7aULL,0x63d92b26b65bb746ULL,0xffcd82badd44867cULL,0xa71b4a9ef8c081b8ULL,0x6c1676a7736c8785ULL,0xbe2c06169d8932d0ULL,+	0x7bfa1a4b9eca1c9fULL,0x960bc1dc8ce5e535ULL,0xe267d9f317eec30aULL,0x06fb89ef6c257d38ULL,0x2328999ad364a26dULL,0x69b794cb26eaab58ULL,0xad28ab1fb85ba596ULL,0x05dcbff356d0aa94ULL,+	0x53376d282bcffbc4ULL,0x708817a706eadb7aULL,0x6ff50e05cd35ae69ULL,0x63b5fb7574bc7fdeULL,0x71c9e953e7fe08c4ULL,0xb4d8bfd4f583ca18ULL,0xde8d788245e81c5cULL,0xa5f5e93ce0474138ULL,+	0x426c160f293c9f31ULL,0x8eb56333e864d7a7ULL,0xbe1164023ebbba30ULL,0x64c2bae32fd5a302ULL,0x800601e1265aff7bULL,0x52d8a88066fd4b14ULL,0x5aba20e746075a9cULL,0xdaa32bf87e1234c6ULL,+	0x80f9bdef694db7e0ULL,0xedca8787b9fcddc6ULL,0x51981c3403b8dce1ULL,0x4274dcf170e10ba1ULL,0xf72743b86def6d1aULL,0xd25b1670ebdb1866ULL,0xc4491e8c050c6f58ULL,0x2be2b2ab87fbd7f5ULL,+	0x3e0e5c9dd111f8ecULL,0xbcc33f8db7c4e760ULL,0x702f9a91bd392a51ULL,0x7da4a795c132e92dULL,0x1a0b0ae30bb1151bULL,0x54febac802e32251ULL,0xea3a5082694e9e78ULL,0xe58ffec1e4fe40b8ULL,+	0xfbb8349d29c4120bULL,0x9f94391fc0d0d915ULL,0xc4074fa75410ba51ULL,0xa66adbf6150a5911ULL,0xc164543c34bfca38ULL,0xe0f27560b9e1ccfcULL,0x99da0f53e820219cULL,0xe8234498c6b4997aULL,+	0x7b23c513516e19e4ULL,0x56e2e847c5c4d593ULL,0x9f727d735ce71ef6ULL,0x5b6304a6f79a44c5ULL,0x6638a7363ab7e433ULL,0x1adea470fe742f83ULL,0xe054b8545b7fc19fULL,0xf935381aba1d0698ULL,+	0xb5504f9d918e4936ULL,0x65035ef6b2513982ULL,0x0553a0c26f4d9cb9ULL,0x6cb10d56bea85509ULL,0x48d957b7a242da11ULL,0x16a4d3dd672b7268ULL,0x3d7e637c8502a96bULL,0x27c7032b730d463bULL,+	0x55366b7d5846426fULL,0xe7d09e89247d441dULL,0x510b404d736fbf48ULL,0x7fa003d0e784bd7dULL,0x25f7614f17fd9596ULL,0x49e0e0a135cb98dbULL,0x2c65957b2e83a76aULL,0x5d40da8dcddbe0f8ULL,+	0x37f68bb4a595939dULL,0x0355647928740217ULL,0x8e740e7c84ad7612ULL,0xd89bc8439044695fULL,0xf7f3da5d85a9184dULL,0x562563bb9fc0b074ULL,0x06d2e6aaf88a888eULL,0x612d8643161fbe7cULL,+	0x9fb3bba354530bb2ULL,0xbde3ef77cb0869eaULL,0x89bc90460b431163ULL,0x4d03d7d2e4819a35ULL,0x33ae4f9e43b6a782ULL,0x216db3079c88a686ULL,0x91dd88e000ffedd9ULL,0xb280da9f12bd4840ULL,+	0x458f86913e538cd7ULL,0xa7001f6c8e08ad53ULL,0x52b8c6e6bf5d15ffULL,0x548234a4011215ddULL,0xff5a9d2d3d5b4045ULL,0xb0ffeeb64a904190ULL,0x55a3aca448607f8bULL,0x8cbd665c30a0672aULL,+	0xa37f3573f37f5937ULL,0xeb0f6c7dd1e4fca5ULL,0x2965a554ac8ab0fcULL,0x17fbf56c274676acULL,0x2e2f6bd9acf7d720ULL,0x41fc8f8810224766ULL,0x517a14b385d53befULL,0xdae327a57d76a7d1ULL,+	0x515d5c891f5f82dcULL,0x9a7f67d76361079eULL,0xa8da81e311a35330ULL,0xe44990c44b18be1bULL,0xc7d5ed95af103e59ULL,0xece8aba78dac9261ULL,0xbe82b0999394b8d3ULL,0x6830f09a16adfe83ULL,+	0x43c41ac194d7d9b1ULL,0x5bafdd82c82e7f17ULL,0xdf0614c15fda0fcaULL,0x74b043a7a8ae37adULL,0x3ba6afa19e71734cULL,0x15d5437e9c450f2eULL,0x4a5883fe67e242b1ULL,0x5143bdc22c1953c2ULL,+	0xa2a9ce7c6b53f5f9ULL,0x642465951b176d99ULL,0xb1298d36b95c081bULL,0x53505bb81d9a9ee6ULL,0x3f6f9e61f2ba70b0ULL,0xd07e16c98afad453ULL,0x9f1694bbe7eb4a6aULL,0xdfebced93cb0bc8eULL,+	0xc676d7f2b1f3390bULL,0x9f7a1b8ca5b61272ULL,0x4ebebfc9c2e127a9ULL,0x4602500c5dd997bfULL,0x7f09771c4711230fULL,0x058eb37c020f09c1ULL,0xab693d4bfee5e38bULL,0x9289eb1f4653cbc0ULL,+	0xa44d2b391770f5a7ULL,0xe4d4d7910e44eb82ULL,0x42e69d1e3f69712aULL,0xbf11c4d6ac6a820eULL,0xb5e7f3e542c4224cULL,0xd6b4e81c449d941cULL,0x5d72bd165450e878ULL,0x6a61e28aee25ac54ULL,+	0x54da9dc7ab952578ULL,0xb5423df226e84d0bULL,0xa8b64eeb9b872042ULL,0xac2057825990f6dfULL,0x4ff696eb21f4c77aULL,0x1a79c3e4aab273afULL,0x29bc922e9436b3f1ULL,0xff807ef8d6d9a27aULL,+	0xc7f3a8f833f6746cULL,0x21e46f65fea990caULL,0x915fd5c5caddb0a9ULL,0xbd41f01678614555ULL,0x346f4434426ffb58ULL,0x8055943614dbc204ULL,0xf3dd20fe5a969b7fULL,0x9d59e956e899a39aULL,+	0x3c2f0ba9b733aa5fULL,0xdece47cbf05af235ULL,0xf8e3f715a2ac82a5ULL,0xc97ba6412203f18aULL,0xc3af550409c11060ULL,0x56ea2c0546af512dULL,0xfac28daff3f28146ULL,0x87fab43a959ef494ULL,+	0x09891641d4c5105fULL,0x1ae80f8e6d7fbd65ULL,0x9d67225fbee6bdb0ULL,0x3b433b597fc4d860ULL,0x44e66db693e85638ULL,0xf7b59252e3e9862fULL,0xdb785157665c32ecULL,0x702fefd7ae362f50ULL,+	0xfe756a5c97290293ULL,0xbf04a19cd388acbfULL,0xfbbbb9cf5e916bdaULL,0xf489527391f93becULL,0xdee07ec32a5923d7ULL,0xc7bc949bfde0c370ULL,0xbd5121750419d8fcULL,0x54f5d4763fdcc93fULL,+	0x3754475d0fefb0c3ULL,0xd48fb56b46d7c35dULL,0xa070b633363798a4ULL,0xae89f3d28fdb98e6ULL,0x970b89c86363d14cULL,0x8981752167abd27dULL,0x9bf7d47444d5a021ULL,0xb3083bafcac72aeeULL,+	0x0acda2ffcc5e62e9ULL,0x3b8b7d755edb02a4ULL,0xa700741c66120c76ULL,0xf77e847f7d974064ULL,0x0d310678a5e3d464ULL,0xde68b1f346bf35a3ULL,0xcae83028d32f9043ULL,0x724e4717517cb0cbULL,+	0x389741debe949a44ULL,0x638e9388546a4fa5ULL,0x3fe6419ca0047bdcULL,0x7047f648aaea57caULL,0x54e48a9041fbab17ULL,0xda8e0b28576bdba2ULL,0xe807eebcc72afddcULL,0x07d3336df42577bfULL,+	0xfae8563b4f3011dbULL,0xda33776536610c03ULL,0xad4f6f2a6381af0bULL,0xc277984cd95378ecULL,0x5ab0a10c5751d2b2ULL,0x70a18bb97a4bf3f4ULL,0x38d07ad4eae3caf3ULL,0xe8ef552fec430361ULL,+	0x62a8c244bfe20925ULL,0x91c19ac38fdce867ULL,0x5a96a5d5dd387063ULL,0x61d587d421d324f6ULL,0xe87673a2a37173eaULL,0x2384800853778b65ULL,0x10f8441e05bab43eULL,0xfa11fe124621efbeULL,+	0xedd0389a8391d54bULL,0xdac74c08c8afe546ULL,0x8525a4ad5be60bbdULL,0x2419ac9690ad7b87ULL,0x078a3a0277eee51bULL,0xe86ecf367239768dULL,0xcb0a259d8fd48035ULL,0x94db43cb9d29ca5bULL,+	0x047b772e81685d7bULL,0x23f27d81bf34a976ULL,0xc27608e2915f48efULL,0x3b0b43faa521d5c3ULL,0x7613fb2663ca7284ULL,0x7f5729b41d4db837ULL,0x87b14898583b526bULL,0x00b732a6bbadd3d1ULL,+	0x4fffa25b90f8550aULL,0x254db3d31c8dae9dULL,0x58cef963c1fbb232ULL,0xad1cb481a4bfdf0fULL,0x84d26ea1958773c2ULL,0x58622664010114f1ULL,0xeadb3a87f051e67eULL,0xf3185722b69e45c6ULL,+	0x8e02f4262048e396ULL,0x436b50b6383d9de4ULL,0xf78d3481471e85adULL,0x8b01ea6ad005c8d6ULL,0xd3c7afee97015c07ULL,0x46cdf1a94e3ba2aeULL,0x7a42e50183d3a1d2ULL,0xd54b5268b541dff4ULL,+	0xe120c4f948a48e22ULL,0xf24977eed0335a96ULL,0x9af3442336151c7bULL,0xe2815a7fd36648d8ULL,0xb4d2deba66e0a6b2ULL,0x783a84cf817515e8ULL,0x78424c0bff3ff24aULL,0x12dd1bb638e1c5d9ULL,+	0x3f24cf304e23e9bcULL,0x4387f816126e3624ULL,0x26a46a033b0b6d61ULL,0xaf1bc8458b2d777cULL,0x25c401ba527de79cULL,0x0e1346d44261bbb6ULL,0x4b96c44b287b4bc7ULL,0x658493c75254562fULL,+	0xfcb0e9d8fdd41d98ULL,0x8be980c1bcc7c7fbULL,0xa72e86506f8fa4d9ULL,0x356b14ad748cb88dULL,0xfd9fccefea6178f2ULL,0xc84a620d78bb0e35ULL,0xbc75367c62f391a9ULL,0x6a83a5c623100c05ULL,+	0x23f949feb8a24a20ULL,0x17ebfed1f52ca53fULL,0x9b691bbebcfb4853ULL,0x5617ff6b6278a05dULL,0x241b34c5e3c99ebdULL,0xfc64242e1784156aULL,0x4206482f695d67dfULL,0xb967ce0eee27c011ULL,+	0xb4480f0441c23fa3ULL,0xb4712eb0c1989a2eULL,0x3ccbba0f93a29ca7ULL,0x6e205c14d619428cULL,0x90db7957b3641686ULL,0x0432691d45ac8b4eULL,0x07a759acf64e0350ULL,0x0514d89c9c972517ULL,+	0xe3b22c6bc4fe3c39ULL,0xba4a81536c7bebdfULL,0xf23ab6b725693459ULL,0x53bc377014922b11ULL,0x4645c8ab5afc60dbULL,0xaa02235520b9f2a3ULL,0x52a2954cce0fc507ULL,0x8c2731bb7ce1c2e7ULL,+	0x5066efb6d9790ed6ULL,0xa77a0cbca6aa793bULL,0x1a915f3c223e042eULL,0x1c5def0469c5874bULL,0x0e83007873b6c1daULL,0x55cf85d2fcd8557aULL,0x0f7c7c760460f3b1ULL,0x87052acb46e58063ULL,+	0x6a7091c2e48fb889ULL,0x26882c137b8a9d06ULL,0xa24986631b82a0e2ULL,0x844ed7363518152dULL,0x282f476fd86e27c7ULL,0xa04edaca04afefdcULL,0x8b256ebc6119e34dULL,0x56a413e90787d78bULL,+	0x16eab6a20d645fd6ULL,0x632cbd8df61d3148ULL,0xcc1bf7cf62079ae9ULL,0x257ee5c7f33eccbbULL,0xbf6b34a81680ac73ULL,0xaa084e8872c77aa0ULL,0x7b5a864e05a0a1d1ULL,0x0641f6db359a1b16ULL,+	0xf01d095dc8385050ULL,0x0d54a5d5df4b441cULL,0x2a37ccb40927706aULL,0xdf008f5445d7eb7eULL,0x74eb34f35bf716c7ULL,0x57a65b58641bd6caULL,0xef345e4835e6fa02ULL,0x191f913b88342a09ULL,+	0x1554d46da670ff1dULL,0x24833d88cb97a1ccULL,0x8fa6ab3cded97493ULL,0x215e037189926498ULL,0x549bd592e56d74ffULL,0x58a8caf543b5e1ecULL,0x3c6087a323e93cb9ULL,0x8b0549875648b83cULL,+	0x82ee061d5a74be50ULL,0xe41781c4dea16ff5ULL,0xe0b0c81e99bfc8a2ULL,0x624f4d690b547e2dULL,0x3a83545dbdcc9ae4ULL,0x2573dbb6409b1e8eULL,0x482960c4a6c93539ULL,0xf01059ad5ae18798ULL,+	0xc431a238013ff83bULL,0x7c0018b2fad69d08ULL,0x99aeb52a4c9589eaULL,0x121f41ab9b1cf19fULL,0x0cfbbcbaef0f5958ULL,0x8deb3aeb7be8fbdcULL,0x12b954081f15aa31ULL,0x5acc09b34c0c06fdULL,+	0x775cbfa86d518ffbULL,0xdecee1f6930f124bULL,0x9a402804f5e81d0fULL,0x0e8225c52a0eeb2fULL,0x884a5d39fee9e867ULL,0x9540428ffb505454ULL,0xb2bf2e20107a70d1ULL,0xd9917c3ba010b2aaULL,+	0xa98f42fa3d843d53ULL,0x33777cc613ef927aULL,0xc440cdbecb84ca74ULL,0x8c22f9631dc7c5ddULL,0x4bc82b70c8d94708ULL,0x7e0b43fcc814364fULL,0x286d4e2486f59b7eULL,0x1abc895e4d6bf4c4ULL,+	0x38151e274d559d96ULL,0x4f18c0d3b8db6c01ULL,0x49a3aa836f9921afULL,0xdbeab27b8c046029ULL,0x242b9eaa7040bf3bULL,0x39c479e51614b091ULL,0x338ede2b0e4baf5dULL,0x5bb192b7f0a53945ULL,+	0x896d572337e440d7ULL,0x685c5fd9ade23f68ULL,0xb5b1a26dc2c64918ULL,0xb9390e30dad6580cULL,0x87911c4e7dee5b9bULL,0xb90c5053deb04f6eULL,0x37b942a18f065aa6ULL,0x34acdf2a1ca0928dULL,+	0x733b64d39de40ca3ULL,0x1d4b6d6fd2f3857eULL,0xbe2be8e9b2ed92f7ULL,0x64ca7047b77da248ULL,0xc65dae9b8da99315ULL,0x9c1451750fc698a4ULL,0x8a296b94ff958c27ULL,0x38684e0843950097ULL,+	0x7872e34b3390ff23ULL,0x968ce4abde7d18efULL,0x9b4a745e627fe7b1ULL,0x9607b0a0caff3e2aULL,0x1b05818eeb40e3a5ULL,0x6ac62204c0fa8d7aULL,0xb5b9058571ed4809ULL,0xb2432ef0f7cb65f2ULL,+	0x715c9f973112795fULL,0xe8244437984e6ee1ULL,0x55cb4858ecb66bcdULL,0x7c136735abaffbeeULL,0x546615955dbec38eULL,0x51c0782c388ad153ULL,0x9ba4c53ac6e0952fULL,0x27e6782a1b21dfa8ULL,+	0xfeb09740e2c2bf15ULL,0x627a2205a9e99704ULL,0xec8d73d0c2fbc565ULL,0x223eed8fc20c8de8ULL,0x1ee32583a8363b49ULL,0x1a0b6cb9c9c2b0a6ULL,0x49f7c3d290dbc85cULL,0xa8dfbb971ef4c1acULL,+	0xc16c236e846e364fULL,0x7f33527cdea50ca0ULL,0xc48107750926b86dULL,0x6c2a36090598e70cULL,0xa6755e52f024e924ULL,0xe0fa07a49db4afcaULL,0x15c3ce7d66831790ULL,0x5b4ef350a6cbb0d6ULL,+	0x42806b2da6dc1d29ULL,0xd3030009f871e144ULL,0xa1feb333aaf49276ULL,0xb5583b9ec70bc04bULL,0x1db0be7895695f20ULL,0xfc84181189d012b5ULL,0x6409f27205f61643ULL,0x40d34174d5883128ULL,+	0x05214c050f15dde9ULL,0xa47a76a80d5f2b82ULL,0xbb254d3062e82b62ULL,0x11a05fe03ec955eeULL,0x7eaff46e9d529b36ULL,0x55ab13018f9e3df6ULL,0xc463e37199317698ULL,0xfd251438ccda47adULL,+	0x8c3c669c72ba075bULL,0x89f78b55ba469015ULL,0x5706aade3e9f8ba8ULL,0x6d8bd565b32d7ed7ULL,0x25f4e63b805f08d6ULL,0x7f48200dc3bcc1b5ULL,0x4e801968b025d847ULL,0x74afac0487cbe0a8ULL,+	0xe2a37598a9d82abfULL,0x5f188ccbe6c170f5ULL,0x816822005066b087ULL,0xda22c212c7155adaULL,0x151e5d3afbddb479ULL,0x4b606b846d715b99ULL,0x4a73b54bf997cb2eULL,0x9a1bfe433ecd8b66ULL,+	0x79732522cccc18adULL,0xaadf3f8df1a6e027ULL,0xf7382c9317c2354dULL,0x5ce1680cd818b689ULL,0x359ebbfcd9ecbee9ULL,0x4330689c1cae62acULL,0xb55ce5b4c51ac38aULL,0x7921dfeafe238ee8ULL,+	0xe13122f3dbfb894eULL,0xbe9b79f6ce274b18ULL,0x85a49de5ca58aadfULL,0x2495775811487351ULL,0x111def61bb939099ULL,0x1d6a974a26d13694ULL,0x4474b4ced3fc253bULL,0x3a1485e64c5db15eULL,+	0x65994ddb0f5f27caULL,0xe85461fba80d59ffULL,0xff05481a66601023ULL,0xc665427afc9ebbfbULL,0xb0571a697587fd52ULL,0x935289f88d49efceULL,0x61becc60ea420688ULL,0xb22639d913a786afULL,+	0x5afddab61430c9abULL,0x0bdd41d32238e997ULL,0xf0947430418042aeULL,0x71f9addacdddc4cbULL,0x7090c016c52dd907ULL,0xd9bdf44d29e2047fULL,0xe6f1fe801b1011a6ULL,0xb63accbcd9acdc78ULL,+	0x264c76680448087cULL,0xac30903f71432daeULL,0x3851b26600f9bf47ULL,0x400ed3116cdd6d03ULL,0x045e79fef8fd2424ULL,0xfdfd974afa6da98bULL,0x45c9f6410c1e673aULL,0x76f2e7335b2c5168ULL,+	0x7817acab4baef62eULL,0x9f5a2202a85b91e8ULL,0x9666ebe66ce57610ULL,0x32ad31f3f73bfe03ULL,0x628330a425bcf4d6ULL,0xea950593515056e6ULL,0x59811c89e1332156ULL,0xc89cf1fe8c11b2d7ULL,+	0x889e5acbc46d7ce1ULL,0x9a515bb78b085877ULL,0xfac1a03d0b7a5050ULL,0x7d3e738af2926035ULL,0x861cc2ce2a6cb0ebULL,0x6f2e29558f7adc79ULL,0x61c4d45133016376ULL,0xd9fd2c805ad59090ULL,+	0x0ad7337ac0b7eff3ULL,0x8552225ec5e48b3cULL,0xe6f78b0c73f13a5fULL,0x5e70062e82349cbeULL,0x6b8d5048e7073969ULL,0x392d2a29c33cb3d2ULL,0xee4f727c4ecaa20fULL,0xa068c99e2ccde707ULL,+	0x1888d65861a023efULL,0x1d72aab4b9e5246eULL,0xa9a26348e5563ec0ULL,0xa0971963c3439a43ULL,0x567dd54badb9b5b7ULL,0x73fac1a1c45a524bULL,0x8fe97ef7fe38e608ULL,0x608748d23f384f48ULL,+	0xebde86ec1ed66f18ULL,0x225d906bd61fce43ULL,0x5cab07d6e8bed74dULL,0x16e4617f27855ab7ULL,0x6568aaddb2fbc3ddULL,0xedb5484f8aeddf5bULL,0x878f20e86dcf2fadULL,0x3516497c615f5699ULL,+	0xef0a3fecfa181e69ULL,0x9ea02f8130d69a98ULL,0xb2e9cf8e66eab95dULL,0x520f2beb24720021ULL,0x621c540a1df84361ULL,0x1203772171fa6d5dULL,0x6e3c7b510ff5f6ffULL,0x817a069babb2bef3ULL,+	0x83572fb6b294cda6ULL,0x6ce9bf75b9039f34ULL,0x20e012f0095cbb21ULL,0xa0aecc1bd063f0daULL,0x57c21c3af02909e5ULL,0xc7d59ecf48ce9cdcULL,0x2732b8448ae336f8ULL,0x056e37233f4f85f4ULL,+	0x8a10b53189e800caULL,0x50fe0c17145208fdULL,0x9e43c0d3b714ba37ULL,0x427d200e34189accULL,0x05dee24fe616e2c0ULL,0x9c25f4c8ee1854c1ULL,0x4d3222a58f342a73ULL,0x0807804fa027c952ULL,+	0xc222653a4f0d56f3ULL,0x961e4047ca28b805ULL,0x2c03f8b04a73434bULL,0x4c966787ab712a19ULL,0xcc196c42864fee42ULL,0xc1be93da5b0ece5cULL,0xa87d9f22c131c159ULL,0x2bb6d593dce45655ULL,+	0x22c49ec9b809b7ceULL,0x8a41486be2c72c2cULL,0x813b9420fea0bf36ULL,0xb3d36ee9a66dac69ULL,0x6fddc08a328cc987ULL,0x0a3bcd2c3a326461ULL,0x7103c49dd810dbbaULL,0xf9d81a284b78a4c4ULL,+	0x3de865ade4d55941ULL,0xdedafa5e30384087ULL,0x6f414abb4ef18b9bULL,0x9ee9ea42faee5268ULL,0x260faa1637a55a4aULL,0xeb19a514015f93b9ULL,0x51d7ebd29e9c3598ULL,0x523fc56d1932178eULL,+	0x501d070cb98fe684ULL,0xd60fbe9a124a1458ULL,0xa45761c892bc6b3fULL,0xf5384858fe6f27cbULL,0x4b0271f7b59e763bULL,0x3d4606a95b5a8e5eULL,0x1eda5d9b05a48292ULL,0xda7731d0e6fec446ULL,+	0xa3e3369390d45871ULL,0xe976404006166d8dULL,0xb5c3368289a90403ULL,0x4bd1798372f1d637ULL,0xa616679ed5d2c53aULL,0x5ec4bcd8fdcf3b87ULL,0xae6d7613b66a694eULL,0x7460fc76e3fc27e5ULL,+	0x70469b8295caabeeULL,0xde024ca5889501e3ULL,0x6bdadc06076ed265ULL,0x0cb1236b5a0ef8b2ULL,0x4065ddbf0972ebf9ULL,0xf1dd387522aca432ULL,0xa88b97cf744aff76ULL,0xd1359afdfe8e3d24ULL,+	0x52a3ba2b91502cf3ULL,0x2c3832a8084db75dULL,0x04a12dddde30b1c9ULL,0x7802eabce31fd60cULL,0x33707327a37fddabULL,0x65d6f2abfaafa973ULL,0x3525c5b811e6f91aULL,0x76aeb0c95f46530bULL,+	0xe8815ff62f93a675ULL,0xa6ec968405f48679ULL,0x6dcbb556358ae884ULL,0x0af61472e19e3873ULL,0x72334372a5f696beULL,0xc65e57ea6f22fb70ULL,0x268da30c946cea90ULL,0x136a8a8765681b2aULL,+	0xad5e81dc0f9f44d4ULL,0xf09a69602c46585aULL,0xd1649164c447d1b1ULL,0x3b4b36c8879dc8b1ULL,0x20d4177b3b6b234cULL,0x096a25051730d9d0ULL,0x0611b9b8ef80531dULL,0xba904b3b64bb495dULL,+	0x1192d9d493a3147aULL,0x9f30a5dc9a565545ULL,0x90b1f9cb6ef07212ULL,0x299585460d87fc13ULL,0xd3323effc17db9baULL,0xcb18548ccb1644a8ULL,0x18a306d44f49ffbcULL,0x28d658f14c2e8684ULL,+	0x44ba60cda99f8c71ULL,0x67b7abdb4bf742ffULL,0x66310f9c914b3f99ULL,0xae430a32f412c161ULL,0x1e6776d388ace52fULL,0x4bc0fa2452d7067dULL,0x03c286aa8f07cd1bULL,0x4cb8f38ca985b2c1ULL,+	0x83ccbe808c3bff36ULL,0x005a0bd25263e575ULL,0x460d7dda259bdcd1ULL,0x4a1c5642fa5cab6bULL,0x2b7bdbb99fe4fc88ULL,0x09418e28cc97bbb5ULL,0xd8274fb4a12321aeULL,0xb137007d5c87b64eULL,+	0x80531fe1c63c4962ULL,0x50541e89981fdb25ULL,0xdc1291a1fd4c2b6bULL,0xc0693a17a6df4fcaULL,0xb2c4604e0117f203ULL,0x245f19630a99b8d0ULL,0xaedc20aac6212c44ULL,0xb1ed4e56520f52a8ULL,+	0xb5560fb6700a1acdULL,0xe823fd73fd999681ULL,0xda915d1f6cb4e1baULL,0x0d0301186ebe00a3ULL,0x744fb0c989fca8cdULL,0x970d01dbf9da0e0bULL,0x0ad8c5647931d76fULL,0xb15737bff659b96aULL,+	0xa12b384ece53c2d0ULL,0x779d897d5e4606daULL,0xa53e47b073ec12b0ULL,0x462dbbba5756f1adULL,0x69fe09f2cafe37b6ULL,0x273d1ebfecce2e17ULL,0x8ac1d5383cf607fdULL,0x8035f7ff12e10c25ULL,+	0xca442d5a2093c22aULL,0xebd0bd31d5703aedULL,0x308f2afd653287b6ULL,0x9bb88bac0d1bc8baULL,0xfbaf853875c1e3b2ULL,0xbd2ac950ca11447cULL,0x286d816cea5c4c8dULL,0xdc3aa80028dc3208ULL,+	0x854d34c77e6c5520ULL,0xc27df9efdcb9ea58ULL,0x405f2369d686666dULL,0x29d1febf0417aa85ULL,0x9846819e93470afeULL,0x3e6a9669e2a27f9eULL,0x24d008a2e31e6504ULL,0xdba7cecf9cb7680aULL,+	0x26a43e41d07fa53dULL,0x3154a78a74e35bc5ULL,0x7b768924e0da2f8cULL,0xba964a2b23613f9aULL,0x5a548d35ba1d16c4ULL,0x2e1bfed1fb54d057ULL,0xff992136bc640205ULL,0xf39cb9148156df29ULL,+	0xc913e64699b444adULL,0xddfce99dc40504c5ULL,0x58482a99d42e53dbULL,0x9aaf2c25d1aff537ULL,0xee90f7962664cf67ULL,0x74ab5c99f1393e2bULL,0xfad0faeae6225bb0ULL,0xc355648c2d63dd6cULL,+	0xe4e31d271d91cf9dULL,0xcb35d4fdb377b20aULL,0x74de1e45055e1327ULL,0x3298e31b28703e75ULL,0x55087237de013339ULL,0x32cbf30123d101c6ULL,0xc70dba22e8aab0dcULL,0x4a52623d3d155bb9ULL,+	0xecaff541338d6e43ULL,0x56f7dd734541d5ccULL,0xb5d426de96bc88caULL,0x48d94f6b9ed3a2c3ULL,0x6354a3bb2ef8279cULL,0xd575465b0b1867f2ULL,0xef99b0ff95225151ULL,0xf3e19d88f94500d8ULL,+	0xdbf435acc85dcf57ULL,0x61745658c88f5415ULL,0x26367e9a17c55807ULL,0x22d077a5ca90c56fULL,0xfbf72258a2e04e76ULL,0xba965d3e6e06e405ULL,0x5724d06fe3e6f954ULL,0x3e47d47581251a74ULL,+	0xb8ba0151e0fb82f7ULL,0x0d160726d5668ac2ULL,0x622ba25814d711b0ULL,0x6addf5577f3fe2f2ULL,0x2b831e1c6b9c9435ULL,0xce3a060ab73826bdULL,0x93fa11c11c240f89ULL,0x4f9cc8d8956e303aULL,+	0x4b9331f6641f82c9ULL,0xd97c7c54dffec756ULL,0xf5ee6d1f1a9158abULL,0x054493a385c3da7fULL,0xa57a05f5eb7d96dfULL,0xa3afd447e4473a39ULL,0x42a4d9c488e16d55ULL,0x83e144f5f5f876aeULL,+	0x92a83268e32dd620ULL,0x913ec99f627849a2ULL,0xedd8fdfa2c378882ULL,0xaf96f33eee6f8cfeULL,0xc06737e5dc3fa8a5ULL,0x236bb531b0b03a1dULL,0x33e59f2989f037b0ULL,0x13f9b5a7d9a12a53ULL,+	0x50d8ae9559029aa6ULL,0xd74e292c5a4db2edULL,0x0b9c3355848f373dULL,0xec018db6ac45ab38ULL,0x1f44690269cc53a8ULL,0x8c4b628d1a879864ULL,0x1c743d284b13475eULL,0xbf4a933873de19f6ULL,+	0x4a8a4f47f0cefa69ULL,0xdc8e4cbaa4546866ULL,0x359ba69b23f603c1ULL,0xdab4d601187b7ac5ULL,0xa6ca4337c1ebc8d9ULL,0x9fa6585452b4074bULL,0x1a4b4f81902fb733ULL,0xd2bb5d7aa525deaaULL,+	0xeb2e92d5f81f9567ULL,0x54cb95ea4d698470ULL,0x5f2acb28e04c81ebULL,0x1c1ebfc4f8ceec64ULL,0x8f799fac06e07423ULL,0x72225f9937fa0c85ULL,0xb0cd861634f4db44ULL,0x5ec36159752c9091ULL,+	0x0d0df6ce51efb310ULL,0xcb5b2eb4958df5beULL,0xd6459e2936158e59ULL,0x82aae2b91466e336ULL,0xfb658a39411aa636ULL,0x7152ecc5d4c0a933ULL,0xf10c758a49f026b7ULL,0xf4837f97cb09311fULL,+	0x994f523a626332d5ULL,0x7bc388335561bb44ULL,0x005ed4b03d845ea2ULL,0xd39d3ee1c2a1f08aULL,0x6561fdd3e7676b0dULL,0x620e35fffb706017ULL,0x36ce424ff264f9a8ULL,0xc4c3419fda2681f7ULL,+	0x00f831769bb81648ULL,0xd69eb485653120d0ULL,0xd17d75f44ccabc62ULL,0x34a07f82b749fcb1ULL,0x2c3af787bbfb5554ULL,0xb06ed4d062e283f8ULL,0x5722889fa19213a0ULL,0x162b085edcf3c7b4ULL,+	0x36d90ddaeb300f7aULL,0x9dcf7dfcedb5e801ULL,0x645cb26874d5244cULL,0xa127ee79348e3aa2ULL,0x488acc53575f1dbbULL,0x95037e8580e6161eULL,0x57e59283292650d0ULL,0xabe67d9914938216ULL,+	0x32670d2f7189e71fULL,0xc64387485ecf91e7ULL,0x15758e57db757a21ULL,0x427d09f8290a9ce5ULL,0x846a308f38384a7aULL,0xaac3acb4b0732b99ULL,0x9e94100917845819ULL,0x95cba111a7ce5e03ULL,+	0xeb81aa377378058eULL,0x41c746a104411154ULL,0xa10c73bcfb828ac7ULL,0x6439be919d972b29ULL,0x4bf3b4b043a2fbadULL,0x39e6dadf82b5e840ULL,0x4f7164086397bd4cULL,0x0f7de5687f1eeccbULL,+	0xdb332a73f37ec3c3ULL,0xc65259bddd59eba0ULL,0x2291709cdb4d3257ULL,0x9a793b25bd389390ULL,0xf39fe34be43756f0ULL,0x2f76bdce9afb56c9ULL,0x9f37867a61208b27ULL,0xea1d4307089972c3ULL,+	0xd0a744878a429f4fULL,0x0649712bdb516609ULL,0xb826ba57e769b5dfULL,0x82335df21fc7aaf2ULL,0x2389f0675c93d995ULL,0x59ac367a68677be6ULL,0xa77985ff21d9951bULL,0x038956fb85011cceULL,+	0x97b7851aaaca5e9bULL,0x518aa52156713b97ULL,0x3357e8c7150a61f6ULL,0x7842e7e2ec2c2b69ULL,0x8dffaf656868a548ULL,0xd963bd82e068fc81ULL,0x64da5c8b65917733ULL,0x927090ff7b247328ULL,+	0xd6ffdb942f6d0d97ULL,0x05c3ee41443b9373ULL,0xb2e541ebffd36db6ULL,0xb7415a96ce1dcc3eULL,0xe383682e163aa2f6ULL,0x46febdd42f3af218ULL,0x90a0507ebafdbadbULL,0x4ca8ab4dce52e21aULL,+	0xa3f0832e1b7cfb73ULL,0x7a8afe523ec354c9ULL,0xae91c97e378eadcaULL,0x7449c599ac3b32bbULL,0xa619c3710b1c4655ULL,0x692e4c6af79da87eULL,0xff3f5d86de38d96aULL,0xc5320f421c08c0ecULL,+	0x92a6f2bc8fec5decULL,0xa71383ff84d6786cULL,0x87588c06dbffa084ULL,0x0d85f5ca6857e715ULL,0xe87311b3b6c774d4ULL,0x672357c84c3521a8ULL,0xe5fe74615b29fe0fULL,0x02bc51105b7158cdULL,+	0x37a01e48a105fc8eULL,0x769d754a289ba48cULL,0xc08c6fe1d51c2180ULL,0xb032dd33b7bd1387ULL,0x953826db020b0aa6ULL,0x05137e800664c73cULL,0xc66302c4660cf95dULL,0x99004e11b2cef28aULL,+	0x824f5b284f973536ULL,0xb43e299ed35b04eaULL,0xc72c88f74da03089ULL,0x8269d57a45a2e42cULL,0x7c1e63fc6607b38eULL,0xe89e2aaf29390b0cULL,0xc7c740da1bee2869ULL,0x8556f6fcaf3fb974ULL,+	0x1a0a3995a4b6bed1ULL,0x2dab579597095c54ULL,0x06c6a1ff2aa73ce9ULL,0xadd0a54b4de438a7ULL,0x160b6b1afca906cdULL,0x25fc601629de10ddULL,0x348e9c99d3633da3ULL,0x1fe3f746158a4d5aULL,+	0xc253edc88be85c1dULL,0xdd3d0e483ca09cb6ULL,0xb997f6879ae3055aULL,0x0c929ad007431dbfULL,0xcef1621584d2db42ULL,0xb50df3ef078828cdULL,0x4589da9d6dbd4b66ULL,0xbc4fd2e3d99c2b04ULL,+	0x214bc9a7d298c241ULL,0xe3b697ba56807cfdULL,0xef1c78024564eadbULL,0xdde8cdcfb48149c5ULL,0x946bf0a75a4d2604ULL,0x27154d7f6c1538afULL,0x95cc9230de5b1fccULL,0xd88519e966864f82ULL,+	0xb828dd1a7cb1282cULL,0xa08d7626be46973aULL,0x6baf8d40e708d6b2ULL,0x72571fa14daeb3f3ULL,0x85b1732ff22dfd98ULL,0x87ab01a70087108dULL,0xaaaafea85988207aULL,0xccc832f869f00755ULL,+	0x488f1185ca8d9d1aULL,0xadf2c77dd987ded2ULL,0x5f3039f060c46124ULL,0xe5d70b7571e095f4ULL,0x82d586506260e70fULL,0x39d75ea7f750d105ULL,0x8cf3d0b175bac364ULL,0xf3a7564d21d01329ULL,+	0x94ab4700ec3128c2ULL,0x6c76d8628e383f49ULL,0xdc36b150c03024ebULL,0xfb43947753daac69ULL,0xfc68764a8dc79623ULL,0x5b86995db440fbb2ULL,0xd66879bfccc5ee0dULL,0x0522894295aa8bd3ULL,+	0xb24aa43e3fcd3efcULL,0xdd26c034b8088e9aULL,0xa5ef4dc9bd3d46eaULL,0xa2f99d588a4c6a6fULL,0xddabd3552f1da46cULL,0x72c3f8ce1afacdd1ULL,0xd90c4eee92d40578ULL,0xd28bb41fca623b94ULL,+	0x5e7c3becee8314f3ULL,0x1c068aeddbea298fULL,0x08d381f17c80acecULL,0x03b56be8e330495bULL,0xaeffb8f29222882dULL,0x95ff38f6c4af8bf7ULL,0x50e32d351fc57d8cULL,0x6635be5217b444f0ULL,+	0x242792d2e7417ce1ULL,0xff42bc71970ee7f5ULL,0x1ff4dc6d5c67a41eULL,0x77709b7b20882a58ULL,0x3554731dbe217f2cULL,0x2af2a8cd5bb72177ULL,0x58eee769591dd059ULL,0xbb2930c94bba6477ULL,+	0x174a9126cecdaa7aULL,0xfc8c7e0e0b13247bULL,0x29c110d23484c1c4ULL,0xf8eb8757831dfc3bULL,0x022f0212c0067452ULL,0x3f6f69ee7b9b926cULL,0x09032da0ef42daf4ULL,0x79f00ade83f80de4ULL,+	0x1e6adddaf176f2c0ULL,0x01ca4604e2572658ULL,0x0a404ded85342ffbULL,0x8cf60f96441838d6ULL,0x9bbc691cc9071c4aULL,0xfd58874434442803ULL,0x97101c85809c0d81ULL,0xa7fb754c8c456f7fULL,+	0xf8559ff4c1e99d81ULL,0x08e1a7d6a3c617c0ULL,0xb398fd43248c6ba7ULL,0x6ffedd91d1283794ULL,0x8a6a59d2d629d208ULL,0xa9d141d53490530eULL,0x42f6fc1838505989ULL,0x09bf250d479d94eeULL,+	0x6af7a1d5af71013fULL,0xe68216e50bedc946ULL,0xf4cba30bd27370a0ULL,0x7981afbf870421ccULL,0x02496a679449f0e1ULL,0x86cfc4be0a47edaeULL,0x3073c936b1feca22ULL,0xf569461203f8f8fbULL,+	0xec14f9e12cb7191eULL,0x78ea1bd8e5b08ea6ULL,0x3c65aa9b46332bb9ULL,0x84cc22b3bf80ce25ULL,0x0098e9e9d49d5bf1ULL,0xcd4ec1c619087da4ULL,0x3c9d07c5aef6e357ULL,0x839a02689f8f64b8ULL,+	0xbcadd6715bde48f8ULL,0xc97038732189bc7dULL,0x5d45299ec709ee8aULL,0xd1287ee2845aaff8ULL,0x7d1f8874db1dbf1fULL,0xea46588b990c88d6ULL,0x60ba649a84368313ULL,0xd5fdcbce60d543aeULL,+	0xf795643037577dd8ULL,0x83b82af429c5fe88ULL,0x9c1bea26cdbdc132ULL,0x589fa0869c04339eULL,0x033e9538b13799dfULL,0x85fa8b21d295d034ULL,0xdf17f73fbd9ddccaULL,0xf32bd122ddb66334ULL,+	0xcf3de9959890272dULL,0x75f3432a3e713a10ULL,0x5e13479fe28227b8ULL,0xb8561ea9fefacdc8ULL,0xa6a297a08332aafdULL,0x9b0d8bb573809b62ULL,0xd2fa1cfd0c63036fULL,0x7a16eb55bd64bda8ULL,+	0x4cc34ec13cf48283ULL,0xb09daa259c8a705eULL,0xd1e9d0d05b7d4f84ULL,0x4df6ef64db38929dULL,0xe16b0763aa21ba46ULL,0xc6b1d178a293f8fbULL,0x0ff5b602d520aabfULL,0x94d671bdc339397aULL,+	0xf7e48e8a2ac13e27ULL,0x4494f6df4eb1a9f5ULL,0xedbf84eb981f0a62ULL,0x49badc32536438f0ULL,0x50bea541004f7571ULL,0xbac67d10df1c94eeULL,0x253d73a1b727bc31ULL,0xb3d01cf230686e28ULL,+	0xd433e50f6d3549cfULL,0x6f33696ffacd665eULL,0x695bfdacce11fcb4ULL,0x810ee252af7c9860ULL,0x65450fe17159bb2cULL,0xf7dfbebe758b357bULL,0x2b057e74d69fea72ULL,0xd485717a92731745ULL,+	0x896c42e8ee36860cULL,0xdaf04dfd4113c22dULL,0x1adbb7b744104213ULL,0xe5fd5fa11fd394eaULL,0x68235d941a4e0551ULL,0x6772cfbe18d10151ULL,0x276071e309984523ULL,0xe4e879de5a56ba98ULL,+	0x6c8d0aa9b898fd52ULL,0x2fb38a57be9af1a7ULL,0xe1f2b9a93b4f03f8ULL,0x2b1aad44c3f0cc6fULL,0x58b5332e7cf2c084ULL,0x1c57d96f0367d26dULL,0x2297eabdfa6e4a8dULL,0x65a947ee4a0e2b6aULL,+	0xaaafafb0285b9491ULL,0x01a0be881e4c705eULL,0xff1d4f5d2ad9caabULL,0x6e349a4ac37a233fULL,0xcf1c12464a1c6a16ULL,0xd99e6b6629383260ULL,0xea3d43665f6d5471ULL,0x36974d04ff8cc89bULL,+	0xf535b616fdd5b854ULL,0x592549c85728719fULL,0xe231468606921cadULL,0x98c8ce34311b1ef8ULL,0x28b937e7e9090b36ULL,0x67fc3ab90bf7bbb7ULL,0x12337097a9d87974ULL,0x3e5adca1f970e3feULL,+	0xc26c49a1cfe89d80ULL,0xb42c026dda9c8371ULL,0xca6c013adad066d2ULL,0xfb8f722856a4f3eeULL,0x08b579ecd850935bULL,0x34c1a74cd631e1b3ULL,0xcb5fe596ac198534ULL,0x39ff21f6e1f24f25ULL,+	0xcdcc68a7b3f85ff0ULL,0xacd21cdd1a888044ULL,0xb6719b2e05dbe894ULL,0xfae1d3d88b8260d4ULL,0xedfedece8a1c5d92ULL,0xbca01a94dc52077eULL,0xc085549c16dd13edULL,0xdc5c3bae495ebaadULL,+	0x27f29e148f929057ULL,0x7a64ae06c0c853dfULL,0x256cd18358e9c5ceULL,0x9d9cce82ded092a5ULL,0xcc6e59796e93b7c7ULL,0xe1e4709231bb9e27ULL,0xb70b3083aa9e29a0ULL,0xbf181a753785e644ULL,+	0xcc17063fbe7b643aULL,0x7872e1c846085760ULL,0x86b0fffbb4214c9eULL,0xb18bbc0e72bf3638ULL,0x8b17de0c722591c9ULL,0x1edeab1948c29e0cULL,0x9fbfd98ef4304f20ULL,0x2d1dbb6b9c77ffb6ULL,+	0xf53f2c658ead09f7ULL,0x1335e1d59780d14dULL,0x69cc20e0cd1b66bcULL,0x9b670a37bbe0bfc8ULL,0xce53dc8128efbeedULL,0x0c74e77c8326a6e5ULL,0x3604e0d2b88e9a63ULL,0xbab38fca13dc2248ULL,+	0x255616d3c7141771ULL,0xa86691ab2f226b66ULL,0xda19fea4b3ca63a9ULL,0xfc05dc42ae672f2bULL,0xa9c6e786718ba28fULL,0x07b7995b9c66b984ULL,0x0f434f551b3702f2ULL,0xd6f6212fda84eeffULL,+	0x8ed6e8c85c0a3f1eULL,0xbcad24927c87c37fULL,0xfdfb62bb9ee3b78dULL,0xeba8e477cbceba46ULL,0x37d38cb0eeaede4bULL,0x0bc498e87976deb6ULL,0xb2944c046b6147fbULL,0x8b123f35f71f9609ULL,+	0x4b0e7987b5b41d78ULL,0xea7df9074bf0c4f8ULL,0xb4d03560fab80ecdULL,0x6cf306f6fb1db7e5ULL,0x0d59fb5689fd4773ULL,0xab254f4000f9be33ULL,0x18a09a9277352da4ULL,0xf81862f5641ea3efULL,+	0xa155dcc7de79dc24ULL,0xf1168a32558f69cdULL,0xbac215950d1850dfULL,0x15c8295bb204c848ULL,0xf661aa367d8184ffULL,0xc396228e30447bdbULL,0x11cd5143bde4a59eULL,0xe3a26e3b6beab5e6ULL,+	0xb59b01579f759d01ULL,0xa2923d2f7eae4fdeULL,0x18327757690ba8c0ULL,0x4bf7e38b44f51443ULL,0xb6812563b413fc26ULL,0xedb7d36379e53b36ULL,0x4fa585c4c389f66dULL,0x8e1adc3154bd3416ULL,+	0xd3b3a13f1402b9d0ULL,0x573441c32c7bc863ULL,0x4b301ec4578c3e6eULL,0xc26fc9c40adaf57eULL,0x96e71bfd7493cea3ULL,0xd05d4b3f1af81456ULL,0xdaca2a8a6a8c608fULL,0x53ef07f60725b276ULL,+	0xa6b5c9d646ac49d2ULL,0x42c77c0b83137aa9ULL,0x24d000fc68225a38ULL,0x0f63cfc82fe1e907ULL,0x22d1b01bc6441f95ULL,0x7d38f719ec8e448fULL,0x9b33fa5f787fb1baULL,0x94dcfda1190158dfULL,+	0x211cde10296c36efULL,0x7ee8967282c4da77ULL,0xb617d270a57836daULL,0xf0cd9c319cb7560bULL,0x01fdcbf7e455fe90ULL,0x3fb53cbb7e7334f3ULL,0x781e2ea44e7de4ecULL,0x8adab3ad0b384fd0ULL,+	0x01778a2b599ff0f9ULL,0x68a923d78104fc6bULL,0x5bfa44dfda694ff3ULL,0x4f7199dbf7667f12ULL,0xc06d8ff6e46f2a79ULL,0x08b5deade9f8131dULL,0x02519a59abb4ce7cULL,0xc4f710bcb42aec3eULL,+	0x3014368b4ed80940ULL,0x67e6d0567a6fceddULL,0x7c208c49ca97579fULL,0xfe3d7a81a23597f6ULL,0x5e2032027e096ae2ULL,0xb1f3e1e724b39366ULL,0x26da26f32fdcdffcULL,0x79422f1d6097be83ULL,+	0x50549c748c878145ULL,0x67f14edf39c63565ULL,0x22ddf78c9bcf2d5eULL,0xffaa842f68201d10ULL,0x47d94a9dd1b2de28ULL,0xc09c4be8054be414ULL,0xac80e178ca82755bULL,0xe3251d105697c3bdULL,+	0x2bbe09d35001417bULL,0x795e84ee5962ed5eULL,0x5b79d1ca279f46c3ULL,0x1f7f8a3b83836a2eULL,0x692200b14a64dc32ULL,0xc84243350f84f739ULL,0xf110da07cc9155c0ULL,0xee8fbe61594b0507ULL,+	0x2f6a5391035703ccULL,0x9899bf6a40c7e24dULL,0xc3f7f248bbfcbb9aULL,0xf65027ded555875bULL,0xa7a16b69ffff3b37ULL,0x67b6eb54145b4431ULL,0x19d7e1d249afd679ULL,0xbd819bab110fccdfULL,+	0x263a2cfb9db3b381ULL,0x9c3a2deed4df0a4bULL,0x728d06e97d04e61fULL,0x8b1adfbc42449325ULL,0x6ec1d9397e053a1bULL,0xee2be5c766daf707ULL,0x80ba1e14810ac7abULL,0xdd2ae778f530f174ULL,+	0x3e708e703b9f0426ULL,0xe5b02fb60c84f17cULL,0x2f4ff35be3b70a0bULL,0x781b3c5f9b15565dULL,0xe76c636a6e124c3aULL,0xbde81eba8b496784ULL,0xa412f8e2443f0370ULL,0x15d42362999be45dULL,+	0x0f503ae2a4af76c1ULL,0x550e66dc08276fe7ULL,0x11e0c1fcbf3a33c6ULL,0x42be231006629f85ULL,0xedf7743e516ace49ULL,0xce436668436a2262ULL,0xe1ac7036446ca192ULL,0x73631cb5476fe13eULL,+	0xd47f82d4a160bcfaULL,0x258fb075b8ceb1f2ULL,0x4f818e8fdf5a8d25ULL,0x6685475e6fd31c9dULL,0xcef6385ae1e9b13fULL,0xe0a42594f0508bdbULL,0x5ad7ae16aef1f90dULL,0x45a155ef63f8a81eULL,+	0x8ca407c28034b95eULL,0xc93eb97617bdc560ULL,0x4ec24e8d339807e8ULL,0x91b734d6dd64a4ebULL,0xd0fece398f668b26ULL,0x4822cc4b141823d5ULL,0xb953bc32f09e4e00ULL,0xca0a7c6006d861aeULL,+	0x4c74d4470f33e712ULL,0x3cec1e0625a87cb0ULL,0x5cec0610e5962db4ULL,0xd971af1571a256aaULL,0xa044c983a2ef4ac9ULL,0xcaa1da63d74e9d00ULL,0xfb972d834673d881ULL,0x50747c5a03a26c8bULL,+	0x04349982a3e25566ULL,0xeef9075e18e1b896ULL,0x4c7bead092b2d24bULL,0xd99f72fb0a21ba55ULL,0xb93e09315005e541ULL,0x2a7a98389ece3205ULL,0xeb388ed11462f2f6ULL,0xb488b15a2e3460a6ULL,+	0x43f6cd67969d56afULL,0x9e0d872cdfc58a8bULL,0x401c1509a4e70377ULL,0x103d1a1308ad646cULL,0x078ee37e9d062427ULL,0x4e69c5acb9bef78cULL,0x521ec00136e66142ULL,0x8de1ecb2a634cd82ULL,+	0x0435d97a205b9d8bULL,0x6eb8f064056756d4ULL,0xd5e88a8bb6f8210eULL,0x070ef12dec9fd9eaULL,0x4d8495053bcc876aULL,0x12a75338a7404ce3ULL,0xd22b49e1b8a1db5eULL,0xec1f205114bfa5adULL,+	0x43ed81b5c4e83d33ULL,0xd9f358795efd488bULL,0x164a620f9deb4d0fULL,0xc6927bdbac6a7394ULL,0x45c28df79f9e0f03ULL,0x2868661efcd7e1a9ULL,0x7cf4e8d0ffa348f1ULL,0x6bd4c284398538e0ULL,+	0x56036e8c06d75fc1ULL,0x2dcf7bb73249a89fULL,0x81dd1d3de245e7ddULL,0xf578dc4bebd6e2a7ULL,0x4c028903df2ce7a0ULL,0xaee362889c39afacULL,0xdc847c31146404abULL,0x6304c0d8a4e97818ULL,+	0xfb6836c327d02dccULL,0x5ad009827a68bcc2ULL,0x1b24b44c005e912dULL,0xcc83d20f811fdcfeULL,0x36527ec1666fba0cULL,0x6994819714754635ULL,0xfcdcb1a8556da9c2ULL,0xa593426781a732b2ULL,+	0xe4ac8b33070d3aabULL,0x2643672b9a2cd5e5ULL,0x52eff79b1cfc9173ULL,0x665ca49b90a7c13fULL,0x5a8dda59b3efb998ULL,0x8a5b922d052f1341ULL,0xae9ebbab3cf9a530ULL,0x35986e7bf56da4d7ULL,+	0x63ee4cbd8088b454ULL,0xdb7f32f79a9e0c8aULL,0xb377d4186b2447cbULL,0xe3e982aad370219bULL,0x06ccc1e4c2a2a593ULL,0x72c368650773f24fULL,0xa13b4da795859423ULL,0x8bbf1d3375040c8fULL,+	0x03c187d0ad886aacULL,0x5c16878ab771b645ULL,0xb07dfc6fc74045abULL,0x2c6360bf7800caedULL,0x24295bb5b9c972a3ULL,0xc9e6f88e7c9a6dbaULL,0x90ffbf2492a79aa6ULL,0xde29d50a41c26ac2ULL,+	0xb1f0fb68d84d835dULL,0xc90caf39861dc1e6ULL,0x12e5b0467594f8d7ULL,0x26897ae265012b92ULL,0xbcf68a08a4d6755dULL,0x403ee41c0991fbdaULL,0x733e343e3bbf17e8ULL,0xd2c7980d679b3d65ULL,+	0x534acf4fda79e5acULL,0x68b83b3a8630215fULL,0x5c748b2ed085756eULL,0xb0317258e5d37cb2ULL,0x6735841ac5ccc2c4ULL,0x7d7dc96b3d9d5069ULL,0xa147e410fd1754bdULL,0x65296e94d399ddd5ULL,+	0x1e71c9a1deb8568bULL,0xa35daea080fb3d32ULL,0xe8b6f2662cf8fb81ULL,0x6d51afe89490696aULL,0x81beac6e51803a19ULL,0xe3d24b7f86219080ULL,0x727cfd9ddf6f463cULL,0x8c6865ca72284ee8ULL,+	0xe00df169d23233f3ULL,0x3e32279677cb637fULL,0x1f897c0e1da0cf6cULL,0xa651f5d831d6bbddULL,0xdd61af191a230c76ULL,0xbd527272cdaa5e4aULL,0xca753636d0abcd7eULL,0x78bdd37c370bd8dcULL,+	0xcddb27c17078c432ULL,0xe1961b9cb77fedb7ULL,0x1edc2f5cc2290570ULL,0x2c3fefca19cbd886ULL,0xcf880a36c2af389aULL,0x96c610fdbda71ceaULL,0xf03977a932aa8463ULL,0x8eb7763f8586d90aULL,+	0x831ab3edf0290a8fULL,0xcae81966cb47c387ULL,0xaad7dece184efb4fULL,0xdcfc53b34749110eULL,0x6698f23c4cb632f9ULL,0xc42a1ad6b91f8067ULL,0xb116a81d6284180aULL,0xebedf5f8e901326fULL,+	0x91633f0ab2cf8940ULL,0x72b0b1786f948f51ULL,0x2d28dc30782653c8ULL,0x88829849db903a05ULL,0xb8095d0c6a19d2bbULL,0x4b9e7f0c86f782cbULL,0x7af739882d907064ULL,0xd12be0fe8b32643cULL,+	0x9561f28b638a7e81ULL,0x54155cdf5980ddc3ULL,0xb2db4a96d26f247aULL,0x9d774e4e4787d100ULL,0x1a9e6e2e078637d2ULL,0x1c363e2d5e0ae06aULL,0x7493483ee9cfa354ULL,0x76843cb37f74b98dULL,+	0x0491f1bc789a283bULL,0x72d3ac3d880836f4ULL,0xaa1c5ea388e5402dULL,0x1b192421d5cc473dULL,0x5c0b99989dc84cacULL,0xb0a8482d9c6e75b8ULL,0x639961d03a191ce2ULL,0xda3bc8656d837930ULL,+	0xd7e0c4cdb30cfb3aULL,0x6d09b8c16c9db4c8ULL,0x40ba1a4207c8d9dfULL,0x6fd495f71c52c66dULL,0xfb0e169f275264daULL,0x80c2b746e57d8362ULL,0xedd987f749ad7222ULL,0xfdc229af4398ec7bULL,+	0xb0d1ed8452666a58ULL,0x4bcb6e00e6a9c3c2ULL,0x3c57411c26906408ULL,0xcfc2075513556400ULL,0xa08b1c505294dba3ULL,0xa30ba2868b7dd31eULL,0xd70ba90e991eca74ULL,0x094e142ce762c2b9ULL,+	0xb81d783e979f3925ULL,0x1efd130aaf4c89a7ULL,0x525c2144fd1bf7faULL,0x4b2969041b265a9eULL,0xed8e9634b9db65b6ULL,0x35c82e3203599d8aULL,0xdaa7a54f403563f3ULL,0x9df088ad022c38abULL,+	0xe5cfb066bb3fd30aULL,0x429169daeff0354eULL,0x809cf8523524e36cULL,0x136f4fb30155be1dULL,0x4826af011fbba712ULL,0x6ef0f0b4506ba1a1ULL,0xd9928b3177aea73eULL,0xe2bf6af25eaa244eULL,+	0x8d084f124237b64bULL,0x688ebe99e3ecfd07ULL,0x57b8a70cf6845dd8ULL,0x808fc59c5da4a325ULL,0xa9032b2ba3585862ULL,0xb66825d5edf29386ULL,0xb5a5a8db431ec29bULL,0xbb143a983a1e8dc8ULL,+	0x35ee94ce12ae381bULL,0x3a7f176c86ccda90ULL,0xc63a657e4606eacaULL,0x9ae5a38043cd04dfULL,0x9bec8d15ed251b46ULL,0x1f5d6d30caca5e64ULL,0x347b3b359ff20f07ULL,0x4d65f034f7e4b286ULL,+	0x9e93ba24f111661eULL,0xedced484b105eb04ULL,0x96dc9ba1f424b578ULL,0xbf8f66b7e83e9069ULL,0x872d4df4d7ed8216ULL,0xbf07f3778e2cbecfULL,0x4281d89998e73754ULL,0xfec85fbb8aab8708ULL,+	0x9a3c0deea5ba5b0bULL,0xe6a116ce42d05299ULL,0xae9775fee9b02d42ULL,0x72b05200a1545cb6ULL,0xbc506f7d31a3b4eaULL,0xe58930788bbd9b32ULL,0xc8bc5f37e4b12a97ULL,0x6b000c064a73b671ULL,+	0x13b5bf22765fa7d0ULL,0x59805bf01d6a5370ULL,0x67a5e29d4280db98ULL,0x4f53916f776b1ce3ULL,0x714ff61f33ddf626ULL,0x4206238ea085d103ULL,0x1c50d4b7e5809ee3ULL,0x999f450d85f8eb1dULL,+	0x658a6051e4c79e9bULL,0x1394cb73c66a9feaULL,0x27f31ed5c6be7b23ULL,0xf4c88f365aa6f8feULL,0x0fb0721f4aaa499eULL,0x68b3a7d5e3fb2a6bULL,0xa788097d3a92851dULL,0x060e7f8ae96f4913ULL,+	0x82eebe731a3a93bcULL,0x42bbf465a21adc1aULL,0xc10b6fa4ef030efdULL,0x247aa4c787b097bbULL,0x8b8dc632f60c77daULL,0x6ffbc26ac223523eULL,0xa4f6ff11344579cfULL,0x5825653c980250f6ULL,+	0xb2dd097ebc1aa2b9ULL,0x0788939337a0333aULL,0x1cf55e7137a0db38ULL,0x2648487f792c1613ULL,0xdad013363fcef261ULL,0x6239c81d0eabf129ULL,0x8ee761de9d276be2ULL,0x406a7a341eda6ad3ULL,+	0x4bf367ba4a493b31ULL,0x54f20a529bf7f026ULL,0xb696e0629795914bULL,0xcddab96d8bf236acULL,0x4ff2c70aed25ea13ULL,0xfa1d09eb81cbbbe7ULL,0x88fc8c87468544c5ULL,0x847a670d696b3317ULL,+	0xf133421e64bcb626ULL,0xaea638c826dee0b5ULL,0xd6e7680bb310346cULL,0xe06f4097d5d4ced3ULL,0x099614527512a30bULL,0xf3d867fde589a59aULL,0x2e73254f52d0c180ULL,0x9063d8a3333c74acULL,+	0xeda6c595d314e7bcULL,0x2ee7464b467899edULL,0x1cef423c0a1ed5d3ULL,0x217e76ea69cc7613ULL,0x27ccce1fe7cda917ULL,0x12d8016b8a893f16ULL,0xbcd6de849fc74f6bULL,0xfa5817e2f3144e61ULL,+	0x1f3541640821ee4cULL,0x1583eab40bc61992ULL,0x7490caf61d72879fULL,0x998ad9f3f76ae7b2ULL,0x1e181950a41157f7ULL,0xa9d7e1e6e8da3a7eULL,0x963784eb8426b95fULL,0x0ee4ed6e542e2a10ULL,+	0xb79d4cc5ac751e7bULL,0x93f96472fd4211bdULL,0x8c72d3d2c8de4fc6ULL,0x7b69cbf5df44f064ULL,0x3da90ca2f4bf94e1ULL,0x1a5325f8f12894e2ULL,0x0a437f6c7917d60bULL,0x9be7048696c9cb5dULL,+	0x949c9976e1337c26ULL,0x6faadebdd73d68e5ULL,0x9e158614f1b768d9ULL,0x22dfa5579cc4f069ULL,0xccd6da17be93c6d6ULL,0x24866c61a504f5b9ULL,0x2121353c8d694da1ULL,0x1c6ca5800140b8c6ULL,+	0xf1604a7dd4b79bb8ULL,0xaee806fb52c878c8ULL,0x34144f118d47b8e8ULL,0x72edf52b949f9054ULL,0xebfca84e2127015aULL,0x9051d0c09cb7cef3ULL,0x86e8fe58296deec8ULL,0x33b2818841010d74ULL,+	0xbd5660ed9aed9f40ULL,0x70ca6ad1532a8c99ULL,0xc4978bfb95c371eaULL,0xe5464d0d7003109dULL,0x1af32fdfd9e535efULL,0xabf57ea798c9185bULL,0xed7a741712b42488ULL,0x8e0296a7e97286faULL,+	0x01079383171b445fULL,0x9bcf21e38131ad4cULL,0x8cdfe205c93987e8ULL,0xe63f4152c92e8c8fULL,0x729462a930add43dULL,0x62ebb143c980f05aULL,0x4f3954e53b06e968ULL,0xfe1d75ad242cf6b1ULL,+	0x8b57416e1f017d5eULL,0x375333967674e99bULL,0x6e6d94c0e8f488a0ULL,0xb93a787adc16f95eULL,0xc3ac51a2dcc99cccULL,0xc134b4139aa47c1dULL,0xf28fcdafafdfd8d5ULL,0x0d57bd8e10b831edULL,+	0x9276fbccf0bcfc46ULL,0x3a822aceb5cffee6ULL,0x328ed2fec75d915bULL,0xa145c113c359476cULL,0xf61a81538be17bcdULL,0x01e867c3aa6c3d8fULL,0x5634e15d6516c82fULL,0xc1437bd26948b9b0ULL,+	0xd2fcd2006c19d4c7ULL,0xa0f3c437e1b1e976ULL,0xf0545ff694f237e8ULL,0xdd10ec3fc0bf8bb1ULL,0x4f89696cac7cd3e1ULL,0xed3714ec5f24bfe6ULL,0x363eb1d85faf7706ULL,0xfcbd604dc027cc32ULL,+	0x5f95c6c7af8685c8ULL,0xd4c1c8ce2f8f01aaULL,0xc44bbe322574692aULL,0xb8003478d4a4a068ULL,0x7c8fc6e52eca3cdbULL,0xea1db16bec04d399ULL,0xb05bc82e8f2bc5cfULL,0x763d517ff44793d2ULL,+	0x16ce8eddc355363bULL,0x4af2f70ff8820d6eULL,0xcb7ed4d27661a508ULL,0x41d3444edd195472ULL,0x17fea2b438da9649ULL,0x9bf69356aeb4a200ULL,0xa13b5f916ab19c3dULL,0xc0519c14dc9360a6ULL,+	0xc2571ae92e42e171ULL,0xcb31ab63ed41ccf9ULL,0x37f3c576b5c8854fULL,0x66e5191bc62392a1ULL,0x71565a1c6cd5683bULL,0x484b0283606fe689ULL,0xf3a25d6767e2fda6ULL,0x87ba21de8a65c0a4ULL,+	0xde74e49ca70684d1ULL,0x3ae8766133e80c3dULL,0x5984a2a916a5c34dULL,0x09a83eccb8298c35ULL,0x9a19867caa4ca4c0ULL,0x02085610b375b8ffULL,0xf296328bf70396dcULL,0x9c9ddc4cde6fae63ULL,+	0x4451c1b808bd98d0ULL,0x644b1cd46575f240ULL,0x6907eb337375d270ULL,0x56c8bebdfa2286bdULL,0xc713d2acc4632b46ULL,0x17da427aafd60242ULL,0x313065b7c95c7546ULL,0xf8239898bf17a3deULL,+	0x94683d260b083b6eULL,0x0a3752eb06f6a54dULL,0x48bedc23752074ddULL,0x637622fc3e822593ULL,0xea0005136be55d3bULL,0x9f5e12f4324d006dULL,0x529486a964fc0270ULL,0x09ba0d0c923399e6ULL,+	0x363858473a977080ULL,0x4cf8e1b80c6a6ab6ULL,0x919a5c6c0482261eULL,0x517a9ad0e5ce4806ULL,0x2792d40c056aa7aaULL,0x4c7c6adae56c61b0ULL,0xf19cb178a4b19e0cULL,0x046d5c4fe4ba267fULL,+	0xd3e926ab121550b3ULL,0xe4975e4ac147ce84ULL,0x7a8be0f95eff722aULL,0x71e4702c6fd4f2a0ULL,0x13b92acf3cb7b280ULL,0xc588716d28272d73ULL,0x862c7bf3daa9fe5cULL,0x78c008f2e2a79e42ULL,+	0xf3b7963f4c830320ULL,0x842c7aa0903203e3ULL,0xaf22ca0ae7327afbULL,0x38e13092967609b6ULL,0x73b8fb62757558f1ULL,0x3cc3e831f7eca8c1ULL,0xe4174474f6331627ULL,0xa77989cac3c40234ULL,+	0xae8317f4b0166f7aULL,0xfbd3e3f7ceec74e6ULL,0xfdb516ace0874bfdULL,0x3d846019c681f3a3ULL,0x0b12ee5c7c1620b0ULL,0xba68b4dd2b63c501ULL,0xac03cd326668c51eULL,0x2a6279f74e0bcb5bULL,+	0xfd8e139f8f5fcda8ULL,0xf3e558c4bdee5bfdULL,0xd76cbaf4e33f9f77ULL,0x3a4c97a471771969ULL,0xda27e84bf6dce6a7ULL,0xff373d9613e6c2d1ULL,0xf115193cd759a6e9ULL,0x3f9b702563d2262cULL,+	0x12536fea87baa627ULL,0x58c1fec1f72aa680ULL,0x6c29b637601e5dc9ULL,0x9e3c3c1cde9e01b9ULL,0xefc8127b2bcfe0b0ULL,0x351071022a12f50dULL,0x6ccd6cb14879b397ULL,0xf792f804f8a82f21ULL,+	0x8c3184911a335cc8ULL,0x563459ba6a5913e4ULL,0x1b920d61c7b32919ULL,0x805ab8b6a02425adULL,0x2ac512da8d006086ULL,0x6ca4846abcf5c0fdULL,0xafea51d8ac2138d7ULL,0xcb647545344cd443ULL,+	0xa3f4f521e447f2c4ULL,0x81b8da7a604291f0ULL,0xd680bc467d5926deULL,0x84f21fd534a1202fULL,0x1d1e31814e9df3d8ULL,0x1ca4861a39ab8d34ULL,0x809ddeec5b19aa4aULL,0x59f72f7e4d329366ULL,+	0x9f1b2466cdedca85ULL,0x140bb7101a09538cULL,0xac8ae8515e11115dULL,0x0d63ff676f03f59eULL,0x755e55517d234afbULL,0x61c2db4e7e208fc1ULL,0xaa9859cef28a4b5dULL,0xbdd6d4fc34af030fULL,+	0x3f39e67f906151e5ULL,0xcea27f5f55e10649ULL,0xdca1d4e1c17cf7b7ULL,0x0c326d122fe2362dULL,0x05f7ac337dd35df3ULL,0x0c3b7639c396dbdfULL,0x0912f5ac03b7db1cULL,0x9dea4b705c9ed4a9ULL,+	0x511053e453544774ULL,0x834d0ecc3adba2bcULL,0x4215d7f7bae371f5ULL,0xfcfd57bf6c8663bcULL,0xded2383dd6901b1dULL,0x3b49fbb4b5587dc3ULL,0xfd44a08d07625f62ULL,0x3ee4d65b9de9b762ULL,+	0x3e56fe5bdf53aad0ULL,0x51314de5e4604a67ULL,0x386ad98607a261a0ULL,0x8b7e021217afcc91ULL,0xcbf411273b72aec5ULL,0x13c85d05c4f9f509ULL,0xeda56845b6484b57ULL,0x13cb1642b3d0995bULL,+	0x5a994b6e717815deULL,0xd995c7a0a7e131d1ULL,0xc8b46df226c023aaULL,0x8cfd094d702afcedULL,0x7bc743cdced6a886ULL,0xb7d70ec41fcabe75ULL,0x2a6c9e47ddac9390ULL,0x720694259310aa90ULL,+	0xa607b3263e8f793cULL,0xa541166ecde3b289ULL,0xf44ff2924a915b21ULL,0x68bea906b58ecda6ULL,0xd85b37b440292897ULL,0xd2a508ae4b768423ULL,0xd10bb79da413bbbeULL,0x0262f481061491f4ULL,+	0xd17e80f55464d0ebULL,0x89d3e1a767a613a7ULL,0x77791260c8c97dadULL,0xec2ff21fe4f0cbe7ULL,0xed984ac2e9e6bc10ULL,0xe3c53de877cba305ULL,0x9bbaf9b283624fdcULL,0x5e9451cd1485c0ecULL,+	0xfdfc9f63b7abad11ULL,0xc7ec3a9263a46189ULL,0x49ebee42f67037b7ULL,0x8247f504cdac1710ULL,0xfc518f8d397583e5ULL,0xe7d24de70c3f8c2eULL,0x354832669c5edcb1ULL,0x94bba483f8c2cefcULL,+	0x1f13756db1761ec8ULL,0xe53c8b98a4b97e55ULL,0xb2aee3f84096cc28ULL,0x48c361a0920f1a8dULL,0xa98b672d8c31190aULL,0x7bc1e7d1001855d4ULL,0x242cfb07bf3f4b2aULL,0x9bf44a3f32a28bc4ULL,+	0xeba8976299da550cULL,0xb2c1781a16baa042ULL,0x3068b082788c2f9dULL,0xc0fa414594869a9eULL,0x73bd9e39d50b693fULL,0xb79e2a9c988e2c5eULL,0xf1cb8de40f8f9f62ULL,0x415b04ee1ea50c7bULL,+	0x64e5137d0d63d1faULL,0x658fc05202a9d89fULL,0x4889487450436309ULL,0xe9ae30f8d598da61ULL,0x2ed710d1818baf91ULL,0xe27e9e068b6a0c20ULL,0x1e28dcfb1c1a6b44ULL,0x883acb64d6ac57dcULL,+	0x8735728dc2c6ff70ULL,0x79d6122fc5dc2235ULL,0x23f5d00319e277f9ULL,0x7ee84e25dded8cc7ULL,0x91a8afb063cd880aULL,0x3f3ea7c63574af60ULL,0x0cfcdc8402de7f42ULL,0x62d0792fb31aa152ULL,+	0xb02c83f9dec31a21ULL,0x988c8b236ad9d573ULL,0x53e983aea57be365ULL,0xe968734d646f834eULL,0x9137ea8f5da6309bULL,0x10f3a624c1f1ce16ULL,0x782a9ea2ca440921ULL,0xdf94739e5b46f1b5ULL,+	0x1df165a434a35ea8ULL,0x3418e0f74d4412f6ULL,0x5af1f8af518836c3ULL,0x42ceef4d130e1965ULL,0x5560ca0b543a1957ULL,0xc33761e5886cb123ULL,0x66624b1ffe98ed30ULL,0xf772f4bf1090997dULL,+	0x56f8410ef4f8b16aULL,0x97241afec47b266aULL,0x0a406b8e6d9c87c1ULL,0x803f3e02cd42ab1bULL,0x7f0309a804dbec69ULL,0xa83b85f73bbad05fULL,0xc6097273ad8e197fULL,0xc097440e5067adc1ULL,+	0xc507b6dd418e7dddULL,0x39888d93472f19d6ULL,0x7eae26be0c27eb4dULL,0x17b53ed3fbabb884ULL,0xfc27021b2b01ae4fULL,0x88462e87cf488682ULL,0xbee096ec215e2d87ULL,0xeb2fea9ad242e29bULL,+	0xbbcc00c756b95bceULL,0x5ec03906616da680ULL,0x79162ee672214252ULL,0x43132b6386a892d2ULL,0x4bdd3ff22f3263bfULL,0xd5b3733c9cd0a142ULL,0x592eaa8244415ccbULL,0x663e89248d5474eaULL,+	0x0d38ab35ce7c42d4ULL,0x9fd493ef82feab10ULL,0x46056b6d82111b45ULL,0xda11dae173efc5c3ULL,0xdc7402785545a7fbULL,0xbdb2601c40d507e6ULL,0x121dfeeb7066fa58ULL,0x214369a839ae8c2aULL,+	0x3f747fa0b311898cULL,0xe2a272e4cd0eac65ULL,0x4bba5851f914d0bcULL,0x7a1a9660c4a43ee3ULL,0xe5a367cea1c8cde9ULL,0x9d958ba97271abe3ULL,0xf3ff7eb63d1615cdULL,0xa2280dcef5ae20b0ULL,+	0x8c0ed566d4312483ULL,0x5179a95d643e216fULL,0xcc185fec17044493ULL,0xb306333954991a21ULL,0xd801ecdb0081a726ULL,0x0149b0c64fa89bbbULL,0xafe9065a4391b6b9ULL,0xedc92786d633f3a3ULL,+	0xd6d9d9e37a6a308bULL,0x623758304c2767d3ULL,0x874a8bc6f38cbeb6ULL,0xd94d3f1accb6fd9eULL,0x92a9735bba21f248ULL,0x272ad0e56cd1efb0ULL,0x7437b69c05b03284ULL,0xe7f047026948c225ULL,+	0x9ae868a9e9adfe1cULL,0x3984403d314e39bbULL,0xb5875720f2fe378fULL,0x33f901e0ba44a628ULL,0xea1125fe3652438cULL,0xae9ec4e69dd1f20bULL,0x1e740d9ebebf7fbdULL,0x6dbd3ddc42dbe79cULL,+	0x266344a43794f8dcULL,0xdcca923a483c5c36ULL,0x2d6b6bbf3f9d10a0ULL,0xb320c5ca81d9bdf3ULL,0x620e28ff47b50a95ULL,0x933e3b01cef03371ULL,0xf081bf8599100153ULL,0x183be9a0c3a8c8d6ULL,+	0x8a9443d77613aa81ULL,0x8010080085fe6584ULL,0x70fc4dbc7fb10288ULL,0xf58280d3e86beee8ULL,0x14fdd82f7c978c38ULL,0xdf1204c10de44d7bULL,0xa08a1c844160252fULL,0x591554cac17646a5ULL,+	0x7ddc81ea77b46a08ULL,0xcf5a6477c7480699ULL,0x43a8cb346633f683ULL,0x1b867e6b92363c60ULL,0x439211141f60558eULL,0xcdbcdd632f41450eULL,0x7fc04601cc630e8bULL,0xea7c66d597038b43ULL,+	0x34fc8820fbeee3f9ULL,0x93e5349049091afdULL,0x764b9be59a31f35cULL,0x71f3786457e3d924ULL,0x02fb34e0943aa75eULL,0xa18c9c58ab8ff6e4ULL,0x080f31b133cf0d19ULL,0x5c9682db083518a7ULL,+	0xb6c185c341dca566ULL,0x7de7fedad8622aa3ULL,0x99e84d92901b6dfbULL,0x30a02b0e7c4ad288ULL,0xc7c81daa2fd3cf36ULL,0xd1319547df89e59fULL,0xb2be8184cd496733ULL,0xd5f449eb93d3412bULL,+	0xc492ec644cd8f64cULL,0x58a2d790279d7b51ULL,0x0ced1fc51fc75256ULL,0x3e658aed8f433017ULL,0x0b61942e05da59ebULL,0xba3d60a30ddc3722ULL,0x7c311cd1742e7f87ULL,0x6473ffeef6b01b6eULL,+	0x8303604f692ac542ULL,0xf079ffe1227b91d3ULL,0x19f63e6315aaf9bdULL,0xf99ee565f1f344fbULL,0x8a1d661fd6219199ULL,0x8c883bc6d48ce41cULL,0x1065118f3c74d904ULL,0x713889ee0faf8b1bULL,+	0xc035f697960eb8c7ULL,0xf1599f2ce2de04d3ULL,0x892450f8d2ad9228ULL,0x7d48129bb829c1abULL,0x24d785e13a50afc9ULL,0x2745ba2763a96ee0ULL,0x956534013bfb6d7bULL,0x536202671bad2a42ULL,+	0x972b3f8f81a1b3beULL,0x4f3ce145ce2764a0ULL,0xe2d0f1cc28c4f5f7ULL,0xdeee0c0dc7f3985bULL,0x7df4adc0d39e25c3ULL,0x40619820c467a080ULL,0x440ebc9361cf5a58ULL,0x527729a6422ad600ULL,+	0xa691398a4a9eb3f0ULL,0x56c1dbff3b99a48fULL,0x9a87e1b91b4b5b32ULL,0xad6396145378b5feULL,0x437a243ec26b5302ULL,0x0275878c3ccb4c10ULL,0x0e81e4a21de07015ULL,0x0c6265c9850df3c0ULL,+	0xca6c0937b1b76ba6ULL,0x1a2eab854d2026dcULL,0xb1715e1519d9ae0aULL,0xf1ad9199bac4a026ULL,0x35b3dfb807ea7b0eULL,0xedf5496f3ed9eb89ULL,0x8932e5ff2d6d08abULL,0xf314874e25bd2731ULL,+	0xc8327149a8c25ff6ULL,0x29bf2556782e6569ULL,0x9012f5c6cd68fc38ULL,0x3e67e8bd3b982ad5ULL,0x5e3a75386ecdca88ULL,0xf297eaa6c1753a04ULL,0x10121e5405db3256ULL,0xab9697d4f0851055ULL,+	0xefb26a753f73f449ULL,0x1d1c94f88d44fc79ULL,0x49f0fbc53bc0dc4dULL,0xb747ea0b3698a0d0ULL,0x5218c3fe228d291eULL,0x35b804b543c129d6ULL,0xfac859b8d1acc516ULL,0x6c10697d95d6e668ULL,+	0xe5d27171f6bdf1bfULL,0x0b77b876facb0d8fULL,0xda95471d8496a31bULL,0x46a50dbb3f16b103ULL,0x2a4f3f977b865bffULL,0x848195e66b1c198cULL,0x491ad08821702ea6ULL,0x3f20b43749035228ULL,+	0xc38e438f0876fd4eULL,0x45f0c30783d2f383ULL,0x203cc2ecb10934cbULL,0x6a8f24392c9d46eeULL,0xf16b431b65ccde7bULL,0x41e2cd1827e76a6fULL,0xb9c8cf8f4e3484d7ULL,0x64426efd8315244aULL,+	0xe6ec98093a69fc01ULL,0x7e20fecbfaa9dfc2ULL,0x5cfdbb07f56f2a55ULL,0xb1cd68680bbdbfdfULL,0x247b4995986eb9edULL,0x74785bf53dd0955eULL,0x88f74f61c0c7a201ULL,0x8861a15b5d01a80dULL,+	0x1c0a8e44fc94dea3ULL,0x34c8cdbfdad6a0b0ULL,0x919c384004113cefULL,0xfd32fba415490ffaULL,0x58d190f6795dcfb7ULL,0xfef01b0383588bafULL,0x9e6d1d63ca1fc1c0ULL,0x53173f96f0a41ac9ULL,+	0x54637e4182997cc1ULL,0x08c5a96ce3720c9cULL,0x78bce01c11de5d45ULL,0x49d623e50dfdd75aULL,0x8c72a4680fb2a3acULL,0xcc53bbff319c25afULL,0x198eba7978a92421ULL,0xcd61f28ba3bdecf3ULL,+	0x2b1d402aba16f73bULL,0x2fb310148cf9b9fcULL,0x2d51e60e446ef7bfULL,0xc731021bb91e1745ULL,0x9d3b47244fee99d4ULL,0x4bca48b6fac5c1eaULL,0x70f5f514bbea9af7ULL,0x751f55a5974c283aULL,+	0x23899fe8662595c2ULL,0x495d672711a80773ULL,0x86c971d2b0d1d43bULL,0xb518637c93b7a65fULL,0x30e453bad98c99ceULL,0xba6e0d4a14d39f5bULL,0xf7db02a6431ce415ULL,0xcd909c7cf6e1d823ULL,+	0x6e30251acb452fdbULL,0x31ee696550f30650ULL,0xb0b3e508933548d9ULL,0xb8949a4ff4b0ef5bULL,0x208b83263c88f3bdULL,0xab147c30db1d9989ULL,0xed6515fd44d4df03ULL,0x17a12f75e72eb0c5ULL,+	0x25914f7881fdad90ULL,0xcf638f560d2cf6abULL,0xb90bc03fcc054de5ULL,0x932811a718b06350ULL,0x2f00b3309bbd11ffULL,0x76108a6fb4044974ULL,0x801bb9e0a851d266ULL,0x0dd099bebf8990c1ULL,+	0x14c6dd8a58d6cd46ULL,0x9cb633b58e6634d2ULL,0xc1305047f81bc328ULL,0x12ede0e226a177e5ULL,0x332cca62065a6f4fULL,0xc3a47ecd67be487bULL,0x741eb1870f47ed1cULL,0x99e66e58e7598b14ULL,+	0xebd6a6777b0ac93dULL,0xa6e37b0d78f5e0d7ULL,0x2516c09676f5492bULL,0x1e4bf8889ac05f3aULL,0xcdb42ce04df0ba2bULL,0x935d5cfd5062341bULL,0x8a30333382acac20ULL,0x429438c45198b00eULL,+	0xfb2838be67e573e0ULL,0x05891db94084c44bULL,0x9131137396c1c2c5ULL,0x6aebfa3fd958444bULL,0xac9cdce9e56e55c1ULL,0x7148ced32caa46d0ULL,0x2e10c7efb61fe8ebULL,0x9fd835daff97cf4dULL,+	0x6c626f56c1770616ULL,0x5351909e09da9a2dULL,0xe58e6825a3730e45ULL,0x9d8c8bc003ef0a79ULL,0x543f78b6056becfdULL,0x33f13253a090b36dULL,0x82ad4997794432f9ULL,0x1386493c4721f502ULL,+	0x3794eefa5abea82aULL,0x8dc611b993fe62d4ULL,0x69f1af37281ef606ULL,0x6af546c839839e69ULL,0x625578c7c977ec23ULL,0xa8de294cbd5c0576ULL,0xe2ddaf0f7cd1a4c0ULL,0x8243fc704f95f4d4ULL,+	0xe566f400b008733aULL,0xcba0697d512e1f57ULL,0x9537c2b240509cd0ULL,0x5f989c6957353d8cULL,0x7dbec9724c3c2b2fULL,0x90e02fa8ff031fa8ULL,0xf4d15c53cfd5d11fULL,0xb3404fae48314dfcULL,+	0xa36da109081e9387ULL,0xfb9780d78c935828ULL,0xd5940332e540b015ULL,0xc9d7b51be0f466faULL,0xfaadcd41d6d9f671ULL,0xba6c1e28b1a2ac17ULL,0x066a7833ed201e5fULL,0x19d99719f90f462bULL,+	0xf02cc3a9f327a07fULL,0xefb27a9b4490937dULL,0x81451e96b1b3afa5ULL,0x67e24de891883be4ULL,0x1ad65d4770869e54ULL,0xd36291a464a3856aULL,0x070a1abf7132e880ULL,0x9511d0a30e28dfdfULL,+	0xfdeed650f8d1cac4ULL,0xeb99194b6d16bda5ULL,0xb53b19f71cabbe46ULL,0x5f45af5039b9276cULL,0xd0784c6126ee9d77ULL,0xf7a1558b0c02ca5dULL,0xb61d6c59f032e720ULL,0xae3ffb95470cf3f7ULL,+	0x9b185facc72a4be5ULL,0xf66de2364d848089ULL,0xba14d07c717afea9ULL,0x25bfbfc02d551c1cULL,0x2cef0ecd4cdf3d88ULL,0x8cee2aa3647f73c4ULL,0xc10a7d3d722d67f7ULL,0x090037a294564a21ULL,+	0xe6567987fa9ced27ULL,0x36b2d8842a9e5dbcULL,0xf4bdeec6380d8e8cULL,0xb5e6a6b0dbc300d0ULL,0x7ba7e9b9592bef36ULL,0x2b373c4fee81b749ULL,0x484b5e01f0ce596bULL,0x7cc51c62c0bf54ccULL,+	0x6ac07bb84f3815c4ULL,0xddb9f6241aa9017eULL,0x31e30228ca85720aULL,0xe59d63f57cb75838ULL,0x69e18e777baad2d0ULL,0x2cfdb784d42f5d73ULL,0x025dd53df5774983ULL,0x2f80e7cee042cd52ULL,+	0x4bf56bafec695bb0ULL,0x22da1ca8f13c78adULL,0x0f9c4b131182abb0ULL,0x02ea555aae7d249eULL,0x868583f25e05d9e3ULL,0xe09cdcdf70382afaULL,0xdf072ec787080408ULL,0x0a317847cbf75658ULL,+	0x43f18d7f4d6ee4abULL,0xd3ac8cde9570c3dcULL,0x527e49070b8c9b2aULL,0x716709a7c5a4c0f1ULL,0x930852b0916a26b1ULL,0x3cc17fcf4e071177ULL,0x34f5e3d459694868ULL,0xee0341aba28f655dULL,+	0xf431f462060b5f61ULL,0xa56f46b47bd057c2ULL,0x348dca6c47e1bf65ULL,0x9a38783e41bcf1ffULL,0x7a5d33a9da710718ULL,0x5a7799872e0aeaf6ULL,0xca87314d2d29d187ULL,0xfa0edc3ec687d733ULL,+	0x1c894849cb198ac7ULL,0xa884a93d0f264665ULL,0x2da964ef9b200678ULL,0x3c351b87009834e6ULL,0xafb2ef9fe2c4b44bULL,0x580f6c473326790cULL,0xb84805210b02264aULL,0x8ba6f9e242a194e2ULL,+	0x39d934abd3c095f1ULL,0x04b261bee4b76d71ULL,0x1d2e6970e73e6984ULL,0x879fb23b5e5fcb11ULL,0x11506c72dfd75490ULL,0x3a97d08561bcf1c1ULL,0x43201d82bf5e7007ULL,0x7f0ac52f798232a7ULL,+	0xb25101fb319d7682ULL,0xb02931290a982feeULL,0x51c1c9b90261b344ULL,0x0e008c5bbfd371faULL,0xd866dd1c0278ca33ULL,0x666f76a6e5aa53b1ULL,0xe5cfb7796013a2cfULL,0x1d3a1aada3521836ULL,+	0x76b4131a567193ecULL,0xaf3c305ae5f6e70bULL,0x9587bd39031eebddULL,0x5709def871bbe831ULL,0x570599830eb2b669ULL,0x4d80ce1b875b7029ULL,0x838a7da80364ac16ULL,0x2f431d23be1c83abULL,+	0xe781276638235d4eULL,0x1c62bd67496e3298ULL,0x8378660c3f175bc8ULL,0x4d04e18917afdd4dULL,0x32a8160185a8068cULL,0xdb58e4e192b29a85ULL,0xe8a65b86c70d8a3bULL,0x5f0e6f4e98a0403bULL,+	0x2c2492b73f894ae0ULL,0xf59df3e5b75f18ceULL,0x7cb740d28f53cad0ULL,0x3eb585fbc4f01294ULL,0x17da0c8632c7f717ULL,0xeb8c795baf943f4cULL,0x4ee23fb5f67c51d2ULL,0xef18757568889949ULL,+	0x3ea011a4e822f0d0ULL,0xbc647ad15a8704f8ULL,0xbb315b3550c6820fULL,0x863dec3db7e76becULL,0x01ff5d3af017bfc7ULL,0x20054439976b8229ULL,0x067fca370bbd0d3bULL,0xf63dde647f5e3d0fULL,+	0x75d9bc15adf7cccfULL,0x81a3e5d6dfa1e1b0ULL,0x8c39e444249bc17eULL,0xf37dccb28ea7fd43ULL,0xda654873907fba12ULL,0x35daa6da4a372904ULL,0x0564cfc66283a6c5ULL,0xd09fa4f64a9395bfULL,+	0xb510b3b56aa39dffULL,0x59b43da29f8e4d8cULL,0xa8ce31fd9e4c4b9fULL,0x0e20be26c1303c01ULL,0x18187182e8ee47c9ULL,0xd9687cdb7db98101ULL,0x7a520e4da1e14ff6ULL,0x429808ba8836d572ULL,+	0x174d46996d16768eULL,0x9fc4ff6a628bf217ULL,0x77705a94154e490dULL,0x9d96dd288d2d997aULL,0x77e2d9d8ce5d72c4ULL,0x9d06c5a4c11c714fULL,0x02aa513679e4a03eULL,0x1386b3c2030ff28bULL,+	0x26a42d69ea40dc3aULL,0xdc84ad22aecc018fULL,0x25c36c7b3270f04aULL,0x46ba6d4750fa72edULL,0x6c37d1c593e58a8eULL,0xa2394731120c088cULL,0xc3be4263cb6e86daULL,0x2c417d367126d038ULL,+	0xcca523bb440e2229ULL,0x324673a273ef4d04ULL,0xaf3adf343e11ec39ULL,0x6136d7f1dc5968d3ULL,0x7a7b2899b053a927ULL,0x3eaa2661ae067ecdULL,0x8549b9c802779cd9ULL,0x061d7940c53385eaULL,+	0xe07141fcaaa2902bULL,0x539ad799e4f69ad3ULL,0xa6453f94813f9ffdULL,0xc58d3c48375bc2f7ULL,0xb3326fad5dc64e96ULL,0x3aafcaa9b240e354ULL,0x1d1b0903aca1e7a9ULL,0x4ceb97671211b8a0ULL,+	0x1eb4de4687032d58ULL,0xc54f3d835e2c79e0ULL,0x07818df45d04ef23ULL,0x55faa9c8673d41b4ULL,0xced64f6f89b95355ULL,0x4860d2eab7415c84ULL,0x5fdb9bd2050ebad3ULL,0xdb53e0cc6685a5bfULL,+	0x919fca5fabfae1caULL,0x937afaac1a21459bULL,0x9e0ca91c1f66a4d2ULL,0x194cc7f323ec1331ULL,0xad25143a8aa11690ULL,0xbe40ad8d09b59e08ULL,0x37d60d9be750860aULL,0x6c53b008c6bf434cULL,+	0x832d7080eb6b242dULL,0xd30bd0233b71e246ULL,0x7027991bbe31139dULL,0x68797e91462e4e53ULL,0x423fe20a6b4e185aULL,0x82f2c67e42d9b707ULL,0x25c817684cf7811bULL,0xbd53005e045bb95dULL,+	0xe5f649be9d8e68fdULL,0xdb0f05331b044320ULL,0xf6fde9b3e0c33398ULL,0x92f4209b66c8cfaeULL,0xe9d1afcc1a739d4bULL,0x09aea75fa28ab8deULL,0x14375fb5eac6f1d0ULL,0x6420b560708f7aa5ULL,+	0x9eae499c6254dc41ULL,0x7e2939247a837e7eULL,0x74aec08c090524a7ULL,0xf82b92198d6f55f2ULL,0x493c962e1402cec5ULL,0x9f17ca17fa2f30e7ULL,0xbcd783e8e9b879cbULL,0xea3d8c145a6f145fULL,+	0xdede15e75e0dee6eULL,0x74f24872dc628aa2ULL,0xd3e9c4fe7861bb93ULL,0x56d4822a6187b2e0ULL,0xb66417cfc59826f9ULL,0xca2609692408169eULL,0xedf69d06c79ef885ULL,0x00031f8adc7d138fULL,+	0x103c46e60ebcf726ULL,0x4482b8316231470eULL,0x6f6dfaca487c2109ULL,0x2e0ace9762e666efULL,0x3246a9d31f8d1f42ULL,0x1b1e83f1574944d2ULL,0x13dfa63aa57f334bULL,0x0cf8daed9f025d81ULL,+	0x30d78ea800ee11c1ULL,0xeb053cd4b5e3dd75ULL,0x9b65b13ed58c43c5ULL,0xc3ad49bdbd151663ULL,0x99fd8e41b6427990ULL,0x12cf15bd707eae1eULL,0x29ad4f1b1aabb71eULL,0x5143e74d07545d0eULL,+	0x30266336c88bdee1ULL,0x25f293065876767cULL,0x9c078571c6731996ULL,0xc88690b2ed552951ULL,0x274f2c2d852705b4ULL,0xb0bf8d444e09552dULL,0x7628beeb986575d1ULL,0x407be2387f864651ULL,+	0x0e5e3049a639fc6bULL,0xe75c35d986003625ULL,0x0cf35bd85dcc1646ULL,0x8bcaced26c26273aULL,0xe22ecf1db5536742ULL,0x013dd8971a9e068bULL,0x17f411cb8a7909c5ULL,0x5757ac98861dd506ULL,+	0x85de1f0d1e935abbULL,0xdefd10b4154de37aULL,0xb8d9e392369cebb5ULL,0x54d5ef9b761324beULL,0x4d6341ba74f17e26ULL,0xc0a0e3c878c1dde4ULL,0xa6d7758187d918fdULL,0x6687601502ca3a13ULL,+	0xc7313e9cf36658f0ULL,0xc433ef1c71f8057eULL,0x853262461b6a835aULL,0xc8f053987c86394cULL,0xff398cdfe983c4a1ULL,0xbf5e816203b7b931ULL,0x93193c46b7b9045bULL,0x1e4ebf5da4a6e46bULL,+	0xf9942a6043a24fe7ULL,0x29c1191effb3492bULL,0x9f662449902fde05ULL,0xc792a7ac6713c32dULL,0x2fd88ad8b737982cULL,0x7e3a0319a21e60e3ULL,0x09b0de447383591aULL,0x6df141ee8310a456ULL,+	0xaec1a039e6d6f471ULL,0x14b2ba0f1198d12eULL,0xebc1a1603aeee5acULL,0x401f4836e0b964ceULL,0x2ee437964fd03f66ULL,0x3fdb4e49dd8f3f12ULL,0x6ef267f629380f18ULL,0x3e8e96708da64d16ULL,+	0xbc19180c207674f1ULL,0x112e09a733ae8fdbULL,0x996675546aaeb71eULL,0x79432af1e101b1c7ULL,0xd5eb558fde2ddec6ULL,0x81392d1f5357753fULL,0xa7a76b973ae1158aULL,0x416fbbff4a899991ULL,+	0x9e65fdfd0d4a9dcfULL,0x7bc29e48944ddf12ULL,0xbc1a92d93c856866ULL,0x273c69056e98dfe2ULL,0x69fce418cdfaa6b8ULL,0x606bd8235061c69fULL,0x42d495a06af75e27ULL,0x8ed3d5056d873a1fULL,+	0xaf5528416ab25b6aULL,0xc6c0ffc72b1a4523ULL,0xab18827b21c99e03ULL,0x060e86489034691bULL,0x5207f90f93c7f398ULL,0x9f4a96cb82f8d10bULL,0xdd71cd793ad0f9e3ULL,0x84f435d2fc3a54f5ULL,+	0x4b03c55b8e33787fULL,0xef42f975a6384673ULL,0xff7304f75051b9f0ULL,0x18aca1dc741c87c2ULL,0x56f120a72d4bfe80ULL,0xfd823b3d053e732cULL,0x11bccfe47537ca16ULL,0xdf6c9c741b5a996bULL,+	0xee7332c7904fc3faULL,0x14a23f45c7e3636aULL,0xc38659c3f091d9aaULL,0x4a995e5db12d8540ULL,0x20a53becf3a5598aULL,0x56534b17b1eaa995ULL,0x9ed3dca4bf04e03cULL,0x716c563ad8d56268ULL,+	0x5043dea7e0f222c2ULL,0x309d42ac72e65142ULL,0x94fe9ddd9216cd30ULL,0xd6539c7d0f87feecULL,0x03c5a57c432ac7d7ULL,0x72692cf0327fda10ULL,0xec28c85f280698deULL,0x2331fb467ec283b1ULL,+	0xa0158eeae457a477ULL,0xd19857dbee6ddc05ULL,0xb326522418c41671ULL,0x3ffdfc7e3c2c0d58ULL,0x3a3a525426ee7cdaULL,0x341b0869df02c3a8ULL,0xa023bf42723bbfc8ULL,0x3d15002a14452691ULL,+	0xc961b2f687500b96ULL,0x795510e72dcd9425ULL,0x0308172978615433ULL,0xe5d0145465445029ULL,0x5bd13302bf690cbeULL,0x44e48831731eca67ULL,0x73306bc72b8038a4ULL,0x351d151ebf57bf02ULL,+	0x5ef7324c85edfa30ULL,0x2597655487d4f3daULL,0x352f5bc0dcb50c86ULL,0x8f6927b04832a96cULL,0xd08ee1ba55f2f94cULL,0x6a996f99344b45faULL,0xe133cb8da8aa455dULL,0x5d0721ec758dc1f7ULL,+	0xf3d44e1f9a876441ULL,0x82bc0c14147a818dULL,0x33ddc1170603eddeULL,0x77163f2e0a25f260ULL,0xb54c02caa3bfaa53ULL,0xbd1b2502e2256982ULL,0xc4e1728c1a6f37dcULL,0xfe36c213814c94a5ULL,+	0xf3cae7e9262a3539ULL,0x78a49d1d6670d59eULL,0x37de0f63c1c5e1b9ULL,0x3072c30c69cb7c1cULL,0x1d278a5277c850e6ULL,0x84f15f8f1f6a3de6ULL,0x46a8bb45592ca7adULL,0x1912e3eee4d424b8ULL,+	0xc1ffe2d490e31734ULL,0x91b1f1267fca007cULL,0x5459b1d0ae3f77e8ULL,0x262b051d46425c88ULL,0xcf5c8f765c51e274ULL,0x997481e2304e6146ULL,0x6fc1198bd84046b5ULL,0x1cb0a6bbe7f7a6bdULL,+	0x6ba7a92079e5fb67ULL,0xe1331feb70aa725eULL,0x5080ccf57df5d837ULL,0xe4cae01d7ff72e21ULL,0xd9243ee60412a77dULL,0x06ff7cacdf449025ULL,0xbe75f7cd23ef5a31ULL,0xbc9578220ddef7a8ULL,+	0xc4737ad11b7f30b0ULL,0x525ab2c63629dcf9ULL,0x963f4cc1186ae160ULL,0x8507671373e6b6e0ULL,0xd9be3180f6998bcfULL,0x93d91da3b1c8d8d8ULL,0xf902ce661b8c0054ULL,0x47e7924d74a8a768ULL,+	0xdc988086365e668bULL,0xada8dcdaaabda5fbULL,0xbc146b4c255f1fbeULL,0x9cfcde29cf34cfc3ULL,0xacbb453e7e85d1e4ULL,0x9ca09679f92358b5ULL,0x15fc2d96240823ffULL,0x8d65adf70c11d11eULL,+	0x323cfd177e19a46fULL,0x0948a7a786161156ULL,0x50d06b977e7d3363ULL,0x41c47ec1a702579dULL,0x9455998e59e9260bULL,0xc865e44446c24260ULL,0x393021ec13bc3744ULL,0x4981994ecd92a14aULL,+	0x8cf7230cb0ce1c55ULL,0x5b534d050bbfb607ULL,0xee1ef1130e16363bULL,0x27e0aa7ab4999e82ULL,0xce1dac2d79362c41ULL,0x67920c9091bb6cb0ULL,0x1e648d632223df24ULL,0x0f7d9eefe32e8f28ULL,+	0x9766f264d66f51c0ULL,0x644317ab5d9ccea5ULL,0xa39b37bcd721e232ULL,0x98bdde0bb9daf737ULL,0xc2ecc758165166bdULL,0x0951a285a2802108ULL,0xe39fbf24997aa66fULL,0x1a2f6862db62da27ULL,+	0x775557f10296f4fdULL,0x1dca76a3ea51b436ULL,0xf3e98f60fb950805ULL,0x31ff32ea831cf7f1ULL,0x643e7bf18d2c714bULL,0x64b5c3392e9d2acaULL,0xa9fd9ccc6adc2d23ULL,0xfc2397eccc721b9bULL,+	0xab651fc764465367ULL,0xe43870152b098f57ULL,0x0a91d519d382376fULL,0xb5afdb0a53cad929ULL,0x457e1875138d5523ULL,0xa92becae1aecacfbULL,0x0762f6e811484f49ULL,0x114b5c86dda16c2bULL,+	0x6943f39afa833834ULL,0x22951722a6328562ULL,0x81d63dd54170fc10ULL,0x9f5fa58faecc2e6dULL,0xb66c8725e77d9a3bULL,0x11235cea6384ebe0ULL,0x06a8c1185845e24aULL,0x0137b286ebd093b1ULL,+	0xdb567d6ac42bd6d2ULL,0x6df86468bb1f96aeULL,0x0efe5b1a4843b28eULL,0x961bbb056379b240ULL,0xb6caf5f070a6a26bULL,0x70686c0d328e6e39ULL,0x80da06cf895fc8d3ULL,0x804d8810b363fdc9ULL,+	0x63b99ce74462007dULL,0xb8ab48a54cb5f5b7ULL,0x9ec673d2f55edde7ULL,0xd1567f748cfaefdaULL,0x46381b6b0887bcecULL,0x694497cee178f3c2ULL,0x5e6525e31e6266cbULL,0x5931de26697d6413ULL,+	0x14e49da11f17a34cULL,0x5420ab39235a1456ULL,0xb76372412f50363bULL,0x7b15d623c3fabb6eULL,0xa0ef40b1e274e49cULL,0x5cf5074496b1860aULL,0xd6583fbf66afe5a4ULL,0x44240510f47e3e9aULL,+	0x142b55021a93507aULL,0xb4cd11878d3c06cfULL,0xdf70e76a91ec3f40ULL,0x484e81ad4e7553c2ULL,0x830f87b5272e9d6eULL,0xea1c93e5c6ff514aULL,0x67cc2adcc4192a8eULL,0xc77e27e242f4535aULL,+	0xb5358b1e48ac2840ULL,0x18311294ecba9477ULL,0xda58f990a6946b43ULL,0x3098baf99ab41819ULL,0x66c4c1584198da52ULL,0xab4fc17c146bfd1bULL,0x2f0a4c3cbf36a908ULL,0x2ae9e34b58cf7838ULL,+	0x45eb40ec0ccced58ULL,0x25cd4b9c0da44f98ULL,0x43e06458871812c6ULL,0x99f80d5516cef651ULL,0x571340c9ce6dc153ULL,0x138d5117d8665521ULL,0xacdb45bc4e07014dULL,0x2f34bb3884b60b91ULL,+	0x417499e84a34f239ULL,0x15fdb83cb90402d5ULL,0xb75f46bf433aa832ULL,0xb61e15af63215db1ULL,0xaabe59d4a127f89aULL,0x5d541e0c07e816daULL,0xaaba0659a618b692ULL,0x5532773317266026ULL,+	0x8cda9cf2d0c05199ULL,0x502fbc22fae78454ULL,0xc0bda9dff572a182ULL,0x5f9b71b86158b372ULL,0xe0f33a592b82dd07ULL,0x763027359523032eULL,0x7fe1a721c4505a32ULL,0x7b6e3e82f796409fULL,+	0x023c155d3f6effc7ULL,0x1fbd69ff9c90f0c7ULL,0xe5d7da8abeec2c5dULL,0x8813872bd7e86273ULL,0x9f3bc2c655f5e228ULL,0x11482869b0923b41ULL,0x65d75c741aa307caULL,0xda92c2577f24eee5ULL,+	0x26357732edc665d1ULL,0x9fb5b731a939ef1bULL,0x7db720fb94968089ULL,0x36f75f2c33138c52ULL,0xf8b793ec48d3cb97ULL,0x8dff1d456d261726ULL,0xfb65791b885c4ffbULL,0xf7c79e2df1a3a870ULL,+	0x08dd1028754c92e1ULL,0xca90b57acf0fef34ULL,0x1a9b84ac8af55919ULL,0xaa95e0e1ed93686bULL,0x46737315167021a4ULL,0x6cb6a0da20d5ff98ULL,0xecc4801a1092e706ULL,0xedcab23a3c5e61a6ULL,+	0xb4c66356ea37ba9eULL,0x1adc84150afff55dULL,0xf0080ef9596cc862ULL,0x756c85b86d647ab6ULL,0xc9db94aa1db9c215ULL,0x2dd36db12013b1a5ULL,0xde6ac61c4286c903ULL,0x3fd32f88c76cf884ULL,+	0x7f1290fca06d107eULL,0x697261fdb7661137ULL,0x1bb5be4e947b4b38ULL,0xb49826b63bb79130ULL,0x019ddfe85ba8bffbULL,0xb1af79007e3fa8e4ULL,0x72e1bdf201bcfe7fULL,0x2ed3ca8fd1169aeaULL,+	0x3befda8cd745fff1ULL,0x70b9e9b669b9924eULL,0xa5df48cfd1511381ULL,0x84f93fe2d06bc535ULL,0xaa42c5a9b279a6c3ULL,0x651da6c4d8f96132ULL,0xb0368c8b01b6aea5ULL,0x64e44c47ac7862a2ULL,+	0xe17a9947d9de99a8ULL,0xc2e61b2dc93477bdULL,0x57f684d41d19e287ULL,0x843c2122fe358135ULL,0xe2d3e2e904f7e8abULL,0xbf93ffe9b5f27aeeULL,0x29830d1d7b1858c4ULL,0xa8f449648106adbfULL,+	0xe3417bc035d0b34aULL,0x440b386b8327c0a7ULL,0x8fb7262dac0362d1ULL,0x2c41114ce0cdf943ULL,0x2ba5cef1ad95a0b1ULL,0xc09b37a867d54362ULL,0x26d6cdd201e486c9ULL,0x20477abf42ff9297ULL,+	0xa004dcb3292a9287ULL,0xddc15cf677b092c7ULL,0x083a8464806c0605ULL,0x4a68df703db997b0ULL,0x9c134e4505bf7dd0ULL,0xa4e63d398ccf7f8cULL,0xa6e6517f41b5f8afULL,0xaa8b9342ad7bc1ccULL,+	0xc41764717af715d2ULL,0xe2f7f594d0134a96ULL,0x2c1873efa41ec956ULL,0xe4e7b4f677821304ULL,0xe5c8ff9788d5374aULL,0x2b915e6380823d5bULL,0xea6bc755b2ee8fe2ULL,0x6657624ce7112651ULL,+	0xd6f800e07442f1d5ULL,0x475607d166e0e3abULL,0x82807f16b7c64047ULL,0x8858e1e3a749883dULL,0x5859120b8231ee10ULL,0x1b80e7eb638a1eceULL,0xcb72525ac6aa73a4ULL,0xa7cdea3d844423acULL,+	0x57477b11e51732d2ULL,0xdfd6eb282538fc0eULL,0x5c43b0cc3b39eec5ULL,0x6af12778cb36cc57ULL,0x70b0852d06c425aeULL,0x6df92f8c5c221b9bULL,0x6c8d4f9ece826d9cULL,0xf59aba7bb49359c3ULL,+	0xd2eb2cf152bfda05ULL,0xe0e4c4e96197b98cULL,0x1d35076cf8a1726fULL,0x6c06085b2db11e3dULL,0x15c0c4d74463ba14ULL,0x9d292f830030238cULL,0x1311ee8b3727536dULL,0xfeea86efbeaedc1eULL,+	0xa7f96054afa05dd8ULL,0x26dfcf21fcaf119eULL,0xe20ef2e30564bb59ULL,0xef4dca5061cb02b8ULL,0xcda7838a65d30672ULL,0x8b08d534fd657e86ULL,0x4c5b439546d595c8ULL,0x39b58725425cb836ULL,+	0xfda853931a62cc26ULL,0x23c69b9650c0e052ULL,0xa227df15bfc633f3ULL,0x2ac788481bae7d48ULL,0x487878f9187d073dULL,0x6c2be919967f807dULL,0x765861d8336e6d8fULL,0x88b8974cce528a43ULL,+	0xc37e2c2e421d3aa4ULL,0xf926407ce84fa840ULL,0x18abc03d1454e41cULL,0x26605ecd3f7af644ULL,0x242341a6d6a5eabfULL,0x1edb84f4216b668eULL,0xd836edb804010102ULL,0x5b337ce7945e1d8cULL,+	0x666ba2dccc78cf66ULL,0xb30181746fdbff77ULL,0x8d4dd0db168d4668ULL,0x259455d01dab3a2aULL,0xf58564c5cde3acecULL,0x7714192513adb276ULL,0x527d725d8a303f65ULL,0x55deb6c9e6f38f7bULL,+	0x864d05d73272d838ULL,0xe22924f9fa6295c5ULL,0x8189593f6c2fda32ULL,0x330d7189b184b544ULL,0x79efa62cbde1f714ULL,0x35771c94e5cb1a63ULL,0x2f4826b8641c8332ULL,0x00a894fbc8cee854ULL,+	0xdcdacd0a1058a318ULL,0x369cf3f578053a9aULL,0xc6c3de5031c68de2ULL,0x4653a5763c4b6d9fULL,0x1688dd5aaa4e5c97ULL,0x5be80aa1b7ab3c74ULL,0x70cefe7cbc65c283ULL,0x57f95f1306867091ULL,+	0xc240b6de34eaacdaULL,0xd9e116e82ba0f1deULL,0xcbe45ec779438e55ULL,0x91787c9d96f752d7ULL,0x897f532bf129ac2fULL,0xd307b7c85a36e22cULL,0x91940675749fb8f3ULL,0xd14f95d0157fdb28ULL,+	0x1625360416df4285ULL,0xb0c9babbd0c56ae2ULL,0x73032b19cfc5cfc3ULL,0xe497e5c309752056ULL,0x12096bb4164bda96ULL,0x1ee42419a0b74da1ULL,0x8fc36243403826baULL,0x0c8f0069dc09e660ULL,+	0xc44b74a15b0ec6f5ULL,0x47989fe45289b2b8ULL,0x745f848458d6fc73ULL,0xec362a6ff61c70abULL,0x070c98a7b3a8ad41ULL,0x73a20fc07b63db51ULL,0xed2c2173f44c35f4ULL,0x8a56149d9acc9dcaULL,+	0xa395c36f35d33ae7ULL,0x200ea12350bb5a94ULL,0x20c789bd0bafe84bULL,0x243ef52d0919276aULL,0x3934c577e23ae233ULL,0xb93807afa460d1ecULL,0xb72a53b1f8fa76a4ULL,0xd8914cb0c3ca4491ULL,+	0x4c076b86d23ddc82ULL,0x03fd344c7e0143f0ULL,0xa95362ff317af2c5ULL,0x0add3db7e18b7a4fULL,0x9c673e3f8260e01bULL,0xfbeb49e554a1cc91ULL,0x91351bf292f2e433ULL,0xc755e7ec851141ebULL,+	0x2bf5db47f23206d5ULL,0x2f6d34201d260152ULL,0x17b876533f8ff89aULL,0x5157c30c378fa458ULL,0x7517c5c52d4fb936ULL,0xef22f7ace6518cdcULL,0xdeb483e6bf847a64ULL,0xf508455892e0fa89ULL,+	0xab9659d8df7304d4ULL,0xb71bcf1bff210e8eULL,0xa9a2438bd73fbd60ULL,0x4595cd1f5d11b4deULL,0x9c0d329a4835859dULL,0x4a0f0d2d7dbb6e56ULL,0xc6038e5edf928a4eULL,0xc94296218f5ad154ULL,+	0x08a33840a70c6ec4ULL,0x9e8819f7e0311195ULL,0xed209d96708ab202ULL,0x10d7c4e7ce943a27ULL,0x372fb317a29b49a1ULL,0x57a67fb346627d1fULL,0xf912561e7cdf39ecULL,0xfa3ce6f26f7c8f17ULL,+	0x91213462f23f2d92ULL,0x6cab71bd60b94078ULL,0x6bdd0a63176cde20ULL,0x54c9b20cee4d54bcULL,0x3cd2d8aa9f2ac02fULL,0x03f8e617206eedb0ULL,0xc7f68e1693086434ULL,0x831469c592dd3db9ULL,+	0xfe7d7465653f3c5fULL,0x283dd45ef040feb1ULL,0x91fe599bf3b7edfeULL,0x5ff039ad80379311ULL,0xbf76995b4e96fa49ULL,0x2640b6b2a3e25094ULL,0x8b096341c1c83f74ULL,0xd2bec884fa560ac3ULL,+	0x8521df248f981354ULL,0x587e23ec3588a259ULL,0xcbedf281d7a0992cULL,0x06930a5538961407ULL,0x09320debbe5bbe21ULL,0xa7ffa5b52491817fULL,0xe6c8b4d909065160ULL,0xac4f3992fff6d2a9ULL,+	0x2e4eb2a3cc53da66ULL,0x04708a71f17a9b4eULL,0xdbfdc7b20de05b2cULL,0x4cdc9aee4907a201ULL,0xe5cc8dfc6c475566ULL,0x2b83cbfb47be1691ULL,0x695833a74c05c3fdULL,0xee938a7243b0deb0ULL,+	0x7aa7a1583ae9c1bdULL,0xe0af6d98e37ce240ULL,0xe54342d928ab38b4ULL,0xe8b750070a1c98caULL,0xefce86afe02358f2ULL,0x31b8b856ea921228ULL,0x052a19120a1c67fcULL,0xb4069ea4e3aead59ULL,+	0x3b826205c6ffcfe6ULL,0x2eb31256aa39418aULL,0xc4b4a9e1c18521fbULL,0x48614dd8db610615ULL,0x04c362bbeea7475fULL,0x916ab969a8ead162ULL,0x6a7975bc0310a876ULL,0xea4e7d11ecc77ec4ULL,+	0x3232d6e27fa03cb3ULL,0xdb938e5b0fdd7d88ULL,0x04c1d2cd2ccbfc5dULL,0xd2f45c12af3a580fULL,0x592620b57883e614ULL,0x5fd27e68be7c5f26ULL,0x139e45a91567e1e3ULL,0x2cc71d2d44d8aaafULL,+	0x6ef493c706ba8e6fULL,0x05c07bf8123deffdULL,0xcebf9b4d9c2b5a4fULL,0xf958147e2d038e9cULL,0x5af5cee03e5561c4ULL,0x8d0b5a7a794a6afcULL,0x9de22df13772168bULL,0xe5d249c5d84097fdULL,+	0x4a9090cde36d0757ULL,0xf722d7b1d9a29382ULL,0xfb7fb04c04b48ddfULL,0x628ad2a7ebe16f43ULL,0xcd3fbfb520226040ULL,0x6c34ecb15104b6c4ULL,0x30c0754ec903c188ULL,0xec336b082d23cab0ULL,+	0x74c70b4295c69248ULL,0x8813259dfed90303ULL,0xa3e330684b8cc87aULL,0xe689371c111a7a95ULL,0x52bfbbf7fbbbc20bULL,0x73a8543d65f9a6e2ULL,0x67bb2fdd7e413e6dULL,0xa066b3efc5cf2032ULL,+	0x473d62a21e206ee5ULL,0xf1e274808c49a633ULL,0x87ab956ce9f6b2c3ULL,0x61830b4862b606eaULL,0x67cd6846e78e815fULL,0xfe40139f4c02082aULL,0x52bbbfcb952ec365ULL,0x74c116426b9836abULL,+	0xcf61b89c44f48971ULL,0xe2d700f76d660683ULL,0x72ef285cd1d431bdULL,0x0593b24e9bdebf4aULL,0x4084bc5b0561f8a1ULL,0x84ce74d0aa16f256ULL,0x9cbc79d309f6d277ULL,0xa94fe2fe4139bdfeULL,+	0x9f51439e558df019ULL,0x230da4baac712b27ULL,0x518919e355185a24ULL,0x4dcefcdd84b78f50ULL,0xa7d90fb2a47d4c5aULL,0x55ac9abfb30e009eULL,0xfd2fc35974eed273ULL,0xb72d824cdbea8fafULL,+	0x549db2b5ef7d9289ULL,0x2480d4a8197f015aULL,0x61d5590bc40493b6ULL,0x3a55b52e6f780331ULL,0x40eb8115309eadb0ULL,0xdea7de5a92e5c625ULL,0x64d631f0cc6a3d5aULL,0x9d5e9d7c93e8dd61ULL,+	0x196860411e84e0e5ULL,0xa5db84d3aea34c93ULL,0xf9d5bb197073a732ULL,0xb8d2fe566bcfd7c0ULL,0x45775f36f3eb82faULL,0x8cb20cccfdff8b58ULL,0x1659b65f8374c110ULL,0xb8b4a422330c789aULL,+	0xc925ff87aedbae9fULL,0x7daf0eb936880a54ULL,0x9284ddf59c4d0e71ULL,0x1581cf93316f8cf5ULL,0x3eeca8873ac1f452ULL,0xb417fce9fb6aeffeULL,0xa5918046eefb8dc3ULL,0x73d318ac02209400ULL,+	0xc4f4cda3af2ebc2fULL,0xa0af843dcb4efe24ULL,0x53b857c19ccd10b1ULL,0xddc9d1eb914d3e04ULL,0x7bdec8bb62771debULL,0x829277aa91c5aa81ULL,0x7af18dd6832391aeULL,0x1740f316c71a84caULL,+	0xdd4a12d8e12b31f8ULL,0x577e29bc177736e6ULL,0x2353722ba88935e8ULL,0xca1d3729015f286dULL,0x86c7b6a239a3e035ULL,0x6e5250bfd3b03a9fULL,0x79d98930fd0d536eULL,0x8c4cbbabfa0c3832ULL,+	0x2d500910cab91f1eULL,0xbedd9e444d1cd216ULL,0xd634b74fedd02252ULL,0xbd60f8e11258617aULL,0xd8c7537b9e05614aULL,0xfd26c766e7af5fc5ULL,0x0660b581582bd926ULL,0x87019244acf07fc8ULL,+	0xafa26ccfaf64ecb6ULL,0xe6054f974bd72775ULL,0xbbcbab5b140f695aULL,0xbc71b4a4e348efdeULL,0xfc2e52becc96d963ULL,0x150abf5f5e5d9018ULL,0xbd182fa604568771ULL,0x35b4c06170339f83ULL,+	0x8928e99aeeaf8c49ULL,0xee7aa73d6e24d728ULL,0x4c5007c2e72b156cULL,0x5fcf57c5ed408a1dULL,0x9f719e39b6057604ULL,0x7d343c01c2868bbfULL,0x2cca254b7e103e2dULL,0xe6eb38a9f131bea2ULL,+	0xc624a04e5f40ff52ULL,0x3611d7cffcd2914aULL,0xb7e8b42b1fd3bfd6ULL,0x6cde40fdfa85063aULL,0x7811c449178b7e5bULL,0x4cb609312972cc13ULL,0xf579125e33b46135ULL,0xca102ef788a4e56eULL,+	0x0ba4e3520a981b0dULL,0x1c354cb3bd1a41a4ULL,0x1aabaa3adf9fab9cULL,0x0701a7d153c418d5ULL,0xdd1a7cefdcf2b921ULL,0x6ceef0b3bcf48061ULL,0x1083b598de25cce6ULL,0x890a54c7e90a5e34ULL,+	0xc535956640ac1807ULL,0xd4c1e56684da0b1eULL,0xc4b33f97e0b82421ULL,0xd0bd23177b41be00ULL,0x53a4b42e147b72e1ULL,0xf8d39f5ff777104cULL,0x36e64e64d3fb530dULL,0xa7a6ba6756074fddULL,+	0x405718db4f6d01b1ULL,0xe73c6bc28f11e8a0ULL,0xac11bb8ca0591a3bULL,0x12d09a5a0acc4531ULL,0xcbf174eee7de13f4ULL,0x177e2be6044fd682ULL,0x65f574cb1c48af70ULL,0xce5966929961cb7cULL,+	0x989fe84ebc6fab9fULL,0xe70ce6b1c80f6474ULL,0xbe9ff3053b02a1bcULL,0x12ef486699c0afd3ULL,0x22d957f9e3411a26ULL,0x0b41d8817b485b98ULL,0x820e56d04ae20119ULL,0x81e3d01f328528e0ULL,+	0xc59eed6c048752a1ULL,0x41f2702ea01341b4ULL,0x6e35903b9dc6b092ULL,0x4291aba81f5b5b23ULL,0x8173aa70a653d61dULL,0xd1b648d44f2eb51eULL,0x31b7ce065ab93f8fULL,0xa55408ee99e2f4feULL,+	0x1fa4ed0985e34160ULL,0xa26d7dc37a03cde2ULL,0x1dac0848fa84df2dULL,0x5e9a28d61b697108ULL,0xa88004d914ea0ea1ULL,0xa8d154283ffe5520ULL,0x4f422dae639b139cULL,0xeedccc0dd4b5a861ULL,+	0xb33e624f8be762b4ULL,0x2a9ee4d1058e3413ULL,0x968e636967d805faULL,0x9848949b7db8bfd7ULL,0x5308d7e5d23a8417ULL,0x892f3b1df3e29da5ULL,0xc95c139e3dee471fULL,0x8631594dd757e089ULL,+	0x1083e2ea1f095615ULL,0x0a28ad7714e68c33ULL,0x6bfc02523d8818beULL,0xb585113af35850cdULL,0x7d935f0b30df8aa1ULL,0xaddda07c4ab7e3acULL,0x92c34299552f00cbULL,0xc33ed1de2909df6cULL,+	0x2dc40d483e07113cULL,0x6e4a5d397d8b63aeULL,0x5582a94b79684c2bULL,0x932b33d4622da26cULL,0xf534f6510dbbf08dULL,0x211d07c964c23a52ULL,0x0eeece0fee5bdc9bULL,0xdf178168f7015558ULL,+	0xabe7905a83cdd60eULL,0x50602fb5a1170184ULL,0x689886cdb023642aULL,0xd568d090a6e1fb00ULL,0x5b1922c70259217fULL,0x93831cd9c43141e4ULL,0xdfca35870c95f86eULL,0xdec2057a568ae828ULL,+	0x568f8925913cc16dULL,0x18bc5b6de1a26f5aULL,0xdfa413bef5f499aeULL,0xf8835decc3f0ae84ULL,0xb6e60bd865a40ab0ULL,0x65596439194b377eULL,0xbcd8562592084a69ULL,0x5ce433b94f23ede0ULL,+	0x860d523d42e06189ULL,0xbf0779414e3aff13ULL,0x0b616dcac1b20650ULL,0xe66dd6d12131300dULL,0xd4a0fd67ff99abdeULL,0xc9903550c7aac50dULL,0x022ecf8b7c46b2d7ULL,0x3333b1e83abf92afULL,+	0xc0da65e784d6365dULL,0xbcb7443f8f759fb8ULL,0x35c712b17ae81930ULL,0x80428dff4c6e08abULL,0xf19dafefa4faf843ULL,0xced8538dffa9855fULL,0x20ac409cbe3ac7ceULL,0x358c1fb6882da71eULL,+	0xefecdef7be42a582ULL,0xd3fc608065046be6ULL,0xc9af13c809e8dba9ULL,0x1e6c9847641491ffULL,0x3b574925d30c31f7ULL,0xb7eb72baac2a2122ULL,0x776a0dacef0859e7ULL,0x06fec31421900942ULL,+	0x324794b07e50122bULL,0xdd744f8b4af07ca5ULL,0x30a12f08d63fc97bULL,0x39650f1a76626d9dULL,0x101b47f71fa38477ULL,0x3d815f19d4dc124fULL,0x1569ae95b26eb58aULL,0xc3cde18895fb1887ULL,+	0x7ec62fbbf4737f21ULL,0xd8dba5ab6209f5acULL,0x24b5d7a9a5f9adbeULL,0x707d28f7a61dc768ULL,0x7711460bcaa999eaULL,0xba7b174d1c92e4ccULL,0x3c4bab6618d4bf2dULL,0xb8f0c980eb8bd279ULL,+	0x9d658932790691bfULL,0xed61058906b736aeULL,0x712c2f04c0d63b6eULL,0x5cf06fd5c63d488fULL,0x97363facd9588e41ULL,0x1f9bf7622b93257eULL,0xa9d1ffc4667acaceULL,0x1cf4a1aa0a061ecfULL,+	0x28d675b2c0519a23ULL,0x9ebf94fe4f6952e3ULL,0xf28bb767a2294a8aULL,0x85512b4dfe0af3f5ULL,0x18958ba899b16a0dULL,0x95c2430cba7548a7ULL,0xb30d1b10a16be615ULL,0xe3ebbb9785bfb74cULL,+	0x07b53f5eb2e63645ULL,0xbe57e54784c84232ULL,0xd779c2167214d5cfULL,0x617969cd029a3acaULL,0xd17668cd8a7017a0ULL,0x77b4d19abe9b7ee8ULL,0x58fd0e939c161776ULL,0xa8c4f4efd5968a72ULL,+	0x81eeb865d2fdca23ULL,0x5a15ee08cc8ef895ULL,0x768fa10a01905614ULL,0xeff5b8ef880ee19bULL,0xf0c0cabbcb1c8a0eULL,0x2e1ee9cdb8c838f9ULL,0x0587d8b88a4a14c0ULL,0xf6f278962ff698e5ULL,+	0x519d34b3bf44da80ULL,0x283834f95ab32e66ULL,0x6e6087976278a000ULL,0x1e62960e627312f6ULL,0x9b87b27be6901c55ULL,0x80e7853824fdbc1fULL,0xbbbc09512facc27dULL,0x06394239ac143b5aULL,+	0x9c4b646e9e2fce99ULL,0x68a210811e80857fULL,0x06d54e443643b52aULL,0xde8d6d630d8eb843ULL,0x7032156342146a0aULL,0x8ba826f25eaa3622ULL,0x227a58bd86138787ULL,0x43b6c03c10281d37ULL,+	0x02b37a952f41deffULL,0x0e44a59ae63b89b7ULL,0x673257dc143ff951ULL,0x19c02205d752baf4ULL,0x46c23069c4b7d692ULL,0x2e6392c3fd1502acULL,0x6057b1a21b220846ULL,0xe51ff9460c1b5b63ULL,+	0x6e85cb51566c5c43ULL,0xcff9c9193597f046ULL,0x9354e90c4994d94aULL,0xe0a393322147927dULL,0x8427fac10dc1eb2bULL,0x88cfd8c22ff319faULL,0xe2d4e68401965274ULL,0xfa2e067d67aaa746ULL,+	0xb6d92a7f3e5f9f11ULL,0x9afe153ad6cb3b8eULL,0x4d1a6dd7ddf800bdULL,0xf6c13cc0caf17e19ULL,0x15f6c58e325fc3eeULL,0x71095400a31dc3b2ULL,0x168e7c07afa3d3e7ULL,0x3f8417a194c7ae2dULL,+	0xec234772813b230dULL,0x634d0f5f17344427ULL,0x11548ab1d77fc56aULL,0x7fab1750ce06af77ULL,0xb62c10a74f7c4f83ULL,0xa7d2edc4220a67d9ULL,0x1c404170921209a0ULL,0x0b9815a0face59f0ULL,+	0x2842589b319540c3ULL,0x18490f59a283d6f8ULL,0xa2731f84daae9fcbULL,0x3db6d960c3683ba0ULL,0xc85c63bb14611069ULL,0xb19436af0788bf05ULL,0x905459df347460d2ULL,0x73f6e094e11a7db1ULL,+	0xdc7f938eb6357f37ULL,0xc5d00f792bd8aa62ULL,0xc878dcb92ca979fcULL,0x37e83ed9eb023a99ULL,0x6b23e2731560bf3dULL,0x1086e4591d0fae61ULL,0x782483169a9414bdULL,0x1b956bc0f0ea9ea1ULL,+	0x7b85bb91c31b9c38ULL,0x0c5aa90b48ef57b5ULL,0xdedeb169af3bab6fULL,0xe610ad732d373685ULL,0xf13870df02ba8e15ULL,0x0337edb68ca7f771ULL,0xe4acf747b62c036cULL,0xd921d576b6b94e81ULL,+	0xdbc864392c422f7aULL,0xfb635362ed348898ULL,0x83084668c45bfcd1ULL,0xc357c9e32b315e11ULL,0xb173b5405b2e5b8cULL,0x7e946931e102b9a4ULL,0x17c890eb7b0fb199ULL,0xec225a83d61b662bULL,+	0xf306a3c8ee3c76cbULL,0x3cf11623d32a1f6eULL,0xe6d5ab646863e956ULL,0x3b8a4cbe5c005c26ULL,0xdcd529a59ce6bb27ULL,0xc4afaa5204d4b16fULL,0xb0624a267923798dULL,0x85e56df66b307fabULL,+	0x0281893c2bf29698ULL,0x91fc19a4d7ce7603ULL,0x75a5dca3ad9a558fULL,0x40ceb3fa4d50bf77ULL,0x1baf6060bc9ba369ULL,0x927e1037597888c2ULL,0xd936bf1986a34c07ULL,0xd4cf10c1c34ae980ULL,+	0x3a3e5334859dd614ULL,0x9c475b5b18d0c8eeULL,0x63080d1f07cd51d5ULL,0xc9c0d0a6b88b4326ULL,0x1ac98691c234296fULL,0x2a0a83a494887fb6ULL,0x565114270cea9cf2ULL,0x5230a6e8a24802f5ULL,+	0xf7a2bf0f72e3d5c1ULL,0x377174464f21439eULL,0xfedcbf259ce30334ULL,0xe0030a787ce202f9ULL,0x6f2d9ebf1202e9caULL,0xe79dde6c75e6e591ULL,0xf52072aff1dac4f8ULL,0x6c8d087ebb9b404dULL,+	0xad0fc73dbce913afULL,0x909e587b458a07cbULL,0x1300da84d4f00c8aULL,0x425cd048b54466acULL,0xb59cb9be90e9d8bfULL,0x991616db3e431b0eULL,0xd3aa117a531aecffULL,0x91af92d359f4dc3bULL,+	0x9b1ec292e93fda29ULL,0x76bb6c17e97d91bcULL,0x7509d95faface1e6ULL,0x3653fe47be855ae3ULL,0x73180b280f680e75ULL,0x75eefd1beeb6c26cULL,0xa4cdf29fb66d4236ULL,0x2d70a9976b5821d8ULL,+	0x7a3ee20720445c36ULL,0x71d1ac8259877174ULL,0x0fc539f7949f73e9ULL,0xd05cf3d7982e3081ULL,0x8758e20b7b1c7129ULL,0xffadcc20569e61f2ULL,0xb05d3a2f59544c2dULL,0xbe16f5c19fff5e53ULL,+	0x73cf65b8aad58135ULL,0x622c2119037aa5beULL,0x79373b3f646fd6a0ULL,0x0e029db50d3978cfULL,0x8bdfc43794fba037ULL,0xaefbd687620797a6ULL,0x3fa5382bbd30d38eULL,0x7627cfbf585d7464ULL,+	0xb2330fef4e4ca463ULL,0xbcef72873566cc63ULL,0xd161d2cacf780900ULL,0x135dc5395b54827dULL,0x638f052e27bf1bc6ULL,0x10a224f007dfa06cULL,0xe973586d6d3321daULL,0x8b0c573826152c8fULL,+	0x9910ba6b23a5d896ULL,0x1fe19e357fe4364eULL,0x6e1da8c39a33c677ULL,0x15b4488b29fd9fd0ULL,0x1f4392541a1f22bfULL,0x920a8a70ab8163e8ULL,0x3fd1b24907e5658eULL,0xf2c4f79cb6ec839bULL,+	0x8b5c619c76497ee8ULL,0x5d2b0ac6c717370eULL,0x98204cb64fcf68e1ULL,0x0bdec21162bc6792ULL,0x6973ccefa63b1011ULL,0xf9e3fa97e0de1ac5ULL,0x5efb693e3d0e0c8bULL,0x037248e9d2d4fcb4ULL,+	0xd3694e2ab20364e4ULL,0x62699718e770b20dULL,0x6183291b6ed77d1cULL,0x69aada7f6d6180a5ULL,0x51f9054bf185509bULL,0xfd7e845678701077ULL,0xd6a2308a7fb96d8dULL,0xd53e48d228dc87eeULL,+	0x80802dc91ec34f9eULL,0xd8772d3533810603ULL,0x3f06d66c530cb4f3ULL,0x7be5ed0dc475c129ULL,0xcb9e3c1931e82b10ULL,0xc63d2857c9ff6b4cULL,0xb92118c692a1b45eULL,0x0aec44147285bbcaULL,+	0x37071afbe8316c45ULL,0x982be4fd46700b8bULL,0x8d5d177c64ff8578ULL,0x5ec40582c9a82fa7ULL,0x5518e37bcfa86678ULL,0x24e809f49f031284ULL,0x312f39604bbbb74cULL,0xad4b4f6fc0c14de6ULL,+	0xa3d6f4868eb6e843ULL,0x6415834bac4ab3abULL,0x028a81514f3cf2dcULL,0xf3b4962f5f3e6c3eULL,0x9119ae90987dd2f2ULL,0x437a6d8ae10bce55ULL,0xc31cdd6b9b149ed6ULL,0x1b77791d06871332ULL,+	0x9c34b650e16e05e9ULL,0x965a774094e74640ULL,0xa3fd22fbcea3f029ULL,0x1eb6a9688f95277cULL,0x2520a63d7bad84f6ULL,0xad917201f58f2feeULL,0xea92c1669b840d48ULL,0x12109c4aacef5cbdULL,+	0xfc189ae71e29a3efULL,0xcbe906f04c93302eULL,0xd0107914ceaae10eULL,0xb7a23f34b68e19f8ULL,0xe9d875c2efd2119dULL,0x03198c6efcadc9c8ULL,0x65591bf64da17113ULL,0x3cf0bbf83d443038ULL,+	0xab293027aad991c1ULL,0x598d0bf8849be4b7ULL,0x8c94a21ab972da90ULL,0xada4cfdd7ecfa840ULL,0x93d4b9c0fbcec63aULL,0x7ca617a203219a34ULL,0x900424eb6a652a55ULL,0xaf9346e9eb8562e0ULL,+	0xc3e04d7902381461ULL,0xb1643ab5911bc478ULL,0xc92becfa390b3ef2ULL,0x54476778acd2f1b6ULL,0x8daa0c4d66bf3aafULL,0x2bc1287b2c21c65aULL,0xee182910b5a13ac3ULL,0xbb04730090b0790aULL,+	0x83766947d17d4e0bULL,0xc5772beefdc3a47bULL,0x765a50db1a6fd0ffULL,0x17f904ba45b0995eULL,0xcee643832883487eULL,0xf56db7f3c270aaedULL,0x6738d94f46cb1fd9ULL,0xc8fa426a142fd4d5ULL,+	0xae485bb72b724759ULL,0x945353e1b2d4c63aULL,0x82159d07de7d6f2cULL,0x389caef34ec5b109ULL,0x4a8ebb53db65ef14ULL,0x2dc2cb7edd99de43ULL,0x816fa3ed83f2405fULL,0x73429bb9c14208a3ULL,+	0x08ed8febd56daf06ULL,0x8d98277b4a837f69ULL,0x9947c636a9b6e05aULL,0x58c8a77ac0d58abdULL,0xf45496a45f121e4fULL,0x16cd67c71076d3d3ULL,0xecbd1958e3fb0c5dULL,0xfbe185ec38e1eb47ULL,+	0xb7ef9760ff79d2eeULL,0xdd4d06aff39e7832ULL,0xfd025001b905b499ULL,0x98fe1c61f5b61d31ULL,0xa9f83980c5f12805ULL,0x376e3b783009cd9aULL,0xa322b09f514eb16dULL,0x08e213122c3832dfULL,+	0xaea68626dc4ad4f4ULL,0x5dc516824ddbc0b6ULL,0xa76697bd602e9065ULL,0xbeeb3ea58c37888eULL,0x1ec4a2f214569113ULL,0xe48b820ca35f4484ULL,0x9fb560949ae44df2ULL,0x6ca1346292cc09fdULL,+	0xb618d590b01e6e27ULL,0x047e2ccde180b2dcULL,0xd1b299b504aea4a9ULL,0x412c9e1e9fa403a4ULL,0x88d28a3679407552ULL,0x49c50136f332b8e3ULL,0x3a1b6fcce668de19ULL,0x178851bc75122b97ULL,+	0x197dd46d95a7b1a2ULL,0x9c4e7ad63c6341fbULL,0x426eca29484c2eceULL,0x9211e489de7f4f8aULL,0x14997f6ec78ef1f4ULL,0x2b2c091006574586ULL,0x17286a6e1c3eede8ULL,0x25f92e470f60e018ULL,+	0xb4e370af3aeac968ULL,0xe4f7fee9c4b63266ULL,0xb4acd4c2e3ac5664ULL,0xf8910bd2ceb38cbfULL,0x1c3ae50cc9c0726eULL,0x15309569d97b40bfULL,0x70884b7ffd5a5a1bULL,0x3890896aef8314cdULL,+	0x090d7d205ffe7b37ULL,0x3b7f3efb1747d2daULL,0xa2cb525fb54fc519ULL,0x6e220932f66a971eULL,0xddc160dfb486d440ULL,0x7fcfec463fe13465ULL,0x83da7e4e76e4c151ULL,0xd6fa48a1d8d302b5ULL,+	0x5ced3c9f82e4c634ULL,0x8efb83143a4464f8ULL,0xe706381b7a1dca25ULL,0x6cd15a3c5a2a412bULL,0x9347a8fdbfcd8fb5ULL,0x31db2eef6e54cd22ULL,0xc4aeb11ef8d8932fULL,0x11e7c1ed344411afULL,+	0xae4065ef12045cf9ULL,0x6fcb2caf9ccce8bdULL,0x1fa0ba4ef2cf6525ULL,0xf683125dcb72c312ULL,0xa01da4eae312410eULL,0x67e286776cd8e830ULL,0xabd9575298fb3f07ULL,0x05f11e11eef649a5ULL,+	0x996884f5903fa271ULL,0xe6da0fd2b9da921eULL,0xa6f2f2695db01e54ULL,0x1ee3e9bd6876214eULL,0xa26e181ce27a9497ULL,0x36d254e48e215e04ULL,0x42f32a6c252cabcaULL,0x9948148780b57614ULL,+	0xea961058fa28a7e0ULL,0xc726cf250bf5ec74ULL,0xe74d55c8db229666ULL,0x0bd9abbfa57f5799ULL,0x7479ef074dfc47b3ULL,0xd9c65fc30c52f91dULL,0x8e0283fe36a8bde2ULL,0xa32a8b5e7d4b7280ULL,+	0xab41b43a43228d83ULL,0x24ae1c304ad63f99ULL,0x8e525f1a46a51229ULL,0x14af860fcd26d2b4ULL,0xd6baef613f714aa1ULL,0xf51865adeb78795eULL,0xd3e21fcee6a9d694ULL,0x82ceb1dd8a37b527ULL,+	0x8605d27d48307682ULL,0x745aaba3e10566daULL,0xe57cae36bff2d7abULL,0x91332ba14b127823ULL,0xcb5c3638f3429f43ULL,0x43a21c4fec462929ULL,0xe9bc95352b18b7bdULL,0x8c2addf3e78cb0c6ULL,+	0x4a665bfd2f9fd51aULL,0x7f2f1fe2481b97f7ULL,0xcad05d69ad36ce50ULL,0x314fc2a4844f4dedULL,0xd5593d8cb55fc5c6ULL,0xe3510ce8bfb1e23dULL,0xf9b7be6937453cceULL,0xd3541b7969fae631ULL,+	0xb013cbcad7154cedULL,0x949b28573f52651aULL,0x03b41f6e9f5e642eULL,0xc3a3462986ed94a4ULL,0xf3c86cd6222b24dcULL,0x7578fe8a028c9c26ULL,0xaac7bfa12edad3b6ULL,0x4112c5d78847940aULL,+	0x99296525eca445dfULL,0xf1af24f22cdfa4c6ULL,0xf5b4eb61eba6d3bcULL,0x4560910c98972cc7ULL,0x54751c32093eaa32ULL,0x018313497d3c67bbULL,0x3bd90ce62d871110ULL,0x75fc863a538baa7eULL,+	0x8fe1f8b64ae3a278ULL,0x160c5306137cdf65ULL,0x22f029e733be0492ULL,0x79a680427a75cd82ULL,0x1d8c094a5b3f3adaULL,0x5d723bbe165c3250ULL,0x08b958ffa5792e22ULL,0x829fa986b11c1eaaULL,+	0x711b8a4176a9f05dULL,0x06ca4e4b9011d488ULL,0x543bc62ba248a65eULL,0x017535ffc9290894ULL,0x840b84ce406851d7ULL,0xafa3acdf90e960b4ULL,0xac3394af7128fd34ULL,0x54eb4d5b2ac0f92cULL,+	0x923ea73e4a14f836ULL,0xc8cc8c57f0946328ULL,0xce3f117fe917a4dfULL,0x6372f933f72ce929ULL,0x75000249c29ba567ULL,0xcbd437e68c829ccaULL,0x6c63aaabdd02ec7aULL,0xe9b2f90c7b42bd17ULL,+	0xdb09e87355dbd4b3ULL,0x1f8799286639bbb1ULL,0xb83e47e51c651962ULL,0xd4ef0fb6c43fb574ULL,0x27d3b9d8f1bfb12aULL,0x6ab877e86e5e8b72ULL,0x8eebdc9d157b9014ULL,0x4c2110053aa5cb64ULL,+};
cbits/p256/p256_ec.c view
@@ -33,8 +33,18 @@ // See http://www.imperialviolet.org/2010/12/04/ecc.html ([1]) for background.  #include "p256/p256_gf.h"+#include "crypton_bzero.h" +#ifdef CRYPTON_S2N_BIGNUM+#include "p256/p256_s2n.h"+#include "p256/p256_verify.h"+/* the memcpy below is the two representations being the same thing */+#if P256_BITSPERDIGIT != 64 || P256_NDIGITS != 4+#error "CRYPTON_S2N_BIGNUM wants the 64-bit crypton_p256_int"+#endif+#endif + /* Field element operations: */  /* felem_inv calculates |out| = |in|^{-1}@@ -43,69 +53,95 @@  *   a^p = a (mod p)  *   a^{p-1} = 1 (mod p)  *   a^{p-2} = a^{-1} (mod p)- */+ *+ * The exponent is built left to right from the shape of p - 2, which for+ * this prime is+ *+ *   ffffffff 00000001 00000000 00000000 00000000 ffffffff ffffffff fffffffd+ *   \__32 ones__/ \_31 zeros, one 1_/ \______ 96 zeros ______/ \_94 ones, 0, 1_/+ *+ * A run of k zeros is k squarings; a run of k ones is k squarings and one+ * multiplication by a^(2^k - 1), which is why the powers below are kept.  The+ * whole chain is 255 squarings, which is the least an exponent of 256 bits+ * can be done in, and 13 multiplications.+ *+ * The chain this replaces built the low 94 ones in a second accumulator and+ * multiplied the two at the end, which cost 32 squarings more than the 255. */ static void felem_inv(felem out, const felem in) {-  felem ftmp, ftmp2;-  /* each e_I will hold |in|^{2^I - 1} */-  felem e2, e4, e8, e16, e32, e64;+  felem ftmp, x2, x4, x8, x16, x32;   unsigned i; -  felem_square(ftmp, in); /* 2^1 */-  felem_mul(ftmp, in, ftmp); /* 2^2 - 2^0 */-  felem_assign(e2, ftmp);-  felem_square(ftmp, ftmp); /* 2^3 - 2^1 */-  felem_square(ftmp, ftmp); /* 2^4 - 2^2 */-  felem_mul(ftmp, ftmp, e2); /* 2^4 - 2^0 */-  felem_assign(e4, ftmp);-  felem_square(ftmp, ftmp); /* 2^5 - 2^1 */-  felem_square(ftmp, ftmp); /* 2^6 - 2^2 */-  felem_square(ftmp, ftmp); /* 2^7 - 2^3 */-  felem_square(ftmp, ftmp); /* 2^8 - 2^4 */-  felem_mul(ftmp, ftmp, e4); /* 2^8 - 2^0 */-  felem_assign(e8, ftmp);+  /* x{k} holds in^(2^k - 1), a run of k ones. */+  felem_square(ftmp, in);+  felem_mul(x2, ftmp, in); /* 2^2 - 1 */++  felem_square(ftmp, x2);+  felem_square(ftmp, ftmp);+  felem_mul(x4, ftmp, x2); /* 2^4 - 1 */++  felem_assign(ftmp, x4);+  for (i = 0; i < 4; i++) {+    felem_square(ftmp, ftmp);+  }+  felem_mul(x8, ftmp, x4); /* 2^8 - 1 */++  felem_assign(ftmp, x8);   for (i = 0; i < 8; i++) {     felem_square(ftmp, ftmp);-  } /* 2^16 - 2^8 */-  felem_mul(ftmp, ftmp, e8); /* 2^16 - 2^0 */-  felem_assign(e16, ftmp);+  }+  felem_mul(x16, ftmp, x8); /* 2^16 - 1 */++  felem_assign(ftmp, x16);   for (i = 0; i < 16; i++) {     felem_square(ftmp, ftmp);-  } /* 2^32 - 2^16 */-  felem_mul(ftmp, ftmp, e16); /* 2^32 - 2^0 */-  felem_assign(e32, ftmp);+  }+  felem_mul(x32, ftmp, x16); /* 2^32 - 1 */++  /* The top 32 ones. */+  felem_assign(ftmp, x32);++  /* 31 zeros and a one: the 00000001 word. */   for (i = 0; i < 32; i++) {     felem_square(ftmp, ftmp);-  } /* 2^64 - 2^32 */-  felem_assign(e64, ftmp);-  felem_mul(ftmp, ftmp, in); /* 2^64 - 2^32 + 2^0 */-  for (i = 0; i < 192; i++) {+  }+  felem_mul(ftmp, ftmp, in);++  /* 96 zeros. */+  for (i = 0; i < 96; i++) {     felem_square(ftmp, ftmp);-  } /* 2^256 - 2^224 + 2^192 */+  } -  felem_mul(ftmp2, e64, e32); /* 2^64 - 2^0 */+  /* 94 ones, as 32 + 32 + 16 + 8 + 4 + 2. */+  for (i = 0; i < 32; i++) {+    felem_square(ftmp, ftmp);+  }+  felem_mul(ftmp, ftmp, x32);+  for (i = 0; i < 32; i++) {+    felem_square(ftmp, ftmp);+  }+  felem_mul(ftmp, ftmp, x32);   for (i = 0; i < 16; i++) {-    felem_square(ftmp2, ftmp2);-  } /* 2^80 - 2^16 */-  felem_mul(ftmp2, ftmp2, e16); /* 2^80 - 2^0 */+    felem_square(ftmp, ftmp);+  }+  felem_mul(ftmp, ftmp, x16);   for (i = 0; i < 8; i++) {-    felem_square(ftmp2, ftmp2);-  } /* 2^88 - 2^8 */-  felem_mul(ftmp2, ftmp2, e8); /* 2^88 - 2^0 */+    felem_square(ftmp, ftmp);+  }+  felem_mul(ftmp, ftmp, x8);   for (i = 0; i < 4; i++) {-    felem_square(ftmp2, ftmp2);-  } /* 2^92 - 2^4 */-  felem_mul(ftmp2, ftmp2, e4); /* 2^92 - 2^0 */-  felem_square(ftmp2, ftmp2); /* 2^93 - 2^1 */-  felem_square(ftmp2, ftmp2); /* 2^94 - 2^2 */-  felem_mul(ftmp2, ftmp2, e2); /* 2^94 - 2^0 */-  felem_square(ftmp2, ftmp2); /* 2^95 - 2^1 */-  felem_square(ftmp2, ftmp2); /* 2^96 - 2^2 */-  felem_mul(ftmp2, ftmp2, in); /* 2^96 - 3 */+    felem_square(ftmp, ftmp);+  }+  felem_mul(ftmp, ftmp, x4);+  felem_square(ftmp, ftmp);+  felem_square(ftmp, ftmp);+  felem_mul(ftmp, ftmp, x2); -  felem_mul(out, ftmp2, ftmp); /* 2^256 - 2^224 + 2^192 + 2^96 - 3 */+  /* A zero and a one: the d of fffffffd. */+  felem_square(ftmp, ftmp);+  felem_square(ftmp, ftmp);+  felem_mul(out, ftmp, in); } - /* Group operations:  *  * Elements of the elliptic curve group are represented in Jacobian@@ -298,7 +334,8 @@ }  /* select_affine_point sets {out_x,out_y} to the index'th entry of table.- * On entry: index < 16, table[0] must be zero. */+ * On entry: index < 16.  Every entry is a point of its own -- the signed+ * representation has no zero digit -- so the scan starts at zero. */ static void select_affine_point(felem out_x, felem out_y, const limb* table,                                 limb index) {   limb i, j;@@ -306,7 +343,7 @@   memset(out_x, 0, sizeof(felem));   memset(out_y, 0, sizeof(felem)); -  for (i = 1; i < 16; i++) {+  for (i = 0; i < 16; i++) {     limb mask = i ^ index;     mask |= mask >> 2;     mask |= mask >> 1;@@ -321,95 +358,306 @@   } } -/* select_jacobian_point sets {out_x,out_y,out_z} to the index'th entry of- * table. On entry: index < 16, table[0] must be zero. */-static void select_jacobian_point(felem out_x, felem out_y, felem out_z,-                                  const limb* table, limb index) {-  limb i, j;+/* words_are_zero returns 1 when |v| is zero and 0 otherwise, without a+ * branch. */+static u32 words_are_zero(u32 v) {+  v |= v >> 16;+  v |= v >> 8;+  v |= v >> 4;+  v |= v >> 2;+  v |= v >> 1;+  return (v & 1) ^ 1;+} -  memset(out_x, 0, sizeof(felem));-  memset(out_y, 0, sizeof(felem));-  memset(out_z, 0, sizeof(felem));+/* The comb: five teeth to a block, the teeth of a block 52 apart and the two+ * blocks 26 from each other, so 26 steps cover all 260 bits between them.+ *+ *   first block    i, 52+i, 104+i, 156+i, 208+i+ *   second block   26+i, 78+i, 130+i, 182+i, 234+i+ *+ * Five teeth would want a table of 32, but the signed representation makes+ * every digit +-1, so the thirty-two values come in pairs that differ by sign+ * and sixteen entries serve: kPrecomputed holds the ones whose top tooth is+ * positive and the other sign is a negated y.  That is the whole of the gain+ * over the four-tooth unsigned comb this replaces, which took 32 steps for+ * the same 256 bits: 25 doublings and 52 mixed additions against 31 and 64.+ */+#define COMB_TEETH 5+#define COMB_STEPS 26+#define COMB_SPAN (2 * COMB_STEPS) /* 52, the distance between a block's teeth */+#define COMB_WORDS 9               /* 260 bits of signs, with room to add into */+#define COMB_BIT(t, q) (((t)[(q) >> 5] >> ((q) & 31)) & 1) -  /* The implicit value at index 0 is all zero. We don't need to perform that-   * iteration of the loop because we already set out_* to zero. */-  table += 3 * NLIMBS;+/* comb_recode writes into |out| the value whose bits are the signs of the+ * scalar's all-bits-set representation: digit j is +1 where bit j of |out| is+ * set and -1 where it is not.+ *+ * For an odd k below 2^260 there is exactly one such representation, and it+ * is (k + 2^260 - 1) / 2 read as bits -- an addition and a shift, and that is+ * all the recoding is.  An even scalar has the order added to make it odd,+ * which changes the scalar and not the point it selects.+ *+ * Constant time in the scalar: every branch below is on a loop counter. */+static void comb_recode(u32 out[COMB_WORDS],+                        const crypton_p256_int* scalar) {+  u32 k[COMB_WORDS], ord[COMB_WORDS];+  u64 carry;+  int i; -  // Hit all entries to obscure cache profiling.-  for (i = 1; i < 16; i++) {-    limb mask = i ^ index;-    mask |= mask >> 2;-    mask |= mask >> 1;-    mask &= 1;-    mask--;-    for (j = 0; j < NLIMBS; j++, table++) {-      out_x[j] |= *table & mask;-    }-    for (j = 0; j < NLIMBS; j++, table++) {-      out_y[j] |= *table & mask;-    }-    for (j = 0; j < NLIMBS; j++, table++) {-      out_z[j] |= *table & mask;+  for (i = 0; i < COMB_WORDS; i++) {+    k[i] = 0;+    ord[i] = 0;+  }+  for (i = 0; i < 256; i += 32) {+    k[i >> 5] = (u32)(P256_DIGIT(scalar, i / P256_BITSPERDIGIT)+                      >> (i % P256_BITSPERDIGIT));+    ord[i >> 5] = (u32)(P256_DIGIT(&crypton_SECP256r1_n, i / P256_BITSPERDIGIT)+                        >> (i % P256_BITSPERDIGIT));+  }++  /* an even scalar becomes odd by taking on the order */+  {+    u32 addmask = (u32)0 - (u32)((k[0] & 1) ^ 1);++    carry = 0;+    for (i = 0; i < COMB_WORDS; i++) {+      u64 v = (u64)k[i] + (u64)(ord[i] & addmask) + carry;+      k[i] = (u32)v;+      carry = v >> 32;     }   }++  /* out = (k + 2^260 - 1) >> 1 */+  carry = 0;+  for (i = 0; i < COMB_WORDS; i++) {+    u64 v = (u64)k[i] + (u64)(i < 8 ? 0xffffffffu : 0xfu) + carry;+    out[i] = (u32)v;+    carry = v >> 32;+  }+  for (i = 0; i < COMB_WORDS - 1; i++) {+    out[i] = (out[i] >> 1) | (out[i + 1] << 31);+  }+  out[COMB_WORDS - 1] >>= 1;+ } +/* point_add_complete_mixed sets {x3,y3,z3} = {x1,y1,z1} + {x2,y2}, where the+ * accumulator is in projective coordinates and the added point is affine.+ *+ * This is Renes-Costello-Batina algorithm 5, for a curve with a = -3.  It is+ * complete: it is right when the two points are the same, when either is the+ * point at infinity, and when they are each other's negation, which is what+ * point_add_mixed cannot say.  It costs 11 multiplications and two by b,+ * against point_add_mixed's 8 multiplications and 3 squarings.+ *+ * The table this comb reads is affine, and an affine point has Z = 1, so+ * Z = Z^2 = Z^3 and the same three numbers are the point in projective+ * coordinates and in Jacobian ones.  That is what lets a comb built on+ * Jacobian arithmetic step into this formula for one addition and back out.+ */+static void point_add_complete_mixed(felem x3, felem y3, felem z3,+                                     const felem x1, const felem y1,+                                     const felem z1, const felem x2,+                                     const felem y2) {+  felem t0, t1, t2, t3, t4, xx, yy, zz;++  felem_mul(t0, x1, x2);+  felem_mul(t1, y1, y2);+  felem_sum(t3, x2, y2);+  felem_sum(t4, x1, y1);+  felem_mul(t3, t3, t4);+  felem_sum(t4, t0, t1);+  felem_diff(t3, t3, t4);+  felem_mul(t4, y2, z1);+  felem_sum(t4, t4, y1);+  felem_mul(yy, x2, z1);+  felem_sum(yy, yy, x1);+  felem_mul(zz, kB, z1);+  felem_diff(xx, yy, zz);+  felem_sum(zz, xx, xx);+  felem_sum(xx, xx, zz);+  felem_diff(zz, t1, xx);+  felem_sum(xx, t1, xx);+  felem_mul(yy, kB, yy);+  felem_sum(t1, z1, z1);+  felem_sum(t2, t1, z1);+  felem_diff(yy, yy, t2);+  felem_diff(yy, yy, t0);+  felem_sum(t1, yy, yy);+  felem_sum(yy, t1, yy);+  felem_sum(t1, t0, t0);+  felem_sum(t0, t1, t0);+  felem_diff(t0, t0, t2);+  felem_mul(t1, t4, yy);+  felem_mul(t2, t0, yy);+  felem_mul(yy, xx, zz);+  felem_sum(y3, yy, t2);+  felem_mul(xx, t3, xx);+  felem_diff(x3, xx, t1);+  felem_mul(zz, t4, zz);+  felem_mul(t1, t3, t0);+  felem_sum(z3, zz, t1);+}++/* point_add_complete sets {x3,y3,z3} = {x1,y1,z1} + {x2,y2,z2}, both in+ * projective coordinates.+ *+ * Renes-Costello-Batina algorithm 4, for a = -3, and complete for the same+ * reasons as the mixed one above.  It costs 12 multiplications and two by b,+ * against point_add's 11 multiplications and 5 squarings.+ */+static void point_add_complete(felem x3, felem y3, felem z3, const felem x1,+                               const felem y1, const felem z1, const felem x2,+                               const felem y2, const felem z2) {+  felem t0, t1, t2, t3, t4, xx, yy, zz;++  felem_mul(t0, x1, x2);+  felem_mul(t1, y1, y2);+  felem_mul(t2, z1, z2);+  felem_sum(t3, x1, y1);+  felem_sum(t4, x2, y2);+  felem_mul(t3, t3, t4);+  felem_sum(t4, t0, t1);+  felem_diff(t3, t3, t4);+  felem_sum(t4, y1, z1);+  felem_sum(xx, y2, z2);+  felem_mul(t4, t4, xx);+  felem_sum(xx, t1, t2);+  felem_diff(t4, t4, xx);+  felem_sum(xx, x1, z1);+  felem_sum(yy, x2, z2);+  felem_mul(xx, xx, yy);+  felem_sum(yy, t0, t2);+  felem_diff(yy, xx, yy);+  felem_mul(zz, kB, t2);+  felem_diff(xx, yy, zz);+  felem_sum(zz, xx, xx);+  felem_sum(xx, xx, zz);+  felem_diff(zz, t1, xx);+  felem_sum(xx, t1, xx);+  felem_mul(yy, kB, yy);+  felem_sum(t1, t2, t2);+  felem_sum(t2, t1, t2);+  felem_diff(yy, yy, t2);+  felem_diff(yy, yy, t0);+  felem_sum(t1, yy, yy);+  felem_sum(yy, t1, yy);+  felem_sum(t1, t0, t0);+  felem_sum(t0, t1, t0);+  felem_diff(t0, t0, t2);+  felem_mul(t1, t4, yy);+  felem_mul(t2, t0, yy);+  felem_mul(yy, xx, zz);+  felem_sum(y3, yy, t2);+  felem_mul(xx, t3, xx);+  felem_diff(x3, xx, t1);+  felem_mul(zz, t4, zz);+  felem_mul(t1, t3, t0);+  felem_sum(z3, zz, t1);+}++/* jacobian_to_projective sets {x2,y2,z2} to the projective form of the+ * Jacobian point {x1,y1,z1}: (X/Z^2, Y/Z^3) is (XZ : Y : Z^3). */+static void jacobian_to_projective(felem x2, felem y2, felem z2,+                                   const felem x1, const felem y1,+                                   const felem z1) {+  felem zz, zzz;++  felem_square(zz, z1);+  felem_mul(zzz, zz, z1);+  felem_mul(x2, x1, z1);+  memcpy(y2, y1, sizeof(felem));+  memcpy(z2, zzz, sizeof(felem));+}++/* projective_to_jacobian is the other way: (X/Z) is (XZ : YZ^2 : Z). */+static void projective_to_jacobian(felem x2, felem y2, felem z2,+                                   const felem x1, const felem y1,+                                   const felem z1) {+  felem zz;++  felem_square(zz, z1);+  felem_mul(x2, x1, z1);+  felem_mul(y2, y1, zz);+  memcpy(z2, z1, sizeof(felem));+}+ /* scalar_base_mult sets {nx,ny,nz} = scalar*G where scalar is a little-endian  * number. Note that the value of scalar must be less than the order of the  * group. */ static void scalar_base_mult(felem nx, felem ny, felem nz,                              const crypton_p256_int* scalar) {-  int i, j;-  limb n_is_infinity_mask = -1, p_is_noninfinite_mask, mask;-  u32 table_offset;+  u32 rec[COMB_WORDS];+  limb n_is_infinity_mask = -1, mask;+  felem px, py, negy, tx, ty, tz, jx, jy, jz, cx, cy, cz;+  int i, blk, j; -  felem px, py;-  felem tx, ty, tz;+  comb_recode(rec, scalar);    memset(nx, 0, sizeof(felem));   memset(ny, 0, sizeof(felem));   memset(nz, 0, sizeof(felem)); -  /* The loop adds bits at positions 0, 64, 128 and 192, followed by-   * positions 32,96,160 and 224 and does this 32 times. */-  for (i = 0; i < 32; i++) {-    if (i) {+  for (i = COMB_STEPS - 1; i >= 0; i--) {+    if (i != COMB_STEPS - 1) {       point_double(nx, ny, nz, nx, ny, nz);     }-    table_offset = 0;-    for (j = 0; j <= 32; j += 32) {-      char bit0 = crypton_p256_get_bit(scalar, 31 - i + j);-      char bit1 = crypton_p256_get_bit(scalar, 95 - i + j);-      char bit2 = crypton_p256_get_bit(scalar, 159 - i + j);-      char bit3 = crypton_p256_get_bit(scalar, 223 - i + j);-      limb index = bit0 | (bit1 << 1) | (bit2 << 2) | (bit3 << 3); -      select_affine_point(px, py, kPrecomputed + table_offset, index);-      table_offset += 30 * NLIMBS;+    for (blk = 0; blk < 2; blk++) {+      u32 base = (u32)(blk * COMB_STEPS + i);+      u32 top = COMB_BIT(rec, base + (COMB_TEETH - 1) * COMB_SPAN);+      limb index = 0; -      /* Since scalar is less than the order of the group, we know that-       * {nx,ny,nz} != {px,py,1}, unless both are zero, which we handle-       * below. */-      point_add_mixed(tx, ty, tz, nx, ny, nz, px, py);-      /* The result of point_add_mixed is incorrect if {nx,ny,nz} is zero-       * (a.k.a.  the point at infinity). We handle that situation by-       * copying the point from the table. */+      for (j = 0; j < COMB_TEETH - 1; j++) {+        /* a tooth agreeing with the top one is a set bit of the index */+        index |= (limb)(COMB_BIT(rec, base + (u32)j * COMB_SPAN) ^ top ^ 1)+                 << j;+      }++      select_affine_point(px, py, kPrecomputed + blk * 16 * 2 * NLIMBS, index);+      felem_diff(negy, kZero, py);+      copy_conditional(py, negy, (limb)0 - (limb)(top ^ 1));++      /* The last addition is the one that can be a point added to itself:+       * entering it the accumulator is (k' - B)*G and what it adds is B*G,+       * where B is the second block's digit at step zero, so the two meet+       * when k' = 2B modulo the order.  One scalar below the order does+       * that, and point_add_mixed cannot answer it.+       *+       * So that one addition goes through the complete formula instead.+       * The table is affine, so the point just selected is the same three+       * numbers read as projective coordinates as read as Jacobian ones --+       * which is what lets a comb built on Jacobian arithmetic step into+       * the formula for an addition and back out of it.  The accumulator+       * is not affine and is converted.+       *+       * Measured on an M4, thread CPU time, the whole base point+       * multiplication is 17.34 us this way against 17.30 us with an extra+       * doubling and a mask, which is inside the spread of either. */+      if (i == 0 && blk == 1) {+        jacobian_to_projective(jx, jy, jz, nx, ny, nz);+        point_add_complete_mixed(cx, cy, cz, jx, jy, jz, px, py);+        projective_to_jacobian(tx, ty, tz, cx, cy, cz);+      } else {+        point_add_mixed(tx, ty, tz, nx, ny, nz, px, py);+      }++      /* The accumulator is the infinity until the first of these, and+       * point_add_mixed cannot start from it; the point itself is the sum. */       copy_conditional(nx, px, n_is_infinity_mask);       copy_conditional(ny, py, n_is_infinity_mask);       copy_conditional(nz, kOne, n_is_infinity_mask);--      /* Equally, the result is also wrong if the point from the table is-       * zero, which happens when the index is zero. We handle that by-       * only copying from {tx,ty,tz} to {nx,ny,nz} if index != 0. */-      p_is_noninfinite_mask = NON_ZERO_TO_ALL_ONES(index);-      mask = p_is_noninfinite_mask & ~n_is_infinity_mask;+      mask = ~n_is_infinity_mask;       copy_conditional(nx, tx, mask);       copy_conditional(ny, ty, mask);       copy_conditional(nz, tz, mask);-      /* If p was not zero, then n is now non-zero. */-      n_is_infinity_mask &= ~p_is_noninfinite_mask;+      n_is_infinity_mask = 0;     }   }++  /* The recoded scalar is the private key in another representation, so it+   * does not stay on the stack. */+  crypton_bzero(rec, sizeof(rec)); }  /* point_to_affine converts a Jacobian point to an affine point. If the input@@ -424,67 +672,309 @@   felem_mul(y_out, ny, z_inv); } -/* scalar_base_mult sets {nx,ny,nz} = scalar*{x,y}. */-static void scalar_mult(felem nx, felem ny, felem nz, const felem x,-                        const felem y, const crypton_p256_int* scalar) {-  int i;-  felem px, py, pz, tx, ty, tz;-  felem precomp[16][3];-  limb n_is_infinity_mask, index, p_is_noninfinite_mask, mask;+/* point_add_mixed_pm sets {xp,yp,zp} = {x1,y1,z1} + {x2,y2} and+ * {xm,ym,zm} = {x1,y1,z1} - {x2,y2}, where {x2,y2} is affine.+ *+ * Negating the second point changes the sign of s2 and so of r, and nothing+ * else: z1z1, tmp, u2, z1z1z1, h, i, j, v, the output z and the product y1*j+ * are common to the two.  What the second point costs over the first is one+ * squaring (r*r) and one multiplication (by r), rather than another eleven.+ *+ * The same restrictions as point_add_mixed: this does not handle P+P,+ * infinity+P nor P+infinity. */+static void point_add_mixed_pm(felem xp, felem yp, felem zp,+                               felem xm, felem ym, felem zm,+                               const felem x1, const felem y1, const felem z1,+                               const felem x2, const felem y2) {+  felem z1z1, z1z1z1, s2, u2, h, i, j, r, rr, v, y1j, tmp; -  /* We precompute 0,1,2,... times {x,y}. */-  memset(precomp, 0, sizeof(felem) * 3);-  memcpy(&precomp[1][0], x, sizeof(felem));-  memcpy(&precomp[1][1], y, sizeof(felem));-  memcpy(&precomp[1][2], kOne, sizeof(felem));+  felem_square(z1z1, z1);+  felem_sum(tmp, z1, z1); -  for (i = 2; i < 16; i += 2) {-    point_double(precomp[i][0], precomp[i][1], precomp[i][2],-                 precomp[i / 2][0], precomp[i / 2][1], precomp[i / 2][2]);+  felem_mul(u2, x2, z1z1);+  felem_mul(z1z1z1, z1, z1z1);+  felem_mul(s2, y2, z1z1z1);+  felem_diff(h, u2, x1);+  felem_sum(i, h, h);+  felem_square(i, i);+  felem_mul(j, h, i);+  felem_mul(v, x1, i);+  felem_mul(y1j, y1, j); -    point_add_mixed(precomp[i + 1][0], precomp[i + 1][1], precomp[i + 1][2],-                    precomp[i][0], precomp[i][1], precomp[i][2], x, y);+  /* The two points share their z. */+  felem_mul(zp, tmp, h);+  felem_assign(zm, zp);++  /* X + P */+  felem_diff(r, s2, y1);+  felem_sum(r, r, r);+  felem_square(rr, r);+  felem_diff(xp, rr, j);+  felem_diff(xp, xp, v);+  felem_diff(xp, xp, v);+  felem_diff(tmp, v, xp);+  felem_mul(yp, tmp, r);+  felem_diff(yp, yp, y1j);+  felem_diff(yp, yp, y1j);++  /* X - P.  Negating the point negates s2, so r becomes -q where+   * q = 2*(s2 + y1).  The square is the same either way, and the sign is+   * carried into y by taking (xm - v) where the other took (v - xp):+   *   xm = q^2 - j - 2v+   *   ym = (v - xm)*(-q) - 2*y1*j = (xm - v)*q - 2*y1*j+   * so no field negation is needed. */+  felem_sum(r, s2, y1);+  felem_sum(r, r, r);+  felem_square(rr, r);+  felem_diff(xm, rr, j);+  felem_diff(xm, xm, v);+  felem_diff(xm, xm, v);+  felem_diff(tmp, xm, v);+  felem_mul(ym, tmp, r);+  felem_diff(ym, ym, y1j);+  felem_diff(ym, ym, y1j);+}++/* select_jacobian_odd sets {out_x,out_y,out_z} to the index'th of the 16+ * entries of table, for index < 16.  There is no implicit infinity at index+ * zero, as the unsigned window this replaces had: every entry is a real+ * point, which is what lets the signed representation below do without the+ * infinity masks. */+static void select_jacobian_odd(felem out_x, felem out_y, felem out_z,+                                const limb* table, limb index) {+  limb i, j;++  memset(out_x, 0, sizeof(felem));+  memset(out_y, 0, sizeof(felem));+  memset(out_z, 0, sizeof(felem));++  for (i = 0; i < 16; i++) {+    limb mask = i ^ index;+    mask |= mask >> 2;+    mask |= mask >> 1;+    mask &= 1;+    mask--;+    for (j = 0; j < NLIMBS; j++, table++) {+      out_x[j] |= *table & mask;+    }+    for (j = 0; j < NLIMBS; j++, table++) {+      out_y[j] |= *table & mask;+    }+    for (j = 0; j < NLIMBS; j++, table++) {+      out_z[j] |= *table & mask;+    }   }+} -  memset(nx, 0, sizeof(felem));-  memset(ny, 0, sizeof(felem));-  memset(nz, 0, sizeof(felem));-  n_is_infinity_mask = -1;+/* The scalar, recoded: 52 signed odd digits, each in {+-1,+-3,...,+-31}, so+ * that scalar = sum d_i * 32^i.  A digit is one byte: the low four bits are+ * the table index (|d|-1)/2, and bit four is set when d is negative.  One+ * byte rather than a byte and a word because this is the private key in+ * another form and has to be wiped afterwards. */+#define SABS_DIGITS 52+#define SABS_INDEX(b) ((limb)((b) & 15))+#define SABS_NEGMASK(b) ((limb)0 - (limb)((b) >> 4))+typedef struct {+  u8 digit[SABS_DIGITS];+} sabs_scalar; -  /* We add in a window of four bits each iteration and do this 64 times. */-  for (i = 0; i < 256; i += 4) {-    if (i) {-      point_double(nx, ny, nz, nx, ny, nz);-      point_double(nx, ny, nz, nx, ny, nz);-      point_double(nx, ny, nz, nx, ny, nz);-      point_double(nx, ny, nz, nx, ny, nz);++/* sabs_recode writes the signed representation of |scalar| into |out|.+ *+ * The recoding is the regular one of Joye and Tunstall: take the low six bits,+ * subtract 32, and carry the difference upwards.  It needs an odd input, which+ * is arranged by adding the group order to an even scalar -- that changes the+ * scalar but not the point it selects, the order being the order.  A zero+ * scalar is replaced by one and the caller is told, since zero times a point+ * is the infinity this code deliberately cannot represent.+ *+ * Constant time in the scalar: every branch below is on a loop counter. */+static limb sabs_recode(sabs_scalar* out,+                        const crypton_p256_int* scalar) {+  u32 k[9], n[9];+  u32 nonzero;+  limb is_zero_mask;+  int i, b;++  for (i = 0; i < 9; i++) {+    k[i] = 0;+    n[i] = 0;+  }+  /* A word at a time.  Bit at a time would be 512 calls into another+   * translation unit, which the compiler cannot inline away. */+  for (b = 0; b < 256; b += 32) {+    k[b >> 5] = (u32)(P256_DIGIT(scalar, b / P256_BITSPERDIGIT)+                      >> (b % P256_BITSPERDIGIT));+    n[b >> 5] = (u32)(P256_DIGIT(&crypton_SECP256r1_n, b / P256_BITSPERDIGIT)+                      >> (b % P256_BITSPERDIGIT));+  }++  /* Replace a zero scalar by one, and report it. */+  nonzero = 0;+  for (i = 0; i < 9; i++) {+    nonzero |= k[i];+  }+  {+    u32 z = words_are_zero(nonzero);+    k[0] |= z;+    is_zero_mask = (limb)0 - (limb)z;+  }++  /* An even scalar becomes odd by adding the order.  The sum is below 2^257,+   * which is why nine words and fifty-two digits are enough. */+  {+    u32 addmask = (u32)0 - (u32)((k[0] & 1) ^ 1);+    u64 carry = 0;+    for (i = 0; i < 9; i++) {+      u64 t = (u64)k[i] + (u64)(n[i] & addmask) + carry;+      k[i] = (u32)t;+      carry = t >> 32;     }+  } -    index = (crypton_p256_get_bit(scalar, 255 - i - 0) << 3) |-            (crypton_p256_get_bit(scalar, 255 - i - 1) << 2) |-            (crypton_p256_get_bit(scalar, 255 - i - 2) << 1) |-            crypton_p256_get_bit(scalar, 255 - i - 3);+  for (i = 0; i < SABS_DIGITS - 1; i++) {+    u32 r6 = k[0] & 63;            /* odd, so never 32 */+    u32 hi = (r6 >> 5) & 1;        /* 1 when the digit is positive */+    u32 wabs = ((r6 - 32) & (0u - hi)) | ((32 - r6) & (hi - 1));+    u32 mlo = 32u - r6;            /* two's complement of the digit's negation */+    u32 ext = 0u - hi;             /* its sign extension */+    u64 carry = 0;+    int w; -    /* See the comments in scalar_base_mult about handling infinities. */-    select_jacobian_point(px, py, pz, precomp[0][0], index);-    point_add(tx, ty, tz, nx, ny, nz, px, py, pz);-    copy_conditional(nx, px, n_is_infinity_mask);-    copy_conditional(ny, py, n_is_infinity_mask);-    copy_conditional(nz, pz, n_is_infinity_mask);+    out->digit[i] = (u8)(((wabs - 1) >> 1) | ((hi ^ 1) << 4)); -    p_is_noninfinite_mask = NON_ZERO_TO_ALL_ONES(index);-    mask = p_is_noninfinite_mask & ~n_is_infinity_mask;+    /* k -= digit, i.e. k += -digit, sign extended over the nine words. */+    for (w = 0; w < 9; w++) {+      u64 t = (u64)k[w] + (u64)(w == 0 ? mlo : ext) + carry;+      k[w] = (u32)t;+      carry = t >> 32;+    }+    /* k >>= 5 */+    for (w = 0; w < 8; w++) {+      k[w] = (k[w] >> 5) | (k[w + 1] << 27);+    }+    k[8] >>= 5;+  } -    copy_conditional(nx, tx, mask);-    copy_conditional(ny, ty, mask);-    copy_conditional(nz, tz, mask);-    n_is_infinity_mask &= ~p_is_noninfinite_mask;+  /* What is left is odd, positive and at most five: the scalar is below+   * 2^257 and fifty-one digits have taken 255 bits off it, each leaving a+   * remainder below one. */+  out->digit[SABS_DIGITS - 1] = (u8)((k[0] - 1) >> 1);++  return is_zero_mask;+}++/* scalar_mult sets {nx,ny,nz} = scalar*{x,y}.+ *+ * A five-bit signed window.  The scalar is recoded into 52 digits, every one+ * of them odd and none of them zero, so the table holds only the odd+ * multiples P, 3P, ..., 31P and a negative digit is served by negating y,+ * which is free.  Against the four-bit unsigned window this replaces, the+ * main loop trades 252 doublings and 64 additions for 255 and 51, and --+ * because no digit is zero and no partial sum is the infinity -- it drops the+ * masks that stood in for infinity on every iteration.+ *+ * The table is built so that each pair of neighbouring odd multiples comes+ * out of one doubling and one shared addition:+ *+ *   2P = 2*P                3P  = 2P + P+ *   6P = 2*(3P)             5P  = 6P - P,  7P  = 6P + P+ *   10P = 2*(5P)            9P  = 10P - P, 11P = 10P + P+ *   ...+ *   30P = 2*(15P)           29P = 30P - P, 31P = 30P + P+ *+ * which is eight doublings, one mixed addition and seven shared pairs. */+static void scalar_mult(felem nx, felem ny, felem nz, const felem x,+                        const felem y, const crypton_p256_int* scalar) {+  /* odd[k] is (2k+1)*P, for k in 0..15. */+  felem odd[16][3];+  felem dx, dy, dz, px, py, pz, negy, ddx, ddy, ddz, qx, qy, qz, cx, cy, cz;+  sabs_scalar rec;+  limb is_zero_mask;+  int i, k;++  is_zero_mask = sabs_recode(&rec, scalar);++  felem_assign(odd[0][0], x);+  felem_assign(odd[0][1], y);+  memcpy(odd[0][2], kOne, sizeof(felem));++  /* 3P = 2P + P */+  point_double(dx, dy, dz, x, y, kOne);+  point_add_mixed(odd[1][0], odd[1][1], odd[1][2], dx, dy, dz, x, y);++  /* (4k+2)P from (2k+1)P, then (4k+1)P and (4k+3)P from it. */+  for (k = 1; k < 8; k++) {+    point_double(dx, dy, dz, odd[k][0], odd[k][1], odd[k][2]);+    point_add_mixed_pm(odd[2 * k + 1][0], odd[2 * k + 1][1], odd[2 * k + 1][2],+                       odd[2 * k][0], odd[2 * k][1], odd[2 * k][2],+                       dx, dy, dz, x, y);   }++  /* The top digit initialises the accumulator; it is always positive. */+  select_jacobian_odd(nx, ny, nz, odd[0][0],+                      SABS_INDEX(rec.digit[SABS_DIGITS - 1]));++  for (i = SABS_DIGITS - 2; i >= 0; i--) {+    point_double(nx, ny, nz, nx, ny, nz);+    point_double(nx, ny, nz, nx, ny, nz);+    point_double(nx, ny, nz, nx, ny, nz);+    point_double(nx, ny, nz, nx, ny, nz);+    point_double(nx, ny, nz, nx, ny, nz);++    select_jacobian_odd(px, py, pz, odd[0][0], SABS_INDEX(rec.digit[i]));+    felem_diff(negy, kZero, py);+    copy_conditional(py, negy, SABS_NEGMASK(rec.digit[i]));++    /* On the last step alone the accumulator can be the very point being+     * added -- the accumulator is (k' - d0)*P and it adds d0*P, so the two+     * meet when k' = 2*d0 modulo the order, which the scalar 30 does with a+     * digit of 15 -- and point_add answers the infinity where the truth is+     * twice that point.  That one addition goes through the complete formula+     * instead, with both points converted to projective coordinates and the+     * answer converted back.  One of fifty-one iterations pays for it, and+     * it comes to a field multiplication less than the doubling and the mask+     * it replaces.+     *+     * point_add finishes with z before it touches x, and with each of x and+     * y before the next, so on every other step the accumulator can be its+     * own output. */+    if (i == 0) {+      jacobian_to_projective(ddx, ddy, ddz, nx, ny, nz);+      jacobian_to_projective(qx, qy, qz, px, py, pz);+      point_add_complete(cx, cy, cz, ddx, ddy, ddz, qx, qy, qz);+      projective_to_jacobian(nx, ny, nz, cx, cy, cz);+    } else {+      point_add(nx, ny, nz, nx, ny, nz, px, py, pz);+    }+  }++  /* Zero was replaced by one on the way in; put the infinity back.  All+   * three coordinates, not just z: crypton_p256_points_mul_vartime reads the+   * comment above it as saying the whole point is zero. */+  for (i = 0; i < NLIMBS; i++) {+    nx[i] &= ~is_zero_mask;+    ny[i] &= ~is_zero_mask;+    nz[i] &= ~is_zero_mask;+  }++  /* The recoded scalar is the private key in another representation, so it+   * does not stay on the stack. */+  crypton_bzero(&rec, sizeof(rec)); }  /* crypton_p256_base_point_mul sets {out_x,out_y} = nG, where n is < the  * order of the group. */ void crypton_p256_base_point_mul(const crypton_p256_int* n, crypton_p256_int* out_x, crypton_p256_int* out_y) {+#ifdef CRYPTON_S2N_BIGNUM+  /* The assembly, four times faster, and constant time as this has to be:+   * it is how a public key is derived from a private one. */+  uint64_t res[8];++  crypton_s2n_p256_scalarmulbase(res, P256_DIGITS(n));+  memcpy(P256_DIGITS(out_x), res, P256_NBYTES);+  memcpy(P256_DIGITS(out_y), res + P256_NDIGITS, P256_NBYTES);+#else   felem x, y, z;    scalar_base_mult(x, y, z, n);@@ -496,8 +986,29 @@     from_montgomery(out_x, x_affine);     from_montgomery(out_y, y_affine);   }+#endif } +#ifdef CRYPTON_S2N_BIGNUM+/* An affine point from the assembly as the Jacobian triple the addition+ * below wants, keeping this file's convention that the point at infinity is+ * all three coordinates zero -- the assembly says it with (0, 0). */+static void s2n_lift(felem x, felem y, felem z, const uint64_t res[8]) {+  crypton_p256_int t;+  uint64_t any = 0;+  int i;++  for (i = 0; i < 2 * P256_NDIGITS; i++)+    any |= res[i];++  memcpy(P256_DIGITS(&t), res, P256_NBYTES);+  to_montgomery(x, &t);+  memcpy(P256_DIGITS(&t), res + P256_NDIGITS, P256_NBYTES);+  to_montgomery(y, &t);+  memcpy(z, any != 0 ? kOne : kZero, sizeof(felem));+}+#endif+ /* crypton_p256_points_mul_vartime sets {out_x,out_y} = n1*G + n2*{in_x,in_y}, where  * n1 and n2 are < the order of the group.  *@@ -516,10 +1027,48 @@     return;   } +#ifdef CRYPTON_S2N_BIGNUM+  {+    /* Both scalars at once, in variable time, which is what the two+     * multiplications below are not: they are constant time, and pay for it,+     * over values that are all public here.  It gives up on the one case the+     * assembly's addition does not cover -- adding a point to itself -- and+     * then the constant-time pair answers instead. */+    uint64_t jr[3 * P256_NDIGITS];++    if (crypton_p256_verify_mul(jr, P256_DIGITS(n1), P256_DIGITS(n2),+                                P256_DIGITS(in_x), P256_DIGITS(in_y))) {+      crypton_p256_int t;++      memcpy(P256_DIGITS(&t), jr, P256_NBYTES);+      to_montgomery(x1, &t);+      memcpy(P256_DIGITS(&t), jr + P256_NDIGITS, P256_NBYTES);+      to_montgomery(y1, &t);+      memcpy(P256_DIGITS(&t), jr + 2 * P256_NDIGITS, P256_NBYTES);+      to_montgomery(z1, &t);++      point_to_affine(px, py, x1, y1, z1);+      from_montgomery(out_x, px);+      from_montgomery(out_y, py);+      return;+    }+  }+  {+    uint64_t r1[8], r2[8], pt[2 * P256_NDIGITS];++    memcpy(pt, P256_DIGITS(in_x), P256_NBYTES);+    memcpy(pt + P256_NDIGITS, P256_DIGITS(in_y), P256_NBYTES);+    crypton_s2n_p256_scalarmulbase(r1, P256_DIGITS(n1));+    crypton_s2n_p256_scalarmul(r2, P256_DIGITS(n2), pt);+    s2n_lift(x1, y1, z1, r1);+    s2n_lift(x2, y2, z2, r2);+  }+#else   to_montgomery(px, in_x);   to_montgomery(py, in_y);   scalar_base_mult(x1, y1, z1, n1);   scalar_mult(x2, y2, z2, px, py, n2);+#endif    if (crypton_p256_is_zero(n2) != 0) {     /* If n2 == 0, then {x2,y2,z2} is zero and the result is just@@ -581,6 +1130,19 @@ void crypton_p256e_point_mul(const crypton_p256_int* n,     const crypton_p256_int* in_x, const crypton_p256_int* in_y,     crypton_p256_int* out_x, crypton_p256_int* out_y) {+#ifdef CRYPTON_S2N_BIGNUM+  /* The vendored assembly, which answers the same thing two and a half to+   * three times faster.  Both sides are four little-endian 64-bit words of+   * an ordinary affine coordinate, so the points cross as they are, and+   * both give (0, 0) for the point at infinity.  See cbits/s2n/README.md. */+  uint64_t point[8], res[8];++  memcpy(point, P256_DIGITS(in_x), P256_NBYTES);+  memcpy(point + P256_NDIGITS, P256_DIGITS(in_y), P256_NBYTES);+  crypton_s2n_p256_scalarmul(res, P256_DIGITS(n), point);+  memcpy(P256_DIGITS(out_x), res, P256_NBYTES);+  memcpy(P256_DIGITS(out_y), res + P256_NDIGITS, P256_NBYTES);+#else   felem x, y, z, px, py;    to_montgomery(px, in_x);@@ -589,4 +1151,5 @@   point_to_affine(px, py, x, y, z);   from_montgomery(out_x, px);   from_montgomery(out_y, py);+#endif }
+ cbits/p256/p256_s2n.c view
@@ -0,0 +1,61 @@+/*+ * Choosing between s2n-bignum's two forms of each routine.  The reasoning+ * is in cbits/s2n/README.md; in short, on ARM the choice is a+ * microarchitecture one that no feature bit answers, and on x86-64 it is+ * exactly a feature bit.+ */+#include "p256/p256_s2n.h"+#include "crypton_cpu.h"++extern void p256_scalarmul(uint64_t res[8], const uint64_t scalar[4],+                           const uint64_t point[8]);+extern void p256_scalarmul_alt(uint64_t res[8], const uint64_t scalar[4],+                               const uint64_t point[8]);+extern void p256_scalarmulbase(uint64_t res[8], const uint64_t scalar[4],+                               uint64_t blocksize, const uint64_t *table);+extern void p256_scalarmulbase_alt(uint64_t res[8], const uint64_t scalar[4],+                                   uint64_t blocksize, const uint64_t *table);++extern const uint64_t crypton_p256_s2n_base_blocksize;+extern const uint64_t crypton_p256_s2n_base_table[];++void crypton_s2n_p256_scalarmul(uint64_t res[8], const uint64_t scalar[4],+                                const uint64_t point[8])+{+#if defined(__aarch64__) || defined(__arm64__)+#ifdef __APPLE__+	/* the _alt form is the one written for high multiplier throughput,+	 * and it is 30-40% faster on Apple silicon */+	p256_scalarmul_alt(res, scalar, point);+#else+	p256_scalarmul(res, scalar, point);+#endif+#else+	if (crypton_x86_simd_features() & CRYPTON_X86_ADX)+		p256_scalarmul(res, scalar, point);+	else+		p256_scalarmul_alt(res, scalar, point);+#endif+}++void crypton_s2n_p256_scalarmulbase(uint64_t res[8], const uint64_t scalar[4])+{+#if defined(__aarch64__) || defined(__arm64__)+#ifdef __APPLE__+	p256_scalarmulbase_alt(res, scalar, crypton_p256_s2n_base_blocksize,+	                       crypton_p256_s2n_base_table);+#else+	p256_scalarmulbase(res, scalar, crypton_p256_s2n_base_blocksize,+	                   crypton_p256_s2n_base_table);+#endif+#else+	if (crypton_x86_simd_features() & CRYPTON_X86_ADX)+		p256_scalarmulbase(res, scalar,+		                   crypton_p256_s2n_base_blocksize,+		                   crypton_p256_s2n_base_table);+	else+		p256_scalarmulbase_alt(res, scalar,+		                       crypton_p256_s2n_base_blocksize,+		                       crypton_p256_s2n_base_table);+#endif+}
+ cbits/p256/p256_verify.c view
@@ -0,0 +1,300 @@+/*+ * The double scalar multiplication ECDSA verification does, in variable+ * time.+ *+ * Verification asks for u1*G + u2*Q, and crypton used to work that out as+ * two separate constant-time multiplications and an addition.  Nothing here+ * is secret -- the message, the signature and the public key are all sent in+ * the clear -- so the constant-time work is paid for nothing, and the two+ * multiplications can share their doublings besides.  Measured, those two+ * were 84% of a verification.+ *+ * So this is the usual interleaved windowed form: both scalars in+ * width-5 non-adjacent form, a table of odd multiples of each point, and one+ * pass down the digits with a doubling at every step and an addition where a+ * digit is not zero.  It is s2n-bignum's Jacobian point arithmetic+ * underneath, the same assembly the constant-time paths use.+ *+ * s2n-bignum's p256_montjadd is correct except when its two arguments are+ * the same point -- that is the side condition its proof carries -- so every+ * sum is checked for the sign of that, which is a zero z where neither+ * argument had one.  There the answer is given up on and the caller falls+ * back to the constant-time pair, which has no such condition.  With random+ * inputs it does not happen; it is here because an attacker chooses the+ * public key and the signature.+ */+#include <string.h>++#include "p256/p256.h"++#ifdef CRYPTON_S2N_BIGNUM++#include "crypton_cpu.h"+#include "p256/p256_verify.h"++/* a Jacobian triple in the Montgomery domain, as the assembly keeps them */+#define JAC 12+#define AFF 8++extern void p256_montjadd(uint64_t p3[JAC], const uint64_t p1[JAC],+                          const uint64_t p2[JAC]);+extern void p256_montjadd_alt(uint64_t p3[JAC], const uint64_t p1[JAC],+                              const uint64_t p2[JAC]);+extern void p256_montjdouble(uint64_t p3[JAC], const uint64_t p1[JAC]);+extern void p256_montjdouble_alt(uint64_t p3[JAC], const uint64_t p1[JAC]);+extern void p256_montjmixadd(uint64_t p3[JAC], const uint64_t p1[JAC],+                             const uint64_t p2[AFF]);+extern void p256_montjmixadd_alt(uint64_t p3[JAC], const uint64_t p1[JAC],+                                 const uint64_t p2[AFF]);++/* the odd multiples of the base point, from cbits/p256/gen_base_table.py */+extern const uint64_t crypton_p256_wnaf_width;+extern const uint64_t crypton_p256_wnaf_table[];+extern void bignum_tomont_p256(uint64_t z[4], const uint64_t x[4]);+extern void bignum_demont_p256(uint64_t z[4], const uint64_t x[4]);+extern void bignum_neg_p256(uint64_t z[4], const uint64_t x[4]);+#if !defined(__aarch64__) && !defined(__arm64__)+extern void bignum_tomont_p256_alt(uint64_t z[4], const uint64_t x[4]);+extern void bignum_demont_p256_alt(uint64_t z[4], const uint64_t x[4]);+#endif++/* The same question as everywhere else in cbits/s2n: a microarchitecture one+ * on ARM that no feature bit answers, and exactly a feature bit on x86-64. */+static int use_alt(void)+{+#if defined(__aarch64__) || defined(__arm64__)+#ifdef __APPLE__+	return 1;+#else+	return 0;+#endif+#else+	return (crypton_x86_simd_features() & CRYPTON_X86_ADX) == 0;+#endif+}++static void padd(uint64_t r[JAC], const uint64_t a[JAC], const uint64_t b[JAC],+                 int alt)+{+	if (alt)+		p256_montjadd_alt(r, a, b);+	else+		p256_montjadd(r, a, b);+}++static void pmixadd(uint64_t r[JAC], const uint64_t a[JAC],+                    const uint64_t b[AFF], int alt)+{+	if (alt)+		p256_montjmixadd_alt(r, a, b);+	else+		p256_montjmixadd(r, a, b);+}++static void pdouble(uint64_t r[JAC], const uint64_t a[JAC], int alt)+{+	if (alt)+		p256_montjdouble_alt(r, a);+	else+		p256_montjdouble(r, a);+}++static void tomont(uint64_t z[4], const uint64_t x[4])+{+#if defined(__aarch64__) || defined(__arm64__)+	bignum_tomont_p256(z, x);+#else+	if (use_alt())+		bignum_tomont_p256_alt(z, x);+	else+		bignum_tomont_p256(z, x);+#endif+}++static void demont(uint64_t z[4], const uint64_t x[4])+{+#if defined(__aarch64__) || defined(__arm64__)+	bignum_demont_p256(z, x);+#else+	if (use_alt())+		bignum_demont_p256_alt(z, x);+	else+		bignum_demont_p256(z, x);+#endif+}++static int is_infinity(const uint64_t p[JAC])+{+	return (p[8] | p[9] | p[10] | p[11]) == 0;+}++/*+ * The base point's table is a constant, so its window is as wide as the+ * table is worth carrying: seven bits, thirty-two odd multiples, two+ * kilobytes, and one addition every eight digits.  The public key's table+ * has to be built for each verification, so five bits is the trade there --+ * eight entries, seven point operations to build, and one addition every+ * six digits.+ */+#define WG 7+#define TG (1 << (WG - 2))+#define WQ 5+#define TQ (1 << (WQ - 2))+#define NAF_MAX 258++/*+ * The width-W non-adjacent form of a scalar, one signed digit per bit+ * position: odd or zero, and never two non-zero digits within W of each+ * other.  Returns how many digits were written.+ *+ * The scalar is public, so the loop may look at it.+ */+static int wnaf(int8_t out[NAF_MAX], const uint64_t in[4], int w)+{+	uint64_t k[5];+	int len = 0;++	memcpy(k, in, 32);+	k[4] = 0;++	while (k[0] | k[1] | k[2] | k[3] | k[4]) {+		int d = 0;++		if (k[0] & 1) {+			d = (int) (k[0] & ((1u << w) - 1));+			if (d >= (1 << (w - 1)))+				d -= 1 << w;+			if (d > 0) {+				uint64_t borrow = (uint64_t) d;+				int i;++				for (i = 0; i < 5 && borrow; i++) {+					uint64_t t = k[i];++					k[i] = t - borrow;+					borrow = (k[i] > t);+				}+			} else {+				uint64_t carry = (uint64_t) (-d);+				int i;++				for (i = 0; i < 5 && carry; i++) {+					k[i] += carry;+					carry = (k[i] < carry);+				}+			}+		}+		out[len++] = (int8_t) d;++		{ /* k >>= 1 */+			int i;++			for (i = 0; i < 4; i++)+				k[i] = (k[i] >> 1) | (k[i + 1] << 63);+			k[4] >>= 1;+		}+	}+	return len;+}++/* P, 3P, 5P, ..., (2*TQ-1)P from a Jacobian P */+static int build_table(uint64_t t[TQ][JAC], const uint64_t p[JAC], int alt)+{+	uint64_t twice[JAC];+	int i;++	memcpy(t[0], p, sizeof(uint64_t) * JAC);+	pdouble(twice, p, alt);+	for (i = 1; i < TQ; i++) {+		padd(t[i], t[i - 1], twice, alt);+		/* the table is built from a point and its double, which are+		 * never the same point unless the point has order two, and+		 * P-256 has none */+		if (is_infinity(t[i]) && !is_infinity(t[i - 1])+		    && !is_infinity(twice))+			return 0;+	}+	return 1;+}++/* the table entry for a digit, negated when the digit is */+static void pick(uint64_t out[JAC], const uint64_t t[TQ][JAC], int digit)+{+	int idx = (digit > 0 ? digit : -digit) / 2;++	memcpy(out, t[idx], sizeof(uint64_t) * JAC);+	if (digit < 0)+		bignum_neg_p256(out + 4, out + 4);+}++/* the same from the base point's affine table */+static void pick_affine(uint64_t out[AFF], int digit)+{+	int idx = (digit > 0 ? digit : -digit) / 2;++	memcpy(out, crypton_p256_wnaf_table + (size_t) idx * AFF,+	       sizeof(uint64_t) * AFF);+	if (digit < 0)+		bignum_neg_p256(out + 4, out + 4);+}++int crypton_p256_verify_mul(uint64_t out[JAC], const uint64_t n1[4],+                            const uint64_t n2[4], const uint64_t qx[4],+                            const uint64_t qy[4])+{+	/* the Montgomery form of one, which is the z of an affine point */+	static const uint64_t mont_one[4] = {+		0x0000000000000001ULL, 0xffffffff00000000ULL,+		0xffffffffffffffffULL, 0x00000000fffffffeULL+	};+	uint64_t tq[TQ][JAC];+	uint64_t q[JAC], acc[JAC], addend[JAC], aff[AFF], sum[JAC];+	int8_t naf1[NAF_MAX], naf2[NAF_MAX];+	int len1, len2, len, i;+	/* asked once rather than at every point operation */+	const int alt = use_alt();++	/* the generated table has to be the width this file walks it at */+	if (crypton_p256_wnaf_width != WG)+		return 0;++	tomont(q, qx);+	tomont(q + 4, qy);+	memcpy(q + 8, mont_one, sizeof mont_one);++	if (!build_table(tq, q, alt))+		return 0;++	len1 = wnaf(naf1, n1, WG);+	len2 = wnaf(naf2, n2, WQ);+	len = len1 > len2 ? len1 : len2;++	memset(acc, 0, sizeof acc);+	for (i = len - 1; i >= 0; i--) {+		pdouble(acc, acc, alt);++		if (i < len1 && naf1[i] != 0) {+			/* the base point's entries are affine, which is a+			 * cheaper addition and no table to build */+			pick_affine(aff, naf1[i]);+			pmixadd(sum, acc, aff, alt);+			if (is_infinity(sum) && !is_infinity(acc))+				return 0;+			memcpy(acc, sum, sizeof acc);+		}+		if (i < len2 && naf2[i] != 0) {+			pick(addend, tq, naf2[i]);+			padd(sum, acc, addend, alt);+			if (is_infinity(sum) && !is_infinity(acc))+				return 0;+			memcpy(acc, sum, sizeof acc);+		}+	}++	demont(out, acc);+	demont(out + 4, acc + 4);+	demont(out + 8, acc + 8);+	return 1;+}++#endif
+ cbits/p256/p256_verify.h view
@@ -0,0 +1,19 @@+#ifndef CRYPTON_P256_VERIFY_H+#define CRYPTON_P256_VERIFY_H++#include <stdint.h>++/*+ * n1*G + n2*Q, in variable time, as a Jacobian triple in the plain domain.+ *+ * Returns 1 when the answer is in `out`, and 0 when the walk met the one+ * case s2n-bignum's point addition does not cover -- adding a point to+ * itself -- in which case the caller works the answer out the constant-time+ * way instead.  Nothing here is secret: it is all in the signature and the+ * public key.+ */+int crypton_p256_verify_mul(uint64_t out[12], const uint64_t n1[4],+                            const uint64_t n2[4], const uint64_t qx[4],+                            const uint64_t qy[4]);++#endif
+ cbits/p256/p256_wnaf_table.c view
@@ -0,0 +1,42 @@+/* Generated by cbits/p256/gen_base_table.py; do not hand-edit.+ * The odd multiples of the base point, in Montgomery-affine+ * form, which cbits/p256/p256_verify.c reads.  A verification+ * is public, so this table is walked in variable time. */+#include <stdint.h>++const uint64_t crypton_p256_wnaf_width = 7;++const uint64_t crypton_p256_wnaf_table[256] = {+	0x79e730d418a9143cULL,0x75ba95fc5fedb601ULL,0x79fb732b77622510ULL,0x18905f76a53755c6ULL,0xddf25357ce95560aULL,0x8b4ab8e4ba19e45cULL,0xd2e88688dd21f325ULL,0x8571ff1825885d85ULL,+	0xffac3f904eebc127ULL,0xb027f84a087d81fbULL,0x66ad77dd87cbbc98ULL,0x26936a3fb6ff747eULL,0xb04c5c1fc983a7ebULL,0x583e47ad0861fe1aULL,0x788208311a2ee98eULL,0xd5f06a29e587cc07ULL,+	0xbe1b8aaec45c61f5ULL,0x90ec649a94b9537dULL,0x941cb5aad076c20cULL,0xc9079605890523c8ULL,0xeb309b4ae7ba4f10ULL,0x73c568efe5eb882bULL,0x3540a9877e7a1f68ULL,0x73a076bb2dd1e916ULL,+	0x0746354ea0173b4fULL,0x2bd20213d23c00f7ULL,0xf43eaab50c23bb08ULL,0x13ba5119c3123e03ULL,0x2847d0303f5b9d4dULL,0x6742f2f25da67bddULL,0xef933bdc77c94195ULL,0xeaedd9156e240867ULL,+	0x75c96e8f264e20e8ULL,0xabe6bfed59a7a841ULL,0x2cc09c0444c8eb00ULL,0xe05b3080f0c4e16bULL,0x1eb7777aa45f3314ULL,0x56af7bedce5d45e3ULL,0x2b6e019a88b12f1aULL,0x086659cdfd835f9bULL,+	0xea7d260a6245e404ULL,0x9de407956e7fdfe0ULL,0x1ff3a4158dac1ab5ULL,0x3e7090f1649c9073ULL,0x1a7685612b944e88ULL,0x250f939ee57f61c8ULL,0x0c0daa891ead643dULL,0x68930023e125b88eULL,+	0xccc425634b2ed709ULL,0x0e356769856fd30dULL,0xbcbcd43f559e9811ULL,0x738477ac5395b759ULL,0x35752b90c00ee17fULL,0x68748390742ed2e3ULL,0x7cd06422bd1f5bc1ULL,0xfbc08769c9e7b797ULL,+	0x72bcd8b7bc60055bULL,0x03cc23ee56e27e4bULL,0xee337424e4819370ULL,0xe2aa0e430ad3da09ULL,0x40b8524f6383c45dULL,0xd766355442a41b25ULL,0x64efa6de778a4797ULL,0x2042170a7079adf4ULL,+	0x97091dcbd53c5c9dULL,0xf17624b6ac0a177bULL,0xb0f139752cfe2dffULL,0xc1a35c0a6c7a574eULL,0x227d314693e79987ULL,0x0575bf30e89cb80eULL,0x2f4e247f0d1883bbULL,0xebd512263274c3d0ULL,+	0xfea912baa5659ae8ULL,0x68363aba25e1a16eULL,0xb8842277752c41acULL,0xfe545c282897c3fcULL,0x2d36e9e7dc4c696bULL,0x5806244afba977c5ULL,0x85665e9be39508c1ULL,0xf720ee256d12597bULL,+	0x562e4cecc135b208ULL,0x74e1b2654783f47dULL,0x6d2a506c5a3f3b30ULL,0xecead9f4c16762fcULL,0xf29dd4b2e286e5b9ULL,0x1b0fadc083bb3c61ULL,0x7a75023e7fac29a4ULL,0xc086d5f1c9477fa3ULL,+	0xf4f876532de45068ULL,0x37c7a7e89e2e1f6eULL,0xd0825fa2a3584069ULL,0xaf2cea7c1727bf42ULL,0x0360a4fb9e4785a9ULL,0xe5fda49c27299f4aULL,0x48068e1371ac2f71ULL,0x83d0687b9077666fULL,+	0xa4a319acd837879fULL,0x6fc1b49eed6b67b0ULL,0xe395993332f1f3afULL,0x966742eb65432a2eULL,0x4b8dc9feb4966228ULL,0x96cc631243f43950ULL,0x12068859c9b731eeULL,0x7b948dc356f79968ULL,+	0x042c2af497e2feb4ULL,0xd36a42d7aebf7313ULL,0x49d2c9eb084ffdd7ULL,0x9f8aa54b2ef7c76aULL,0x9200b7ba09895e70ULL,0x3bd0c66fddb7fb58ULL,0x2d97d10878eb4cbbULL,0x2d431068d84bde31ULL,+	0x5e5db46acb66e132ULL,0xf1be963a0d925880ULL,0x944a70270317b9e2ULL,0xe266f95948603d48ULL,0x98db66735c208899ULL,0x90472447a2fb18a3ULL,0x8a966939777c619fULL,0x3798142a2a3be21bULL,+	0xe2f73c696755ff89ULL,0xdd3cf7e7473017e6ULL,0x8ef5689d3cf7600dULL,0x948dc4f8b1fc87b4ULL,0xd9e9fe814ea53299ULL,0x2d921ca298eb6028ULL,0xfaecedfd0c9803fcULL,0xf38ae8914d7b4745ULL,+	0x871514560f664534ULL,0x85ceae7c4b68f103ULL,0xac09c4ae65578ab9ULL,0x33ec6868f044b10cULL,0x6ac4832b3a8ec1f1ULL,0x5509d1285847d5efULL,0xf909604f763f1574ULL,0xb16c4303c32f63c4ULL,+	0xfd16847fdec67ef5ULL,0x742ee464233e76b7ULL,0x0b8e4134efc2b4c8ULL,0xca640b8642a3e521ULL,0x653a01908ceb6aa9ULL,0x313c300c547852d5ULL,0x24e4ab126b237af7ULL,0x2ba901628bb47af8ULL,+	0x00467bc58cce08b5ULL,0xb636458c7f178d55ULL,0xc5748baea677d806ULL,0x2763a387dfa394ebULL,0xa12b448a7d3cebb6ULL,0xe7adda3e6f20d850ULL,0xf63ebce51558462cULL,0x58b36143620088a8ULL,+	0xa9d89488a059c142ULL,0x6f5ae714ff0b9346ULL,0x068f237d16fb3664ULL,0x5853e4c4363186acULL,0xe2d87d2363c52f98ULL,0x2ec4a76681828876ULL,0x47b864fae14e7b1cULL,0x0c0bc0e569192408ULL,+	0x624d60492ed22e91ULL,0x6fdfe0b56f072822ULL,0xeeca111539ce2271ULL,0x98100a4fdb01614fULL,0xb6b0daa2a35c628fULL,0xb6f94d2ec87e9a47ULL,0xc67732591d57d9ceULL,0xf70bfeec03884a7bULL,+	0x4ff23ffd248a7d06ULL,0x80c5bfb4878873faULL,0xb7d9ad9005745981ULL,0x179c85db3db01994ULL,0xba41b06261a6966cULL,0x4d82d052eadce5a8ULL,0x9e91cd3ba5e6a318ULL,0x47795f4f95b2dda0ULL,+	0x1ee426ccd5cd79bfULL,0x0032940b946c6e18ULL,0x1b1e8ae057477f58ULL,0xe94f7d346d823278ULL,0xc747cb96782ba21aULL,0xc5254469f72b33a5ULL,0x772ef6dec7f80c81ULL,0xd73acbfe2cd9e6b5ULL,+	0x283c7513caa76097ULL,0x0a624fa936c83906ULL,0x6b20afec715af2c7ULL,0x4b969974eba78bfdULL,0x220755ccd921d60eULL,0x9b944e107baeca13ULL,0x04819d515ded93d4ULL,0x9bbff86e6dddfd27ULL,+	0x21950b421ff6acd3ULL,0xffe7048453dc6909ULL,0xff4cd0b228766127ULL,0xabdbe6084fb7db2bULL,0x837c92285e1109e8ULL,0x26147d27f4645b5aULL,0x4d78f592f7818ed8ULL,0xd394077ef247fa36ULL,+	0x508cec1c3b3f64c9ULL,0xe20bc0ba1e5edf3fULL,0xda1deb852f4318d4ULL,0xd20ebe0d5c3fa443ULL,0x370b4ea773241ea3ULL,0x61f1511c5e1a5f65ULL,0x99a5e23d82681c62ULL,0xd731e383a2f54c2dULL,+	0x97359638546c4d8dULL,0x5f9c3fc492f24679ULL,0x912e8beda8c8acd9ULL,0xec3a318d306634b0ULL,0x80167f41c31cb264ULL,0x3db82f6f522113f2ULL,0xb155bcd2dcafe197ULL,0xfba1da5943465283ULL,+	0x258bbbf9e7305683ULL,0x31eea5bf07ef5be6ULL,0x0deb0e4a46c814c1ULL,0x5cee8449a7b730ddULL,0xeab495c5a0182bdeULL,0xee759f879e27a6b4ULL,0xc2cf6a6880e518caULL,0x25e8013ff14cf3f4ULL,+	0x3ec832e77acaca28ULL,0x1bfeea57c7385b29ULL,0x068212e3fd1eaf38ULL,0xc13298306acf8cccULL,0xb909f2db2aac9e59ULL,0x5748060db661782aULL,0xc5ab2632c79b7a01ULL,0xda44c6c600017626ULL,+	0x69d44ed65c46aa8eULL,0x2100d5d3a8d063d1ULL,0xcb9727eaa2d17c36ULL,0x4c2bab1b8add53b7ULL,0xa084e90c15426704ULL,0x778afcd3a837ebeaULL,0x6651f7017ce477f8ULL,0xa062499846fb7a8bULL,+	0x3667eb1a7f4c04ccULL,0x59556621a9404f84ULL,0x71cdf6537eceb50aULL,0x994a44a69b8335faULL,0xd7faf819dbeb9b69ULL,0x473c5680eed4350dULL,0xb6658466da44bba2ULL,0x0d1bc780872bdbf3ULL,+	0xb8d3d9319ff91fe5ULL,0x039c4800f0518eedULL,0x95c376329182cb26ULL,0x0763a43482fc568dULL,0x707c04d5383e76baULL,0xac98b930824e8197ULL,0x92bf7c8f91230de0ULL,0x90876a0140959b70ULL,+};
+ cbits/s2n/COMMIT view
@@ -0,0 +1,1 @@+62ec77dc6c2c8cc4c48c768f5f785240b55a47bf
+ cbits/s2n/LICENSE view
@@ -0,0 +1,222 @@+SPDX-License-Identifier: Apache-2.0 OR ISC or MIT-0+++Apache 2.0 license+-------------------------------------+++                                 Apache License+                           Version 2.0, January 2004+                        http://www.apache.org/licenses/++   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION++   1. Definitions.++      "License" shall mean the terms and conditions for use, reproduction,+      and distribution as defined by Sections 1 through 9 of this document.++      "Licensor" shall mean the copyright owner or entity authorized by+      the copyright owner that is granting the License.++      "Legal Entity" shall mean the union of the acting entity and all+      other entities that control, are controlled by, or are under common+      control with that entity. For the purposes of this definition,+      "control" means (i) the power, direct or indirect, to cause the+      direction or management of such entity, whether by contract or+      otherwise, or (ii) ownership of fifty percent (50%) or more of the+      outstanding shares, or (iii) beneficial ownership of such entity.++      "You" (or "Your") shall mean an individual or Legal Entity+      exercising permissions granted by this License.++      "Source" form shall mean the preferred form for making modifications,+      including but not limited to software source code, documentation+      source, and configuration files.++      "Object" form shall mean any form resulting from mechanical+      transformation or translation of a Source form, including but+      not limited to compiled object code, generated documentation,+      and conversions to other media types.++      "Work" shall mean the work of authorship, whether in Source or+      Object form, made available under the License, as indicated by a+      copyright notice that is included in or attached to the work+      (an example is provided in the Appendix below).++      "Derivative Works" shall mean any work, whether in Source or Object+      form, that is based on (or derived from) the Work and for which the+      editorial revisions, annotations, elaborations, or other modifications+      represent, as a whole, an original work of authorship. For the purposes+      of this License, Derivative Works shall not include works that remain+      separable from, or merely link (or bind by name) to the interfaces of,+      the Work and Derivative Works thereof.++      "Contribution" shall mean any work of authorship, including+      the original version of the Work and any modifications or additions+      to that Work or Derivative Works thereof, that is intentionally+      submitted to Licensor for inclusion in the Work by the copyright owner+      or by an individual or Legal Entity authorized to submit on behalf of+      the copyright owner. For the purposes of this definition, "submitted"+      means any form of electronic, verbal, or written communication sent+      to the Licensor or its representatives, including but not limited to+      communication on electronic mailing lists, source code control systems,+      and issue tracking systems that are managed by, or on behalf of, the+      Licensor for the purpose of discussing and improving the Work, but+      excluding communication that is conspicuously marked or otherwise+      designated in writing by the copyright owner as "Not a Contribution."++      "Contributor" shall mean Licensor and any individual or Legal Entity+      on behalf of whom a Contribution has been received by Licensor and+      subsequently incorporated within the Work.++   2. Grant of Copyright License. Subject to the terms and conditions of+      this License, each Contributor hereby grants to You a perpetual,+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable+      copyright license to reproduce, prepare Derivative Works of,+      publicly display, publicly perform, sublicense, and distribute the+      Work and such Derivative Works in Source or Object form.++   3. Grant of Patent License. Subject to the terms and conditions of+      this License, each Contributor hereby grants to You a perpetual,+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable+      (except as stated in this section) patent license to make, have made,+      use, offer to sell, sell, import, and otherwise transfer the Work,+      where such license applies only to those patent claims licensable+      by such Contributor that are necessarily infringed by their+      Contribution(s) alone or by combination of their Contribution(s)+      with the Work to which such Contribution(s) was submitted. If You+      institute patent litigation against any entity (including a+      cross-claim or counterclaim in a lawsuit) alleging that the Work+      or a Contribution incorporated within the Work constitutes direct+      or contributory patent infringement, then any patent licenses+      granted to You under this License for that Work shall terminate+      as of the date such litigation is filed.++   4. Redistribution. You may reproduce and distribute copies of the+      Work or Derivative Works thereof in any medium, with or without+      modifications, and in Source or Object form, provided that You+      meet the following conditions:++      (a) You must give any other recipients of the Work or+          Derivative Works a copy of this License; and++      (b) You must cause any modified files to carry prominent notices+          stating that You changed the files; and++      (c) You must retain, in the Source form of any Derivative Works+          that You distribute, all copyright, patent, trademark, and+          attribution notices from the Source form of the Work,+          excluding those notices that do not pertain to any part of+          the Derivative Works; and++      (d) If the Work includes a "NOTICE" text file as part of its+          distribution, then any Derivative Works that You distribute must+          include a readable copy of the attribution notices contained+          within such NOTICE file, excluding those notices that do not+          pertain to any part of the Derivative Works, in at least one+          of the following places: within a NOTICE text file distributed+          as part of the Derivative Works; within the Source form or+          documentation, if provided along with the Derivative Works; or,+          within a display generated by the Derivative Works, if and+          wherever such third-party notices normally appear. The contents+          of the NOTICE file are for informational purposes only and+          do not modify the License. You may add Your own attribution+          notices within Derivative Works that You distribute, alongside+          or as an addendum to the NOTICE text from the Work, provided+          that such additional attribution notices cannot be construed+          as modifying the License.++      You may add Your own copyright statement to Your modifications and+      may provide additional or different license terms and conditions+      for use, reproduction, or distribution of Your modifications, or+      for any such Derivative Works as a whole, provided Your use,+      reproduction, and distribution of the Work otherwise complies with+      the conditions stated in this License.++   5. Submission of Contributions. Unless You explicitly state otherwise,+      any Contribution intentionally submitted for inclusion in the Work+      by You to the Licensor shall be under the terms and conditions of+      this License, without any additional terms or conditions.+      Notwithstanding the above, nothing herein shall supersede or modify+      the terms of any separate license agreement you may have executed+      with Licensor regarding such Contributions.++   6. Trademarks. This License does not grant permission to use the trade+      names, trademarks, service marks, or product names of the Licensor,+      except as required for reasonable and customary use in describing the+      origin of the Work and reproducing the content of the NOTICE file.++   7. Disclaimer of Warranty. Unless required by applicable law or+      agreed to in writing, Licensor provides the Work (and each+      Contributor provides its Contributions) on an "AS IS" BASIS,+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or+      implied, including, without limitation, any warranties or conditions+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A+      PARTICULAR PURPOSE. You are solely responsible for determining the+      appropriateness of using or redistributing the Work and assume any+      risks associated with Your exercise of permissions under this License.++   8. Limitation of Liability. In no event and under no legal theory,+      whether in tort (including negligence), contract, or otherwise,+      unless required by applicable law (such as deliberate and grossly+      negligent acts) or agreed to in writing, shall any Contributor be+      liable to You for damages, including any direct, indirect, special,+      incidental, or consequential damages of any character arising as a+      result of this License or out of the use or inability to use the+      Work (including but not limited to damages for loss of goodwill,+      work stoppage, computer failure or malfunction, or any and all+      other commercial damages or losses), even if such Contributor+      has been advised of the possibility of such damages.++   9. Accepting Warranty or Additional Liability. While redistributing+      the Work or Derivative Works thereof, You may choose to offer,+      and charge a fee for, acceptance of support, warranty, indemnity,+      or other liability obligations and/or rights consistent with this+      License. However, in accepting such obligations, You may act only+      on Your own behalf and on Your sole responsibility, not on behalf+      of any other Contributor, and only if You agree to indemnify,+      defend, and hold each Contributor harmless for any liability+      incurred by, or claims asserted against, such Contributor by reason+      of your accepting any such warranty or additional liability.++   END OF TERMS AND CONDITIONS+++ISC license+-------------------------------------++Copyright Amazon.com, Inc. or its affiliates.++Permission to use, copy, modify, and/or distribute this software for any+purpose with or without fee is hereby granted, provided that the above+copyright notice and this permission notice appear in all copies.++THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES+WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF+MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR+ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES+WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN+ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF+OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.+++MIT-0 license+-------------------------------------++Copyright 2021-2024 Amazon.com, Inc. or its affiliates.++Permission is hereby granted, free of charge, to any person obtaining a+copy of this software and associated documentation files (the "Software"),+to deal in the Software without restriction, including without limitation+the rights to use, copy, modify, merge, publish, distribute, sublicense,+and/or sell copies of the Software, and to permit persons to whom the+Software is furnished to do so.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER+DEALINGS IN THE SOFTWARE.
+ cbits/s2n/README.md view
@@ -0,0 +1,184 @@+# s2n-bignum++Assembly for the NIST prime curves from AWS's+[s2n-bignum](https://github.com/awslabs/s2n-bignum), vendored here.++`COMMIT` holds the upstream revision these files came from, and `import.sh`+fetches them again.  The files are unmodified: the choice between the two+variants of each routine is made in crypton's own C, not by editing theirs.++## Why++crypton's P-256 is portable C, and on the two architectures s2n-bignum+covers, hand-written assembly beats it by a lot.  Measured against crypton's+own code in the same process, agreeing with it on every scalar tried:++| | crypton | s2n-bignum |+| --- | ---: | ---: |+| Apple M4 | 52.9 us | **20.6** |+| x86-64 with ADX (EPYC 9V74) | 175.9 | **54.5** |+| x86-64, ADX not advertised | 156.1 | **59.4** |++The third row is the `_alt` path, which is what crypton picks where `CPUID`+does not report ADX.  It was measured on a KVM guest whose `CPUID` says so+while the host underneath runs ADX instructions anyway, so it says what the+two implementations cost relative to each other on that path, not what an+actual pre-Broadwell part would do.++Each routine also carries a machine-checked proof in HOL-Light that it+computes what it says, and is written in a constant-time style.++## Licence++`Apache-2.0 OR ISC OR MIT-0`, one of which is on every file and all three in+`LICENSE`.  crypton is BSD-3, so it takes these under **ISC** -- the MIT-0+option would do as well, and the Apache one is the reason OpenSSL's and+BoringSSL's `ecp_nistz256`, which is Apache-2.0 only, cannot be used here.++## Which variant++Both forms of each routine are vendored, because which one is faster is not+the same question on the two architectures:++* **On ARM**, `_alt` is written for parts with high multiplier throughput,+  which is what Apple silicon is, and it wins there by 30-40%.  The plain+  form is for parts that pipeline `UMULH` less well.  There is no feature bit+  for this, so the choice is made at compile time on `__APPLE__`.+* **On x86-64**, the plain form uses `MULX`, `ADCX` and `ADOX`, and `_alt` is+  the fallback for processors without them.  That *is* a feature bit, so the+  choice is made at run time, from `crypton_x86_simd_features()`.++## RSA++`crypton_powm.c`'s modular exponentiation also borrows from here, but only its+innermost step and only on x86-64: `bignum_kmul_16_32`, `bignum_ksqr_16_32`,+`bignum_kmul_32_64`, `bignum_ksqr_32_64` and `bignum_emontredc_8n` replace the+C's Montgomery multiplication and square, and at RSA-3072's 24 limbs the square+is built from `bignum_sqr_8_16` and `bignum_mul_8_16`.  The window, the table and its masked+scan are crypton's throughout.  Sixteen limbs is 1024 bits and thirty-two is+2048: the halves a CRT exponentiation works in for RSA-2048 and RSA-4096, and+the whole thing without CRT.  The reduction wants ADX, so the choice is made at+run time like the other x86-64 ones.++It is twice the C, because the C cannot form the two carry chains `ADCX` and+`ADOX` give.  Measured on an EPYC 7763 at 1024 bits:++| | C | s2n-bignum |+| --- | ---: | ---: |+| multiplication | 0.4832 us | **0.2479** |+| square | 0.3984 | **0.1994** |++### The Montgomery constant++`bignum_montifier` is vendored for **both** architectures.  It is the one+routine here that is not about a particular curve or a particular size: every+modular exponentiation and every setup in `crypton_ecc.c` starts by computing+R^2 mod m, and this computes it.  crypton's C climbs to it by a+binary ladder of Montgomery squarings -- log2 of the modulus' bits of them,+eleven at 2048 -- and the assembly is between two and three and a half times+faster, the gap being the carry chains again:++| | C | s2n-bignum |+| --- | ---: | ---: |+| M4, 1024 bits | 1.853 us | **0.771** |+| M4, 2048 bits | 5.835 | **2.407** |+| an Intel x86-64, 1024 bits | 6.145 | **1.995** |+| an Intel x86-64, 2048 bits | 23.332 | **6.852** |++A CRT signature computes it twice, so an RSA-2048 private operation on the+EPYC 7763 goes from 613.5 to 606.4 microseconds -- a little over one per+cent, which is all a constant computed twice out of some two and a half+thousand multiplications can be.  It is worth much more than that to anything+whose exponent is short, which is why it is here: see `crypton_powm.c`.++The two agree bit for bit; `cbits/tests/powm` requires it at six widths, with+moduli that fill their limbs and moduli that do not, and a build without the+assembly (`-DCRYPTON_NO_MONTIFIER`) is what it is checked against.++**None of the five above is vendored for AArch64**, where they measure level+with the C.  That took three attempts to establish, and the first two were+wrong in opposite directions: a reading when the x86-64 routines went in put+the C 5% ahead, and one on 2026-09-26 put the assembly 4% ahead.  Both were+wall-clock times on a machine with other work on it, where a swing of that size+says nothing.  Measured by thread CPU time, taking the best of twenty-five+batches and alternating the two binaries with the order swapped, one RSA-2048+CRT private operation is 598.9 us through the C and 597.7 through the assembly:+two tenths of a per cent, which is nothing.  The two agree on 200 random cases+at 1024 and 2048 bits, so both were computing the same thing.  Five files for+nothing is not a trade, so the C stays.++The gap to OpenSSL on Apple silicon -- about half its speed -- is assembly and+not the C, and no portable C closes it.  BearSSL's `i62`, which is as far as+portable C is known to go -- 62-bit limbs in 64-bit words, so that a+multiply-accumulate fits an `__int128` with no carry chain at all, and a+five-bit window against crypton's four -- was built and measured the same way+on the same machine: 626.4 us against crypton's C at 668.0, a tie.++`bignum_emontredc_8n_cdiff` was tried too and is not the answer either: it+wants a precomputation this test did not give it, and was slower besides.  What+the window is worth, and why it is four and not five, is in+`cbits/crypton_powm.c` beside the constant.++## Windows++x86-64 Windows takes the assembly as every other x86-64 does.  Two things+stood in the way, and neither is fixed in a vendored file:++* **The ELF directives.**  Each file ends its symbols with `.type` and+  `.size`, which the COFF assemblers refuse -- mingw's gas says "junk at end+  of line" after a warning, clang says "expected absolute expression".+  There is nothing in a PE object for them to say, so on `_WIN32` they are+  emptied, as upstream already does for Mach-O.+* **The calling convention.**  Each file carries a prologue that moves+  Microsoft's argument registers into System V's, under `WINDOWS_ABI`, which+  upstream leaves to the build to define.  On `_WIN32` it is defined.++Both are done in `x86_att/_internal_s2n_bignum_x86_att.h`, which is crypton's+and not upstream's.  Every vendored file includes that name in quotes, and a+quoted include looks first beside the file that asks for it, so this is the+one each gets; it hands on to upstream's in `include/` by a relative path.+**It must not be removed as an upstream file that went missing**: without it+Windows does not assemble.  On Linux and macOS it changes nothing -- the+objects are byte for byte what they were without it.++Not `#include_next`, which is the obvious way to hand on: clang, finding this+header beside the file that asked for it, starts that search from the top+and finds this header again, without end.++Windows on ARM keeps the C.  The AArch64 files have the same directives and+would want the same header, and there is no runner to check one.++## What is not here++s2n-bignum has only x86-64 and AArch64, so crypton's C stays and is what+every other architecture uses.++Its `generic` directory holds a complete variable-size bignum library on both+architectures -- `bignum_montmul`, `bignum_montredc`, `bignum_emontredc`,+`bignum_modinv` and a whole `bignum_modexp` -- and only `bignum_montifier` is+taken from it.  **The rest was measured on an M4 and is slower than what is+here**, which is why it stays out; they agree with crypton's answers in every+case first, so both were computing the same thing.++| nanoseconds, AArch64 | C | s2n-bignum |+| --- | ---: | ---: |+| Montgomery multiply, 1024 bits | **171.7** | 227.9 |+| Montgomery multiply, 2048 bits | **649.6** | 870.3 |+| Montgomery square, 1024 bits | **157.3** | 202.6 |+| Montgomery square, 2048 bits | **539.2** | 870.8 |++That is the inline assembly above doing its job: four limbs to an iteration+with two carry chains, where a size-parametric routine takes them one at a+time.  `bignum_modexp`, which would have replaced `crypton_powm_sec` outright,+loses by more again -- 421.0 microseconds against 246.4 at 1024 bits and 3578+against 1534 at 2048 -- because it has that multiplication inside it and no+window, so it pays about twice the multiplications as well.++The README there says every function is written in a constant-time style, with+one named exception that is none of these, though it also says that unlike+functional correctness this is not machine-checked.++`p384_montjscalarmul` and `p521_jscalarmul`+have no affine wrapper upstream; crypton's is in `cbits/p256/p256_s2n.c`'s+sibling for those curves.  There is no fixed-base routine for P-384 or+P-521 at all, so signing on those curves keeps crypton's comb.
+ cbits/s2n/arm/bignum_deamont_p384.S view
@@ -0,0 +1,151 @@+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0++// ----------------------------------------------------------------------------+// Convert from almost-Montgomery form, z := (x / 2^384) mod p_384+// Input x[6]; output z[6]+//+//    extern void bignum_deamont_p384(uint64_t z[static 6],+//                                    const uint64_t x[static 6]);+//+// Convert a 6-digit bignum x out of its (optionally almost) Montgomery form,+// "almost" meaning any 6-digit input will work, with no range restriction.+//+// Standard ARM ABI: X0 = z, X1 = x+// ----------------------------------------------------------------------------++#include "_internal_s2n_bignum_arm.h"++        S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_deamont_p384)+        S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_deamont_p384)+        S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_deamont_p384)+        S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_deamont_p384_alt)+        S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_deamont_p384_alt)+        S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_deamont_p384_alt)+        .text+        .balign 4++// ---------------------------------------------------------------------------+// Core one-step "short" Montgomery reduction macro. Takes input in+// [d5;d4;d3;d2;d1;d0] and returns result in [d6;d5;d4;d3;d2;d1],+// adding to the existing contents of [d5;d4;d3;d2;d1]. It is fine+// for d6 to be the same register as d0.+//+// We want to add (2^384 - 2^128 - 2^96 + 2^32 - 1) * w+// where w = [d0 + (d0<<32)] mod 2^64+// ---------------------------------------------------------------------------++#define montreds(d6,d5,d4,d3,d2,d1,d0, t3,t2,t1)                            \+/* Our correction multiplier is w = [d0 + (d0<<32)] mod 2^64            */  \+/* Recycle d0 (which we know gets implicitly cancelled) to store it     */  \+        lsl     t1, d0, #32 __LF                                       \+        add     d0, t1, d0 __LF                                        \+/* Now let [t2;t1] = 2^64 * w - w + w_hi where w_hi = floor(w/2^32)     */  \+/* We need to subtract 2^32 * this, and we can ignore its lower 32      */  \+/* bits since by design it will cancel anyway; we only need the w_hi    */  \+/* part to get the carry propagation going.                             */  \+        lsr     t1, d0, #32 __LF                                       \+        subs    t1, t1, d0 __LF                                        \+        sbc     t2, d0, xzr __LF                                       \+/* Now select in t1 the field to subtract from d1                       */  \+        extr    t1, t2, t1, #32 __LF                                   \+/* And now get the terms to subtract from d2 and d3                     */  \+        lsr     t2, t2, #32 __LF                                       \+        adds    t2, t2, d0 __LF                                        \+        adc     t3, xzr, xzr __LF                                      \+/* Do the subtraction of that portion                                   */  \+        subs    d1, d1, t1 __LF                                        \+        sbcs    d2, d2, t2 __LF                                        \+        sbcs    d3, d3, t3 __LF                                        \+        sbcs    d4, d4, xzr __LF                                       \+        sbcs    d5, d5, xzr __LF                                       \+/* Now effectively add 2^384 * w by taking d0 as the input for last sbc */  \+        sbc     d6, d0, xzr++// Input parameters++#define z x0+#define x x1++// Rotating registers for the intermediate windows++#define d0 x2+#define d1 x3+#define d2 x4+#define d3 x5+#define d4 x6+#define d5 x7++// Other temporaries++#define u x8+#define v x9+#define w x10++S2N_BN_SYMBOL(bignum_deamont_p384):++S2N_BN_SYMBOL(bignum_deamont_p384_alt):+        CFI_START++// Set up an initial window with the input x and an extra leading zero++        ldp     d0, d1, [x]+        ldp     d2, d3, [x, #16]+        ldp     d4, d5, [x, #32]++// Systematically scroll left doing 1-step reductions++        montreds(d0,d5,d4,d3,d2,d1,d0, u,v,w)++        montreds(d1,d0,d5,d4,d3,d2,d1, u,v,w)++        montreds(d2,d1,d0,d5,d4,d3,d2, u,v,w)++        montreds(d3,d2,d1,d0,d5,d4,d3, u,v,w)++        montreds(d4,d3,d2,d1,d0,d5,d4, u,v,w)++        montreds(d5,d4,d3,d2,d1,d0,d5, u,v,w)++// Now compare end result in [d5;d4;d3;d2;d1;d0] = dd with p_384 by *adding*+// 2^384 - p_384 = [0;0;0;w;v;u]. This will set CF if+// dd + (2^384 - p_384) >= 2^384, hence iff dd >= p_384++        mov     u, #0xffffffff00000001+        mov     v, #0x00000000ffffffff+        mov     w, #0x0000000000000001++        adds    xzr, d0, u+        adcs    xzr, d1, v+        adcs    xzr, d2, w+        adcs    xzr, d3, xzr+        adcs    xzr, d4, xzr+        adcs    xzr, d5, xzr++// Convert the condition dd >= p_384 into a bitmask in w and do a masked+// subtraction of p_384, via a masked addition of 2^384 - p_384:++        csetm   w, cs+        and     u, u, w+        adds    d0, d0, u+        and     v, v, w+        adcs    d1, d1, v+        and     w, w, #1+        adcs    d2, d2, w+        adcs    d3, d3, xzr+        adcs    d4, d4, xzr+        adc     d5, d5, xzr++// Store it back++        stp     d0, d1, [z]+        stp     d2, d3, [z, #16]+        stp     d4, d5, [z, #32]++        CFI_RET++S2N_BN_SIZE_DIRECTIVE(bignum_deamont_p384)++#if defined(__linux__) && defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/s2n/arm/bignum_demont_p256.S view
@@ -0,0 +1,99 @@+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0++// ----------------------------------------------------------------------------+// Convert from Montgomery form z := (x / 2^256) mod p_256, assuming x reduced+// Input x[4]; output z[4]+//+//    extern void bignum_demont_p256(uint64_t z[static 4],+//                                   const uint64_t x[static 4]);+//+// This assumes the input is < p_256 for correctness. If this is not the case,+// use the variant "bignum_deamont_p256" instead.+//+// Standard ARM ABI: X0 = z, X1 = x+// ----------------------------------------------------------------------------++#include "_internal_s2n_bignum_arm.h"++        S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_demont_p256)+        S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_demont_p256)+        S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_demont_p256)+        S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_demont_p256_alt)+        S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_demont_p256_alt)+        S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_demont_p256_alt)+        .text+        .balign 4++// ---------------------------------------------------------------------------+// Core one-step "short" Montgomery reduction macro. Takes input in+// [d3;d2;d1;d0] and returns result in [d4;d3;d2;d1], adding to the+// existing contents of [d3;d2;d1] and generating d4 from zero, re-using+// d0 as a temporary internally together with t0, t1 and t2.+// It is fine for d4 to be the same register as d0, and it often is.+// ---------------------------------------------------------------------------++#define montreds(d4,d3,d2,d1,d0, t2,t1,t0)                                  \+/* Let w = d0, the original word we use as offset; d0 gets recycled      */ \+/* First let [t2;t1] = 2^32 * w                                          */ \+/* then let [d0;t0] = (2^64 - 2^32 + 1) * w (overwrite old d0)           */ \+        lsl     t1, d0, #32 __LF                                       \+        subs    t0, d0, t1 __LF                                        \+        lsr     t2, d0, #32 __LF                                       \+        sbc     d0, d0, t2 __LF                                        \+/* Hence [d4;..;d1] := [d3;d2;d1;0] + (2^256 - 2^224 + 2^192 + 2^96) * w */ \+        adds    d1, d1, t1 __LF                                        \+        adcs    d2, d2, t2 __LF                                        \+        adcs    d3, d3, t0 __LF                                        \+        adc     d4, d0, xzr++// Input parameters++#define z x0+#define x x1++// Rotating registers for the intermediate windows (with repetitions)++#define d0 x2+#define d1 x3+#define d2 x4+#define d3 x5++// Other temporaries++#define u x6+#define v x7+#define w x8++S2N_BN_SYMBOL(bignum_demont_p256):++S2N_BN_SYMBOL(bignum_demont_p256_alt):+        CFI_START++// Set up an initial window with the input x and an extra leading zero++        ldp     d0, d1, [x]+        ldp     d2, d3, [x, #16]++// Systematically scroll left doing 1-step reductions++        montreds(d0,d3,d2,d1,d0, u,v,w)++        montreds(d1,d0,d3,d2,d1, u,v,w)++        montreds(d2,d1,d0,d3,d2, u,v,w)++        montreds(d3,d2,d1,d0,d3, u,v,w)++// Write back result++        stp     d0, d1, [z]+        stp     d2, d3, [z, #16]++        CFI_RET++S2N_BN_SIZE_DIRECTIVE(bignum_demont_p256)++#if defined(__linux__) && defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/s2n/arm/bignum_inv_p521.S view
@@ -0,0 +1,1701 @@+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0++// ----------------------------------------------------------------------------+// Modular inverse modulo p_521 =  2^521 - 1+// Input x[9]; output z[9]+//+// extern void bignum_inv_p521(uint64_t z[static 9],const uint64_t x[static 9]);+//+// Assuming the 9-digit input x is coprime to p_521, i.e. is not divisible+// by it, returns z < p_521 such that x * z == 1 (mod p_521). Note that+// x does not need to be reduced modulo p_521, but the output always is.+//+// Standard ARM ABI: X0 = z, X1 = x+// ----------------------------------------------------------------------------++#include "_internal_s2n_bignum_arm.h"++        S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_inv_p521)+        S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_inv_p521)+        S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_inv_p521)++        .text+        .balign 4++// Size in bytes of a 64-bit word++#define N 8++// Used for the return pointer++#define res x20++// Loop counter and d = 2 * delta value for divstep++#define i x21+#define d x22++// Registers used for matrix element magnitudes and signs++#define m00 x10+#define m01 x11+#define m10 x12+#define m11 x13+#define s00 x14+#define s01 x15+#define s10 x16+#define s11 x17++// Initial carries for combinations++#define car0 x9+#define car1 x19++// Input and output, plain registers treated according to pattern++#define reg0 x0, #0+#define reg1 x1, #0+#define reg2 x2, #0+#define reg3 x3, #0+#define reg4 x4, #0++#define x x1, #0+#define z x0, #0++// Pointer-offset pairs for temporaries on stack++#define f sp, #0+#define g sp, #(9*N)+#define u sp, #(18*N)+#define v sp, #(27*N)++// Total size to reserve on the stack++#define NSPACE 36*N++// Very similar to a subroutine call to the s2n-bignum word_divstep59.+// But different in register usage and returning the final matrix in+// registers as follows+//+// [ m00  m01]+// [ m10  m11]++#define divstep59()                                                     \+        and     x4, x2, #0xfffff __LF                                      \+        orr     x4, x4, #0xfffffe0000000000 __LF                           \+        and     x5, x3, #0xfffff __LF                                      \+        orr     x5, x5, #0xc000000000000000 __LF                           \+        tst     x5, #0x1 __LF                                              \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        asr     x5, x5, #1 __LF                                            \+        add     x8, x4, #0x100, lsl #12 __LF                               \+        sbfx    x8, x8, #21, #21 __LF                                      \+        mov     x11, #0x100000 __LF                                        \+        add     x11, x11, x11, lsl #21 __LF                                \+        add     x9, x4, x11 __LF                                           \+        asr     x9, x9, #42 __LF                                           \+        add     x10, x5, #0x100, lsl #12 __LF                              \+        sbfx    x10, x10, #21, #21 __LF                                    \+        add     x11, x5, x11 __LF                                          \+        asr     x11, x11, #42 __LF                                         \+        mul     x6, x8, x2 __LF                                            \+        mul     x7, x9, x3 __LF                                            \+        mul     x2, x10, x2 __LF                                           \+        mul     x3, x11, x3 __LF                                           \+        add     x4, x6, x7 __LF                                            \+        add     x5, x2, x3 __LF                                            \+        asr     x2, x4, #20 __LF                                           \+        asr     x3, x5, #20 __LF                                           \+        and     x4, x2, #0xfffff __LF                                      \+        orr     x4, x4, #0xfffffe0000000000 __LF                           \+        and     x5, x3, #0xfffff __LF                                      \+        orr     x5, x5, #0xc000000000000000 __LF                           \+        tst     x5, #0x1 __LF                                              \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        asr     x5, x5, #1 __LF                                            \+        add     x12, x4, #0x100, lsl #12 __LF                              \+        sbfx    x12, x12, #21, #21 __LF                                    \+        mov     x15, #0x100000 __LF                                        \+        add     x15, x15, x15, lsl #21 __LF                                \+        add     x13, x4, x15 __LF                                          \+        asr     x13, x13, #42 __LF                                         \+        add     x14, x5, #0x100, lsl #12 __LF                              \+        sbfx    x14, x14, #21, #21 __LF                                    \+        add     x15, x5, x15 __LF                                          \+        asr     x15, x15, #42 __LF                                         \+        mul     x6, x12, x2 __LF                                           \+        mul     x7, x13, x3 __LF                                           \+        mul     x2, x14, x2 __LF                                           \+        mul     x3, x15, x3 __LF                                           \+        add     x4, x6, x7 __LF                                            \+        add     x5, x2, x3 __LF                                            \+        asr     x2, x4, #20 __LF                                           \+        asr     x3, x5, #20 __LF                                           \+        and     x4, x2, #0xfffff __LF                                      \+        orr     x4, x4, #0xfffffe0000000000 __LF                           \+        and     x5, x3, #0xfffff __LF                                      \+        orr     x5, x5, #0xc000000000000000 __LF                           \+        tst     x5, #0x1 __LF                                              \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        mul     x2, x12, x8 __LF                                           \+        mul     x3, x12, x9 __LF                                           \+        mul     x6, x14, x8 __LF                                           \+        mul     x7, x14, x9 __LF                                           \+        madd    x8, x13, x10, x2 __LF                                      \+        madd    x9, x13, x11, x3 __LF                                      \+        madd    x16, x15, x10, x6 __LF                                     \+        madd    x17, x15, x11, x7 __LF                                     \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        asr     x5, x5, #1 __LF                                            \+        add     x12, x4, #0x100, lsl #12 __LF                              \+        sbfx    x12, x12, #22, #21 __LF                                    \+        mov     x15, #0x100000 __LF                                        \+        add     x15, x15, x15, lsl #21 __LF                                \+        add     x13, x4, x15 __LF                                          \+        asr     x13, x13, #43 __LF                                         \+        add     x14, x5, #0x100, lsl #12 __LF                              \+        sbfx    x14, x14, #22, #21 __LF                                    \+        add     x15, x5, x15 __LF                                          \+        asr     x15, x15, #43 __LF                                         \+        mneg    x2, x12, x8 __LF                                           \+        mneg    x3, x12, x9 __LF                                           \+        mneg    x4, x14, x8 __LF                                           \+        mneg    x5, x14, x9 __LF                                           \+        msub    m00, x13, x16, x2 __LF                                     \+        msub    m01, x13, x17, x3 __LF                                     \+        msub    m10, x15, x16, x4 __LF                                     \+        msub    m11, x15, x17, x5++// Loading large constants++#define movbig(nn,n3,n2,n1,n0)                                      \+        movz    nn, n0 __LF                                            \+        movk    nn, n1, lsl #16 __LF                                   \+        movk    nn, n2, lsl #32 __LF                                   \+        movk    nn, n3, lsl #48++S2N_BN_SYMBOL(bignum_inv_p521):+        CFI_START++// Save registers and make room for temporaries++        CFI_PUSH2(x19,x20)+        CFI_PUSH2(x21,x22)+        CFI_DEC_SP(NSPACE)++// Save the return pointer for the end so we can overwrite x0 later++        mov     res, x0++// Copy the prime p_521 = 2^521 - 1 into the f variable++        mov     x10, #0xFFFFFFFFFFFFFFFF+        stp     x10, x10, [f]+        stp     x10, x10, [f+16]+        stp     x10, x10, [f+32]+        stp     x10, x10, [f+48]+        mov     x11, #0x1FF+        str     x11, [f+64]++// Copy the input into the g variable, but reduce it strictly mod p_521+// so that g <= f as assumed in the bound proof. This code fragment is+// very similar to bignum_mod_p521_9 complete with carry condensation.++        ldr     x8, [x1, #64]+        lsr     x9, x8, #9++        subs    xzr, xzr, xzr+        ldp     x10, x11, [x1]+        adcs    xzr, x10, x9+        adcs    xzr, x11, xzr+        ldp     x12, x13, [x1, #16]+        and     x7, x12, x13+        adcs    xzr, x7, xzr+        ldp     x14, x15, [x1, #32]+        and     x7, x14, x15+        adcs    xzr, x7, xzr+        ldp     x16, x17, [x1, #48]+        and     x7, x16, x17+        adcs    xzr, x7, xzr+        orr     x7, x8, #~0x1FF+        adcs    x7, x7, xzr++        adcs    x10, x10, x9+        adcs    x11, x11, xzr+        adcs    x12, x12, xzr+        adcs    x13, x13, xzr+        adcs    x14, x14, xzr+        adcs    x15, x15, xzr+        adcs    x16, x16, xzr+        adcs    x17, x17, xzr+        adc     x8, x8, xzr+        and     x8, x8, #0x1FF++        stp     x10, x11, [g]+        stp     x12, x13, [g+16]+        stp     x14, x15, [g+32]+        stp     x16, x17, [g+48]+        str     x8, [g+64]++// Also maintain weakly reduced < 2*p_521 vector [u,v] such that+// [f,g] == x * 2^{1239-59*i} * [u,v] (mod p_521)+// starting with [p_521,x] == x * 2^{1239-59*0} * [0,2^-1239] (mod p_521)+// Note that because (2^{a+521} == 2^a) (mod p_521) we simply have+// (2^-1239 == 2^324) (mod p_521) so the constant initializer is simple.+//+// Based on the standard divstep bound, for inputs <= 2^b we need at least+// n >= (9437 * b + 1) / 4096. Since b is 521, that means 1201 iterations.+// Since we package divstep in multiples of 59 bits, we do 21 blocks of 59+// making *1239* total. (With a bit more effort we could avoid the full 59+// divsteps and use a shorter tail computation, but we keep it simple.)+// Hence, after the 21st iteration we have [f,g] == x * [u,v] and since+// |f| = 1 we get the modular inverse from u by flipping its sign with f.++        stp     xzr, xzr, [u]+        stp     xzr, xzr, [u+16]+        stp     xzr, xzr, [u+32]+        stp     xzr, xzr, [u+48]+        str     xzr, [u+64]++        mov     x10, #16+        stp     xzr, xzr, [v]+        stp     xzr, xzr, [v+16]+        stp     xzr, x10, [v+32]+        stp     xzr, xzr, [v+48]+        str     xzr, [v+64]++// Start of main loop. We jump into the middle so that the divstep+// portion is common to the special 21st iteration after a uniform+// first 20.++        mov     i, #21+        mov     d, #1+        b       Lbignum_inv_p521_midloop++Lbignum_inv_p521_loop:++// Separate the matrix elements into sign-magnitude pairs++        cmp     m00, xzr+        csetm   s00, mi+        cneg    m00, m00, mi++        cmp     m01, xzr+        csetm   s01, mi+        cneg    m01, m01, mi++        cmp     m10, xzr+        csetm   s10, mi+        cneg    m10, m10, mi++        cmp     m11, xzr+        csetm   s11, mi+        cneg    m11, m11, mi++// Adjust the initial values to allow for complement instead of negation+// This initial offset is the same for [f,g] and [u,v] compositions.+// Save it in stable registers for the [u,v] part and do [f,g] first.++        and     x0, m00, s00+        and     x1, m01, s01+        add     car0, x0, x1++        and     x0, m10, s10+        and     x1, m11, s11+        add     car1, x0, x1++// Now the computation of the updated f and g values. This maintains a+// 2-word carry between stages so we can conveniently insert the shift+// right by 59 before storing back, and not overwrite digits we need+// again of the old f and g values.+//+// Digit 0 of [f,g]++        ldr     x7, [f]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x4, car0, x0+        adc     x2, xzr, x1+        ldr     x8, [g]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x4, x4, x0+        adc     x2, x2, x1++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x5, car1, x0+        adc     x3, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x5, x5, x0+        adc     x3, x3, x1++// Digit 1 of [f,g]++        ldr     x7, [f+N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x2, x2, x0+        adc     x6, xzr, x1+        ldr     x8, [g+N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x2, x2, x0+        adc     x6, x6, x1+        extr    x4, x2, x4, #59+        str     x4, [f]++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x3, x3, x0+        adc     x4, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x3, x3, x0+        adc     x4, x4, x1+        extr    x5, x3, x5, #59+        str     x5, [g]++// Digit 2 of [f,g]++        ldr     x7, [f+2*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x6, x6, x0+        adc     x5, xzr, x1+        ldr     x8, [g+2*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x6, x6, x0+        adc     x5, x5, x1+        extr    x2, x6, x2, #59+        str     x2, [f+N]++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x4, x4, x0+        adc     x2, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x4, x4, x0+        adc     x2, x2, x1+        extr    x3, x4, x3, #59+        str     x3, [g+N]++// Digit 3 of [f,g]++        ldr     x7, [f+3*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x5, x5, x0+        adc     x3, xzr, x1+        ldr     x8, [g+3*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x5, x5, x0+        adc     x3, x3, x1+        extr    x6, x5, x6, #59+        str     x6, [f+2*N]++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x2, x2, x0+        adc     x6, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x2, x2, x0+        adc     x6, x6, x1+        extr    x4, x2, x4, #59+        str     x4, [g+2*N]++// Digit 4 of [f,g]++        ldr     x7, [f+4*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x3, x3, x0+        adc     x4, xzr, x1+        ldr     x8, [g+4*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x3, x3, x0+        adc     x4, x4, x1+        extr    x5, x3, x5, #59+        str     x5, [f+3*N]++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x6, x6, x0+        adc     x5, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x6, x6, x0+        adc     x5, x5, x1+        extr    x2, x6, x2, #59+        str     x2, [g+3*N]++// Digit 5 of [f,g]++        ldr     x7, [f+5*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x4, x4, x0+        adc     x2, xzr, x1+        ldr     x8, [g+5*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x4, x4, x0+        adc     x2, x2, x1+        extr    x3, x4, x3, #59+        str     x3, [f+4*N]++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x5, x5, x0+        adc     x3, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x5, x5, x0+        adc     x3, x3, x1+        extr    x6, x5, x6, #59+        str     x6, [g+4*N]++// Digit 6 of [f,g]++        ldr     x7, [f+6*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x2, x2, x0+        adc     x6, xzr, x1+        ldr     x8, [g+6*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x2, x2, x0+        adc     x6, x6, x1+        extr    x4, x2, x4, #59+        str     x4, [f+5*N]++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x3, x3, x0+        adc     x4, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x3, x3, x0+        adc     x4, x4, x1+        extr    x5, x3, x5, #59+        str     x5, [g+5*N]++// Digit 7 of [f,g]++        ldr     x7, [f+7*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x6, x6, x0+        adc     x5, xzr, x1+        ldr     x8, [g+7*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x6, x6, x0+        adc     x5, x5, x1+        extr    x2, x6, x2, #59+        str     x2, [f+6*N]++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x4, x4, x0+        adc     x2, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x4, x4, x0+        adc     x2, x2, x1+        extr    x3, x4, x3, #59+        str     x3, [g+6*N]++// Digits 8 and 9 of [f,g]++        ldr     x7, [f+8*N]+        eor     x1, x7, s00+        asr     x3, x1, #63+        and     x3, x3, m00+        neg     x3, x3+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x5, x5, x0+        adc     x3, x3, x1+        ldr     x8, [g+8*N]+        eor     x1, x8, s01+        asr     x0, x1, #63+        and     x0, x0, m01+        sub     x3, x3, x0+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x5, x5, x0+        adc     x3, x3, x1+        extr    x6, x5, x6, #59+        str     x6, [f+7*N]+        extr    x5, x3, x5, #59+        str     x5, [f+8*N]++        eor     x1, x7, s10+        asr     x5, x1, #63+        and     x5, x5, m10+        neg     x5, x5+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x2, x2, x0+        adc     x5, x5, x1+        eor     x1, x8, s11+        asr     x0, x1, #63+        and     x0, x0, m11+        sub     x5, x5, x0+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x2, x2, x0+        adc     x5, x5, x1+        extr    x4, x2, x4, #59+        str     x4, [g+7*N]+        extr    x2, x5, x2, #59+        str     x2, [g+8*N]++// Now the computation of the updated u and v values and their+// modular reductions. A very similar accumulation except that+// the top words of u and v are unsigned and we don't shift.+//+// Digit 0 of [u,v]++        ldr     x7, [u]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x4, car0, x0+        adc     x2, xzr, x1+        ldr     x8, [v]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x4, x4, x0+        str     x4, [u]+        adc     x2, x2, x1++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x5, car1, x0+        adc     x3, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x5, x5, x0+        str     x5, [v]+        adc     x3, x3, x1++// Digit 1 of [u,v]++        ldr     x7, [u+N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x2, x2, x0+        adc     x6, xzr, x1+        ldr     x8, [v+N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x2, x2, x0+        str     x2, [u+N]+        adc     x6, x6, x1++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x3, x3, x0+        adc     x4, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x3, x3, x0+        str     x3, [v+N]+        adc     x4, x4, x1++// Digit 2 of [u,v]++        ldr     x7, [u+2*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x6, x6, x0+        adc     x5, xzr, x1+        ldr     x8, [v+2*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x6, x6, x0+        str     x6, [u+2*N]+        adc     x5, x5, x1++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x4, x4, x0+        adc     x2, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x4, x4, x0+        str     x4, [v+2*N]+        adc     x2, x2, x1++// Digit 3 of [u,v]++        ldr     x7, [u+3*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x5, x5, x0+        adc     x3, xzr, x1+        ldr     x8, [v+3*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x5, x5, x0+        str     x5, [u+3*N]+        adc     x3, x3, x1++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x2, x2, x0+        adc     x6, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x2, x2, x0+        str     x2, [v+3*N]+        adc     x6, x6, x1++// Digit 4 of [u,v]++        ldr     x7, [u+4*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x3, x3, x0+        adc     x4, xzr, x1+        ldr     x8, [v+4*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x3, x3, x0+        str     x3, [u+4*N]+        adc     x4, x4, x1++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x6, x6, x0+        adc     x5, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x6, x6, x0+        str     x6, [v+4*N]+        adc     x5, x5, x1++// Digit 5 of [u,v]++        ldr     x7, [u+5*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x4, x4, x0+        adc     x2, xzr, x1+        ldr     x8, [v+5*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x4, x4, x0+        str     x4, [u+5*N]+        adc     x2, x2, x1++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x5, x5, x0+        adc     x3, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x5, x5, x0+        str     x5, [v+5*N]+        adc     x3, x3, x1++// Digit 6 of [u,v]++        ldr     x7, [u+6*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x2, x2, x0+        adc     x6, xzr, x1+        ldr     x8, [v+6*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x2, x2, x0+        str     x2, [u+6*N]+        adc     x6, x6, x1++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x3, x3, x0+        adc     x4, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x3, x3, x0+        str     x3, [v+6*N]+        adc     x4, x4, x1++// Digit 7 of [u,v]++        ldr     x7, [u+7*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x6, x6, x0+        adc     x5, xzr, x1+        ldr     x8, [v+7*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x6, x6, x0+        str     x6, [u+7*N]+        adc     x5, x5, x1++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x4, x4, x0+        adc     x2, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x4, x4, x0+        str     x4, [v+7*N]+        adc     x2, x2, x1++// Digits 8 and 9 of u (top is unsigned)++        ldr     x7, [u+8*N]+        eor     x1, x7, s00+        and     x3, s00, m00+        neg     x3, x3+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x5, x5, x0+        adc     x3, x3, x1+        ldr     x8, [v+8*N]+        eor     x1, x8, s01+        and     x0, s01, m01+        sub     x3, x3, x0+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x5, x5, x0+        adc     x3, x3, x1++// Modular reduction of u, reloading as needed from u[0],...,u[7],x5,x3++        extr    x6, x3, x5, #9+        ldp     x0, x1, [u]+        add     x6, x6, x3, asr #63+        sub     x5, x5, x6, lsl #9+        adds    x0, x0, x6+        asr     x6, x6, #63+        adcs    x1, x1, x6+        stp     x0, x1, [u]+        ldp     x0, x1, [u+16]+        adcs    x0, x0, x6+        adcs    x1, x1, x6+        stp     x0, x1, [u+16]+        ldp     x0, x1, [u+32]+        adcs    x0, x0, x6+        adcs    x1, x1, x6+        stp     x0, x1, [u+32]+        ldp     x0, x1, [u+48]+        adcs    x0, x0, x6+        adcs    x1, x1, x6+        stp     x0, x1, [u+48]+        adc     x5, x5, x6+        str     x5, [u+64]++// Digits 8 and 9 of v (top is unsigned)++        eor     x1, x7, s10+        and     x5, s10, m10+        neg     x5, x5+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x2, x2, x0+        adc     x5, x5, x1+        eor     x1, x8, s11+        and     x0, s11, m11+        sub     x5, x5, x0+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x2, x2, x0+        adc     x5, x5, x1++// Modular reduction of v, reloading as needed from v[0],...,v[7],x2,x5++        extr    x6, x5, x2, #9+        ldp     x0, x1, [v]+        add     x6, x6, x5, asr #63+        sub     x2, x2, x6, lsl #9+        adds    x0, x0, x6+        asr     x6, x6, #63+        adcs    x1, x1, x6+        stp     x0, x1, [v]+        ldp     x0, x1, [v+16]+        adcs    x0, x0, x6+        adcs    x1, x1, x6+        stp     x0, x1, [v+16]+        ldp     x0, x1, [v+32]+        adcs    x0, x0, x6+        adcs    x1, x1, x6+        stp     x0, x1, [v+32]+        ldp     x0, x1, [v+48]+        adcs    x0, x0, x6+        adcs    x1, x1, x6+        stp     x0, x1, [v+48]+        adc     x2, x2, x6+        str     x2, [v+64]++Lbignum_inv_p521_midloop:++        mov     x1, d+        ldr     x2, [f]+        ldr     x3, [g]+        divstep59()+        mov     d, x1++// Next iteration++        subs    i, i, #1+        bne     Lbignum_inv_p521_loop++// The 21st and last iteration does not need anything except the+// u value and the sign of f; the latter can be obtained from the+// lowest word of f. So it's done differently from the main loop.+// Find the sign of the new f. For this we just need one digit+// since we know (for in-scope cases) that f is either +1 or -1.+// We don't explicitly shift right by 59 either, but looking at+// bit 63 (or any bit >= 60) of the unshifted result is enough+// to distinguish -1 from +1; this is then made into a mask.++        ldr     x0, [f]+        ldr     x1, [g]+        mul     x0, x0, m00+        madd    x1, x1, m01, x0+        asr     x0, x1, #63++// Now separate out the matrix into sign-magnitude pairs+// and adjust each one based on the sign of f.+//+// Note that at this point we expect |f|=1 and we got its+// sign above, so then since [f,0] == x * [u,v] (mod p_521)+// we want to flip the sign of u according to that of f.++        cmp     m00, xzr+        csetm   s00, mi+        cneg    m00, m00, mi+        eor     s00, s00, x0++        cmp     m01, xzr+        csetm   s01, mi+        cneg    m01, m01, mi+        eor     s01, s01, x0++        cmp     m10, xzr+        csetm   s10, mi+        cneg    m10, m10, mi+        eor     s10, s10, x0++        cmp     m11, xzr+        csetm   s11, mi+        cneg    m11, m11, mi+        eor     s11, s11, x0++// Adjust the initial value to allow for complement instead of negation++        and     x0, m00, s00+        and     x1, m01, s01+        add     car0, x0, x1++// Digit 0 of [u]++        ldr     x7, [u]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x4, car0, x0+        adc     x2, xzr, x1+        ldr     x8, [v]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x4, x4, x0+        str     x4, [u]+        adc     x2, x2, x1++// Digit 1 of [u]++        ldr     x7, [u+N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x2, x2, x0+        adc     x6, xzr, x1+        ldr     x8, [v+N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x2, x2, x0+        str     x2, [u+N]+        adc     x6, x6, x1++// Digit 2 of [u]++        ldr     x7, [u+2*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x6, x6, x0+        adc     x5, xzr, x1+        ldr     x8, [v+2*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x6, x6, x0+        str     x6, [u+2*N]+        adc     x5, x5, x1++// Digit 3 of [u]++        ldr     x7, [u+3*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x5, x5, x0+        adc     x3, xzr, x1+        ldr     x8, [v+3*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x5, x5, x0+        str     x5, [u+3*N]+        adc     x3, x3, x1++// Digit 4 of [u]++        ldr     x7, [u+4*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x3, x3, x0+        adc     x4, xzr, x1+        ldr     x8, [v+4*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x3, x3, x0+        str     x3, [u+4*N]+        adc     x4, x4, x1++// Digit 5 of [u]++        ldr     x7, [u+5*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x4, x4, x0+        adc     x2, xzr, x1+        ldr     x8, [v+5*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x4, x4, x0+        str     x4, [u+5*N]+        adc     x2, x2, x1++// Digit 6 of [u]++        ldr     x7, [u+6*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x2, x2, x0+        adc     x6, xzr, x1+        ldr     x8, [v+6*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x2, x2, x0+        str     x2, [u+6*N]+        adc     x6, x6, x1++// Digit 7 of [u]++        ldr     x7, [u+7*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x6, x6, x0+        adc     x5, xzr, x1+        ldr     x8, [v+7*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x6, x6, x0+        str     x6, [u+7*N]+        adc     x5, x5, x1++// Digits 8 and 9 of u (top is unsigned)++        ldr     x7, [u+8*N]+        eor     x1, x7, s00+        and     x3, s00, m00+        neg     x3, x3+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x5, x5, x0+        adc     x3, x3, x1+        ldr     x8, [v+8*N]+        eor     x1, x8, s01+        and     x0, s01, m01+        sub     x3, x3, x0+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x5, x5, x0+        adc     x3, x3, x1++// Modular reduction of u, reloading as needed from u[0],...,u[7],x5,x3++        extr    x6, x3, x5, #9+        ldp     x10, x11, [u]+        add     x6, x6, x3, asr #63+        sub     x5, x5, x6, lsl #9+        adds    x10, x10, x6+        asr     x6, x6, #63+        adcs    x11, x11, x6+        ldp     x12, x13, [u+16]+        adcs    x12, x12, x6+        adcs    x13, x13, x6+        ldp     x14, x15, [u+32]+        adcs    x14, x14, x6+        adcs    x15, x15, x6+        ldp     x16, x17, [u+48]+        adcs    x16, x16, x6+        adcs    x17, x17, x6+        adc     x19, x5, x6++// Further strict reduction ready for the output, which just means+// a conditional subtraction of p_521++        subs    x0, x10, #-1+        adcs    x1, x11, xzr+        adcs    x2, x12, xzr+        adcs    x3, x13, xzr+        adcs    x4, x14, xzr+        adcs    x5, x15, xzr+        adcs    x6, x16, xzr+        adcs    x7, x17, xzr+        mov     x8, #0x1FF+        sbcs    x8, x19, x8++        csel    x0, x0, x10, cs+        csel    x1, x1, x11, cs+        csel    x2, x2, x12, cs+        csel    x3, x3, x13, cs+        csel    x4, x4, x14, cs+        csel    x5, x5, x15, cs+        csel    x6, x6, x16, cs+        csel    x7, x7, x17, cs+        csel    x8, x8, x19, cs++// Store it back to the final output++        stp     x0, x1, [res]+        stp     x2, x3, [res, #16]+        stp     x4, x5, [res, #32]+        stp     x6, x7, [res, #48]+        str     x8, [res, #64]++// Restore stack and registers++        CFI_INC_SP(NSPACE)+        CFI_POP2(x21,x22)+        CFI_POP2(x19,x20)+        CFI_RET++S2N_BN_SIZE_DIRECTIVE(bignum_inv_p521)++#if defined(__linux__) && defined(__ELF__)+.section .note.GNU-stack, "", %progbits+#endif
+ cbits/s2n/arm/bignum_modinv.S view
@@ -0,0 +1,613 @@+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0++// ----------------------------------------------------------------------------+// Invert modulo m, z = (1/a) mod b, assuming b is an odd number > 1, coprime a+// Inputs a[k], b[k]; output z[k]; temporary buffer t[>=3*k]+//+//    extern void bignum_modinv(uint64_t k, uint64_t *z, const uint64_t *a,+//                              const uint64_t *b, uint64_t *t);+//+// k-digit (digit=64 bits) "z := a^-1 mod b" (modular inverse of a modulo b)+// using t as a temporary buffer (t at least 3*k words = 24*k bytes), and+// assuming that a and b are coprime *and* that b is an odd number > 1.+//+// Standard ARM ABI: X0 = k, X1 = z, X2 = a, X3 = b, X4 = t+// ----------------------------------------------------------------------------++#include "_internal_s2n_bignum_arm.h"++        S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_modinv)+        S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_modinv)+        S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_modinv)+        .text+        .balign 4++// We get CHUNKSIZE bits per outer iteration, 64 minus a few for proxy errors++#define CHUNKSIZE 58++// Pervasive variables++#define k x0+#define z x1+#define b x3+#define w x4++// This one is recycled after initial copying in of a as outer loop counter++#define a x2+#define t x2++// Additional variables; later ones are currently rather high regs++#define l x5++#define m x21+#define n x22++// The matrix of update factors to apply to m and n+// Also used a couple of additional temporary variables for the swapping loop+// Also used as an extra down-counter in corrective negation loops++#define m_m x6+#define m_n x7+#define n_m x8+#define n_n x9++#define j x6++// General temporary variables and loop counters++#define i x10+#define t1 x11+#define t2 x12++// High and low proxies for the inner loop+// Then re-used for high and carry words during actual cross-multiplications++#define m_hi x13+#define n_hi x14+#define m_lo x15+#define n_lo x16++#define h1 x13+#define h2 x14+#define l1 x15+#define l2 x16++#define c1 x17+#define c2 x19++// Negated modular inverse for Montgomery++#define v x20++// Some more intuitive names for temp regs in initial word-level negmodinv.+// These just use t1 and t2 again, though carefully since t1 = initial b[0]++#define one t2+#define e1 t2+#define e2 t1+#define e4 t2+#define e8 t1++S2N_BN_SYMBOL(bignum_modinv):+        CFI_START++// We make use of registers beyond the modifiable++        CFI_PUSH2(x19,x20)+        CFI_PUSH2(x21,x22)++// If k = 0 then do nothing (this is out of scope anyway)++        cbz     k, Lbignum_modinv_end++// Set up the additional two buffers m and n beyond w in temp space++        lsl     i, k, #3+        add     m, w, i+        add     n, m, i++// Initialize the main buffers with their starting values:+// m = a, n = b, w = b (to be tweaked to b - 1) and z = 0++        mov     i, xzr+Lbignum_modinv_copyloop:+        ldr     t1, [a, i, lsl #3]+        ldr     t2, [b, i, lsl #3]+        str     t1, [m, i, lsl #3]+        str     t2, [n, i, lsl #3]+        str     t2, [w, i, lsl #3]+        str     xzr, [z, i, lsl #3]+        add     i, i, #1+        cmp     i, k+        bcc     Lbignum_modinv_copyloop++// Tweak down w to b - 1 (this crude approach is safe as b needs to be odd+// for it to be in scope). We have then established the congruence invariant:+//+//   a * w == -m (mod b)+//   a * z == n (mod b)+//+// This, with the bound w <= b and z <= b, is maintained round the outer loop++        ldr     t1, [w]+        sub     t2, t1, #1+        str     t2, [w]++// Compute v = negated modular inverse of b mod 2^64, reusing t1 from above+// This is used for Montgomery reduction operations each time round the loop++        lsl     v, t1, #2+        sub     v, t1, v+        eor     v, v, #2+        mov     one, #1+        madd    e1, t1, v, one+        mul     e2, e1, e1+        madd    v, e1, v, v+        mul     e4, e2, e2+        madd    v, e2, v, v+        mul     e8, e4, e4+        madd    v, e4, v, v+        madd    v, e8, v, v++// Set up the outer loop count of 128 * k+// The invariant is that m * n < 2^t at all times.++        lsl     t, k, #7++// Start of the main outer loop iterated t / CHUNKSIZE times++Lbignum_modinv_outerloop:++// We need only bother with sharper l = min k (ceil(t/64)) digits+// for the computations on m and n (but we still need k for w and z).+// Either both m and n fit in l digits, or m has become zero and so+// nothing happens in the loop anyway and this makes no difference.++        add     i, t, #63+        lsr     l, i, #6+        cmp     l, k+        csel    l, k, l, cs++// Select upper and lower proxies for both m and n to drive the inner+// loop. The lower proxies are simply the lowest digits themselves,+// m_lo = m[0] and n_lo = n[0], while the upper proxies are bitfields+// of the two inputs selected so their top bit (63) aligns with the+// most significant bit of *either* of the two inputs.++        mov     h1, xzr // Previous high and low for m+        mov     l1, xzr+        mov     h2, xzr // Previous high and low for n+        mov     l2, xzr+        mov     c2, xzr // Mask flag: previous word of one was nonzero+        // and in this case h1 and h2 are those words+        mov     i, xzr+Lbignum_modinv_toploop:+        ldr     t1, [m, i, lsl #3]+        ldr     t2, [n, i, lsl #3]+        orr     c1, t1, t2+        cmp     c1, xzr+        and     c1, c2, h1+        csel    l1, c1, l1, ne+        and     c1, c2, h2+        csel    l2, c1, l2, ne+        csel    h1, t1, h1, ne+        csel    h2, t2, h2, ne+        csetm   c2, ne+        add     i, i, #1+        cmp     i, l+        bcc     Lbignum_modinv_toploop++        orr     t1, h1, h2+        clz     t2, t1+        negs    c1, t2+        lsl     h1, h1, t2+        csel    l1, l1, xzr, ne+        lsl     h2, h2, t2+        csel    l2, l2, xzr, ne+        lsr     l1, l1, c1+        lsr     l2, l2, c1+        orr     m_hi, h1, l1+        orr     n_hi, h2, l2++        ldr     m_lo, [m]+        ldr     n_lo, [n]++// Now the inner loop, with i as loop counter from CHUNKSIZE down.+// This records a matrix of updates to apply to the initial+// values of m and n with, at stage j:+//+//     sgn * m' = (m_m * m - m_n * n) / 2^j+//    -sgn * n' = (n_m * m - n_n * n) / 2^j+//+// where "sgn" is either +1 or -1, and we lose track of which except+// that both instance above are the same. This throwing away the sign+// costs nothing (since we have to correct in general anyway because+// of the proxied comparison) and makes things a bit simpler. But it+// is simply the parity of the number of times the first condition,+// used as the swapping criterion, fires in this loop.++        mov     m_m, #1+        mov     m_n, xzr+        mov     n_m, xzr+        mov     n_n, #1++        mov     i, #CHUNKSIZE++// Conceptually in the inner loop we follow these steps:+//+// * If m_lo is odd and m_hi < n_hi, then swap the four pairs+//    (m_hi,n_hi); (m_lo,n_lo); (m_m,n_m); (m_n,n_n)+//+// * Now, if m_lo is odd (old or new, doesn't matter as initial n_lo is odd)+//    m_hi := m_hi - n_hi, m_lo := m_lo - n_lo+//    m_m  := m_m + n_m, m_n := m_n + n_n+//+// * Halve and double them+//     m_hi := m_hi / 2, m_lo := m_lo / 2+//     n_m := n_m * 2, n_n := n_n * 2+//+// The actual computation computes updates before actually swapping and+// then corrects as needed. It also maintains the invariant ~ZF <=> odd(m_lo),+// since it seems to reduce the dependent latency. Set that up first.++        ands    xzr, m_lo, #1++Lbignum_modinv_innerloop:++// At the start of the loop ~ZF <=> m_lo is odd; mask values accordingly+// Set the flags for m_hi - [~ZF] * n_hi so we know to flip things.++        csel    t1, n_hi, xzr, ne+        csel    t2, n_lo, xzr, ne+        csel    c1, n_m, xzr, ne+        csel    c2, n_n, xzr, ne+        ccmp    m_hi, n_hi, #0x2, ne++// Compute subtractive updates, trivial in the case ZF <=> even(m_lo).++        sub     t1, m_hi, t1+        sub     t2, m_lo, t2++// If the subtraction borrows, swap things appropriately, negating where+// we've already subtracted so things are as if we actually swapped first.++        csel    n_hi, n_hi, m_hi, cs+        cneg    t1, t1, cc+        csel    n_lo, n_lo, m_lo, cs+        cneg    m_lo, t2, cc+        csel    n_m, n_m, m_m, cs+        csel    n_n, n_n, m_n, cs++// Update and shift while setting oddness flag for next iteration+// We look at bit 1 of t2 (m_lo before possible negation), which is+// safe because it is even.++        ands    xzr, t2, #2+        add     m_m, m_m, c1+        add     m_n, m_n, c2+        lsr     m_hi, t1, #1+        lsr     m_lo, m_lo, #1+        add     n_m, n_m, n_m+        add     n_n, n_n, n_n++// Next iteration; don't disturb the flags since they are used at entry++        sub     i, i, #1+        cbnz    i, Lbignum_modinv_innerloop++// Apply the update to w and z, using addition in this case, and also take+// the chance to shift an additional 6 = 64-CHUNKSIZE bits to be ready for a+// Montgomery multiplication. Because we know that m_m + m_n <= 2^58 and+// w, z <= b < 2^{64k}, we know that both of these fit in k+1 words.+// We do this before the m-n update to allow us to play with c1 and c2 here.+//+//    h1::w = 2^6 * (m_m * w + m_n * z)+//    h2::z = 2^6 * (n_m * w + n_n * z)+//+// with c1 and c2 recording previous words for the shifting part++        mov     h1, xzr+        mov     h2, xzr+        mov     c1, xzr+        mov     c2, xzr++        mov     i, xzr+Lbignum_modinv_congloop:+        ldr     t1, [w, i, lsl #3]+        ldr     t2, [z, i, lsl #3]++        mul     l1, m_m, t1+        mul     l2, m_n, t2+        adds    l1, l1, h1+        umulh   h1, m_m, t1+        adc     h1, h1, xzr+        adds    l1, l1, l2+        extr    c1, l1, c1, #CHUNKSIZE+        str     c1, [w, i, lsl #3]+        mov     c1, l1+        umulh   l1, m_n, t2+        adc     h1, h1, l1++        mul     l1, n_m, t1+        mul     l2, n_n, t2+        adds    l1, l1, h2+        umulh   h2, n_m, t1+        adc     h2, h2, xzr+        adds    l1, l1, l2+        extr    c2, l1, c2, #CHUNKSIZE+        str     c2, [z, i, lsl #3]+        mov     c2, l1+        umulh   l1, n_n, t2+        adc     h2, h2, l1++        add     i, i, #1+        cmp     i, k+        bcc     Lbignum_modinv_congloop++        extr    h1, h1, c1, #CHUNKSIZE+        extr    h2, h2, c2, #CHUNKSIZE++// Do a Montgomery reduction of h1::w++        ldr     t1, [w]+        mul     c1, t1, v+        ldr     t2, [b]+        mul     l1, c1, t2+        umulh   l2, c1, t2+        adds    t1, t1, l1      // Will be zero but want the carry++        mov     i, #1+        sub     t1, k, #1+        cbz     t1, Lbignum_modinv_wmontend+Lbignum_modinv_wmontloop:+        ldr     t1, [b, i, lsl #3]+        ldr     t2, [w, i, lsl #3]+        mul     l1, c1, t1+        adcs    t2, t2, l2+        umulh   l2, c1, t1+        adc     l2, l2, xzr+        adds    t2, t2, l1+        sub     l1, i, #1+        str     t2, [w, l1, lsl #3]+        add     i, i, #1+        sub     t1, i, k+        cbnz    t1, Lbignum_modinv_wmontloop+Lbignum_modinv_wmontend:+        adcs    l2, l2, h1+        adc     h1, xzr, xzr+        sub     l1, i, #1+        str     l2, [w, l1, lsl #3]++        subs    i, xzr, xzr+Lbignum_modinv_wcmploop:+        ldr     t1, [w, i, lsl #3]+        ldr     t2, [b, i, lsl #3]+        sbcs    xzr, t1, t2+        add     i, i, #1+        sub     t1, i, k+        cbnz    t1, Lbignum_modinv_wcmploop++        sbcs    xzr, h1, xzr+        csetm   h1, cs++        subs    i, xzr, xzr+Lbignum_modinv_wcorrloop:+        ldr     t1, [w, i, lsl #3]+        ldr     t2, [b, i, lsl #3]+        and     t2, t2, h1+        sbcs    t1, t1, t2+        str     t1, [w, i, lsl #3]+        add     i, i, #1+        sub     t1, i, k+        cbnz    t1, Lbignum_modinv_wcorrloop++// Do a Montgomery reduction of h2::z++        ldr     t1, [z]+        mul     c1, t1, v+        ldr     t2, [b]+        mul     l1, c1, t2+        umulh   l2, c1, t2+        adds    t1, t1, l1      // Will be zero but want the carry++        mov     i, #1+        sub     t1, k, #1+        cbz     t1, Lbignum_modinv_zmontend+Lbignum_modinv_zmontloop:+        ldr     t1, [b, i, lsl #3]+        ldr     t2, [z, i, lsl #3]+        mul     l1, c1, t1+        adcs    t2, t2, l2+        umulh   l2, c1, t1+        adc     l2, l2, xzr+        adds    t2, t2, l1+        sub     l1, i, #1+        str     t2, [z, l1, lsl #3]+        add     i, i, #1+        sub     t1, i, k+        cbnz    t1, Lbignum_modinv_zmontloop+Lbignum_modinv_zmontend:+        adcs    l2, l2, h2+        adc     h2, xzr, xzr+        sub     l1, i, #1+        str     l2, [z, l1, lsl #3]++        subs    i, xzr, xzr+Lbignum_modinv_zcmploop:+        ldr     t1, [z, i, lsl #3]+        ldr     t2, [b, i, lsl #3]+        sbcs    xzr, t1, t2+        add     i, i, #1+        sub     t1, i, k+        cbnz    t1, Lbignum_modinv_zcmploop++        sbcs    xzr, h2, xzr+        csetm   h2, cs++        subs    i, xzr, xzr+Lbignum_modinv_zcorrloop:+        ldr     t1, [z, i, lsl #3]+        ldr     t2, [b, i, lsl #3]+        and     t2, t2, h2+        sbcs    t1, t1, t2+        str     t1, [z, i, lsl #3]+        add     i, i, #1+        sub     t1, i, k+        cbnz    t1, Lbignum_modinv_zcorrloop++// Now actually compute the updates to m and n corresponding to the matrix,+// and correct the signs if they have gone negative. First we compute the+// (k+1)-sized updates with the following invariant (here c1 and c2 are in+// fact carry bitmasks, either 0 or -1):+//+//    c1::h1::m = m_m * m - m_n * n+//    c2::h2::n = n_m * m - n_n * n++        mov     h1, xzr+        mov     h2, xzr+        mov     c1, xzr+        mov     c2, xzr+        mov     i, xzr+Lbignum_modinv_crossloop:+        ldr     t1, [m, i, lsl #3]+        ldr     t2, [n, i, lsl #3]++        mul     l1, m_m, t1+        mul     l2, m_n, t2+        adds    l1, l1, h1+        umulh   h1, m_m, t1+        adc     h1, h1, xzr+        subs    l1, l1, l2+        str     l1, [m, i, lsl #3]+        umulh   l1, m_n, t2+        sub     c1, l1, c1+        sbcs    h1, h1, c1+        csetm   c1, cc++        mul     l1, n_m, t1+        mul     l2, n_n, t2+        adds    l1, l1, h2+        umulh   h2, n_m, t1+        adc     h2, h2, xzr+        subs    l1, l1, l2+        str     l1, [n, i, lsl #3]+        umulh   l1, n_n, t2+        sub     c2, l1, c2+        sbcs    h2, h2, c2+        csetm   c2, cc++        add     i, i, #1+        cmp     i, l+        bcc     Lbignum_modinv_crossloop++// Write back m optionally negated and shifted right CHUNKSIZE bits++        adds    xzr, c1, c1++        ldr     l1, [m]+        mov     i, xzr+        sub     j, l, #1+        cbz     j, Lbignum_modinv_negskip1++Lbignum_modinv_negloop1:+        add     t1, i, #8+        ldr     t2, [m, t1]+        extr    l1, t2, l1, #CHUNKSIZE+        eor     l1, l1, c1+        adcs    l1, l1, xzr+        str     l1, [m, i]+        mov     l1, t2+        add     i, i, #8+        sub     j, j, #1+        cbnz    j, Lbignum_modinv_negloop1+Lbignum_modinv_negskip1:+        extr    l1, h1, l1, #CHUNKSIZE+        eor     l1, l1, c1+        adcs    l1, l1, xzr+        str     l1, [m, i]++// Write back n optionally negated and shifted right CHUNKSIZE bits++        adds    xzr, c2, c2++        ldr     l1, [n]+        mov     i, xzr+        sub     j, l, #1+        cbz     j, Lbignum_modinv_negskip2+Lbignum_modinv_negloop2:+        add     t1, i, #8+        ldr     t2, [n, t1]+        extr    l1, t2, l1, #CHUNKSIZE+        eor     l1, l1, c2+        adcs    l1, l1, xzr+        str     l1, [n, i]+        mov     l1, t2+        add     i, i, #8+        sub     j, j, #1+        cbnz    j, Lbignum_modinv_negloop2+Lbignum_modinv_negskip2:+        extr    l1, h2, l1, #CHUNKSIZE+        eor     l1, l1, c2+        adcs    l1, l1, xzr+        str     l1, [n, i]++// Finally, use the signs c1 and c2 to do optional modular negations of+// w and z respectively, flipping c2 to make signs work. We don't make+// any checks for zero values, but we certainly retain w <= b and z <= b.+// This is enough for the Montgomery step in the next iteration to give+// strict reduction w < b amd z < b, and anyway when we terminate we+// could not have z = b since it violates the coprimality assumption for+// in-scope cases.++        mov     i, xzr+        adds    xzr, c1, c1+Lbignum_modinv_wfliploop:+        ldr     t1, [b, i, lsl #3]+        ldr     t2, [w, i, lsl #3]+        and     t1, t1, c1+        eor     t2, t2, c1+        adcs    t1, t1, t2+        str     t1, [w, i, lsl #3]+        add     i, i, #1+        sub     t1, i, k+        cbnz    t1, Lbignum_modinv_wfliploop++        mvn     c2, c2++        mov     i, xzr+        adds    xzr, c2, c2+Lbignum_modinv_zfliploop:+        ldr     t1, [b, i, lsl #3]+        ldr     t2, [z, i, lsl #3]+        and     t1, t1, c2+        eor     t2, t2, c2+        adcs    t1, t1, t2+        str     t1, [z, i, lsl #3]+        add     i, i, #1+        sub     t1, i, k+        cbnz    t1, Lbignum_modinv_zfliploop++// End of main loop. We can stop if t' <= 0 since then m * n < 2^0, which+// since n is odd and m and n are coprime (in the in-scope cases) means+// m = 0, n = 1 and hence from the congruence invariant a * z == 1 (mod b).+// Moreover we do in fact need to maintain strictly t > 0 in the main loop,+// or the computation of the optimized digit bound l could collapse to 0.++        subs    t, t, #CHUNKSIZE+        bhi     Lbignum_modinv_outerloop++Lbignum_modinv_end:+        CFI_POP2(x21,x22)+        CFI_POP2(x19,x20)++        CFI_RET++S2N_BN_SIZE_DIRECTIVE(bignum_modinv)++#if defined(__linux__) && defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/s2n/arm/bignum_montifier.S view
@@ -0,0 +1,462 @@+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0++// ----------------------------------------------------------------------------+// Compute "montification" constant z := 2^{128k} mod m+// Input m[k]; output z[k]; temporary buffer t[>=k]+//+//    extern void bignum_montifier(uint64_t k, uint64_t *z, const uint64_t *m,+//                                 uint64_t *t);+//+// The last argument points to a temporary buffer t that should have size >= k.+// This is called "montifier" because given any other k-digit number x,+// whether or not it's reduced modulo m, it can be mapped to its Montgomery+// representation (2^{64k} * x) mod m just by Montgomery multiplication by z.+//+// Standard ARM ABI: X0 = k, X1 = z, X2 = m, X3 = t+// ----------------------------------------------------------------------------++#include "_internal_s2n_bignum_arm.h"++        S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montifier)+        S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montifier)+        S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montifier)+        .text+        .balign 4++#define k x0+#define z x1+#define m x2+#define t x3++// Some variables+// Modular inverse w is aliased to i, but we never use them together++#define i x4+#define w x4+#define j x5+#define h x6+#define a x7+#define l x8+#define c x9+#define b x10+#define d x11++// Some aliases for the values b and d++#define r x10+#define q x11+++S2N_BN_SYMBOL(bignum_montifier):+        CFI_START++// If k = 0 the whole operation is trivial++        cbz     k, Lbignum_montifier_end++// Copy the input m into the temporary buffer t. The temporary register+// c matters since we want it to hold the highest digit, ready for the+// normalization phase.++        mov     i, xzr+Lbignum_montifier_copyinloop:+        ldr     c, [m, i, lsl #3]+        str     c, [t, i, lsl #3]+        add     i, i, #1+        cmp     i, k+        bcc     Lbignum_montifier_copyinloop++// Do a rather stupid but constant-time digit normalization, conditionally+// shifting left (k-1) times based on whether the top word is zero.+// With careful binary striding this could be O(k*log(k)) instead of O(k^2)+// while still retaining the constant-time style.+// The "cmp c, xzr" sets the zeroness predicate (ZF) for the entire inner loop++        subs    i, k, #1+        beq     Lbignum_montifier_normalized+Lbignum_montifier_normloop:+        mov     j, xzr+        cmp     c, xzr+        mov     a, xzr+Lbignum_montifier_shufloop:+        mov     c, a+        ldr     a, [t, j, lsl #3]+        csel    c, c, a, eq+        str     c, [t, j, lsl #3]+        add     j, j, #1+        sub     d, j, k+        cbnz    d, Lbignum_montifier_shufloop+        subs    i, i, #1+        bne     Lbignum_montifier_normloop++// We now have the top digit nonzero, assuming the input was nonzero,+// and as per the invariant of the loop above, c holds that digit. So+// now just count c's leading zeros and shift t bitwise that many bits.++Lbignum_montifier_normalized:+        clz     c, c++        mov     b, xzr+        mov     i, xzr+        ands    xzr, c, #63+        csetm   l, ne+        neg     d, c+Lbignum_montifier_bitloop:+        ldr     j, [t, i, lsl #3]+        lsl     a, j, c+        orr     a, a, b+        lsr     b, j, d+        and     b, b, l+        str     a, [t, i, lsl #3]+        add     i, i, #1+        cmp     i, k+        bcc     Lbignum_montifier_bitloop++// Let h be the high word of n, which in all the in-scope cases is >= 2^63.+// Now successively form q = 2^i div h and r = 2^i mod h as i goes from+// 64 to 126. We avoid just using division out of constant-time concerns+// (at the least we would need to fix up h = 0 for out-of-scope inputs) and+// don't bother with Newton-Raphson, since this stupid simple loop doesn't+// contribute much of the overall runtime at typical sizes.++        sub     h, k, #1+        ldr     h, [t, h, lsl #3]+        mov     q, #1+        neg     r, h+        mov     i, #62+Lbignum_montifier_estloop:+        add     q, q, q+        mov     a, h+        sub     a, a, r+        cmp     r, a    // CF <=> r >= h - r <=> 2 * r >= h+        csetm   a, cs+        sub     q, q, a+        add     r, r, r+        and     a, a, h+        sub     r, r, a+        subs    i, i, #1+        bne     Lbignum_montifier_estloop++// Strictly speaking the above loop doesn't quite give the true remainder+// and quotient in the special case r = h = 2^63, so fix it up. We get+// q = 2^63 - 1 and r = 2^63 and really want q = 2^63 and r = 0. This is+// supererogatory, because the main property of q used below still holds+// in this case unless the initial m = 1, and then anyway the overall+// specification (congruence modulo m) holds degenerately. But it seems+// nicer to get a "true" quotient and remainder.++        cmp     r, h+        csinc   q, q, q, ne++// So now we have q and r with 2^126 = q * h + r (imagining r = 0 in the+// fixed-up case above: note that we never actually use the computed+// value of r below and so didn't adjust it). And we can assume the ranges+// q <= 2^63 and r < h < 2^64.+//+// The idea is to use q as a first quotient estimate for a remainder+// of 2^{p+62} mod n, where p = 64 * k. We have, splitting n into the+// high and low parts h and l:+//+// 2^{p+62} - q * n = 2^{p+62} - q * (2^{p-64} * h + l)+//                  = 2^{p+62} - (2^{p-64} * (q * h) + q * l)+//                  = 2^{p+62} - 2^{p-64} * (2^126 - r) - q * l+//                  = 2^{p-64} * r - q * l+//+// Note that 2^{p-64} * r < 2^{p-64} * h <= n+// and also  q * l < 2^63 * 2^{p-64} = 2^{p-1} <= n+// so |diff| = |2^{p-64} * r - q * l| < n.+//+// If in fact diff >= 0 then it is already 2^{p+62} mod n.+// otherwise diff + n is the right answer.+//+// To (maybe?) make the computation slightly easier we actually flip+// the sign and compute d = q * n - 2^{p+62}. Then the answer is either+// -d (when negative) or n - d; in either case we effectively negate d.+// This negating tweak in fact spoils the result for cases where+// 2^{p+62} mod n = 0, when we get n instead. However the only case+// where this can happen is m = 1, when the whole spec holds trivially,+// and actually the remainder of the logic below works anyway since+// the latter part of the code only needs a congruence for the k-digit+// result, not strict modular reduction (the doublings will maintain+// the non-strict inequality).++        mov     c, xzr+        adds    i, xzr, xzr+Lbignum_montifier_mulloop:+        ldr     a, [t, i, lsl #3]+        mul     l, q, a+        adcs    l, l, c+        umulh   c, q, a+        str     l, [z, i, lsl #3]+        add     i, i, #1+        sub     a, i, k+        cbnz    a, Lbignum_montifier_mulloop++        adc     c, c, xzr+        mov     a, #0x4000000000000000+        subs    c, c, a+        csetm   q, cs++// Now do [c] * n - d for our final answer++        subs    i, xzr, xzr+Lbignum_montifier_remloop:+        ldr     a, [t, i, lsl #3]+        ldr     b, [z, i, lsl #3]+        and     a, a, q+        sbcs    a, a, b+        str     a, [z, i, lsl #3]+        add     i, i, #1+        sub     a, i, k+        cbnz    a, Lbignum_montifier_remloop++// Now still need to do a couple of modular doublings to get us all the+// way up to 2^{p+64} == r from the initial 2^{p+62} == r (mod n).++        mov     c, xzr+        subs    j, xzr, xzr+Lbignum_montifier_dubloop1:+        ldr     a, [z, j, lsl #3]+        extr    c, a, c, #63+        ldr     b, [t, j, lsl #3]+        sbcs    c, c, b+        str     c, [z, j, lsl #3]+        mov     c, a+        add     j, j, #1+        sub     a, j, k+        cbnz    a, Lbignum_montifier_dubloop1+        lsr     c, c, #63+        sbc     c, c, xzr+        adds    j, xzr, xzr+Lbignum_montifier_corrloop1:+        ldr     a, [z, j, lsl #3]+        ldr     b, [t, j, lsl #3]+        and     b, b, c+        adcs    a, a, b+        str     a, [z, j, lsl #3]+        add     j, j, #1+        sub     a, j, k+        cbnz    a, Lbignum_montifier_corrloop1++// This is not exactly the same: we also copy output to t giving the+// initialization t_1 = r == 2^{p+64} mod n for the main loop next.++        mov     c, xzr+        subs    j, xzr, xzr+Lbignum_montifier_dubloop2:+        ldr     a, [z, j, lsl #3]+        extr    c, a, c, #63+        ldr     b, [t, j, lsl #3]+        sbcs    c, c, b+        str     c, [z, j, lsl #3]+        mov     c, a+        add     j, j, #1+        sub     a, j, k+        cbnz    a, Lbignum_montifier_dubloop2+        lsr     c, c, #63+        sbc     c, c, xzr+        adds    j, xzr, xzr+Lbignum_montifier_corrloop2:+        ldr     a, [z, j, lsl #3]+        ldr     b, [t, j, lsl #3]+        and     b, b, c+        adcs    a, a, b+        str     a, [z, j, lsl #3]+        str     a, [t, j, lsl #3]+        add     j, j, #1+        sub     a, j, k+        cbnz    a, Lbignum_montifier_corrloop2++// We then successively generate (k+1)-digit values satisfying+// t_i == 2^{p+64*i} mod n, each of which is stored in h::t. Finish+// initialization by zeroing h initially++        mov     h, xzr++// Then if t_i = 2^{p} * h + l+// we have t_{i+1} == 2^64 * t_i+//         = (2^{p+64} * h) + (2^64 * l)+//        == r * h + l<<64+// Do this 2*k more times so we end up == 2^{192*k+64}, one more than we want+//+// Writing B = 2^{64k}, the possible correction of adding r, which for+// a (k+1)-digit result is equivalent to subtracting q = 2^{64*(k+1)} - r+// would give the overall worst-case value minus q of+// [ B * (B^k - 1) + (B - 1) * r ] - [B^{k+1} - r]+// = B * (r - 1) < B^{k+1} so we keep inside k+1 digits as required.+//+// This implementation makes the shift implicit by starting b with the+// "previous" digit (initially 0) to offset things by 1.++        add     i, k, k+Lbignum_montifier_modloop:+        mov     j, xzr+        mov     b, xzr+        adds    c, xzr, xzr+Lbignum_montifier_cmaloop:+        ldr     a, [z, j, lsl #3]+        mul     l, h, a+        adcs    b, b, c+        umulh   c, h, a+        adc     c, c, xzr+        adds    l, b, l+        ldr     b, [t, j, lsl #3]+        str     l, [t, j, lsl #3]+        add     j, j, #1+        sub     a, j, k+        cbnz    a, Lbignum_montifier_cmaloop++        adcs    h, b, c++        csetm   l, cs++        adds    j, xzr, xzr+Lbignum_montifier_oaloop:+        ldr     a, [t, j, lsl #3]+        ldr     b, [z, j, lsl #3]+        and     b, b, l+        adcs    a, a, b+        str     a, [t, j, lsl #3]+        add     j, j, #1+        sub     a, j, k+        cbnz    a, Lbignum_montifier_oaloop+        adc     h, h, xzr++        subs    i, i, #1+        bne     Lbignum_montifier_modloop++// Compute the negated modular inverse w (same register as i, not used again).++        ldr     a, [m]+        lsl     w, a, #2+        sub     w, a, w+        eor     w, w, #2+        mov     l, #1+        madd    c, a, w, l+        mul     b, c, c+        madd    w, c, w, w+        mul     c, b, b+        madd    w, b, w, w+        mul     b, c, c+        madd    w, c, w, w+        madd    w, b, w, w++// Now do one almost-Montgomery reduction w.r.t. the original m+// which lops off one 2^64 from the congruence and, with the usual+// almost-Montgomery correction, gets us back inside k digits for+// the end result.++        ldr     b, [t]+        mul     d, b, w++        mul     l, d, a+        umulh   c, d, a+        mov     j, #1+        sub     a, k, #1+        adds    xzr, b, l+        cbz     a, Lbignum_montifier_amontend++Lbignum_montifier_amontloop:+        ldr     a, [m, j, lsl #3]+        ldr     b, [t, j, lsl #3]+        mul     l, d, a+        adcs    b, b, c+        umulh   c, d, a+        adc     c, c, xzr+        adds    b, b, l+        sub     a, j, #1+        str     b, [t, a, lsl #3]+        add     j, j, #1+        sub     a, j, k+        cbnz    a, Lbignum_montifier_amontloop+Lbignum_montifier_amontend:+        adcs    h, h, c+        csetm   l, cs+        sub     a, k, #1+        str     h, [t, a, lsl #3]++        subs    j, xzr, xzr+Lbignum_montifier_osloop:+        ldr     a, [t, j, lsl #3]+        ldr     b, [m, j, lsl #3]+        and     b, b, l+        sbcs    a, a, b+        str     a, [z, j, lsl #3]+        add     j, j, #1+        sub     a, j, k+        cbnz    a, Lbignum_montifier_osloop++// So far, the code(basically a variant of bignum_amontifier) has produced+// a k-digit value z == 2^{192k} (mod m), not necessarily fully reduced mod m.+// We now do a short Montgomery reduction (similar to bignum_demont) so that+// we achieve full reduction mod m while lopping 2^{64k} off the congruence.+// We recycle h as the somewhat strangely-named outer loop counter.++        mov     h, k++Lbignum_montifier_montouterloop:+        ldr     b, [z]+        mul     d, b, w+        ldr     a, [m]+        mul     l, d, a+        umulh   c, d, a+        mov     j, #1+        sub     a, k, #1+        adds    xzr, b, l+        cbz     a, Lbignum_montifier_montend+Lbignum_montifier_montloop:+        ldr     a, [m, j, lsl #3]+        ldr     b, [z, j, lsl #3]+        mul     l, d, a+        adcs    b, b, c+        umulh   c, d, a+        adc     c, c, xzr+        adds    b, b, l+        sub     a, j, #1+        str     b, [z, a, lsl #3]+        add     j, j, #1+        sub     a, j, k+        cbnz    a, Lbignum_montifier_montloop+Lbignum_montifier_montend:+        adc     c, c, xzr+        sub     a, k, #1+        str     c, [z, a, lsl #3]++        subs    h, h, #1+        bne     Lbignum_montifier_montouterloop++// Now do a comparison of z with m to set a final correction mask+// indicating that z >= m and so we need to subtract m.++        subs    j, xzr, xzr+Lbignum_montifier_cmploop:+        ldr     a, [z, j, lsl #3]+        ldr     b, [m, j, lsl #3]+        sbcs    xzr, a, b+        add     j, j, #1+        sub     a, j, k+        cbnz    a, Lbignum_montifier_cmploop+        csetm   h, cs++// Now do a masked subtraction of m for the final reduced result.++        subs    j, xzr, xzr+Lbignum_montifier_corrloop:+        ldr     a, [z, j, lsl #3]+        ldr     b, [m, j, lsl #3]+        and     b, b, h+        sbcs    a, a, b+        str     a, [z, j, lsl #3]+        add     j, j, #1+        sub     a, j, k+        cbnz    a, Lbignum_montifier_corrloop++Lbignum_montifier_end:+        CFI_RET++S2N_BN_SIZE_DIRECTIVE(bignum_montifier)++#if defined(__linux__) && defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/s2n/arm/bignum_montinv_p384.S view
@@ -0,0 +1,1493 @@+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0++// ----------------------------------------------------------------------------+// Montgomery inverse modulo p_384 = 2^384 - 2^128 - 2^96 + 2^32 - 1+// Input x[6]; output z[6]+//+// extern void bignum_montinv_p384(uint64_t z[static 6],+//                                 const uint64_t x[static 6]);+//+// If the 6-digit input x is coprime to p_384, i.e. is not divisible+// by it, returns z < p_384 such that x * z == 2^768 (mod p_384). This+// is effectively "Montgomery inverse" because if we consider x and z as+// Montgomery forms of X and Z, i.e. x == 2^384 * X and z == 2^384 * Z+// (both mod p_384) then X * Z == 1 (mod p_384). That is, this function+// gives the analog of the modular inverse bignum_inv_p384 but with both+// input and output in the Montgomery domain. Note that x does not need+// to be reduced modulo p_384, but the output always is. If the input+// is divisible (i.e. is 0 or p_384), then there can be no solution to+// the congruence x * z == 2^768 (mod p_384), and z = 0 is returned.+//+// Standard ARM ABI: X0 = z, X1 = x+// ----------------------------------------------------------------------------++#include "_internal_s2n_bignum_arm.h"++        S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montinv_p384)+        S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montinv_p384)+        S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montinv_p384)++        .text+        .balign 4++// Size in bytes of a 64-bit word++#define N 8++// Used for the return pointer++#define res x20++// Loop counter and d = 2 * delta value for divstep++#define i x21+#define d x22++// Registers used for matrix element magnitudes and signs++#define m00 x10+#define m01 x11+#define m10 x12+#define m11 x13+#define s00 x14+#define s01 x15+#define s10 x16+#define s11 x17++// Initial carries for combinations++#define car0 x9+#define car1 x19++// Input and output, plain registers treated according to pattern++#define reg0 x0, #0+#define reg1 x1, #0+#define reg2 x2, #0+#define reg3 x3, #0+#define reg4 x4, #0++#define x x1, #0+#define z x0, #0++// Pointer-offset pairs for temporaries on stack+// The u and v variables are 6 words each as expected, but the f and g+// variables are 8 words each -- they need to have at least one extra+// word for a sign word, and to preserve alignment we "round up" to 8.+// In fact, we currently keep an extra word in u and v as well.++#define f sp, #0+#define g sp, #(8*N)+#define u sp, #(16*N)+#define v sp, #(24*N)++// Total size to reserve on the stack++#define NSPACE 32*N++// ---------------------------------------------------------------------------+// Core signed almost-Montgomery reduction macro. Takes input in+// [d6;d5;d4;d3;d2;d1;d0] and returns result in [d6;d5d4;d3;d2;d1], adding+// to the existing [d6;d5;d4;d3;d2;d1], and re-using d0 as a temporary+// internally as well as t0, t1, t2. This is almost-Montgomery, i.e. the+// result fits in 6 digits but is not necessarily strictly reduced mod p_384.+// ---------------------------------------------------------------------------++#define amontred(d6,d5,d4,d3,d2,d1,d0, t3,t2,t1)                            \+/* We only know the input is -2^444 < x < 2^444. To do traditional  */      \+/* unsigned Montgomery reduction, start by adding 2^61 * p_384.     */      \+        mov     t1, #0xe000000000000000 __LF                           \+        adds    d0, d0, t1 __LF                                        \+        mov     t2, #0x000000001fffffff __LF                           \+        adcs    d1, d1, t2 __LF                                        \+        mov     t3, #0xffffffffe0000000 __LF                           \+        bic     t3, t3, #0x2000000000000000 __LF                       \+        adcs    d2, d2, t3 __LF                                        \+        sbcs    d3, d3, xzr __LF                                       \+        sbcs    d4, d4, xzr __LF                                       \+        sbcs    d5, d5, xzr __LF                                       \+        mov     t1, #0x1fffffffffffffff __LF                           \+        adc     d6, d6, t1 __LF                                        \+/* Our correction multiplier is w = [d0 + (d0<<32)] mod 2^64  */    \+/* Store it back into d0 since we no longer need that digit.  */    \+        add     d0, d0, d0, lsl #32 __LF                               \+/* Now let [t3;t2;t1;-] = (2^384 - p_384) * w                 */    \+/* We know the lowest word will cancel d0 so we don't need it */    \+        mov     t1, #0xffffffff00000001 __LF                           \+        umulh   t1, t1, d0 __LF                                        \+        mov     t2, #0x00000000ffffffff __LF                           \+        mul     t3, t2, d0 __LF                                        \+        umulh   t2, t2, d0 __LF                                        \+        adds    t1, t1, t3 __LF                                        \+        adcs    t2, t2, d0 __LF                                        \+        cset    t3, cs __LF                                            \+/* Now x + p_384 * w = (x + 2^384 * w) - (2^384 - p_384) * w */     \+/* We catch the net top carry from add-subtract in the digit d0 */  \+        adds    d6, d6, d0 __LF                                        \+        cset    d0, cs __LF                                            \+        subs    d1, d1, t1 __LF                                        \+        sbcs    d2, d2, t2 __LF                                        \+        sbcs    d3, d3, t3 __LF                                        \+        sbcs    d4, d4, xzr __LF                                       \+        sbcs    d5, d5, xzr __LF                                       \+        sbcs    d6, d6, xzr __LF                                       \+        sbcs    d0, d0, xzr __LF                                       \+/* Now if d0 is nonzero we subtract p_384 (almost-Montgomery) */    \+        neg     d0, d0 __LF                                            \+        and     t1, d0, #0x00000000ffffffff __LF                       \+        and     t2, d0, #0xffffffff00000000 __LF                       \+        and     t3, d0, #0xfffffffffffffffe __LF                       \+        subs    d1, d1, t1 __LF                                        \+        sbcs    d2, d2, t2 __LF                                        \+        sbcs    d3, d3, t3 __LF                                        \+        sbcs    d4, d4, d0 __LF                                        \+        sbcs    d5, d5, d0 __LF                                        \+        sbc     d6, d6, d0++// Very similar to a subroutine call to the s2n-bignum word_divstep59.+// But different in register usage and returning the final matrix in+// registers as follows+//+// [ m00  m01]+// [ m10  m11]++#define divstep59()                                                     \+        and     x4, x2, #0xfffff __LF                                      \+        orr     x4, x4, #0xfffffe0000000000 __LF                           \+        and     x5, x3, #0xfffff __LF                                      \+        orr     x5, x5, #0xc000000000000000 __LF                           \+        tst     x5, #0x1 __LF                                              \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        asr     x5, x5, #1 __LF                                            \+        add     x8, x4, #0x100, lsl #12 __LF                               \+        sbfx    x8, x8, #21, #21 __LF                                      \+        mov     x11, #0x100000 __LF                                        \+        add     x11, x11, x11, lsl #21 __LF                                \+        add     x9, x4, x11 __LF                                           \+        asr     x9, x9, #42 __LF                                           \+        add     x10, x5, #0x100, lsl #12 __LF                              \+        sbfx    x10, x10, #21, #21 __LF                                    \+        add     x11, x5, x11 __LF                                          \+        asr     x11, x11, #42 __LF                                         \+        mul     x6, x8, x2 __LF                                            \+        mul     x7, x9, x3 __LF                                            \+        mul     x2, x10, x2 __LF                                           \+        mul     x3, x11, x3 __LF                                           \+        add     x4, x6, x7 __LF                                            \+        add     x5, x2, x3 __LF                                            \+        asr     x2, x4, #20 __LF                                           \+        asr     x3, x5, #20 __LF                                           \+        and     x4, x2, #0xfffff __LF                                      \+        orr     x4, x4, #0xfffffe0000000000 __LF                           \+        and     x5, x3, #0xfffff __LF                                      \+        orr     x5, x5, #0xc000000000000000 __LF                           \+        tst     x5, #0x1 __LF                                              \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        asr     x5, x5, #1 __LF                                            \+        add     x12, x4, #0x100, lsl #12 __LF                              \+        sbfx    x12, x12, #21, #21 __LF                                    \+        mov     x15, #0x100000 __LF                                        \+        add     x15, x15, x15, lsl #21 __LF                                \+        add     x13, x4, x15 __LF                                          \+        asr     x13, x13, #42 __LF                                         \+        add     x14, x5, #0x100, lsl #12 __LF                              \+        sbfx    x14, x14, #21, #21 __LF                                    \+        add     x15, x5, x15 __LF                                          \+        asr     x15, x15, #42 __LF                                         \+        mul     x6, x12, x2 __LF                                           \+        mul     x7, x13, x3 __LF                                           \+        mul     x2, x14, x2 __LF                                           \+        mul     x3, x15, x3 __LF                                           \+        add     x4, x6, x7 __LF                                            \+        add     x5, x2, x3 __LF                                            \+        asr     x2, x4, #20 __LF                                           \+        asr     x3, x5, #20 __LF                                           \+        and     x4, x2, #0xfffff __LF                                      \+        orr     x4, x4, #0xfffffe0000000000 __LF                           \+        and     x5, x3, #0xfffff __LF                                      \+        orr     x5, x5, #0xc000000000000000 __LF                           \+        tst     x5, #0x1 __LF                                              \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        mul     x2, x12, x8 __LF                                           \+        mul     x3, x12, x9 __LF                                           \+        mul     x6, x14, x8 __LF                                           \+        mul     x7, x14, x9 __LF                                           \+        madd    x8, x13, x10, x2 __LF                                      \+        madd    x9, x13, x11, x3 __LF                                      \+        madd    x16, x15, x10, x6 __LF                                     \+        madd    x17, x15, x11, x7 __LF                                     \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        tst     x5, #0x2 __LF                                              \+        asr     x5, x5, #1 __LF                                            \+        csel    x6, x4, xzr, ne __LF                                       \+        ccmp    x1, xzr, #0x8, ne __LF                                     \+        cneg    x1, x1, ge __LF                                            \+        cneg    x6, x6, ge __LF                                            \+        csel    x4, x5, x4, ge __LF                                        \+        add     x5, x5, x6 __LF                                            \+        add     x1, x1, #0x2 __LF                                          \+        asr     x5, x5, #1 __LF                                            \+        add     x12, x4, #0x100, lsl #12 __LF                              \+        sbfx    x12, x12, #22, #21 __LF                                    \+        mov     x15, #0x100000 __LF                                        \+        add     x15, x15, x15, lsl #21 __LF                                \+        add     x13, x4, x15 __LF                                          \+        asr     x13, x13, #43 __LF                                         \+        add     x14, x5, #0x100, lsl #12 __LF                              \+        sbfx    x14, x14, #22, #21 __LF                                    \+        add     x15, x5, x15 __LF                                          \+        asr     x15, x15, #43 __LF                                         \+        mneg    x2, x12, x8 __LF                                           \+        mneg    x3, x12, x9 __LF                                           \+        mneg    x4, x14, x8 __LF                                           \+        mneg    x5, x14, x9 __LF                                           \+        msub    m00, x13, x16, x2 __LF                                     \+        msub    m01, x13, x17, x3 __LF                                     \+        msub    m10, x15, x16, x4 __LF                                     \+        msub    m11, x15, x17, x5++S2N_BN_SYMBOL(bignum_montinv_p384):+        CFI_START++// Save registers and make room for temporaries++        CFI_PUSH2(x19,x20)+        CFI_PUSH2(x21,x22)+        CFI_PUSH2(x23,x24)+        CFI_DEC_SP(NSPACE)++// Save the return pointer for the end so we can overwrite x0 later++        mov     res, x0++// Copy the prime and input into the main f and g variables respectively.+// Make sure x is reduced so that g <= f as assumed in the bound proof.++        mov     x10, #0x00000000ffffffff+        mov     x11, #0xffffffff00000000+        mov     x12, #0xfffffffffffffffe+        mov     x15, #0xffffffffffffffff+        stp     x10, x11, [f]+        stp     x12, x15, [f+2*N]+        stp     x15, x15, [f+4*N]+        str     xzr, [f+6*N]++        ldp     x2, x3, [x1]+        subs    x10, x2, x10+        sbcs    x11, x3, x11+        ldp     x4, x5, [x1, #(2*N)]+        sbcs    x12, x4, x12+        sbcs    x13, x5, x15+        ldp     x6, x7, [x1, #(4*N)]+        sbcs    x14, x6, x15+        sbcs    x15, x7, x15++        csel    x2, x2, x10, cc+        csel    x3, x3, x11, cc+        csel    x4, x4, x12, cc+        csel    x5, x5, x13, cc+        csel    x6, x6, x14, cc+        csel    x7, x7, x15, cc++        stp     x2, x3, [g]+        stp     x4, x5, [g+2*N]+        stp     x6, x7, [g+4*N]+        str     xzr, [g+6*N]++// Also maintain reduced < 2^384 vector [u,v] such that+// [f,g] == x * 2^{5*i-843} * [u,v] (mod p_384)+// starting with [p_384,x] == x * 2^{5*0-843} * [0,2^843] (mod p_384)+// The weird-looking 5*i modifications come in because we are doing+// 64-bit word-sized Montgomery reductions at each stage, which is+// 5 bits more than the 59-bit requirement to keep things stable.+// After the 15th and last iteration and sign adjustment, when+// f == 1 for in-scope cases, we have x * 2^{75-843} * u == 1, i.e.+// x * u == 2^768 as required.++        stp     xzr, xzr, [u]+        stp     xzr, xzr, [u+2*N]+        stp     xzr, xzr, [u+4*N]++// The starting constant 2^843 mod p_384 is+// 0x0000000000000800:00001000000007ff:fffff00000000000+//  :00001000000007ff:fffff00000000800:0000000000000000+// where colons separate 64-bit subwords, least significant at the right.+// Not all of these are single loads on ARM so this is a bit dynamic++        mov     x12, #0xfffff00000000000+        orr     x10, x12, #0x0000000000000800+        stp     xzr, x10, [v]+        mov     x11, #0x00000000000007ff+        orr     x11, x11, #0x0000100000000000+        stp     x11, x12, [v+2*N]+        mov     x12, #0x0000000000000800+        stp     x11, x12, [v+4*N]++// Start of main loop. We jump into the middle so that the divstep+// portion is common to the special fifteenth iteration after a uniform+// first 14.++        mov     i, #15+        mov     d, #1+        b       Lbignum_montinv_p384_midloop++Lbignum_montinv_p384_loop:++// Separate the matrix elements into sign-magnitude pairs++        cmp     m00, xzr+        csetm   s00, mi+        cneg    m00, m00, mi++        cmp     m01, xzr+        csetm   s01, mi+        cneg    m01, m01, mi++        cmp     m10, xzr+        csetm   s10, mi+        cneg    m10, m10, mi++        cmp     m11, xzr+        csetm   s11, mi+        cneg    m11, m11, mi++// Adjust the initial values to allow for complement instead of negation+// This initial offset is the same for [f,g] and [u,v] compositions.+// Save it in stable registers for the [u,v] part and do [f,g] first.++        and     x0, m00, s00+        and     x1, m01, s01+        add     car0, x0, x1++        and     x0, m10, s10+        and     x1, m11, s11+        add     car1, x0, x1++// Now the computation of the updated f and g values. This maintains a+// 2-word carry between stages so we can conveniently insert the shift+// right by 59 before storing back, and not overwrite digits we need+// again of the old f and g values.+//+// Digit 0 of [f,g]++        ldr     x7, [f]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x4, car0, x0+        adc     x2, xzr, x1+        ldr     x8, [g]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x4, x4, x0+        adc     x2, x2, x1++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x5, car1, x0+        adc     x3, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x5, x5, x0+        adc     x3, x3, x1++// Digit 1 of [f,g]++        ldr     x7, [f+N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x2, x2, x0+        adc     x6, xzr, x1+        ldr     x8, [g+N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x2, x2, x0+        adc     x6, x6, x1+        extr    x4, x2, x4, #59+        str     x4, [f]++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x3, x3, x0+        adc     x4, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x3, x3, x0+        adc     x4, x4, x1+        extr    x5, x3, x5, #59+        str     x5, [g]++// Digit 2 of [f,g]++        ldr     x7, [f+2*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x6, x6, x0+        adc     x5, xzr, x1+        ldr     x8, [g+2*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x6, x6, x0+        adc     x5, x5, x1+        extr    x2, x6, x2, #59+        str     x2, [f+N]++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x4, x4, x0+        adc     x2, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x4, x4, x0+        adc     x2, x2, x1+        extr    x3, x4, x3, #59+        str     x3, [g+N]++// Digit 3 of [f,g]++        ldr     x7, [f+3*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x5, x5, x0+        adc     x3, xzr, x1+        ldr     x8, [g+3*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x5, x5, x0+        adc     x3, x3, x1+        extr    x6, x5, x6, #59+        str     x6, [f+2*N]++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x2, x2, x0+        adc     x6, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x2, x2, x0+        adc     x6, x6, x1+        extr    x4, x2, x4, #59+        str     x4, [g+2*N]++// Digit 4 of [f,g]++        ldr     x7, [f+4*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x3, x3, x0+        adc     x4, xzr, x1+        ldr     x8, [g+4*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x3, x3, x0+        adc     x4, x4, x1+        extr    x5, x3, x5, #59+        str     x5, [f+3*N]++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x6, x6, x0+        adc     x5, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x6, x6, x0+        adc     x5, x5, x1+        extr    x2, x6, x2, #59+        str     x2, [g+3*N]++// Digits 5 and 6 of [f,g]++        ldr     x7, [f+5*N]+        eor     x1, x7, s00+        ldr     x23, [f+6*N]+        eor     x2, x23, s00+        and     x2, x2, m00+        neg     x2, x2+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x4, x4, x0+        adc     x2, x2, x1+        ldr     x8, [g+5*N]+        eor     x1, x8, s01+        ldr     x24, [g+6*N]+        eor     x0, x24, s01+        and     x0, x0, m01+        sub     x2, x2, x0+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x4, x4, x0+        adc     x2, x2, x1+        extr    x3, x4, x3, #59+        str     x3, [f+4*N]+        extr    x4, x2, x4, #59+        str     x4, [f+5*N]+        asr     x2, x2, #59+        str     x2, [f+6*N]++        eor     x1, x7, s10+        eor     x4, x23, s10+        and     x4, x4, m10+        neg     x4, x4+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x5, x5, x0+        adc     x4, x4, x1+        eor     x1, x8, s11+        eor     x0, x24, s11+        and     x0, x0, m11+        sub     x4, x4, x0+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x5, x5, x0+        adc     x4, x4, x1+        extr    x6, x5, x6, #59+        str     x6, [g+4*N]+        extr    x5, x4, x5, #59+        str     x5, [g+5*N]+        asr     x4, x4, #59+        str     x4, [g+6*N]++// Now the computation of the updated u and v values and their+// Montgomery reductions. A very similar accumulation except that+// the top words of u and v are unsigned and we don't shift.+//+// Digit 0 of [u,v]++        ldr     x7, [u]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x4, car0, x0+        adc     x2, xzr, x1+        ldr     x8, [v]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x4, x4, x0+        str     x4, [u]+        adc     x2, x2, x1++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x5, car1, x0+        adc     x3, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x5, x5, x0+        str     x5, [v]+        adc     x3, x3, x1++// Digit 1 of [u,v]++        ldr     x7, [u+N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x2, x2, x0+        adc     x6, xzr, x1+        ldr     x8, [v+N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x2, x2, x0+        str     x2, [u+N]+        adc     x6, x6, x1++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x3, x3, x0+        adc     x4, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x3, x3, x0+        str     x3, [v+N]+        adc     x4, x4, x1++// Digit 2 of [u,v]++        ldr     x7, [u+2*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x6, x6, x0+        adc     x5, xzr, x1+        ldr     x8, [v+2*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x6, x6, x0+        str     x6, [u+2*N]+        adc     x5, x5, x1++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x4, x4, x0+        adc     x2, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x4, x4, x0+        str     x4, [v+2*N]+        adc     x2, x2, x1++// Digit 3 of [u,v]++        ldr     x7, [u+3*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x5, x5, x0+        adc     x3, xzr, x1+        ldr     x8, [v+3*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x5, x5, x0+        str     x5, [u+3*N]+        adc     x3, x3, x1++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x2, x2, x0+        adc     x6, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x2, x2, x0+        str     x2, [v+3*N]+        adc     x6, x6, x1++// Digit 4 of [u,v]++        ldr     x7, [u+4*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x3, x3, x0+        adc     x4, xzr, x1+        ldr     x8, [v+4*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x3, x3, x0+        str     x3, [u+4*N]+        adc     x4, x4, x1++        eor     x1, x7, s10+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x6, x6, x0+        adc     x5, xzr, x1+        eor     x1, x8, s11+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x6, x6, x0+        str     x6, [v+4*N]+        adc     x5, x5, x1++// Digits 5 and 6 of [u,v] (top is unsigned)++        ldr     x7, [u+5*N]+        eor     x1, x7, s00+        and     x2, s00, m00+        neg     x2, x2+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x4, x4, x0+        adc     x2, x2, x1+        ldr     x8, [v+5*N]+        eor     x1, x8, s01+        and     x0, s01, m01+        sub     x2, x2, x0+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x4, x4, x0+        str     x4, [u+5*N]+        adc     x2, x2, x1+        str     x2, [u+6*N]++        eor     x1, x7, s10+        and     x4, s10, m10+        neg     x4, x4+        mul     x0, x1, m10+        umulh   x1, x1, m10+        adds    x5, x5, x0+        adc     x4, x4, x1+        eor     x1, x8, s11+        and     x0, s11, m11+        sub     x4, x4, x0+        mul     x0, x1, m11+        umulh   x1, x1, m11+        adds    x5, x5, x0+        str     x5, [v+5*N]+        adc     x4, x4, x1+        str     x4, [v+6*N]++// Montgomery reduction of u++        ldp     x0, x1, [u]+        ldp     x2, x3, [u+16]+        ldp     x4, x5, [u+32]+        ldr     x6, [u+48]+        amontred(x6,x5,x4,x3,x2,x1,x0, x9,x8,x7)+        stp     x1, x2, [u]+        stp     x3, x4, [u+16]+        stp     x5, x6, [u+32]++// Montgomery reduction of v++        ldp     x0, x1, [v]+        ldp     x2, x3, [v+16]+        ldp     x4, x5, [v+32]+        ldr     x6, [v+48]+        amontred(x6,x5,x4,x3,x2,x1,x0, x9,x8,x7)+        stp     x1, x2, [v]+        stp     x3, x4, [v+16]+        stp     x5, x6, [v+32]++Lbignum_montinv_p384_midloop:++        mov     x1, d+        ldr     x2, [f]+        ldr     x3, [g]+        divstep59()+        mov     d, x1++// Next iteration++        subs    i, i, #1+        bne     Lbignum_montinv_p384_loop++// The 15th and last iteration does not need anything except the+// u value and the sign of f; the latter can be obtained from the+// lowest word of f. So it's done differently from the main loop.+// Find the sign of the new f. For this we just need one digit+// since we know (for in-scope cases) that f is either +1 or -1.+// We don't explicitly shift right by 59 either, but looking at+// bit 63 (or any bit >= 60) of the unshifted result is enough+// to distinguish -1 from +1; this is then made into a mask.++        ldr     x0, [f]+        ldr     x1, [g]+        mul     x0, x0, m00+        madd    x1, x1, m01, x0+        asr     x0, x1, #63++// Now separate out the matrix into sign-magnitude pairs+// and adjust each one based on the sign of f.+//+// Note that at this point we expect |f|=1 and we got its+// sign above, so then since [f,0] == x * 2^{-768} [u,v] (mod p_384)+// we want to flip the sign of u according to that of f.++        cmp     m00, xzr+        csetm   s00, mi+        cneg    m00, m00, mi+        eor     s00, s00, x0++        cmp     m01, xzr+        csetm   s01, mi+        cneg    m01, m01, mi+        eor     s01, s01, x0++        cmp     m10, xzr+        csetm   s10, mi+        cneg    m10, m10, mi+        eor     s10, s10, x0++        cmp     m11, xzr+        csetm   s11, mi+        cneg    m11, m11, mi+        eor     s11, s11, x0++// Adjust the initial value to allow for complement instead of negation++        and     x0, m00, s00+        and     x1, m01, s01+        add     car0, x0, x1++// Digit 0 of [u]++        ldr     x7, [u]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x4, car0, x0+        adc     x2, xzr, x1+        ldr     x8, [v]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x4, x4, x0+        str     x4, [u]+        adc     x2, x2, x1++// Digit 1 of [u]++        ldr     x7, [u+N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x2, x2, x0+        adc     x6, xzr, x1+        ldr     x8, [v+N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x2, x2, x0+        str     x2, [u+N]+        adc     x6, x6, x1++// Digit 2 of [u]++        ldr     x7, [u+2*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x6, x6, x0+        adc     x5, xzr, x1+        ldr     x8, [v+2*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x6, x6, x0+        str     x6, [u+2*N]+        adc     x5, x5, x1++// Digit 3 of [u]++        ldr     x7, [u+3*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x5, x5, x0+        adc     x3, xzr, x1+        ldr     x8, [v+3*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x5, x5, x0+        str     x5, [u+3*N]+        adc     x3, x3, x1++// Digit 4 of [u]++        ldr     x7, [u+4*N]+        eor     x1, x7, s00+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x3, x3, x0+        adc     x4, xzr, x1+        ldr     x8, [v+4*N]+        eor     x1, x8, s01+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x3, x3, x0+        str     x3, [u+4*N]+        adc     x4, x4, x1++// Digits 5 and 6 of [u] (top is unsigned)++        ldr     x7, [u+5*N]+        eor     x1, x7, s00+        and     x2, s00, m00+        neg     x2, x2+        mul     x0, x1, m00+        umulh   x1, x1, m00+        adds    x4, x4, x0+        adc     x2, x2, x1+        ldr     x8, [v+5*N]+        eor     x1, x8, s01+        and     x0, s01, m01+        sub     x2, x2, x0+        mul     x0, x1, m01+        umulh   x1, x1, m01+        adds    x4, x4, x0+        str     x4, [u+5*N]+        adc     x2, x2, x1+        str     x2, [u+6*N]++// Montgomery reduction of u. This needs to be strict not "almost"+// so it is followed by an optional subtraction of p_384++        ldp     x10, x0, [u]+        ldp     x1, x2, [u+16]+        ldp     x3, x4, [u+32]+        ldr     x5, [u+48]+        amontred(x5,x4,x3,x2,x1,x0,x10, x9,x8,x7)++        mov     x10, #0x00000000ffffffff+        subs    x10, x0, x10+        mov     x11, #0xffffffff00000000+        sbcs    x11, x1, x11+        mov     x12, #0xfffffffffffffffe+        sbcs    x12, x2, x12+        mov     x15, #0xffffffffffffffff+        sbcs    x13, x3, x15+        sbcs    x14, x4, x15+        sbcs    x15, x5, x15++        csel    x0, x0, x10, cc+        csel    x1, x1, x11, cc+        csel    x2, x2, x12, cc+        csel    x3, x3, x13, cc+        csel    x4, x4, x14, cc+        csel    x5, x5, x15, cc++// Store it back to the final output++        stp     x0, x1, [res]+        stp     x2, x3, [res, #16]+        stp     x4, x5, [res, #32]++// Restore stack and registers++        CFI_INC_SP(NSPACE)+        CFI_POP2(x23,x24)+        CFI_POP2(x21,x22)+        CFI_POP2(x19,x20)+        CFI_RET++S2N_BN_SIZE_DIRECTIVE(bignum_montinv_p384)++#if defined(__linux__) && defined(__ELF__)+.section .note.GNU-stack, "", %progbits+#endif
+ cbits/s2n/arm/bignum_montmul_p384.S view
@@ -0,0 +1,891 @@+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0++// ----------------------------------------------------------------------------+// Montgomery multiply, z := (x * y / 2^384) mod p_384+// Inputs x[6], y[6]; output z[6]+//+//    extern void bignum_montmul_p384(uint64_t z[static 6],+//                                    const uint64_t x[static 6],+//                                    const uint64_t y[static 6]);+//+// Does z := (2^{-384} * x * y) mod p_384, assuming that the inputs x and y+// satisfy x * y <= 2^384 * p_384 (in particular this is true if we are in+// the "usual" case x < p_384 and y < p_384).+//+// Standard ARM ABI: X0 = z, X1 = x, X2 = y+// ----------------------------------------------------------------------------++// bignum_montmul_p384 is functionally equivalent to+// unopt/bignum_montmul_p384_base.+// It is written in a way that+// 1. A subset of scalar multiplications in bignum_montmul_p384 are carefully+//    chosen and vectorized+// 2. The vectorized assembly is rescheduled using the SLOTHY superoptimizer.+//    https://github.com/slothy-optimizer/slothy+//+// The output program of step 1. is as follows:+//+//        stp	x19, x20, [sp, #-16]!+//        stp	x21, x22, [sp, #-16]!+//        stp	x23, x24, [sp, #-16]!+//        ldp x3, x21, [x1]+//        ldr q30, [x1]+//        ldp x8, x24, [x1, #16]+//        ldp x5, x10, [x1, #32]+//        ldp x13, x23, [x2]+//        ldr q19, [x2]+//        ldp x6, x14, [x2, #16]+//        ldp x15, x17, [x2, #32]+//        ldr q1, [x1, #32]+//        ldr q28, [x2, #32]+//        uzp1 v5.4S, v19.4S, v30.4S+//        rev64 v19.4S, v19.4S+//        uzp1 v0.4S, v30.4S, v30.4S+//        mul v21.4S, v19.4S, v30.4S+//        uaddlp v19.2D, v21.4S+//        shl v19.2D, v19.2D, #32+//        umlal v19.2D, v0.2S, v5.2S+//        mov x12, v19.d[0]+//        mov x16, v19.d[1]+//        mul x20, x8, x6+//        umulh x4, x3, x13+//        umulh x1, x21, x23+//        umulh x2, x8, x6+//        adds x4, x4, x16+//        adcs x19, x1, x20+//        adc x20, x2, xzr+//        adds x11, x4, x12+//        adcs x16, x19, x4+//        adcs x1, x20, x19+//        adc x2, x20, xzr+//        adds x7, x16, x12+//        adcs x4, x1, x4+//        adcs x9, x2, x19+//        adc x19, x20, xzr+//        subs x2, x3, x21+//        cneg x20, x2, cc+//        csetm x16, cc+//        subs x2, x23, x13+//        cneg x2, x2, cc+//        mul x1, x20, x2+//        umulh x2, x20, x2+//        cinv x16, x16, cc+//        eor x1, x1, x16+//        eor x2, x2, x16+//        cmn x16, #0x1+//        adcs x11, x11, x1+//        adcs x7, x7, x2+//        adcs x4, x4, x16+//        adcs x9, x9, x16+//        adc x19, x19, x16+//        subs x2, x3, x8+//        cneg x20, x2, cc+//        csetm x16, cc+//        subs x2, x6, x13+//        cneg x2, x2, cc+//        mul x1, x20, x2+//        umulh x2, x20, x2+//        cinv x16, x16, cc+//        eor x1, x1, x16+//        eor x2, x2, x16+//        cmn x16, #0x1+//        adcs x7, x7, x1+//        adcs x4, x4, x2+//        adcs x9, x9, x16+//        adc x19, x19, x16+//        subs x2, x21, x8+//        cneg x20, x2, cc+//        csetm x16, cc+//        subs x2, x6, x23+//        cneg x2, x2, cc+//        mul x1, x20, x2+//        umulh x2, x20, x2+//        cinv x16, x16, cc+//        eor x1, x1, x16+//        eor x2, x2, x16+//        cmn x16, #0x1+//        adcs x4, x4, x1+//        adcs x20, x9, x2+//        adc x16, x19, x16+//        lsl x2, x12, #32+//        add x19, x2, x12+//        lsr x2, x19, #32+//        subs x1, x2, x19+//        sbc x2, x19, xzr+//        extr x1, x2, x1, #32+//        lsr x2, x2, #32+//        adds x12, x2, x19+//        adc x2, xzr, xzr+//        subs x1, x11, x1+//        sbcs x7, x7, x12+//        sbcs x4, x4, x2+//        sbcs x20, x20, xzr+//        sbcs x16, x16, xzr+//        sbc x9, x19, xzr+//        lsl x2, x1, #32+//        add x19, x2, x1+//        lsr x2, x19, #32+//        subs x1, x2, x19+//        sbc x2, x19, xzr+//        extr x1, x2, x1, #32+//        lsr x2, x2, #32+//        adds x12, x2, x19+//        adc x2, xzr, xzr+//        subs x1, x7, x1+//        sbcs x4, x4, x12+//        sbcs x20, x20, x2+//        sbcs x16, x16, xzr+//        sbcs x7, x9, xzr+//        sbc x9, x19, xzr+//        lsl x2, x1, #32+//        add x19, x2, x1+//        lsr x2, x19, #32+//        subs x1, x2, x19+//        sbc x2, x19, xzr+//        extr x12, x2, x1, #32+//        lsr x2, x2, #32+//        adds x1, x2, x19+//        adc x2, xzr, xzr+//        subs x4, x4, x12+//        sbcs x20, x20, x1+//        sbcs x16, x16, x2+//        sbcs x12, x7, xzr+//        sbcs x1, x9, xzr+//        sbc x2, x19, xzr+//        stp x4, x20, [x0]                       // @slothy:writes=buffer0+//        stp x16, x12, [x0, #16]                 // @slothy:writes=buffer16+//        stp x1, x2, [x0, #32]                   // @slothy:writes=buffer32+//        mul x22, x24, x14+//        movi v31.2D, #0x00000000ffffffff+//        uzp2 v16.4S, v28.4S, v28.4S+//        xtn v6.2S, v1.2D+//        xtn v30.2S, v28.2D+//        rev64 v28.4S, v28.4S+//        umull v5.2D, v6.2S, v30.2S+//        umull v0.2D, v6.2S, v16.2S+//        uzp2 v19.4S, v1.4S, v1.4S+//        mul v20.4S, v28.4S, v1.4S+//        usra v0.2D, v5.2D, #32+//        umull v1.2D, v19.2S, v16.2S+//        uaddlp v24.2D, v20.4S+//        and v5.16B, v0.16B, v31.16B+//        umlal v5.2D, v19.2S, v30.2S+//        shl v19.2D, v24.2D, #32+//        usra v1.2D, v0.2D, #32+//        umlal v19.2D, v6.2S, v30.2S+//        usra v1.2D, v5.2D, #32+//        mov x20, v19.d[0]+//        mov x16, v19.d[1]+//        umulh x12, x24, x14+//        mov x1, v1.d[0]+//        mov x2, v1.d[1]+//        adds x4, x12, x20+//        adcs x20, x1, x16+//        adc x16, x2, xzr+//        adds x7, x4, x22+//        adcs x12, x20, x4+//        adcs x1, x16, x20+//        adc x2, x16, xzr+//        adds x9, x12, x22+//        adcs x19, x1, x4+//        adcs x4, x2, x20+//        adc x20, x16, xzr+//        subs x2, x24, x5+//        cneg x16, x2, cc+//        csetm x12, cc+//        subs x2, x15, x14+//        cneg x2, x2, cc+//        mul x1, x16, x2+//        umulh x2, x16, x2+//        cinv x12, x12, cc+//        eor x1, x1, x12+//        eor x2, x2, x12+//        cmn x12, #0x1+//        adcs x11, x7, x1+//        adcs x9, x9, x2+//        adcs x19, x19, x12+//        adcs x4, x4, x12+//        adc x20, x20, x12+//        subs x2, x24, x10+//        cneg x16, x2, cc+//        csetm x12, cc+//        subs x2, x17, x14+//        cneg x2, x2, cc+//        mul x1, x16, x2+//        umulh x2, x16, x2+//        cinv x12, x12, cc+//        eor x1, x1, x12+//        eor x2, x2, x12+//        cmn x12, #0x1+//        adcs x7, x9, x1+//        adcs x19, x19, x2+//        adcs x4, x4, x12+//        adc x20, x20, x12+//        subs x2, x5, x10+//        cneg x16, x2, cc+//        csetm x12, cc+//        subs x2, x17, x15+//        cneg x2, x2, cc+//        mul x1, x16, x2+//        umulh x2, x16, x2+//        cinv x16, x12, cc+//        eor x1, x1, x16+//        eor x2, x2, x16+//        cmn x16, #0x1+//        adcs x19, x19, x1+//        adcs x12, x4, x2+//        adc x1, x20, x16+//        subs x2, x24, x3+//        sbcs x24, x5, x21+//        sbcs x21, x10, x8+//        ngc x5, xzr+//        cmn x5, #0x1+//        eor x2, x2, x5+//        adcs x4, x2, xzr+//        eor x2, x24, x5+//        adcs x20, x2, xzr+//        eor x2, x21, x5+//        adc x16, x2, xzr+//        subs x2, x13, x14+//        sbcs x24, x23, x15+//        sbcs x8, x6, x17+//        ngc x21, xzr+//        cmn x21, #0x1+//        eor x2, x2, x21+//        adcs x15, x2, xzr+//        eor x2, x24, x21+//        adcs x14, x2, xzr+//        eor x2, x8, x21+//        adc x6, x2, xzr+//        eor x9, x5, x21+//        ldp x21, x2, [x0]                       // @slothy:reads=buffer0+//        adds x10, x22, x21+//        adcs x5, x11, x2+//        ldp x21, x2, [x0, #16]                  // @slothy:reads=buffer16+//        adcs x24, x7, x21+//        adcs x8, x19, x2+//        ldp x21, x2, [x0, #32]                  // @slothy:reads=buffer32+//        adcs x21, x12, x21+//        adcs x2, x1, x2+//        adc x19, xzr, xzr+//        stp x10, x5, [x0]                       // @slothy:writes=buffer0+//        stp x24, x8, [x0, #16]                  // @slothy:writes=buffer16+//        stp x21, x2, [x0, #32]                  // @slothy:writes=buffer32+//        mul x12, x4, x15+//        mul x5, x20, x14+//        mul x24, x16, x6+//        umulh x8, x4, x15+//        umulh x21, x20, x14+//        umulh x2, x16, x6+//        adds x10, x8, x5+//        adcs x5, x21, x24+//        adc x24, x2, xzr+//        adds x23, x10, x12+//        adcs x8, x5, x10+//        adcs x21, x24, x5+//        adc x2, x24, xzr+//        adds x13, x8, x12+//        adcs x1, x21, x10+//        adcs x10, x2, x5+//        adc x5, x24, xzr+//        subs x2, x4, x20+//        cneg x24, x2, cc+//        csetm x8, cc+//        subs x2, x14, x15+//        cneg x2, x2, cc+//        mul x21, x24, x2+//        umulh x2, x24, x2+//        cinv x8, x8, cc+//        eor x21, x21, x8+//        eor x2, x2, x8+//        cmn x8, #0x1+//        adcs x23, x23, x21+//        adcs x13, x13, x2+//        adcs x1, x1, x8+//        adcs x10, x10, x8+//        adc x5, x5, x8+//        subs x2, x4, x16+//        cneg x24, x2, cc+//        csetm x8, cc+//        subs x2, x6, x15+//        cneg x2, x2, cc+//        mul x21, x24, x2+//        umulh x2, x24, x2+//        cinv x8, x8, cc+//        eor x21, x21, x8+//        eor x2, x2, x8+//        cmn x8, #0x1+//        adcs x4, x13, x21+//        adcs x13, x1, x2+//        adcs x1, x10, x8+//        adc x10, x5, x8+//        subs x2, x20, x16+//        cneg x24, x2, cc+//        csetm x8, cc+//        subs x2, x6, x14+//        cneg x2, x2, cc+//        mul x21, x24, x2+//        umulh x2, x24, x2+//        cinv x5, x8, cc+//        eor x21, x21, x5+//        eor x2, x2, x5+//        cmn x5, #0x1+//        adcs x24, x13, x21+//        adcs x8, x1, x2+//        adc x21, x10, x5+//        ldp x20, x16, [x0]                      // @slothy:reads=buffer0+//        ldp x17, x15, [x0, #16]                 // @slothy:reads=buffer16+//        ldp x14, x6, [x0, #32]                  // @slothy:reads=buffer32+//        cmn x9, #0x1+//        eor x2, x12, x9+//        adcs x12, x2, x20+//        eor x2, x23, x9+//        adcs x23, x2, x16+//        eor x2, x4, x9+//        adcs x13, x2, x17+//        eor x2, x24, x9+//        adcs x10, x2, x15+//        eor x2, x8, x9+//        adcs x5, x2, x14+//        eor x2, x21, x9+//        adcs x24, x2, x6+//        adcs x1, x9, x19+//        adcs x8, x9, xzr+//        adcs x21, x9, xzr+//        adc x2, x9, xzr+//        adds x10, x10, x20+//        adcs x5, x5, x16+//        adcs x24, x24, x17+//        adcs x17, x1, x15+//        adcs x15, x8, x14+//        adcs x14, x21, x6+//        adc x6, x2, x19+//        lsl x2, x12, #32+//        add x1, x2, x12+//        lsr x2, x1, #32+//        subs x21, x2, x1+//        sbc x2, x1, xzr+//        extr x21, x2, x21, #32+//        lsr x2, x2, #32+//        adds x8, x2, x1+//        adc x2, xzr, xzr+//        subs x21, x23, x21+//        sbcs x23, x13, x8+//        sbcs x10, x10, x2+//        sbcs x5, x5, xzr+//        sbcs x24, x24, xzr+//        sbc x13, x1, xzr+//        lsl x2, x21, #32+//        add x1, x2, x21+//        lsr x2, x1, #32+//        subs x21, x2, x1+//        sbc x2, x1, xzr+//        extr x21, x2, x21, #32+//        lsr x2, x2, #32+//        adds x8, x2, x1+//        adc x2, xzr, xzr+//        subs x21, x23, x21+//        sbcs x10, x10, x8+//        sbcs x5, x5, x2+//        sbcs x24, x24, xzr+//        sbcs x23, x13, xzr+//        sbc x13, x1, xzr+//        lsl x2, x21, #32+//        add x1, x2, x21+//        lsr x2, x1, #32+//        subs x21, x2, x1+//        sbc x2, x1, xzr+//        extr x8, x2, x21, #32+//        lsr x2, x2, #32+//        adds x21, x2, x1+//        adc x2, xzr, xzr+//        subs x10, x10, x8+//        sbcs x5, x5, x21+//        sbcs x24, x24, x2+//        sbcs x8, x23, xzr+//        sbcs x21, x13, xzr+//        sbc x2, x1, xzr+//        adds x23, x17, x8+//        adcs x13, x15, x21+//        adcs x1, x14, x2+//        adc x2, x6, xzr+//        add x8, x2, #0x1+//        lsl x2, x8, #32+//        subs x21, x8, x2+//        sbc x2, x2, xzr+//        adds x10, x10, x21+//        adcs x5, x5, x2+//        adcs x24, x24, x8+//        adcs x8, x23, xzr+//        adcs x21, x13, xzr+//        adcs x13, x1, xzr+//        csetm x1, cc+//        mov x2, #0xffffffff+//        and x2, x2, x1+//        adds x10, x10, x2+//        eor x2, x2, x1+//        adcs x5, x5, x2+//        mov x2, #0xfffffffffffffffe+//        and x2, x2, x1+//        adcs x24, x24, x2+//        adcs x8, x8, x1+//        adcs x21, x21, x1+//        adc x2, x13, x1+//        stp x10, x5, [x0]                       // @slothy:writes=buffer0+//        stp x24, x8, [x0, #16]                  // @slothy:writes=buffer16+//        stp x21, x2, [x0, #32]                  // @slothy:writes=buffer32+//        ldp	x23, x24, [sp], #16+//        ldp	x21, x22, [sp], #16+//        ldp	x19, x20, [sp], #16+//        ret+//+// The bash script used for step 2 is as follows:+//+//        # Store the assembly instructions except the last 'ret' and+//        # callee-register store/loads as, say, 'input.S'.+//        export OUTPUTS="[hint_buffer0,hint_buffer16,hint_buffer32]"+//        export RESERVED_REGS="[x18,x25,x26,x27,x28,x29,x30,sp,q8,q9,q10,q11,q12,q13,q14,q15,v8,v9,v10,v11,v12,v13,v14,v15]"+//        <s2n-bignum>/tools/external/slothy.sh input.S my_out_dir+//        # my_out_dir/3.opt.s is the optimized assembly. Its output may differ+//        # from this file since the sequence is non-deterministically chosen.+//        # Please add 'ret' at the end of the output assembly.+++#include "_internal_s2n_bignum_arm.h"++        S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montmul_p384)+        S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montmul_p384)+        S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montmul_p384)+        .text+        .balign 4++S2N_BN_SYMBOL(bignum_montmul_p384):+        CFI_START++// Save some registers++        CFI_PUSH2(x19,x20)+        CFI_PUSH2(x21,x22)+        CFI_PUSH2(x23,x24)++        ldr q3, [x1]+        ldr q25, [x2]+        ldp x13, x23, [x2]+        ldp x3, x21, [x1]+        rev64 v23.4S, v25.4S+        uzp1 v17.4S, v25.4S, v3.4S+        umulh x15, x3, x13+        mul v6.4S, v23.4S, v3.4S+        uzp1 v3.4S, v3.4S, v3.4S+        ldr q27, [x2, #32]+        ldp x8, x24, [x1, #16]+        subs x6, x3, x21+        ldr q0, [x1, #32]+        movi v23.2D, #0x00000000ffffffff+        csetm x10, cc+        umulh x19, x21, x23+        rev64 v4.4S, v27.4S+        uzp2 v25.4S, v27.4S, v27.4S+        cneg x4, x6, cc+        subs x7, x23, x13+        xtn v22.2S, v0.2D+        xtn v24.2S, v27.2D+        cneg x20, x7, cc+        ldp x6, x14, [x2, #16]+        mul v27.4S, v4.4S, v0.4S+        uaddlp v20.2D, v6.4S+        cinv x5, x10, cc+        mul x16, x4, x20+        uzp2 v6.4S, v0.4S, v0.4S+        umull v21.2D, v22.2S, v25.2S+        shl v0.2D, v20.2D, #32+        umlal v0.2D, v3.2S, v17.2S+        mul x22, x8, x6+        umull v1.2D, v6.2S, v25.2S+        subs x12, x3, x8+        umull v20.2D, v22.2S, v24.2S+        cneg x17, x12, cc+        umulh x9, x8, x6+        mov x12, v0.d[1]+        eor x11, x16, x5+        mov x7, v0.d[0]+        csetm x10, cc+        usra v21.2D, v20.2D, #32+        adds x15, x15, x12+        adcs x12, x19, x22+        umulh x20, x4, x20+        adc x19, x9, xzr+        usra v1.2D, v21.2D, #32+        adds x22, x15, x7+        and v26.16B, v21.16B, v23.16B+        adcs x16, x12, x15+        uaddlp v25.2D, v27.4S+        adcs x9, x19, x12+        umlal v26.2D, v6.2S, v24.2S+        adc x4, x19, xzr+        adds x16, x16, x7+        shl v27.2D, v25.2D, #32+        adcs x9, x9, x15+        adcs x4, x4, x12+        eor x12, x20, x5+        adc x15, x19, xzr+        subs x20, x6, x13+        cneg x20, x20, cc+        cinv x10, x10, cc+        cmn x5, #0x1+        mul x19, x17, x20+        adcs x11, x22, x11+        adcs x12, x16, x12+        adcs x9, x9, x5+        umulh x17, x17, x20+        adcs x22, x4, x5+        adc x5, x15, x5+        subs x16, x21, x8+        cneg x20, x16, cc+        eor x19, x19, x10+        csetm x4, cc+        subs x16, x6, x23+        cneg x16, x16, cc+        umlal v27.2D, v22.2S, v24.2S+        mul x15, x20, x16+        cinv x4, x4, cc+        cmn x10, #0x1+        usra v1.2D, v26.2D, #32+        adcs x19, x12, x19+        eor x17, x17, x10+        adcs x9, x9, x17+        adcs x22, x22, x10+        lsl x12, x7, #32+        umulh x20, x20, x16+        eor x16, x15, x4+        ldp x15, x17, [x2, #32]+        add x2, x12, x7+        adc x7, x5, x10+        ldp x5, x10, [x1, #32]+        lsr x1, x2, #32+        eor x12, x20, x4+        subs x1, x1, x2+        sbc x20, x2, xzr+        cmn x4, #0x1+        adcs x9, x9, x16+        extr x1, x20, x1, #32+        lsr x20, x20, #32+        adcs x22, x22, x12+        adc x16, x7, x4+        adds x12, x20, x2+        umulh x7, x24, x14+        adc x4, xzr, xzr+        subs x1, x11, x1+        sbcs x20, x19, x12+        sbcs x12, x9, x4+        lsl x9, x1, #32+        add x1, x9, x1+        sbcs x9, x22, xzr+        mul x22, x24, x14+        sbcs x16, x16, xzr+        lsr x4, x1, #32+        sbc x19, x2, xzr+        subs x4, x4, x1+        sbc x11, x1, xzr+        extr x2, x11, x4, #32+        lsr x4, x11, #32+        adds x4, x4, x1+        adc x11, xzr, xzr+        subs x2, x20, x2+        sbcs x4, x12, x4+        sbcs x20, x9, x11+        lsl x12, x2, #32+        add x2, x12, x2+        sbcs x9, x16, xzr+        lsr x11, x2, #32+        sbcs x19, x19, xzr+        sbc x1, x1, xzr+        subs x16, x11, x2+        sbc x12, x2, xzr+        extr x16, x12, x16, #32+        lsr x12, x12, #32+        adds x11, x12, x2+        adc x12, xzr, xzr+        subs x16, x4, x16+        mov x4, v27.d[0]+        sbcs x11, x20, x11+        sbcs x20, x9, x12+        stp x16, x11, [x0]+        sbcs x11, x19, xzr+        sbcs x9, x1, xzr+        stp x20, x11, [x0, #16]+        mov x1, v1.d[0]+        sbc x20, x2, xzr+        subs x12, x24, x5+        mov x11, v27.d[1]+        cneg x16, x12, cc+        csetm x2, cc+        subs x19, x15, x14+        mov x12, v1.d[1]+        cinv x2, x2, cc+        cneg x19, x19, cc+        stp x9, x20, [x0, #32]+        mul x9, x16, x19+        adds x4, x7, x4+        adcs x11, x1, x11+        adc x1, x12, xzr+        adds x20, x4, x22+        umulh x19, x16, x19+        adcs x7, x11, x4+        eor x16, x9, x2+        adcs x9, x1, x11+        adc x12, x1, xzr+        adds x7, x7, x22+        adcs x4, x9, x4+        adcs x9, x12, x11+        adc x12, x1, xzr+        cmn x2, #0x1+        eor x1, x19, x2+        adcs x11, x20, x16+        adcs x19, x7, x1+        adcs x1, x4, x2+        adcs x20, x9, x2+        adc x2, x12, x2+        subs x12, x24, x10+        cneg x16, x12, cc+        csetm x12, cc+        subs x9, x17, x14+        cinv x12, x12, cc+        cneg x9, x9, cc+        subs x3, x24, x3+        sbcs x21, x5, x21+        mul x24, x16, x9+        sbcs x4, x10, x8+        ngc x8, xzr+        subs x10, x5, x10+        eor x5, x24, x12+        csetm x7, cc+        cneg x24, x10, cc+        subs x10, x17, x15+        cinv x7, x7, cc+        cneg x10, x10, cc+        subs x14, x13, x14+        sbcs x15, x23, x15+        eor x13, x21, x8+        mul x23, x24, x10+        sbcs x17, x6, x17+        eor x6, x3, x8+        ngc x21, xzr+        umulh x9, x16, x9+        cmn x8, #0x1+        eor x3, x23, x7+        adcs x23, x6, xzr+        adcs x13, x13, xzr+        eor x16, x4, x8+        adc x16, x16, xzr+        eor x4, x17, x21+        umulh x17, x24, x10+        cmn x21, #0x1+        eor x24, x14, x21+        eor x6, x15, x21+        adcs x15, x24, xzr+        adcs x14, x6, xzr+        adc x6, x4, xzr+        cmn x12, #0x1+        eor x4, x9, x12+        adcs x19, x19, x5+        umulh x5, x23, x15+        adcs x1, x1, x4+        adcs x10, x20, x12+        eor x4, x17, x7+        ldp x20, x9, [x0]+        adc x2, x2, x12+        cmn x7, #0x1+        adcs x12, x1, x3+        ldp x17, x24, [x0, #16]+        mul x1, x16, x6+        adcs x3, x10, x4+        adc x2, x2, x7+        ldp x7, x4, [x0, #32]+        adds x20, x22, x20+        mul x10, x13, x14+        adcs x11, x11, x9+        eor x9, x8, x21+        adcs x21, x19, x17+        stp x20, x11, [x0]+        adcs x12, x12, x24+        mul x8, x23, x15+        adcs x3, x3, x7+        stp x21, x12, [x0, #16]+        adcs x12, x2, x4+        adc x19, xzr, xzr+        subs x21, x23, x16+        umulh x2, x16, x6+        stp x3, x12, [x0, #32]+        cneg x3, x21, cc+        csetm x24, cc+        umulh x11, x13, x14+        subs x21, x13, x16+        eor x7, x8, x9+        cneg x17, x21, cc+        csetm x16, cc+        subs x21, x6, x15+        cneg x22, x21, cc+        cinv x21, x24, cc+        subs x20, x23, x13+        umulh x12, x3, x22+        cneg x23, x20, cc+        csetm x24, cc+        subs x20, x14, x15+        cinv x24, x24, cc+        mul x22, x3, x22+        cneg x3, x20, cc+        subs x13, x6, x14+        cneg x20, x13, cc+        cinv x15, x16, cc+        adds x13, x5, x10+        mul x4, x23, x3+        adcs x11, x11, x1+        adc x14, x2, xzr+        adds x5, x13, x8+        adcs x16, x11, x13+        umulh x23, x23, x3+        adcs x3, x14, x11+        adc x1, x14, xzr+        adds x10, x16, x8+        adcs x6, x3, x13+        adcs x8, x1, x11+        umulh x13, x17, x20+        eor x1, x4, x24+        adc x4, x14, xzr+        cmn x24, #0x1+        adcs x1, x5, x1+        eor x16, x23, x24+        eor x11, x1, x9+        adcs x23, x10, x16+        eor x2, x22, x21+        adcs x3, x6, x24+        mul x14, x17, x20+        eor x17, x13, x15+        adcs x13, x8, x24+        adc x8, x4, x24+        cmn x21, #0x1+        adcs x6, x23, x2+        mov x16, #0xfffffffffffffffe+        eor x20, x12, x21+        adcs x20, x3, x20+        eor x23, x14, x15+        adcs x2, x13, x21+        adc x8, x8, x21+        cmn x15, #0x1+        ldp x5, x4, [x0]+        ldp x21, x12, [x0, #16]+        adcs x22, x20, x23+        eor x23, x22, x9+        adcs x17, x2, x17+        adc x22, x8, x15+        cmn x9, #0x1+        adcs x15, x7, x5+        ldp x10, x14, [x0, #32]+        eor x1, x6, x9+        lsl x2, x15, #32+        adcs x8, x11, x4+        adcs x13, x1, x21+        eor x1, x22, x9+        adcs x24, x23, x12+        eor x11, x17, x9+        adcs x23, x11, x10+        adcs x7, x1, x14+        adcs x17, x9, x19+        adcs x20, x9, xzr+        add x1, x2, x15+        lsr x3, x1, #32+        adcs x11, x9, xzr+        adc x9, x9, xzr+        subs x3, x3, x1+        sbc x6, x1, xzr+        adds x24, x24, x5+        adcs x4, x23, x4+        extr x3, x6, x3, #32+        lsr x6, x6, #32+        adcs x21, x7, x21+        adcs x15, x17, x12+        adcs x7, x20, x10+        adcs x20, x11, x14+        mov x14, #0xffffffff+        adc x22, x9, x19+        adds x12, x6, x1+        adc x10, xzr, xzr+        subs x3, x8, x3+        sbcs x12, x13, x12+        lsl x9, x3, #32+        add x3, x9, x3+        sbcs x10, x24, x10+        sbcs x24, x4, xzr+        lsr x9, x3, #32+        sbcs x21, x21, xzr+        sbc x1, x1, xzr+        subs x9, x9, x3+        sbc x13, x3, xzr+        extr x9, x13, x9, #32+        lsr x13, x13, #32+        adds x13, x13, x3+        adc x6, xzr, xzr+        subs x12, x12, x9+        sbcs x17, x10, x13+        lsl x2, x12, #32+        sbcs x10, x24, x6+        add x9, x2, x12+        sbcs x6, x21, xzr+        lsr x5, x9, #32+        sbcs x21, x1, xzr+        sbc x13, x3, xzr+        subs x8, x5, x9+        sbc x19, x9, xzr+        lsr x12, x19, #32+        extr x3, x19, x8, #32+        adds x8, x12, x9+        adc x1, xzr, xzr+        subs x2, x17, x3+        sbcs x12, x10, x8+        sbcs x5, x6, x1+        sbcs x3, x21, xzr+        sbcs x19, x13, xzr+        sbc x24, x9, xzr+        adds x23, x15, x3+        adcs x8, x7, x19+        adcs x11, x20, x24+        adc x9, x22, xzr+        add x24, x9, #0x1+        lsl x7, x24, #32+        subs x21, x24, x7+        sbc x10, x7, xzr+        adds x6, x2, x21+        adcs x7, x12, x10+        adcs x24, x5, x24+        adcs x13, x23, xzr+        adcs x8, x8, xzr+        adcs x15, x11, xzr+        csetm x23, cc+        and x11, x16, x23+        and x20, x14, x23+        adds x22, x6, x20+        eor x3, x20, x23+        adcs x5, x7, x3+        adcs x14, x24, x11+        stp x22, x5, [x0]+        adcs x5, x13, x23+        adcs x21, x8, x23+        stp x14, x5, [x0, #16]+        adc x12, x15, x23+        stp x21, x12, [x0, #32]++// Restore registers and return++        CFI_POP2(x23,x24)+        CFI_POP2(x21,x22)+        CFI_POP2(x19,x20)++        CFI_RET++S2N_BN_SIZE_DIRECTIVE(bignum_montmul_p384)++#if defined(__linux__) && defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/s2n/arm/bignum_montmul_p384_alt.S view
@@ -0,0 +1,345 @@+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0++// ----------------------------------------------------------------------------+// Montgomery multiply, z := (x * y / 2^384) mod p_384+// Inputs x[6], y[6]; output z[6]+//+//    extern void bignum_montmul_p384_alt(uint64_t z[static 6],+//                                        const uint64_t x[static 6],+//                                        const uint64_t y[static 6]);+//+// Does z := (2^{-384} * x * y) mod p_384, assuming that the inputs x and y+// satisfy x * y <= 2^384 * p_384 (in particular this is true if we are in+// the "usual" case x < p_384 and y < p_384).+//+// Standard ARM ABI: X0 = z, X1 = x, X2 = y+// ----------------------------------------------------------------------------++#include "_internal_s2n_bignum_arm.h"++        S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montmul_p384_alt)+        S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montmul_p384_alt)+        S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montmul_p384_alt)+        .text+        .balign 4++// ---------------------------------------------------------------------------+// Core one-step "short" Montgomery reduction macro. Takes input in+// [d5;d4;d3;d2;d1;d0] and returns result in [d6;d5;d4;d3;d2;d1],+// adding to the existing contents of [d5;d4;d3;d2;d1]. It is fine+// for d6 to be the same register as d0.+//+// We want to add (2^384 - 2^128 - 2^96 + 2^32 - 1) * w+// where w = [d0 + (d0<<32)] mod 2^64+// ---------------------------------------------------------------------------++#define montreds(d6,d5,d4,d3,d2,d1,d0, t3,t2,t1)                            \+/* Our correction multiplier is w = [d0 + (d0<<32)] mod 2^64            */  \+/* Store it in d6 to make the 2^384 * w contribution already            */  \+        lsl     t1, d0, #32 __LF                                       \+        add     d6, t1, d0 __LF                                        \+/* Now let [t3;t2;t1;-] = (2^384 - p_384) * w                    */         \+/* We know the lowest word will cancel d0 so we don't need it    */         \+        mov     t1, #0xffffffff00000001 __LF                           \+        umulh   t1, t1, d6 __LF                                        \+        mov     t2, #0x00000000ffffffff __LF                           \+        mul     t3, t2, d6 __LF                                        \+        umulh   t2, t2, d6 __LF                                        \+        adds    t1, t1, t3 __LF                                        \+        adcs    t2, t2, d6 __LF                                        \+        adc     t3, xzr, xzr __LF                                      \+/* Now add it, by subtracting from 2^384 * w + x */                         \+        subs    d1, d1, t1 __LF                                        \+        sbcs    d2, d2, t2 __LF                                        \+        sbcs    d3, d3, t3 __LF                                        \+        sbcs    d4, d4, xzr __LF                                       \+        sbcs    d5, d5, xzr __LF                                       \+        sbc     d6, d6, xzr+++#define z x0+#define x x1+#define y x2++// These are repeated mod 2 as we load pairs of inputs++#define a0 x3+#define a1 x4+#define a2 x3+#define a3 x4+#define a4 x3+#define a5 x4++#define b0 x5+#define b1 x6+#define b2 x7+#define b3 x8+#define b4 x9+#define b5 x10++#define l x11++#define u0 x12+#define u1 x13+#define u2 x14+#define u3 x15+#define u4 x16+#define u5 x17+#define u6 x19+#define u7 x20+#define u8 x21+#define u9 x22+#define u10 x2 // same as y+#define u11 x1 // same as x+#define h b5 // same as b5++S2N_BN_SYMBOL(bignum_montmul_p384_alt):+        CFI_START++// Save more registers++        CFI_PUSH2(x19,x20)+        CFI_PUSH2(x21,x22)++// Load operands and set up row 0 = [u6;...;u0] = a0 * [b5;...;b0]++        ldp     a0, a1, [x]+        ldp     b0, b1, [y]++        mul     u0, a0, b0+        umulh   u1, a0, b0+        mul     l, a0, b1+        umulh   u2, a0, b1+        adds    u1, u1, l++        ldp     b2, b3, [y, #16]++        mul     l, a0, b2+        umulh   u3, a0, b2+        adcs    u2, u2, l++        mul     l, a0, b3+        umulh   u4, a0, b3+        adcs    u3, u3, l++        ldp     b4, b5, [y, #32]++        mul     l, a0, b4+        umulh   u5, a0, b4+        adcs    u4, u4, l++        mul     l, a0, b5+        umulh   u6, a0, b5+        adcs    u5, u5, l++        adc     u6, u6, xzr++// Row 1 = [u7;...;u0] = [a1;a0] * [b5;...;b0]++        mul     l, a1, b0+        adds    u1, u1, l+        mul     l, a1, b1+        adcs    u2, u2, l+        mul     l, a1, b2+        adcs    u3, u3, l+        mul     l, a1, b3+        adcs    u4, u4, l+        mul     l, a1, b4+        adcs    u5, u5, l+        mul     l, a1, b5+        adcs    u6, u6, l+        cset    u7, cs++        umulh   l, a1, b0+        adds    u2, u2, l+        umulh   l, a1, b1+        adcs    u3, u3, l+        umulh   l, a1, b2+        adcs    u4, u4, l+        umulh   l, a1, b3+        adcs    u5, u5, l+        umulh   l, a1, b4+        adcs    u6, u6, l+        umulh   l, a1, b5+        adc     u7, u7, l++// Row 2 = [u8;...;u0] = [a2;a1;a0] * [b5;...;b0]++        ldp     a2, a3, [x, #16]++        mul     l, a2, b0+        adds    u2, u2, l+        mul     l, a2, b1+        adcs    u3, u3, l+        mul     l, a2, b2+        adcs    u4, u4, l+        mul     l, a2, b3+        adcs    u5, u5, l+        mul     l, a2, b4+        adcs    u6, u6, l+        mul     l, a2, b5+        adcs    u7, u7, l+        cset    u8, cs++        umulh   l, a2, b0+        adds    u3, u3, l+        umulh   l, a2, b1+        adcs    u4, u4, l+        umulh   l, a2, b2+        adcs    u5, u5, l+        umulh   l, a2, b3+        adcs    u6, u6, l+        umulh   l, a2, b4+        adcs    u7, u7, l+        umulh   l, a2, b5+        adc     u8, u8, l++// Row 3 = [u9;...;u0] = [a3;a2;a1;a0] * [b5;...;b0]++        mul     l, a3, b0+        adds    u3, u3, l+        mul     l, a3, b1+        adcs    u4, u4, l+        mul     l, a3, b2+        adcs    u5, u5, l+        mul     l, a3, b3+        adcs    u6, u6, l+        mul     l, a3, b4+        adcs    u7, u7, l+        mul     l, a3, b5+        adcs    u8, u8, l+        cset    u9, cs++        umulh   l, a3, b0+        adds    u4, u4, l+        umulh   l, a3, b1+        adcs    u5, u5, l+        umulh   l, a3, b2+        adcs    u6, u6, l+        umulh   l, a3, b3+        adcs    u7, u7, l+        umulh   l, a3, b4+        adcs    u8, u8, l+        umulh   l, a3, b5+        adc     u9, u9, l++// Row 4 = [u10;...;u0] = [a4;a3;a2;a1;a0] * [b5;...;b0]++        ldp     a4, a5, [x, #32]++        mul     l, a4, b0+        adds    u4, u4, l+        mul     l, a4, b1+        adcs    u5, u5, l+        mul     l, a4, b2+        adcs    u6, u6, l+        mul     l, a4, b3+        adcs    u7, u7, l+        mul     l, a4, b4+        adcs    u8, u8, l+        mul     l, a4, b5+        adcs    u9, u9, l+        cset    u10, cs++        umulh   l, a4, b0+        adds    u5, u5, l+        umulh   l, a4, b1+        adcs    u6, u6, l+        umulh   l, a4, b2+        adcs    u7, u7, l+        umulh   l, a4, b3+        adcs    u8, u8, l+        umulh   l, a4, b4+        adcs    u9, u9, l+        umulh   l, a4, b5+        adc     u10, u10, l++// Row 5 = [u11;...;u0] = [a5;a4;a3;a2;a1;a0] * [b5;...;b0]++        mul     l, a5, b0+        adds    u5, u5, l+        mul     l, a5, b1+        adcs    u6, u6, l+        mul     l, a5, b2+        adcs    u7, u7, l+        mul     l, a5, b3+        adcs    u8, u8, l+        mul     l, a5, b4+        adcs    u9, u9, l+        mul     l, a5, b5+        adcs    u10, u10, l+        cset    u11, cs++        umulh   l, a5, b0+        adds    u6, u6, l+        umulh   l, a5, b1+        adcs    u7, u7, l+        umulh   l, a5, b2+        adcs    u8, u8, l+        umulh   l, a5, b3+        adcs    u9, u9, l+        umulh   l, a5, b4+        adcs    u10, u10, l+        umulh   l, a5, b5+        adc     u11, u11, l++// Montgomery rotate the low half++        montreds(u0,u5,u4,u3,u2,u1,u0, b0,b1,b2)+        montreds(u1,u0,u5,u4,u3,u2,u1, b0,b1,b2)+        montreds(u2,u1,u0,u5,u4,u3,u2, b0,b1,b2)+        montreds(u3,u2,u1,u0,u5,u4,u3, b0,b1,b2)+        montreds(u4,u3,u2,u1,u0,u5,u4, b0,b1,b2)+        montreds(u5,u4,u3,u2,u1,u0,u5, b0,b1,b2)++// Add up the high and low parts as [h; u5;u4;u3;u2;u1;u0] = z++        adds    u0, u0, u6+        adcs    u1, u1, u7+        adcs    u2, u2, u8+        adcs    u3, u3, u9+        adcs    u4, u4, u10+        adcs    u5, u5, u11+        adc     h, xzr, xzr++// Now add [h; u11;u10;u9;u8;u7;u6] = z + (2^384 - p_384)++        mov     l, #0xffffffff00000001+        adds    u6, u0, l+        mov     l, #0x00000000ffffffff+        adcs    u7, u1, l+        mov     l, #0x0000000000000001+        adcs    u8, u2, l+        adcs    u9, u3, xzr+        adcs    u10, u4, xzr+        adcs    u11, u5, xzr+        adcs    h, h, xzr++// Now z >= p_384 iff h is nonzero, so select accordingly++        csel    u0, u0, u6, eq+        csel    u1, u1, u7, eq+        csel    u2, u2, u8, eq+        csel    u3, u3, u9, eq+        csel    u4, u4, u10, eq+        csel    u5, u5, u11, eq++// Store back final result++        stp     u0, u1, [z]+        stp     u2, u3, [z, #16]+        stp     u4, u5, [z, #32]++// Restore registers++        CFI_POP2(x21,x22)+        CFI_POP2(x19,x20)++        CFI_RET++S2N_BN_SIZE_DIRECTIVE(bignum_montmul_p384_alt)++#if defined(__linux__) && defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/s2n/arm/bignum_montsqr_p384.S view
@@ -0,0 +1,671 @@+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0++// ----------------------------------------------------------------------------+// Montgomery square, z := (x^2 / 2^384) mod p_384+// Input x[6]; output z[6]+//+//    extern void bignum_montsqr_p384(uint64_t z[static 6],+//                                    const uint64_t x[static 6]);+//+// Does z := (x^2 / 2^384) mod p_384, assuming x^2 <= 2^384 * p_384, which is+// guaranteed in particular if x < p_384 initially (the "intended" case).+//+// Standard ARM ABI: X0 = z, X1 = x+// ----------------------------------------------------------------------------++// bignum_montsqr_p384 is functionally equivalent to+// unopt/bignum_montsqr_p384_base.+// It is written in a way that+// 1. A subset of scalar multiplications in bignum_montsqr_p384 are carefully+//    chosen and vectorized+// 2. The vectorized assembly is rescheduled using the SLOTHY superoptimizer.+//    https://github.com/slothy-optimizer/slothy+//+// The output program of step 1. is as follows:+//+//        ldp x9, x2, [x1]+//        ldr q18, [x1]+//        ldr q19, [x1]+//        ldp x4, x6, [x1, #16]+//        ldp x5, x10, [x1, #32]+//        ldr q21, [x1, #32]+//        ldr q28, [x1, #32]+//        mul x12, x9, x2+//        mul x1, x9, x4+//        mul x13, x2, x4+//        movi v0.2D, #0x00000000ffffffff+//        uzp2 v5.4S, v19.4S, v19.4S+//        xtn v25.2S, v18.2D+//        xtn v4.2S, v19.2D+//        rev64 v23.4S, v19.4S+//        umull v20.2D, v25.2S, v4.2S+//        umull v30.2D, v25.2S, v5.2S+//        uzp2 v19.4S, v18.4S, v18.4S+//        mul v22.4S, v23.4S, v18.4S+//        usra v30.2D, v20.2D, #32+//        umull v18.2D, v19.2S, v5.2S+//        uaddlp v22.2D, v22.4S+//        and v20.16B, v30.16B, v0.16B+//        umlal v20.2D, v19.2S, v4.2S+//        shl v19.2D, v22.2D, #32+//        usra v18.2D, v30.2D, #32+//        umlal v19.2D, v25.2S, v4.2S+//        usra v18.2D, v20.2D, #32+//        mov x7, v19.d[0]+//        mov x17, v19.d[1]+//        mul x16, x4, x4+//        umulh x3, x9, x2+//        adds x15, x1, x3+//        umulh x1, x9, x4+//        adcs x13, x13, x1+//        umulh x1, x2, x4+//        adcs x8, x1, xzr+//        mov x11, v18.d[0]+//        mov x14, v18.d[1]+//        umulh x1, x4, x4+//        adds x3, x12, x12+//        adcs x15, x15, x15+//        adcs x13, x13, x13+//        adcs x12, x8, x8+//        adc x1, x1, xzr+//        adds x11, x11, x3+//        adcs x3, x17, x15+//        adcs x17, x14, x13+//        adcs x15, x16, x12+//        adc x13, x1, xzr+//        lsl x1, x7, #32+//        add x16, x1, x7+//        lsr x1, x16, #32+//        subs x12, x1, x16+//        sbc x1, x16, xzr+//        extr x12, x1, x12, #32+//        lsr x1, x1, #32+//        adds x7, x1, x16+//        adc x1, xzr, xzr+//        subs x12, x11, x12+//        sbcs x11, x3, x7+//        sbcs x17, x17, x1+//        sbcs x15, x15, xzr+//        sbcs x13, x13, xzr+//        sbc x3, x16, xzr+//        lsl x1, x12, #32+//        add x16, x1, x12+//        lsr x1, x16, #32+//        subs x12, x1, x16+//        sbc x1, x16, xzr+//        extr x12, x1, x12, #32+//        lsr x1, x1, #32+//        adds x7, x1, x16+//        adc x1, xzr, xzr+//        subs x12, x11, x12+//        sbcs x17, x17, x7+//        sbcs x15, x15, x1+//        sbcs x13, x13, xzr+//        sbcs x11, x3, xzr+//        sbc x3, x16, xzr+//        lsl x1, x12, #32+//        add x16, x1, x12+//        lsr x1, x16, #32+//        subs x12, x1, x16+//        sbc x1, x16, xzr+//        extr x7, x1, x12, #32+//        lsr x1, x1, #32+//        adds x12, x1, x16+//        adc x1, xzr, xzr+//        subs x17, x17, x7+//        sbcs x15, x15, x12+//        sbcs x13, x13, x1+//        sbcs x7, x11, xzr+//        sbcs x12, x3, xzr+//        sbc x1, x16, xzr+//        stp x17, x15, [x0]                     // @slothy:writes=buffer0+//        stp x13, x7, [x0, #16]                 // @slothy:writes=buffer16+//        stp x12, x1, [x0, #32]                 // @slothy:writes=buffer32+//        mul x14, x9, x6+//        mul x15, x2, x5+//        mul x13, x4, x10+//        umulh x7, x9, x6+//        umulh x12, x2, x5+//        umulh x1, x4, x10+//        adds x15, x7, x15+//        adcs x16, x12, x13+//        adc x13, x1, xzr+//        adds x11, x15, x14+//        adcs x7, x16, x15+//        adcs x12, x13, x16+//        adc x1, x13, xzr+//        adds x17, x7, x14+//        adcs x15, x12, x15+//        adcs x3, x1, x16+//        adc x16, x13, xzr+//        subs x1, x9, x2+//        cneg x13, x1, cc+//        csetm x7, cc+//        subs x1, x5, x6+//        cneg x1, x1, cc+//        mul x12, x13, x1+//        umulh x1, x13, x1+//        cinv x7, x7, cc+//        eor x12, x12, x7+//        eor x1, x1, x7+//        cmn x7, #0x1+//        adcs x11, x11, x12+//        adcs x17, x17, x1+//        adcs x15, x15, x7+//        adcs x3, x3, x7+//        adc x16, x16, x7+//        subs x9, x9, x4+//        cneg x13, x9, cc+//        csetm x7, cc+//        subs x1, x10, x6+//        cneg x1, x1, cc+//        mul x12, x13, x1+//        umulh x1, x13, x1+//        cinv x7, x7, cc+//        eor x12, x12, x7+//        eor x1, x1, x7+//        cmn x7, #0x1+//        adcs x17, x17, x12+//        adcs x15, x15, x1+//        adcs x13, x3, x7+//        adc x7, x16, x7+//        subs x2, x2, x4+//        cneg x12, x2, cc+//        csetm x1, cc+//        subs x2, x10, x5+//        cneg x2, x2, cc+//        mul x4, x12, x2+//        umulh x2, x12, x2+//        cinv x1, x1, cc+//        eor x4, x4, x1+//        eor x2, x2, x1+//        cmn x1, #0x1+//        adcs x12, x15, x4+//        adcs x4, x13, x2+//        adc x2, x7, x1+//        adds x1, x14, x14+//        adcs x16, x11, x11+//        adcs x17, x17, x17+//        adcs x15, x12, x12+//        adcs x13, x4, x4+//        adcs x7, x2, x2+//        adc x12, xzr, xzr+//        ldp x4, x2, [x0]                       // @slothy:reads=buffer0+//        adds x1, x1, x4+//        adcs x16, x16, x2+//        ldp x4, x2, [x0, #16]                  // @slothy:reads=buffer16+//        adcs x17, x17, x4+//        adcs x15, x15, x2+//        ldp x4, x2, [x0, #32]                  // @slothy:reads=buffer32+//        adcs x13, x13, x4+//        adcs x7, x7, x2+//        adc x11, x12, xzr+//        lsl x2, x1, #32+//        add x12, x2, x1+//        lsr x2, x12, #32+//        subs x4, x2, x12+//        sbc x2, x12, xzr+//        extr x4, x2, x4, #32+//        lsr x2, x2, #32+//        adds x1, x2, x12+//        adc x2, xzr, xzr+//        subs x4, x16, x4+//        sbcs x16, x17, x1+//        sbcs x17, x15, x2+//        sbcs x15, x13, xzr+//        sbcs x13, x7, xzr+//        sbc x7, x12, xzr+//        lsl x2, x4, #32+//        add x12, x2, x4+//        lsr x2, x12, #32+//        subs x4, x2, x12+//        sbc x2, x12, xzr+//        extr x4, x2, x4, #32+//        lsr x2, x2, #32+//        adds x1, x2, x12+//        adc x2, xzr, xzr+//        subs x4, x16, x4+//        sbcs x16, x17, x1+//        sbcs x17, x15, x2+//        sbcs x15, x13, xzr+//        sbcs x13, x7, xzr+//        sbc x7, x12, xzr+//        lsl x2, x4, #32+//        add x12, x2, x4+//        lsr x2, x12, #32+//        subs x4, x2, x12+//        sbc x2, x12, xzr+//        extr x1, x2, x4, #32+//        lsr x2, x2, #32+//        adds x4, x2, x12+//        adc x2, xzr, xzr+//        subs x3, x16, x1+//        sbcs x17, x17, x4+//        sbcs x15, x15, x2+//        sbcs x1, x13, xzr+//        sbcs x4, x7, xzr+//        sbc x2, x12, xzr+//        adds x13, x11, x1+//        adcs x7, x4, xzr+//        adcs x12, x2, xzr+//        adcs x16, xzr, xzr+//        mul x2, x6, x6+//        adds x3, x3, x2+//        xtn v30.2S, v28.2D+//        shrn v26.2S, v28.2D, #32+//        umull v26.2D, v30.2S, v26.2S+//        shl v19.2D, v26.2D, #33+//        umlal v19.2D, v30.2S, v30.2S+//        mov x1, v19.d[0]+//        mov x4, v19.d[1]+//        umulh x2, x6, x6+//        adcs x17, x17, x2+//        umulh x2, x5, x5+//        adcs x15, x15, x1+//        adcs x13, x13, x2+//        umulh x2, x10, x10+//        adcs x7, x7, x4+//        adcs x12, x12, x2+//        adc x16, x16, xzr+//        dup v28.2D, x6+//        movi v0.2D, #0x00000000ffffffff+//        uzp2 v5.4S, v21.4S, v21.4S+//        xtn v25.2S, v28.2D+//        xtn v4.2S, v21.2D+//        rev64 v19.4S, v21.4S+//        umull v30.2D, v25.2S, v4.2S+//        umull v23.2D, v25.2S, v5.2S+//        uzp2 v20.4S, v28.4S, v28.4S+//        mul v19.4S, v19.4S, v28.4S+//        usra v23.2D, v30.2D, #32+//        umull v18.2D, v20.2S, v5.2S+//        uaddlp v19.2D, v19.4S+//        and v30.16B, v23.16B, v0.16B+//        umlal v30.2D, v20.2S, v4.2S+//        shl v19.2D, v19.2D, #32+//        usra v18.2D, v23.2D, #32+//        umlal v19.2D, v25.2S, v4.2S+//        usra v18.2D, v30.2D, #32+//        mov x6, v19.d[0]+//        mov x1, v19.d[1]+//        mul x4, x5, x10+//        mov x2, v18.d[0]+//        adds x1, x1, x2+//        mov x2, v18.d[1]+//        adcs x4, x4, x2+//        umulh x5, x5, x10+//        adc x2, x5, xzr+//        adds x5, x6, x6+//        adcs x6, x1, x1+//        adcs x1, x4, x4+//        adcs x4, x2, x2+//        adc x2, xzr, xzr+//        adds x17, x17, x5+//        adcs x15, x15, x6+//        adcs x13, x13, x1+//        adcs x7, x7, x4+//        adcs x12, x12, x2+//        adc x2, x16, xzr+//        mov x5, #0xffffffff00000001+//        mov x6, #0xffffffff+//        mov x1, #0x1+//        cmn x3, x5+//        adcs xzr, x17, x6+//        adcs xzr, x15, x1+//        adcs xzr, x13, xzr+//        adcs xzr, x7, xzr+//        adcs xzr, x12, xzr+//        adc x2, x2, xzr+//        neg x4, x2+//        and x2, x5, x4+//        adds x10, x3, x2+//        and x2, x6, x4+//        adcs x5, x17, x2+//        and x2, x1, x4+//        adcs x6, x15, x2+//        adcs x1, x13, xzr+//        adcs x4, x7, xzr+//        adc x2, x12, xzr+//        stp x10, x5, [x0]                      // @slothy:writes=buffer0+//        stp x6, x1, [x0, #16]                  // @slothy:writes=buffer16+//        stp x4, x2, [x0, #32]                  // @slothy:writes=buffer32+//        ret+//+// The bash script used for step 2 is as follows:+//+//        # Store the assembly instructions except the last 'ret' as, say, 'input.S'.+//        export OUTPUTS="[hint_buffer0,hint_buffer16,hint_buffer32]"+//        export RESERVED_REGS="[x18,x19,x20,x21,x22,x23,x24,x25,x26,x27,x28,x29,x30,sp,q8,q9,q10,q11,q12,q13,q14,q15,v8,v9,v10,v11,v12,v13,v14,v15]"+//        <s2n-bignum>/tools/external/slothy.sh input.S my_out_dir+//        # my_out_dir/3.opt.s is the optimized assembly. Its output may differ+//        # from this file since the sequence is non-deterministically chosen.+//        # Please add 'ret' at the end of the output assembly.++++#include "_internal_s2n_bignum_arm.h"++        S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montsqr_p384)+        S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montsqr_p384)+        S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montsqr_p384)+        .text+        .balign 4++S2N_BN_SYMBOL(bignum_montsqr_p384):+        CFI_START++        ldr q1, [x1]+        ldp x9, x2, [x1]+        ldr q0, [x1]+        ldp x4, x6, [x1, #16]+        rev64 v21.4S, v1.4S+        uzp2 v28.4S, v1.4S, v1.4S+        umulh x7, x9, x2+        xtn v17.2S, v1.2D+        mul v27.4S, v21.4S, v0.4S+        ldr q20, [x1, #32]+        xtn v30.2S, v0.2D+        ldr q1, [x1, #32]+        uzp2 v31.4S, v0.4S, v0.4S+        ldp x5, x10, [x1, #32]+        umulh x8, x9, x4+        uaddlp v3.2D, v27.4S+        umull v16.2D, v30.2S, v17.2S+        mul x16, x9, x4+        umull v27.2D, v30.2S, v28.2S+        shrn v0.2S, v20.2D, #32+        xtn v7.2S, v20.2D+        shl v20.2D, v3.2D, #32+        umull v3.2D, v31.2S, v28.2S+        mul x3, x2, x4+        umlal v20.2D, v30.2S, v17.2S+        umull v22.2D, v7.2S, v0.2S+        usra v27.2D, v16.2D, #32+        umulh x11, x2, x4+        movi v21.2D, #0x00000000ffffffff+        uzp2 v28.4S, v1.4S, v1.4S+        adds x15, x16, x7+        and v5.16B, v27.16B, v21.16B+        adcs x3, x3, x8+        usra v3.2D, v27.2D, #32+        dup v29.2D, x6+        adcs x16, x11, xzr+        mov x14, v20.d[0]+        umlal v5.2D, v31.2S, v17.2S+        mul x8, x9, x2+        mov x7, v20.d[1]+        shl v19.2D, v22.2D, #33+        xtn v25.2S, v29.2D+        rev64 v31.4S, v1.4S+        lsl x13, x14, #32+        uzp2 v6.4S, v29.4S, v29.4S+        umlal v19.2D, v7.2S, v7.2S+        usra v3.2D, v5.2D, #32+        adds x1, x8, x8+        umulh x8, x4, x4+        add x12, x13, x14+        mul v17.4S, v31.4S, v29.4S+        xtn v4.2S, v1.2D+        adcs x14, x15, x15+        lsr x13, x12, #32+        adcs x15, x3, x3+        umull v31.2D, v25.2S, v28.2S+        adcs x11, x16, x16+        umull v21.2D, v25.2S, v4.2S+        mov x17, v3.d[0]+        umull v18.2D, v6.2S, v28.2S+        adc x16, x8, xzr+        uaddlp v16.2D, v17.4S+        movi v1.2D, #0x00000000ffffffff+        subs x13, x13, x12+        usra v31.2D, v21.2D, #32+        sbc x8, x12, xzr+        adds x17, x17, x1+        mul x1, x4, x4+        shl v28.2D, v16.2D, #32+        mov x3, v3.d[1]+        adcs x14, x7, x14+        extr x7, x8, x13, #32+        adcs x13, x3, x15+        and v3.16B, v31.16B, v1.16B+        adcs x11, x1, x11+        lsr x1, x8, #32+        umlal v3.2D, v6.2S, v4.2S+        usra v18.2D, v31.2D, #32+        adc x3, x16, xzr+        adds x1, x1, x12+        umlal v28.2D, v25.2S, v4.2S+        adc x16, xzr, xzr+        subs x15, x17, x7+        sbcs x7, x14, x1+        lsl x1, x15, #32+        sbcs x16, x13, x16+        add x8, x1, x15+        usra v18.2D, v3.2D, #32+        sbcs x14, x11, xzr+        lsr x1, x8, #32+        sbcs x17, x3, xzr+        sbc x11, x12, xzr+        subs x13, x1, x8+        umulh x12, x4, x10+        sbc x1, x8, xzr+        extr x13, x1, x13, #32+        lsr x1, x1, #32+        adds x15, x1, x8+        adc x1, xzr, xzr+        subs x7, x7, x13+        sbcs x13, x16, x15+        lsl x3, x7, #32+        umulh x16, x2, x5+        sbcs x15, x14, x1+        add x7, x3, x7+        sbcs x3, x17, xzr+        lsr x1, x7, #32+        sbcs x14, x11, xzr+        sbc x11, x8, xzr+        subs x8, x1, x7+        sbc x1, x7, xzr+        extr x8, x1, x8, #32+        lsr x1, x1, #32+        adds x1, x1, x7+        adc x17, xzr, xzr+        subs x13, x13, x8+        umulh x8, x9, x6+        sbcs x1, x15, x1+        sbcs x15, x3, x17+        sbcs x3, x14, xzr+        mul x17, x2, x5+        sbcs x11, x11, xzr+        stp x13, x1, [x0]                       // @slothy:writes=buffer0+        sbc x14, x7, xzr+        mul x7, x4, x10+        subs x1, x9, x2+        stp x15, x3, [x0, #16]                  // @slothy:writes=buffer16+        csetm x15, cc+        cneg x1, x1, cc+        stp x11, x14, [x0, #32]                 // @slothy:writes=buffer32+        mul x14, x9, x6+        adds x17, x8, x17+        adcs x7, x16, x7+        adc x13, x12, xzr+        subs x12, x5, x6+        cneg x3, x12, cc+        cinv x16, x15, cc+        mul x8, x1, x3+        umulh x1, x1, x3+        eor x12, x8, x16+        adds x11, x17, x14+        adcs x3, x7, x17+        adcs x15, x13, x7+        adc x8, x13, xzr+        adds x3, x3, x14+        adcs x15, x15, x17+        adcs x17, x8, x7+        eor x1, x1, x16+        adc x13, x13, xzr+        subs x9, x9, x4+        csetm x8, cc+        cneg x9, x9, cc+        subs x4, x2, x4+        cneg x4, x4, cc+        csetm x7, cc+        subs x2, x10, x6+        cinv x8, x8, cc+        cneg x2, x2, cc+        cmn x16, #0x1+        adcs x11, x11, x12+        mul x12, x9, x2+        adcs x3, x3, x1+        adcs x15, x15, x16+        umulh x9, x9, x2+        adcs x17, x17, x16+        adc x13, x13, x16+        subs x1, x10, x5+        cinv x2, x7, cc+        cneg x1, x1, cc+        eor x9, x9, x8+        cmn x8, #0x1+        eor x7, x12, x8+        mul x12, x4, x1+        adcs x3, x3, x7+        adcs x7, x15, x9+        adcs x15, x17, x8+        ldp x9, x17, [x0, #16]                  // @slothy:reads=buffer16+        umulh x4, x4, x1+        adc x8, x13, x8+        cmn x2, #0x1+        eor x1, x12, x2+        adcs x1, x7, x1+        ldp x7, x16, [x0]                       // @slothy:reads=buffer0+        eor x12, x4, x2+        adcs x4, x15, x12+        ldp x15, x12, [x0, #32]                 // @slothy:reads=buffer32+        adc x8, x8, x2+        adds x13, x14, x14+        umulh x14, x5, x10+        adcs x2, x11, x11+        adcs x3, x3, x3+        adcs x1, x1, x1+        adcs x4, x4, x4+        adcs x11, x8, x8+        adc x8, xzr, xzr+        adds x13, x13, x7+        adcs x2, x2, x16+        mul x16, x5, x10+        adcs x3, x3, x9+        adcs x1, x1, x17+        umulh x5, x5, x5+        lsl x9, x13, #32+        add x9, x9, x13+        adcs x4, x4, x15+        mov x13, v28.d[1]+        adcs x15, x11, x12+        lsr x7, x9, #32+        adc x11, x8, xzr+        subs x7, x7, x9+        umulh x10, x10, x10+        sbc x17, x9, xzr+        extr x7, x17, x7, #32+        lsr x17, x17, #32+        adds x17, x17, x9+        adc x12, xzr, xzr+        subs x8, x2, x7+        sbcs x17, x3, x17+        lsl x7, x8, #32+        sbcs x2, x1, x12+        add x3, x7, x8+        sbcs x12, x4, xzr+        lsr x1, x3, #32+        sbcs x7, x15, xzr+        sbc x15, x9, xzr+        subs x1, x1, x3+        sbc x4, x3, xzr+        lsr x9, x4, #32+        extr x8, x4, x1, #32+        adds x9, x9, x3+        adc x4, xzr, xzr+        subs x1, x17, x8+        lsl x17, x1, #32+        sbcs x8, x2, x9+        sbcs x9, x12, x4+        add x17, x17, x1+        mov x1, v18.d[1]+        lsr x2, x17, #32+        sbcs x7, x7, xzr+        mov x12, v18.d[0]+        sbcs x15, x15, xzr+        sbc x3, x3, xzr+        subs x4, x2, x17+        sbc x2, x17, xzr+        adds x12, x13, x12+        adcs x16, x16, x1+        lsr x13, x2, #32+        extr x1, x2, x4, #32+        adc x2, x14, xzr+        adds x4, x13, x17+        mul x13, x6, x6+        adc x14, xzr, xzr+        subs x1, x8, x1+        sbcs x4, x9, x4+        mov x9, v28.d[0]+        sbcs x7, x7, x14+        sbcs x8, x15, xzr+        sbcs x3, x3, xzr+        sbc x14, x17, xzr+        adds x17, x9, x9+        adcs x12, x12, x12+        mov x15, v19.d[0]+        adcs x9, x16, x16+        umulh x6, x6, x6+        adcs x16, x2, x2+        adc x2, xzr, xzr+        adds x11, x11, x8+        adcs x3, x3, xzr+        adcs x14, x14, xzr+        adcs x8, xzr, xzr+        adds x13, x1, x13+        mov x1, v19.d[1]+        adcs x6, x4, x6+        mov x4, #0xffffffff+        adcs x15, x7, x15+        adcs x7, x11, x5+        adcs x1, x3, x1+        adcs x14, x14, x10+        adc x11, x8, xzr+        adds x6, x6, x17+        adcs x8, x15, x12+        adcs x3, x7, x9+        adcs x15, x1, x16+        mov x16, #0xffffffff00000001+        adcs x14, x14, x2+        mov x2, #0x1+        adc x17, x11, xzr+        cmn x13, x16+        adcs xzr, x6, x4+        adcs xzr, x8, x2+        adcs xzr, x3, xzr+        adcs xzr, x15, xzr+        adcs xzr, x14, xzr+        adc x1, x17, xzr+        neg x9, x1+        and x1, x16, x9+        adds x11, x13, x1+        and x13, x4, x9+        adcs x5, x6, x13+        and x1, x2, x9+        adcs x7, x8, x1+        stp x11, x5, [x0]                       // @slothy:writes=buffer0+        adcs x11, x3, xzr+        adcs x2, x15, xzr+        stp x7, x11, [x0, #16]                  // @slothy:writes=buffer16+        adc x17, x14, xzr+        stp x2, x17, [x0, #32]                  // depth 72 // @slothy:writes=buffer32++        CFI_RET++S2N_BN_SIZE_DIRECTIVE(bignum_montsqr_p384)++#if defined(__linux__) && defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/s2n/arm/bignum_montsqr_p384_alt.S view
@@ -0,0 +1,273 @@+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0++// ----------------------------------------------------------------------------+// Montgomery square, z := (x^2 / 2^384) mod p_384+// Input x[6]; output z[6]+//+//    extern void bignum_montsqr_p384_alt(uint64_t z[static 6],+//                                        const uint64_t x[static 6]);+//+// Does z := (x^2 / 2^384) mod p_384, assuming x^2 <= 2^384 * p_384, which is+// guaranteed in particular if x < p_384 initially (the "intended" case).+//+// Standard ARM ABI: X0 = z, X1 = x+// ----------------------------------------------------------------------------++#include "_internal_s2n_bignum_arm.h"++        S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montsqr_p384_alt)+        S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montsqr_p384_alt)+        S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montsqr_p384_alt)+        .text+        .balign 4++// ---------------------------------------------------------------------------+// Core one-step "short" Montgomery reduction macro. Takes input in+// [d5;d4;d3;d2;d1;d0] and returns result in [d6;d5;d4;d3;d2;d1],+// adding to the existing contents of [d5;d4;d3;d2;d1]. It is fine+// for d6 to be the same register as d0.+//+// We want to add (2^384 - 2^128 - 2^96 + 2^32 - 1) * w+// where w = [d0 + (d0<<32)] mod 2^64+// ---------------------------------------------------------------------------++#define montreds(d6,d5,d4,d3,d2,d1,d0, t3,t2,t1)                            \+/* Our correction multiplier is w = [d0 + (d0<<32)] mod 2^64            */  \+/* Store it in d6 to make the 2^384 * w contribution already            */  \+        lsl     t1, d0, #32 __LF                                       \+        add     d6, t1, d0 __LF                                        \+/* Now let [t3;t2;t1;-] = (2^384 - p_384) * w                    */         \+/* We know the lowest word will cancel d0 so we don't need it    */         \+        mov     t1, #0xffffffff00000001 __LF                           \+        umulh   t1, t1, d6 __LF                                        \+        mov     t2, #0x00000000ffffffff __LF                           \+        mul     t3, t2, d6 __LF                                        \+        umulh   t2, t2, d6 __LF                                        \+        adds    t1, t1, t3 __LF                                        \+        adcs    t2, t2, d6 __LF                                        \+        adc     t3, xzr, xzr __LF                                      \+/* Now add it, by subtracting from 2^384 * w + x */                         \+        subs    d1, d1, t1 __LF                                        \+        sbcs    d2, d2, t2 __LF                                        \+        sbcs    d3, d3, t3 __LF                                        \+        sbcs    d4, d4, xzr __LF                                       \+        sbcs    d5, d5, xzr __LF                                       \+        sbc     d6, d6, xzr++#define z x0+#define x x1++#define a0 x2+#define a1 x3+#define a2 x4+#define a3 x5+#define a4 x6+#define a5 x7++#define l x8++#define u0 x2 // The same as a0, which is safe+#define u1 x9+#define u2 x10+#define u3 x11+#define u4 x12+#define u5 x13+#define u6 x14+#define u7 x15+#define u8 x16+#define u9 x17+#define u10 x19+#define u11 x20+#define h x6 // same as a4++S2N_BN_SYMBOL(bignum_montsqr_p384_alt):+        CFI_START++// It's convenient to have two more registers to play with++        CFI_PUSH2(x19,x20)++// Load all the elements as [a5;a4;a3;a2;a1;a0], set up an initial+// window [u8;u7; u6;u5; u4;u3; u2;u1] = [34;05;03;01], and then+// chain in the addition of 02 + 12 + 13 + 14 + 15 to that window+// (no carry-out possible since we add it to the top of a product).++        ldp     a0, a1, [x]++        mul     u1, a0, a1+        umulh   u2, a0, a1++        ldp     a2, a3, [x, #16]++        mul     l, a0, a2+        adds    u2, u2, l++        mul     u3, a0, a3+        mul     l, a1, a2+        adcs    u3, u3, l++        umulh   u4, a0, a3+        mul     l, a1, a3+        adcs    u4, u4, l++        ldp     a4, a5, [x, #32]++        mul     u5, a0, a5+        mul     l, a1, a4+        adcs    u5, u5, l++        umulh   u6, a0, a5+        mul     l, a1, a5+        adcs    u6, u6, l++        mul     u7, a3, a4+        adcs    u7, u7, xzr++        umulh   u8, a3, a4+        adc     u8, u8, xzr++        umulh   l, a0, a2+        adds    u3, u3, l+        umulh   l, a1, a2+        adcs    u4, u4, l+        umulh   l, a1, a3+        adcs    u5, u5, l+        umulh   l, a1, a4+        adcs    u6, u6, l+        umulh   l, a1, a5+        adcs    u7, u7, l+        adc     u8, u8, xzr++// Now chain in the 04 + 23 + 24 + 25 + 35 + 45 terms++        mul     l, a0, a4+        adds    u4, u4, l+        mul     l, a2, a3+        adcs    u5, u5, l+        mul     l, a2, a4+        adcs    u6, u6, l+        mul     l, a2, a5+        adcs    u7, u7, l+        mul     l, a3, a5+        adcs    u8, u8, l+        mul     u9, a4, a5+        adcs    u9, u9, xzr+        umulh   u10, a4, a5+        adc     u10, u10, xzr++        umulh   l, a0, a4+        adds    u5, u5, l+        umulh   l, a2, a3+        adcs    u6, u6, l+        umulh   l, a2, a4+        adcs    u7, u7, l+        umulh   l, a2, a5+        adcs    u8, u8, l+        umulh   l, a3, a5+        adcs    u9, u9, l+        adc     u10, u10, xzr++// Double that, with u11 holding the top carry++        adds    u1, u1, u1+        adcs    u2, u2, u2+        adcs    u3, u3, u3+        adcs    u4, u4, u4+        adcs    u5, u5, u5+        adcs    u6, u6, u6+        adcs    u7, u7, u7+        adcs    u8, u8, u8+        adcs    u9, u9, u9+        adcs    u10, u10, u10+        cset    u11, cs++// Add the homogeneous terms 00 + 11 + 22 + 33 + 44 + 55++        umulh   l, a0, a0+        mul     u0, a0, a0+        adds    u1, u1, l++        mul     l, a1, a1+        adcs    u2, u2, l+        umulh   l, a1, a1+        adcs    u3, u3, l++        mul     l, a2, a2+        adcs    u4, u4, l+        umulh   l, a2, a2+        adcs    u5, u5, l++        mul     l, a3, a3+        adcs    u6, u6, l+        umulh   l, a3, a3+        adcs    u7, u7, l++        mul     l, a4, a4+        adcs    u8, u8, l+        umulh   l, a4, a4+        adcs    u9, u9, l++        mul     l, a5, a5+        adcs    u10, u10, l+        umulh   l, a5, a5+        adc     u11, u11, l++// Montgomery rotate the low half++        montreds(u0,u5,u4,u3,u2,u1,u0, a1,a2,a3)+        montreds(u1,u0,u5,u4,u3,u2,u1, a1,a2,a3)+        montreds(u2,u1,u0,u5,u4,u3,u2, a1,a2,a3)+        montreds(u3,u2,u1,u0,u5,u4,u3, a1,a2,a3)+        montreds(u4,u3,u2,u1,u0,u5,u4, a1,a2,a3)+        montreds(u5,u4,u3,u2,u1,u0,u5, a1,a2,a3)++// Add up the high and low parts as [h; u5;u4;u3;u2;u1;u0] = z++        adds    u0, u0, u6+        adcs    u1, u1, u7+        adcs    u2, u2, u8+        adcs    u3, u3, u9+        adcs    u4, u4, u10+        adcs    u5, u5, u11+        adc     h, xzr, xzr++// Now add [h; u11;u10;u9;u8;u7;u6] = z + (2^384 - p_384)++        mov     l, #0xffffffff00000001+        adds    u6, u0, l+        mov     l, #0x00000000ffffffff+        adcs    u7, u1, l+        mov     l, #0x0000000000000001+        adcs    u8, u2, l+        adcs    u9, u3, xzr+        adcs    u10, u4, xzr+        adcs    u11, u5, xzr+        adcs    h, h, xzr++// Now z >= p_384 iff h is nonzero, so select accordingly++        csel    u0, u0, u6, eq+        csel    u1, u1, u7, eq+        csel    u2, u2, u8, eq+        csel    u3, u3, u9, eq+        csel    u4, u4, u10, eq+        csel    u5, u5, u11, eq++// Store back final result++        stp     u0, u1, [z]+        stp     u2, u3, [z, #16]+        stp     u4, u5, [z, #32]++// Restore registers++        CFI_POP2(x19,x20)++        CFI_RET++S2N_BN_SIZE_DIRECTIVE(bignum_montsqr_p384_alt)++#if defined(__linux__) && defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/s2n/arm/bignum_mul_p521.S view
@@ -0,0 +1,1407 @@+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0++// ----------------------------------------------------------------------------+// Multiply modulo p_521, z := (x * y) mod p_521, assuming x and y reduced+// Inputs x[9], y[9]; output z[9]+//+//    extern void bignum_mul_p521(uint64_t z[static 9], const uint64_t x[static 9],+//                                const uint64_t y[static 9]);+//+// Standard ARM ABI: X0 = z, X1 = x, X2 = y+// ----------------------------------------------------------------------------++#include "_internal_s2n_bignum_arm.h"++// bignum_mul_p521 is functionally equivalent to unopt/bignum_mul_p521_base.+// It is written in a way that+// 1. A subset of scalar multiplications in bignum_montmul_p384 are carefully+//    chosen and vectorized+// 2. The vectorized assembly is rescheduled using the SLOTHY superoptimizer.+//    https://github.com/slothy-optimizer/slothy+//+// The output program of step 1. is as follows:+//+//        stp     x19, x20, [sp, #-16]!+//        stp     x21, x22, [sp, #-16]!+//        stp     x23, x24, [sp, #-16]!+//        stp     x25, x26, [sp, #-16]!+//        sub     sp, sp, #80+//        ldp x15, x21, [x1]+//        ldp x10, x17, [x1, #16]+//        ldp x13, x16, [x2]+//        ldr q18, [x1]+//        ldr q28, [x2]+//        ldp x5, x20, [x2, #16]+//        movi v16.2D, #0x00000000ffffffff+//        uzp2 v7.4S, v28.4S, v28.4S+//        xtn v4.2S, v18.2D+//        xtn v1.2S, v28.2D+//        rev64 v27.4S, v28.4S+//        umull v21.2D, v4.2S, v1.2S+//        umull v28.2D, v4.2S, v7.2S+//        uzp2 v5.4S, v18.4S, v18.4S+//        mul v18.4S, v27.4S, v18.4S+//        usra v28.2D, v21.2D, #32+//        umull v29.2D, v5.2S, v7.2S+//        uaddlp v18.2D, v18.4S+//        and v16.16B, v28.16B, v16.16B+//        umlal v16.2D, v5.2S, v1.2S+//        shl v18.2D, v18.2D, #32+//        usra v29.2D, v28.2D, #32+//        umlal v18.2D, v4.2S, v1.2S+//        usra v29.2D, v16.2D, #32+//        mov x8, v18.d[0]+//        mov x9, v18.d[1]+//        mul x6, x10, x5+//        mul x19, x17, x20+//        mov x14, v29.d[0]+//        adds x9, x9, x14+//        mov x14, v29.d[1]+//        adcs x6, x6, x14+//        umulh x14, x10, x5+//        adcs x19, x19, x14+//        umulh x14, x17, x20+//        adc x14, x14, xzr+//        adds x11, x9, x8+//        adcs x9, x6, x9+//        adcs x6, x19, x6+//        adcs x19, x14, x19+//        adc x14, xzr, x14+//        adds x3, x9, x8+//        adcs x24, x6, x11+//        adcs x9, x19, x9+//        adcs x6, x14, x6+//        adcs x19, xzr, x19+//        adc x14, xzr, x14+//        subs x4, x10, x17+//        cneg x4, x4, cc+//        csetm x7, cc+//        subs x23, x20, x5+//        cneg x23, x23, cc+//        mul x22, x4, x23+//        umulh x4, x4, x23+//        cinv x7, x7, cc+//        cmn x7, #0x1+//        eor x23, x22, x7+//        adcs x6, x6, x23+//        eor x4, x4, x7+//        adcs x19, x19, x4+//        adc x14, x14, x7+//        subs x4, x15, x21+//        cneg x4, x4, cc+//        csetm x7, cc+//        subs x23, x16, x13+//        cneg x23, x23, cc+//        mul x22, x4, x23+//        umulh x4, x4, x23+//        cinv x7, x7, cc+//        cmn x7, #0x1+//        eor x23, x22, x7+//        adcs x11, x11, x23+//        eor x4, x4, x7+//        adcs x3, x3, x4+//        adcs x24, x24, x7+//        adcs x9, x9, x7+//        adcs x6, x6, x7+//        adcs x19, x19, x7+//        adc x14, x14, x7+//        subs x4, x21, x17+//        cneg x4, x4, cc+//        csetm x7, cc+//        subs x23, x20, x16+//        cneg x23, x23, cc+//        mul x22, x4, x23+//        umulh x4, x4, x23+//        cinv x7, x7, cc+//        cmn x7, #0x1+//        eor x23, x22, x7+//        adcs x9, x9, x23+//        eor x4, x4, x7+//        adcs x6, x6, x4+//        adcs x19, x19, x7+//        adc x14, x14, x7+//        subs x4, x15, x10+//        cneg x4, x4, cc+//        csetm x7, cc+//        subs x23, x5, x13+//        cneg x23, x23, cc+//        mul x22, x4, x23+//        umulh x4, x4, x23+//        cinv x7, x7, cc+//        cmn x7, #0x1+//        eor x23, x22, x7+//        adcs x3, x3, x23+//        eor x4, x4, x7+//        adcs x24, x24, x4+//        adcs x9, x9, x7+//        adcs x6, x6, x7+//        adcs x19, x19, x7+//        adc x14, x14, x7+//        subs x17, x15, x17+//        cneg x17, x17, cc+//        csetm x4, cc+//        subs x13, x20, x13+//        cneg x13, x13, cc+//        mul x20, x17, x13+//        umulh x17, x17, x13+//        cinv x13, x4, cc+//        cmn x13, #0x1+//        eor x20, x20, x13+//        adcs x20, x24, x20+//        eor x17, x17, x13+//        adcs x17, x9, x17+//        adcs x9, x6, x13+//        adcs x6, x19, x13+//        adc x13, x14, x13+//        subs x21, x21, x10+//        cneg x21, x21, cc+//        csetm x10, cc+//        subs x16, x5, x16+//        cneg x16, x16, cc+//        mul x5, x21, x16+//        umulh x21, x21, x16+//        cinv x10, x10, cc+//        cmn x10, #0x1+//        eor x16, x5, x10+//        adcs x16, x20, x16+//        eor x21, x21, x10+//        adcs x21, x17, x21+//        adcs x17, x9, x10+//        adcs x5, x6, x10+//        adc x10, x13, x10+//        lsl x13, x8, #9+//        extr x20, x11, x8, #55+//        extr x8, x3, x11, #55+//        extr x9, x16, x3, #55+//        lsr x16, x16, #55+//        stp x21, x17, [sp]                       // @slothy:writes=stack0+//        stp x5, x10, [sp, #16]                   // @slothy:writes=stack16+//        stp x13, x20, [sp, #32]                  // @slothy:writes=stack32+//        stp x8, x9, [sp, #48]                    // @slothy:writes=stack48+//        str x16, [sp, #64]                       // @slothy:writes=stack64+//        ldp x21, x10, [x1, #32]+//        ldp x17, x13, [x1, #48]+//        ldp x16, x5, [x2, #32]+//        ldr q18, [x1, #32]+//        ldr q28, [x2, #32]+//        ldp x20, x8, [x2, #48]+//        movi v16.2D, #0x00000000ffffffff+//        uzp2 v7.4S, v28.4S, v28.4S+//        xtn v4.2S, v18.2D+//        xtn v1.2S, v28.2D+//        rev64 v28.4S, v28.4S+//        umull v27.2D, v4.2S, v1.2S+//        umull v29.2D, v4.2S, v7.2S+//        uzp2 v21.4S, v18.4S, v18.4S+//        mul v28.4S, v28.4S, v18.4S+//        usra v29.2D, v27.2D, #32+//        umull v18.2D, v21.2S, v7.2S+//        uaddlp v28.2D, v28.4S+//        and v16.16B, v29.16B, v16.16B+//        umlal v16.2D, v21.2S, v1.2S+//        shl v28.2D, v28.2D, #32+//        usra v18.2D, v29.2D, #32+//        umlal v28.2D, v4.2S, v1.2S+//        usra v18.2D, v16.2D, #32+//        mov x9, v28.d[0]+//        mov x6, v28.d[1]+//        mul x19, x17, x20+//        mul x14, x13, x8+//        mov x11, v18.d[0]+//        adds x6, x6, x11+//        mov x11, v18.d[1]+//        adcs x19, x19, x11+//        umulh x11, x17, x20+//        adcs x14, x14, x11+//        umulh x11, x13, x8+//        adc x11, x11, xzr+//        adds x3, x6, x9+//        adcs x6, x19, x6+//        adcs x19, x14, x19+//        adcs x14, x11, x14+//        adc x11, xzr, x11+//        adds x24, x6, x9+//        adcs x4, x19, x3+//        adcs x6, x14, x6+//        adcs x19, x11, x19+//        adcs x14, xzr, x14+//        adc x11, xzr, x11+//        subs x7, x17, x13+//        cneg x7, x7, cc+//        csetm x23, cc+//        subs x22, x8, x20+//        cneg x22, x22, cc+//        mul x12, x7, x22+//        umulh x7, x7, x22+//        cinv x23, x23, cc+//        cmn x23, #0x1+//        eor x22, x12, x23+//        adcs x19, x19, x22+//        eor x7, x7, x23+//        adcs x14, x14, x7+//        adc x11, x11, x23+//        subs x7, x21, x10+//        cneg x7, x7, cc+//        csetm x23, cc+//        subs x22, x5, x16+//        cneg x22, x22, cc+//        mul x12, x7, x22+//        umulh x7, x7, x22+//        cinv x23, x23, cc+//        cmn x23, #0x1+//        eor x22, x12, x23+//        adcs x3, x3, x22+//        eor x7, x7, x23+//        adcs x24, x24, x7+//        adcs x4, x4, x23+//        adcs x6, x6, x23+//        adcs x19, x19, x23+//        adcs x14, x14, x23+//        adc x11, x11, x23+//        subs x7, x10, x13+//        cneg x7, x7, cc+//        csetm x23, cc+//        subs x22, x8, x5+//        cneg x22, x22, cc+//        mul x12, x7, x22+//        umulh x7, x7, x22+//        cinv x23, x23, cc+//        cmn x23, #0x1+//        eor x22, x12, x23+//        adcs x6, x6, x22+//        eor x7, x7, x23+//        adcs x19, x19, x7+//        adcs x14, x14, x23+//        adc x11, x11, x23+//        subs x7, x21, x17+//        cneg x7, x7, cc+//        csetm x23, cc+//        subs x22, x20, x16+//        cneg x22, x22, cc+//        mul x12, x7, x22+//        umulh x7, x7, x22+//        cinv x23, x23, cc+//        cmn x23, #0x1+//        eor x22, x12, x23+//        adcs x24, x24, x22+//        eor x7, x7, x23+//        adcs x4, x4, x7+//        adcs x6, x6, x23+//        adcs x19, x19, x23+//        adcs x14, x14, x23+//        adc x11, x11, x23+//        subs x7, x21, x13+//        cneg x7, x7, cc+//        csetm x23, cc+//        subs x22, x8, x16+//        cneg x22, x22, cc+//        mul x12, x7, x22+//        umulh x7, x7, x22+//        cinv x23, x23, cc+//        cmn x23, #0x1+//        eor x22, x12, x23+//        adcs x4, x4, x22+//        eor x7, x7, x23+//        adcs x6, x6, x7+//        adcs x19, x19, x23+//        adcs x14, x14, x23+//        adc x11, x11, x23+//        subs x7, x10, x17+//        cneg x7, x7, cc+//        csetm x23, cc+//        subs x22, x20, x5+//        cneg x22, x22, cc+//        mul x12, x7, x22+//        umulh x7, x7, x22+//        cinv x23, x23, cc+//        cmn x23, #0x1+//        eor x22, x12, x23+//        adcs x4, x4, x22+//        eor x7, x7, x23+//        adcs x6, x6, x7+//        adcs x19, x19, x23+//        adcs x14, x14, x23+//        adc x11, x11, x23+//        ldp x7, x23, [sp]                        // @slothy:reads=stack0+//        adds x9, x9, x7+//        adcs x3, x3, x23+//        stp x9, x3, [sp]                         // @slothy:writes=stack0+//        ldp x9, x3, [sp, #16]                    // @slothy:reads=stack16+//        adcs x9, x24, x9+//        adcs x3, x4, x3+//        stp x9, x3, [sp, #16]                    // @slothy:writes=stack16+//        ldp x9, x3, [sp, #32]                    // @slothy:reads=stack32+//        adcs x9, x6, x9+//        adcs x6, x19, x3+//        stp x9, x6, [sp, #32]                    // @slothy:writes=stack32+//        ldp x9, x6, [sp, #48]                    // @slothy:reads=stack48+//        adcs x9, x14, x9+//        adcs x6, x11, x6+//        stp x9, x6, [sp, #48]                    // @slothy:writes=stack48+//        ldr x9, [sp, #64]                        // @slothy:reads=stack64+//        adc x9, x9, xzr+//        str x9, [sp, #64]                        // @slothy:writes=stack64+//        ldp x9, x6, [x1]+//        subs x21, x21, x9+//        sbcs x10, x10, x6+//        ldp x9, x6, [x1, #16]+//        sbcs x17, x17, x9+//        sbcs x13, x13, x6+//        csetm x9, cc+//        ldp x6, x19, [x2]+//        subs x16, x6, x16+//        sbcs x5, x19, x5+//        ldp x6, x19, [x2, #16]+//        sbcs x20, x6, x20+//        sbcs x8, x19, x8+//        csetm x6, cc+//        eor x21, x21, x9+//        subs x21, x21, x9+//        eor x10, x10, x9+//        sbcs x10, x10, x9+//        eor x17, x17, x9+//        sbcs x17, x17, x9+//        eor x13, x13, x9+//        sbc x13, x13, x9+//        eor x16, x16, x6+//        subs x16, x16, x6+//        eor x5, x5, x6+//        sbcs x5, x5, x6+//        eor x20, x20, x6+//        sbcs x20, x20, x6+//        eor x8, x8, x6+//        sbc x8, x8, x6+//        eor x9, x6, x9+//        mul x6, x21, x16+//        mul x19, x10, x5+//        mul x14, x17, x20+//        mul x11, x13, x8+//        umulh x3, x21, x16+//        adds x19, x19, x3+//        umulh x3, x10, x5+//        adcs x14, x14, x3+//        umulh x3, x17, x20+//        adcs x11, x11, x3+//        umulh x3, x13, x8+//        adc x3, x3, xzr+//        adds x24, x19, x6+//        adcs x19, x14, x19+//        adcs x14, x11, x14+//        adcs x11, x3, x11+//        adc x3, xzr, x3+//        adds x4, x19, x6+//        adcs x7, x14, x24+//        adcs x19, x11, x19+//        adcs x14, x3, x14+//        adcs x11, xzr, x11+//        adc x3, xzr, x3+//        subs x23, x17, x13+//        cneg x23, x23, cc+//        csetm x22, cc+//        subs x12, x8, x20+//        cneg x12, x12, cc+//        mul x15, x23, x12+//        umulh x23, x23, x12+//        cinv x22, x22, cc+//        cmn x22, #0x1+//        eor x12, x15, x22+//        adcs x14, x14, x12+//        eor x23, x23, x22+//        adcs x11, x11, x23+//        adc x3, x3, x22+//        subs x23, x21, x10+//        cneg x23, x23, cc+//        csetm x22, cc+//        subs x12, x5, x16+//        cneg x12, x12, cc+//        mul x15, x23, x12+//        umulh x23, x23, x12+//        cinv x22, x22, cc+//        cmn x22, #0x1+//        eor x12, x15, x22+//        adcs x24, x24, x12+//        eor x23, x23, x22+//        adcs x4, x4, x23+//        adcs x7, x7, x22+//        adcs x19, x19, x22+//        adcs x14, x14, x22+//        adcs x11, x11, x22+//        adc x3, x3, x22+//        subs x23, x10, x13+//        cneg x23, x23, cc+//        csetm x22, cc+//        subs x12, x8, x5+//        cneg x12, x12, cc+//        mul x15, x23, x12+//        umulh x23, x23, x12+//        cinv x22, x22, cc+//        cmn x22, #0x1+//        eor x12, x15, x22+//        adcs x19, x19, x12+//        eor x23, x23, x22+//        adcs x14, x14, x23+//        adcs x11, x11, x22+//        adc x3, x3, x22+//        subs x23, x21, x17+//        cneg x23, x23, cc+//        csetm x22, cc+//        subs x12, x20, x16+//        cneg x12, x12, cc+//        mul x15, x23, x12+//        umulh x23, x23, x12+//        cinv x22, x22, cc+//        cmn x22, #0x1+//        eor x12, x15, x22+//        adcs x4, x4, x12+//        eor x23, x23, x22+//        adcs x7, x7, x23+//        adcs x19, x19, x22+//        adcs x14, x14, x22+//        adcs x11, x11, x22+//        adc x3, x3, x22+//        subs x21, x21, x13+//        cneg x21, x21, cc+//        csetm x13, cc+//        subs x16, x8, x16+//        cneg x16, x16, cc+//        mul x8, x21, x16+//        umulh x21, x21, x16+//        cinv x13, x13, cc+//        cmn x13, #0x1+//        eor x16, x8, x13+//        adcs x16, x7, x16+//        eor x21, x21, x13+//        adcs x21, x19, x21+//        adcs x8, x14, x13+//        adcs x19, x11, x13+//        adc x13, x3, x13+//        subs x10, x10, x17+//        cneg x10, x10, cc+//        csetm x17, cc+//        subs x5, x20, x5+//        cneg x5, x5, cc+//        mul x20, x10, x5+//        umulh x10, x10, x5+//        cinv x17, x17, cc+//        cmn x17, #0x1+//        eor x5, x20, x17+//        adcs x16, x16, x5+//        eor x10, x10, x17+//        adcs x21, x21, x10+//        adcs x10, x8, x17+//        adcs x5, x19, x17+//        adc x17, x13, x17+//        ldp x13, x20, [sp]                       // @slothy:reads=stack0+//        ldp x8, x19, [sp, #16]                   // @slothy:reads=stack16+//        eor x6, x6, x9+//        adds x6, x6, x13+//        eor x14, x24, x9+//        adcs x14, x14, x20+//        eor x11, x4, x9+//        adcs x11, x11, x8+//        eor x16, x16, x9+//        adcs x16, x16, x19+//        eor x21, x21, x9+//        ldp x3, x24, [sp, #32]                   // @slothy:reads=stack32+//        ldp x4, x7, [sp, #48]                    // @slothy:reads=stack48+//        ldr x23, [sp, #64]                       // @slothy:reads=stack64+//        adcs x21, x21, x3+//        eor x10, x10, x9+//        adcs x10, x10, x24+//        eor x5, x5, x9+//        adcs x5, x5, x4+//        eor x17, x17, x9+//        adcs x17, x17, x7+//        adc x22, x23, xzr+//        adds x21, x21, x13+//        adcs x10, x10, x20+//        adcs x13, x5, x8+//        adcs x17, x17, x19+//        and x5, x9, #0x1ff+//        lsl x20, x6, #9+//        orr x5, x20, x5+//        adcs x5, x3, x5+//        extr x20, x14, x6, #55+//        adcs x20, x24, x20+//        extr x8, x11, x14, #55+//        adcs x8, x4, x8+//        extr x9, x16, x11, #55+//        adcs x9, x7, x9+//        lsr x16, x16, #55+//        adc x16, x16, x23+//        ldr x6, [x2, #64]+//        ldp x19, x14, [x1]+//        and x11, x19, #0xfffffffffffff+//        mul x11, x6, x11+//        ldr x3, [x1, #64]+//        ldp x24, x4, [x2]+//        and x7, x24, #0xfffffffffffff+//        mul x7, x3, x7+//        add x11, x11, x7+//        extr x19, x14, x19, #52+//        and x19, x19, #0xfffffffffffff+//        mul x19, x6, x19+//        extr x24, x4, x24, #52+//        and x24, x24, #0xfffffffffffff+//        mul x24, x3, x24+//        add x19, x19, x24+//        lsr x24, x11, #52+//        add x19, x19, x24+//        lsl x11, x11, #12+//        extr x11, x19, x11, #12+//        adds x21, x21, x11+//        ldp x11, x24, [x1, #16]+//        ldp x7, x23, [x2, #16]+//        extr x14, x11, x14, #40+//        and x14, x14, #0xfffffffffffff+//        mul x14, x6, x14+//        extr x4, x7, x4, #40+//        and x4, x4, #0xfffffffffffff+//        mul x4, x3, x4+//        add x14, x14, x4+//        lsr x4, x19, #52+//        add x14, x14, x4+//        lsl x19, x19, #12+//        extr x19, x14, x19, #24+//        adcs x10, x10, x19+//        extr x19, x24, x11, #28+//        and x19, x19, #0xfffffffffffff+//        mul x19, x6, x19+//        extr x11, x23, x7, #28+//        and x11, x11, #0xfffffffffffff+//        mul x11, x3, x11+//        add x19, x19, x11+//        lsr x11, x14, #52+//        add x19, x19, x11+//        lsl x14, x14, #12+//        extr x14, x19, x14, #36+//        adcs x13, x13, x14+//        and x14, x10, x13+//        ldp x11, x4, [x1, #32]+//        ldp x7, x12, [x2, #32]+//        extr x24, x11, x24, #16+//        and x24, x24, #0xfffffffffffff+//        mul x24, x6, x24+//        extr x23, x7, x23, #16+//        and x23, x23, #0xfffffffffffff+//        mul x23, x3, x23+//        add x24, x24, x23+//        lsl x23, x22, #48+//        add x24, x24, x23+//        lsr x23, x19, #52+//        add x24, x24, x23+//        lsl x19, x19, #12+//        extr x19, x24, x19, #48+//        adcs x17, x17, x19+//        and x19, x14, x17+//        lsr x14, x11, #4+//        and x14, x14, #0xfffffffffffff+//        mul x14, x6, x14+//        lsr x23, x7, #4+//        and x23, x23, #0xfffffffffffff+//        mul x23, x3, x23+//        add x14, x14, x23+//        lsr x23, x24, #52+//        add x14, x14, x23+//        lsl x24, x24, #12+//        extr x24, x14, x24, #60+//        extr x11, x4, x11, #56+//        and x11, x11, #0xfffffffffffff+//        mul x11, x6, x11+//        extr x7, x12, x7, #56+//        and x7, x7, #0xfffffffffffff+//        mul x7, x3, x7+//        add x11, x11, x7+//        lsr x14, x14, #52+//        add x14, x11, x14+//        lsl x11, x24, #8+//        extr x11, x14, x11, #8+//        adcs x5, x5, x11+//        and x19, x19, x5+//        ldp x11, x24, [x1, #48]+//        ldp x2, x7, [x2, #48]+//        extr x4, x11, x4, #44+//        and x4, x4, #0xfffffffffffff+//        mul x4, x6, x4+//        extr x23, x2, x12, #44+//        and x23, x23, #0xfffffffffffff+//        mul x23, x3, x23+//        add x4, x4, x23+//        lsr x23, x14, #52+//        add x4, x4, x23+//        lsl x14, x14, #12+//        extr x14, x4, x14, #20+//        adcs x20, x20, x14+//        and x19, x19, x20+//        extr x14, x24, x11, #32+//        and x14, x14, #0xfffffffffffff+//        mul x14, x6, x14+//        extr x2, x7, x2, #32+//        and x2, x2, #0xfffffffffffff+//        mul x2, x3, x2+//        add x2, x14, x2+//        lsr x14, x4, #52+//        add x2, x2, x14+//        lsl x14, x4, #12+//        extr x14, x2, x14, #32+//        adcs x8, x8, x14+//        and x19, x19, x8+//        lsr x14, x24, #20+//        mul x14, x6, x14+//        lsr x11, x7, #20+//        mul x11, x3, x11+//        add x14, x14, x11+//        lsr x11, x2, #52+//        add x14, x14, x11+//        lsl x2, x2, #12+//        extr x2, x14, x2, #44+//        adcs x9, x9, x2+//        and x2, x19, x9+//        mul x6, x6, x3+//        lsr x19, x14, #44+//        add x6, x6, x19+//        adc x16, x16, x6+//        lsr x6, x16, #9+//        orr x16, x16, #0xfffffffffffffe00+//        cmp xzr, xzr+//        adcs xzr, x21, x6+//        adcs xzr, x2, xzr+//        adcs xzr, x16, xzr+//        adcs x21, x21, x6+//        adcs x10, x10, xzr+//        adcs x13, x13, xzr+//        adcs x17, x17, xzr+//        adcs x5, x5, xzr+//        adcs x20, x20, xzr+//        adcs x8, x8, xzr+//        adcs x9, x9, xzr+//        adc x16, x16, xzr+//        and x2, x21, #0x1ff+//        extr x21, x10, x21, #9+//        extr x10, x13, x10, #9+//        stp x21, x10, [x0]                       // @slothy:writes=buffer0+//        extr x21, x17, x13, #9+//        extr x10, x5, x17, #9+//        stp x21, x10, [x0, #16]                  // @slothy:writes=buffer16+//        extr x21, x20, x5, #9+//        extr x10, x8, x20, #9+//        stp x21, x10, [x0, #32]                  // @slothy:writes=buffer32+//        extr x21, x9, x8, #9+//        extr x10, x16, x9, #9+//        stp x21, x10, [x0, #48]                  // @slothy:writes=buffer48+//        str x2, [x0, #64]                        // @slothy:writes=buffer64+//        add     sp, sp, #80+//        ldp     x25, x26, [sp], #16+//        ldp     x23, x24, [sp], #16+//        ldp     x21, x22, [sp], #16+//        ldp     x19, x20, [sp], #16+//        ret+//+// The bash script used for step 2 is as follows:+//+//        # Store the assembly instructions except the last 'ret',+//        # callee-register store/loads and add/sub sp #80 as, say, 'input.S'.+//        export OUTPUTS="[hint_buffer0,hint_buffer16,hint_buffer32,hint_buffer48,hint_buffer64]"+//        export RESERVED_REGS="[x18,x27,x28,x29,x30,sp,q8,q9,q10,q11,q12,q13,q14,q15,v8,v9,v10,v11,v12,v13,v14,v15]"+//        <s2n-bignum>/tools/external/slothy.sh input.S my_out_dir+//        # my_out_dir/3.opt.s is the optimized assembly. Its output may differ+//        # from this file since the sequence is non-deterministically chosen.+//        # Please add 'ret' at the end of the output assembly.++        S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_mul_p521)+        S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_mul_p521)+        S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_mul_p521)+        .text+        .balign 4++S2N_BN_SYMBOL(bignum_mul_p521):+        CFI_START++// Save registers and make space for the temporary buffer++        CFI_PUSH2(x19,x20)+        CFI_PUSH2(x21,x22)+        CFI_PUSH2(x23,x24)+        CFI_PUSH2(x25,x26)+        CFI_DEC_SP(80)++        ldr q6, [x2]+        ldp x10, x17, [x1, #16]+        ldr q4, [x1]+        ldr q16, [x2, #32]+        ldp x5, x20, [x2, #16]+        ldr q2, [x1, #32]+        movi v31.2D, #0x00000000ffffffff+        uzp2 v17.4S, v6.4S, v6.4S+        rev64 v7.4S, v6.4S+        ldp x15, x21, [x1]+        xtn v25.2S, v6.2D+        xtn v22.2S, v4.2D+        subs x14, x10, x17+        mul v7.4S, v7.4S, v4.4S+        csetm x8, cc+        rev64 v3.4S, v16.4S+        xtn v1.2S, v16.2D+        ldp x13, x16, [x2]+        mul x26, x10, x5+        uzp2 v16.4S, v16.4S, v16.4S+        uaddlp v26.2D, v7.4S+        cneg x4, x14, cc+        subs x24, x15, x21+        xtn v5.2S, v2.2D+        mul v28.4S, v3.4S, v2.4S+        shl v26.2D, v26.2D, #32+        mul x22, x17, x20+        umull v20.2D, v22.2S, v25.2S+        uzp2 v6.4S, v4.4S, v4.4S+        umull v18.2D, v22.2S, v17.2S+        uzp2 v4.4S, v2.4S, v2.4S+        cneg x14, x24, cc+        csetm x7, cc+        umulh x11, x17, x20+        usra v18.2D, v20.2D, #32+        uaddlp v7.2D, v28.4S+        subs x19, x16, x13+        umlal v26.2D, v22.2S, v25.2S+        cneg x19, x19, cc+        shl v28.2D, v7.2D, #32+        umull v7.2D, v5.2S, v1.2S+        umull v30.2D, v5.2S, v16.2S+        cinv x6, x7, cc+        mul x25, x14, x19+        umlal v28.2D, v5.2S, v1.2S+        umull v21.2D, v6.2S, v17.2S+        umulh x14, x14, x19+        usra v30.2D, v7.2D, #32+        subs x9, x20, x5+        and v29.16B, v18.16B, v31.16B+        cinv x23, x8, cc+        mov x8, v26.d[1]+        cneg x12, x9, cc+        usra v21.2D, v18.2D, #32+        umlal v29.2D, v6.2S, v25.2S+        mul x24, x4, x12+        umull v18.2D, v4.2S, v16.2S+        movi v25.2D, #0x00000000ffffffff+        eor x9, x14, x6+        and v7.16B, v30.16B, v25.16B+        usra v21.2D, v29.2D, #32+        umulh x7, x10, x5+        usra v18.2D, v30.2D, #32+        umlal v7.2D, v4.2S, v1.2S+        mov x19, v21.d[0]+        umulh x3, x4, x12+        mov x14, v21.d[1]+        usra v18.2D, v7.2D, #32+        adds x4, x8, x19+        mov x8, v26.d[0]+        adcs x19, x26, x14+        adcs x14, x22, x7+        adc x12, x11, xzr+        adds x11, x4, x8+        adcs x26, x19, x4+        adcs x22, x14, x19+        eor x4, x24, x23+        adcs x14, x12, x14+        eor x7, x25, x6+        adc x25, xzr, x12+        eor x19, x3, x23+        adds x3, x26, x8+        adcs x24, x22, x11+        adcs x12, x14, x26+        adcs x22, x25, x22+        adcs x26, xzr, x14+        adc x14, xzr, x25+        cmn x23, #0x1+        adcs x22, x22, x4+        adcs x19, x26, x19+        adc x25, x14, x23+        subs x14, x21, x17+        cneg x23, x14, cc+        csetm x26, cc+        subs x4, x20, x16+        cneg x14, x4, cc+        cinv x4, x26, cc+        cmn x6, #0x1+        adcs x11, x11, x7+        mul x7, x23, x14+        adcs x9, x3, x9+        adcs x26, x24, x6+        umulh x3, x23, x14+        adcs x14, x12, x6+        adcs x22, x22, x6+        adcs x12, x19, x6+        extr x24, x11, x8, #55+        adc x6, x25, x6+        subs x19, x15, x17+        csetm x17, cc+        cneg x23, x19, cc+        subs x19, x20, x13+        lsl x25, x8, #9+        eor x8, x7, x4+        cneg x20, x19, cc+        umulh x7, x23, x20+        cinv x19, x17, cc+        subs x17, x15, x10+        csetm x15, cc+        stp x25, x24, [sp, #32]+        cneg x24, x17, cc+        mul x20, x23, x20+        subs x25, x5, x13+        cneg x13, x25, cc+        cinv x15, x15, cc+        mul x25, x24, x13+        subs x21, x21, x10+        csetm x23, cc+        cneg x17, x21, cc+        subs x21, x5, x16+        umulh x13, x24, x13+        cinv x10, x23, cc+        cneg x23, x21, cc+        cmn x4, #0x1+        adcs x14, x14, x8+        eor x21, x3, x4+        adcs x21, x22, x21+        eor x5, x20, x19+        adcs x24, x12, x4+        mul x12, x17, x23+        eor x8, x25, x15+        adc x25, x6, x4+        cmn x15, #0x1+        adcs x6, x9, x8+        ldp x20, x8, [x2, #48]+        eor x9, x13, x15+        adcs x4, x26, x9+        umulh x26, x17, x23+        ldp x17, x13, [x1, #48]+        adcs x9, x14, x15+        adcs x16, x21, x15+        adcs x14, x24, x15+        eor x21, x7, x19+        mul x23, x17, x20+        adc x24, x25, x15+        cmn x19, #0x1+        adcs x7, x4, x5+        adcs x9, x9, x21+        umulh x3, x13, x8+        adcs x16, x16, x19+        adcs x22, x14, x19+        eor x5, x12, x10+        adc x12, x24, x19+        cmn x10, #0x1+        adcs x19, x7, x5+        eor x14, x26, x10+        mov x7, v28.d[1]+        adcs x24, x9, x14+        extr x4, x19, x6, #55+        umulh x15, x17, x20+        mov x14, v18.d[1]+        lsr x9, x19, #55+        adcs x5, x16, x10+        mov x16, v18.d[0]+        adcs x19, x22, x10+        str x9, [sp, #64]+        extr x25, x6, x11, #55+        adc x21, x12, x10+        subs x26, x17, x13+        stp x25, x4, [sp, #48]+        stp x19, x21, [sp, #16]+        csetm x6, cc+        cneg x4, x26, cc+        mul x19, x13, x8+        subs x11, x8, x20+        stp x24, x5, [sp]+        ldp x21, x10, [x1, #32]+        cinv x12, x6, cc+        cneg x6, x11, cc+        mov x9, v28.d[0]+        umulh x25, x4, x6+        adds x22, x7, x16+        ldp x16, x5, [x2, #32]+        adcs x14, x23, x14+        adcs x11, x19, x15+        adc x24, x3, xzr+        adds x3, x22, x9+        adcs x15, x14, x22+        mul x22, x4, x6+        adcs x6, x11, x14+        adcs x4, x24, x11+        eor x14, x25, x12+        adc x26, xzr, x24+        subs x7, x21, x10+        csetm x23, cc+        cneg x19, x7, cc+        subs x24, x5, x16+        cneg x11, x24, cc+        cinv x7, x23, cc+        adds x25, x15, x9+        eor x23, x22, x12+        adcs x22, x6, x3+        mul x24, x19, x11+        adcs x15, x4, x15+        adcs x6, x26, x6+        umulh x19, x19, x11+        adcs x11, xzr, x4+        adc x26, xzr, x26+        cmn x12, #0x1+        adcs x4, x6, x23+        eor x6, x24, x7+        adcs x14, x11, x14+        adc x26, x26, x12+        subs x11, x10, x13+        cneg x12, x11, cc+        csetm x11, cc+        eor x19, x19, x7+        subs x24, x8, x5+        cinv x11, x11, cc+        cneg x24, x24, cc+        cmn x7, #0x1+        adcs x3, x3, x6+        mul x23, x12, x24+        adcs x25, x25, x19+        adcs x6, x22, x7+        umulh x19, x12, x24+        adcs x22, x15, x7+        adcs x12, x4, x7+        eor x24, x23, x11+        adcs x4, x14, x7+        adc x26, x26, x7+        eor x19, x19, x11+        subs x14, x21, x17+        cneg x7, x14, cc+        csetm x14, cc+        subs x23, x20, x16+        cinv x14, x14, cc+        cneg x23, x23, cc+        cmn x11, #0x1+        adcs x22, x22, x24+        mul x24, x7, x23+        adcs x15, x12, x19+        adcs x4, x4, x11+        adc x19, x26, x11+        umulh x26, x7, x23+        subs x7, x21, x13+        eor x11, x24, x14+        cneg x23, x7, cc+        csetm x12, cc+        subs x7, x8, x16+        cneg x7, x7, cc+        cinv x12, x12, cc+        cmn x14, #0x1+        eor x26, x26, x14+        adcs x11, x25, x11+        mul x25, x23, x7+        adcs x26, x6, x26+        adcs x6, x22, x14+        adcs x24, x15, x14+        umulh x23, x23, x7+        adcs x4, x4, x14+        adc x22, x19, x14+        eor x14, x25, x12+        eor x7, x23, x12+        cmn x12, #0x1+        adcs x14, x26, x14+        ldp x19, x25, [x2]+        ldp x15, x23, [x2, #16]+        adcs x26, x6, x7+        adcs x24, x24, x12+        adcs x7, x4, x12+        adc x4, x22, x12+        subs x19, x19, x16+        ldp x16, x22, [x1]+        sbcs x6, x25, x5+        ldp x12, x25, [x1, #16]+        sbcs x15, x15, x20+        sbcs x8, x23, x8+        csetm x23, cc+        subs x21, x21, x16+        eor x16, x19, x23+        sbcs x19, x10, x22+        eor x22, x6, x23+        eor x8, x8, x23+        sbcs x6, x17, x12+        sbcs x13, x13, x25+        csetm x12, cc+        subs x10, x10, x17+        cneg x17, x10, cc+        csetm x25, cc+        subs x5, x20, x5+        eor x10, x19, x12+        cneg x19, x5, cc+        eor x20, x15, x23+        eor x21, x21, x12+        cinv x15, x25, cc+        mul x25, x17, x19+        subs x16, x16, x23+        sbcs x5, x22, x23+        eor x6, x6, x12+        sbcs x20, x20, x23+        eor x22, x13, x12+        sbc x8, x8, x23+        subs x21, x21, x12+        umulh x19, x17, x19+        sbcs x10, x10, x12+        sbcs x17, x6, x12+        eor x6, x19, x15+        eor x19, x25, x15+        umulh x25, x17, x20+        sbc x13, x22, x12+        cmn x15, #0x1+        adcs x22, x14, x19+        adcs x19, x26, x6+        ldp x6, x26, [sp]+        adcs x14, x24, x15+        umulh x24, x21, x16+        adcs x7, x7, x15+        adc x15, x4, x15+        adds x4, x9, x6+        eor x9, x23, x12+        adcs x12, x3, x26+        stp x4, x12, [sp]+        ldp x4, x26, [sp, #16]+        umulh x12, x10, x5+        ldp x6, x23, [sp, #32]+        adcs x3, x11, x4+        mul x4, x13, x8+        adcs x26, x22, x26+        ldp x22, x11, [sp, #48]+        adcs x6, x19, x6+        stp x3, x26, [sp, #16]+        mul x26, x10, x5+        adcs x14, x14, x23+        stp x6, x14, [sp, #32]+        ldr x6, [sp, #64]+        adcs x22, x7, x22+        adcs x14, x15, x11+        mul x11, x17, x20+        adc x19, x6, xzr+        stp x22, x14, [sp, #48]+        adds x14, x26, x24+        str x19, [sp, #64]+        umulh x19, x13, x8+        adcs x7, x11, x12+        adcs x22, x4, x25+        mul x6, x21, x16+        adc x19, x19, xzr+        subs x11, x17, x13+        cneg x12, x11, cc+        csetm x11, cc+        subs x24, x8, x20+        cinv x11, x11, cc+        cneg x24, x24, cc+        adds x4, x14, x6+        adcs x14, x7, x14+        mul x3, x12, x24+        adcs x7, x22, x7+        adcs x22, x19, x22+        umulh x12, x12, x24+        adc x24, xzr, x19+        adds x19, x14, x6+        eor x3, x3, x11+        adcs x26, x7, x4+        adcs x14, x22, x14+        adcs x25, x24, x7+        adcs x23, xzr, x22+        eor x7, x12, x11+        adc x12, xzr, x24+        subs x22, x21, x10+        cneg x24, x22, cc+        csetm x22, cc+        subs x15, x5, x16+        cinv x22, x22, cc+        cneg x15, x15, cc+        cmn x11, #0x1+        adcs x3, x25, x3+        mul x25, x24, x15+        adcs x23, x23, x7+        adc x11, x12, x11+        subs x7, x10, x13+        umulh x15, x24, x15+        cneg x12, x7, cc+        csetm x7, cc+        eor x24, x25, x22+        eor x25, x15, x22+        cmn x22, #0x1+        adcs x24, x4, x24+        adcs x19, x19, x25+        adcs x15, x26, x22+        adcs x4, x14, x22+        adcs x26, x3, x22+        adcs x25, x23, x22+        adc x23, x11, x22+        subs x14, x21, x17+        cneg x3, x14, cc+        csetm x11, cc+        subs x14, x8, x5+        cneg x14, x14, cc+        cinv x7, x7, cc+        subs x13, x21, x13+        cneg x21, x13, cc+        csetm x13, cc+        mul x22, x12, x14+        subs x8, x8, x16+        cinv x13, x13, cc+        umulh x14, x12, x14+        cneg x12, x8, cc+        subs x8, x20, x16+        cneg x8, x8, cc+        cinv x16, x11, cc+        eor x22, x22, x7+        cmn x7, #0x1+        eor x14, x14, x7+        adcs x4, x4, x22+        mul x11, x3, x8+        adcs x22, x26, x14+        adcs x14, x25, x7+        eor x25, x24, x9+        adc x26, x23, x7+        umulh x7, x3, x8+        subs x17, x10, x17+        cneg x24, x17, cc+        eor x3, x11, x16+        csetm x11, cc+        subs x20, x20, x5+        cneg x5, x20, cc+        cinv x11, x11, cc+        cmn x16, #0x1+        mul x17, x21, x12+        eor x8, x7, x16+        adcs x10, x19, x3+        and x19, x9, #0x1ff+        adcs x20, x15, x8+        umulh x15, x21, x12+        eor x12, x10, x9+        eor x8, x6, x9+        adcs x6, x4, x16+        adcs x4, x22, x16+        adcs x21, x14, x16+        adc x7, x26, x16+        mul x10, x24, x5+        cmn x13, #0x1+        ldp x3, x14, [x1]+        eor x17, x17, x13+        umulh x5, x24, x5+        adcs x20, x20, x17+        eor x17, x15, x13+        adcs x16, x6, x17+        eor x22, x10, x11+        adcs x23, x4, x13+        extr x10, x14, x3, #52+        and x26, x3, #0xfffffffffffff+        adcs x24, x21, x13+        and x15, x10, #0xfffffffffffff+        adc x6, x7, x13+        cmn x11, #0x1+        adcs x17, x20, x22+        eor x4, x5, x11+        ldp x21, x10, [sp]+        adcs x7, x16, x4+        eor x16, x17, x9+        eor x13, x7, x9+        ldp x3, x17, [sp, #16]+        adcs x7, x23, x11+        eor x23, x7, x9+        ldp x5, x22, [sp, #32]+        adcs x7, x24, x11+        adc x24, x6, x11+        ldr x6, [x2, #64]+        adds x20, x8, x21+        lsl x11, x20, #9+        eor x4, x7, x9+        orr x7, x11, x19+        eor x8, x24, x9+        adcs x11, x25, x10+        mul x26, x6, x26+        ldp x19, x24, [sp, #48]+        adcs x12, x12, x3+        adcs x16, x16, x17+        adcs x9, x13, x5+        ldr x25, [sp, #64]+        extr x20, x11, x20, #55+        adcs x13, x23, x22+        adcs x4, x4, x19+        extr x23, x12, x11, #55+        adcs x8, x8, x24+        adc x11, x25, xzr+        adds x21, x9, x21+        extr x9, x16, x12, #55+        lsr x12, x16, #55+        adcs x10, x13, x10+        mul x15, x6, x15+        adcs x13, x4, x3+        ldp x16, x4, [x2]+        ldr x3, [x1, #64]+        adcs x17, x8, x17+        adcs x5, x5, x7+        adcs x20, x22, x20+        adcs x8, x19, x23+        and x22, x16, #0xfffffffffffff+        ldp x19, x7, [x1, #16]+        adcs x9, x24, x9+        extr x24, x4, x16, #52+        adc x16, x12, x25+        mul x22, x3, x22+        and x25, x24, #0xfffffffffffff+        extr x14, x19, x14, #40+        and x12, x14, #0xfffffffffffff+        extr x23, x7, x19, #28+        ldp x19, x24, [x2, #16]+        mul x14, x3, x25+        and x23, x23, #0xfffffffffffff+        add x22, x26, x22+        lsl x11, x11, #48+        lsr x26, x22, #52+        lsl x25, x22, #12+        mul x22, x6, x12+        extr x12, x19, x4, #40+        add x4, x15, x14+        mul x15, x6, x23+        add x4, x4, x26+        extr x23, x24, x19, #28+        ldp x14, x19, [x1, #32]+        and x26, x12, #0xfffffffffffff+        extr x12, x4, x25, #12+        and x25, x23, #0xfffffffffffff+        adds x21, x21, x12+        mul x12, x3, x26+        extr x23, x14, x7, #16+        and x23, x23, #0xfffffffffffff+        mul x7, x3, x25+        ldp x25, x26, [x2, #32]+        add x12, x22, x12+        extr x22, x19, x14, #56+        mul x23, x6, x23+        lsr x14, x14, #4+        extr x24, x25, x24, #16+        add x7, x15, x7+        and x15, x24, #0xfffffffffffff+        and x22, x22, #0xfffffffffffff+        lsr x24, x4, #52+        mul x15, x3, x15+        and x14, x14, #0xfffffffffffff+        add x12, x12, x24+        lsl x24, x4, #12+        lsr x4, x12, #52+        extr x24, x12, x24, #24+        adcs x10, x10, x24+        lsl x24, x12, #12+        add x12, x7, x4+        mul x22, x6, x22+        add x4, x23, x15+        extr x7, x12, x24, #36+        adcs x13, x13, x7+        lsl x15, x12, #12+        add x7, x4, x11+        lsr x24, x12, #52+        ldp x23, x11, [x2, #48]+        add x4, x7, x24+        mul x12, x6, x14+        extr x7, x26, x25, #56+        extr x14, x4, x15, #48+        and x2, x7, #0xfffffffffffff+        extr x24, x11, x23, #32+        ldp x15, x7, [x1, #48]+        and x1, x24, #0xfffffffffffff+        lsr x24, x4, #52+        mul x2, x3, x2+        extr x26, x23, x26, #44+        lsr x23, x25, #4+        and x23, x23, #0xfffffffffffff+        and x25, x26, #0xfffffffffffff+        extr x26, x7, x15, #32+        extr x19, x15, x19, #44+        mul x23, x3, x23+        and x15, x26, #0xfffffffffffff+        lsl x26, x4, #12+        and x4, x19, #0xfffffffffffff+        lsr x11, x11, #20+        mul x19, x6, x4+        adcs x17, x17, x14+        add x14, x22, x2+        add x22, x12, x23+        lsr x7, x7, #20+        add x22, x22, x24+        extr x2, x22, x26, #60+        mul x24, x3, x25+        lsr x22, x22, #52+        add x14, x14, x22+        lsl x22, x2, #8+        extr x22, x14, x22, #8+        lsl x2, x14, #12+        mul x1, x3, x1+        adcs x12, x5, x22+        mul x5, x6, x15+        and x26, x10, x13+        and x4, x26, x17+        add x23, x19, x24+        lsr x14, x14, #52+        mul x22, x3, x11+        add x11, x23, x14+        extr x25, x11, x2, #20+        lsl x19, x11, #12+        adcs x25, x20, x25+        and x14, x4, x12+        add x1, x5, x1+        and x14, x14, x25+        mul x15, x6, x7+        add x26, x15, x22+        mul x6, x6, x3+        lsr x22, x11, #52+        add x4, x1, x22+        lsr x1, x4, #52+        extr x3, x4, x19, #32+        lsl x15, x4, #12+        add x7, x26, x1+        adcs x23, x8, x3+        extr x20, x7, x15, #44+        and x3, x14, x23+        lsr x19, x7, #44+        adcs x7, x9, x20+        add x11, x6, x19+        adc x4, x16, x11+        lsr x14, x4, #9+        cmp xzr, xzr+        and x15, x3, x7+        orr x3, x4, #0xfffffffffffffe00+        adcs xzr, x21, x14+        adcs xzr, x15, xzr+        adcs xzr, x3, xzr+        adcs x11, x21, x14+        and x14, x11, #0x1ff+        adcs x1, x10, xzr+        extr x10, x1, x11, #9+        str x14, [x0, #64]+        adcs x14, x13, xzr+        extr x11, x14, x1, #9+        adcs x1, x17, xzr+        extr x4, x1, x14, #9+        stp x10, x11, [x0]+        adcs x11, x12, xzr+        extr x14, x11, x1, #9+        adcs x10, x25, xzr+        extr x11, x10, x11, #9+        stp x4, x14, [x0, #16]+        adcs x14, x23, xzr+        extr x10, x14, x10, #9+        adcs x1, x7, xzr+        stp x11, x10, [x0, #32]+        extr x14, x1, x14, #9+        adc x10, x3, xzr+        extr x26, x10, x1, #9+        stp x14, x26, [x0, #48]++// Restore regs and return++        CFI_INC_SP(80)+        CFI_POP2(x25,x26)+        CFI_POP2(x23,x24)+        CFI_POP2(x21,x22)+        CFI_POP2(x19,x20)+        CFI_RET++S2N_BN_SIZE_DIRECTIVE(bignum_mul_p521)++#if defined(__linux__) && defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/s2n/arm/bignum_mul_p521_alt.S view
@@ -0,0 +1,538 @@+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0++// ----------------------------------------------------------------------------+// Multiply modulo p_521, z := (x * y) mod p_521, assuming x and y reduced+// Inputs x[9], y[9]; output z[9]+//+//    extern void bignum_mul_p521_alt(uint64_t z[static 9],+//                                    const uint64_t x[static 9],+//                                    const uint64_t y[static 9]);+//+// Standard ARM ABI: X0 = z, X1 = x, X2 = y+// ----------------------------------------------------------------------------++#include "_internal_s2n_bignum_arm.h"++        S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_mul_p521_alt)+        S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_mul_p521_alt)+        S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_mul_p521_alt)+        .text+        .balign 4++#define z x0+#define x x1+#define y x2++// These are repeated mod 2 as we load paris of inputs++#define a0 x3+#define a1 x4+#define a2 x3+#define a3 x4+#define a4 x3+#define a5 x4+#define a6 x3+#define a7 x4+#define a8 x3++#define b0 x5+#define b1 x6+#define b2 x7+#define b3 x8+#define b4 x9+#define b5 x10+#define b6 x11+#define b7 x12+#define b8 x13++#define t x14++// These repeat mod 11 as we stash some intermediate results in the+// output buffer.++#define u0 x15+#define u1 x16+#define u2 x17+#define u3 x19+#define u4 x20+#define u5 x21+#define u6 x22+#define u7 x23+#define u8 x24+#define u9 x25+#define u10 x26+#define u11 x15+#define u12 x16+#define u13 x17+#define u14 x19+#define u15 x20+#define u16 x21++S2N_BN_SYMBOL(bignum_mul_p521_alt):+        CFI_START++// Save more registers and make temporary space on stack++        CFI_PUSH2(x19,x20)+        CFI_PUSH2(x21,x22)+        CFI_PUSH2(x23,x24)+        CFI_PUSH2(x25,x26)+        CFI_DEC_SP(64)++// Load operands and set up row 0 = [u9;...;u0] = a0 * [b8;...;b0]++        ldp     a0, a1, [x]+        ldp     b0, b1, [y]++        mul     u0, a0, b0+        umulh   u1, a0, b0+        mul     t, a0, b1+        umulh   u2, a0, b1+        adds    u1, u1, t++        ldp     b2, b3, [y, #16]++        mul     t, a0, b2+        umulh   u3, a0, b2+        adcs    u2, u2, t++        mul     t, a0, b3+        umulh   u4, a0, b3+        adcs    u3, u3, t++        ldp     b4, b5, [y, #32]++        mul     t, a0, b4+        umulh   u5, a0, b4+        adcs    u4, u4, t++        mul     t, a0, b5+        umulh   u6, a0, b5+        adcs    u5, u5, t++        ldp     b6, b7, [y, #48]++        mul     t, a0, b6+        umulh   u7, a0, b6+        adcs    u6, u6, t++        ldr     b8, [y, #64]++        mul     t, a0, b7+        umulh   u8, a0, b7+        adcs    u7, u7, t++        mul     t, a0, b8+        umulh   u9, a0, b8+        adcs    u8, u8, t++        adc     u9, u9, xzr++// Row 1 = [u10;...;u0] = [a1;a0] * [b8;...;b0]++        mul     t, a1, b0+        adds    u1, u1, t+        mul     t, a1, b1+        adcs    u2, u2, t+        mul     t, a1, b2+        adcs    u3, u3, t+        mul     t, a1, b3+        adcs    u4, u4, t+        mul     t, a1, b4+        adcs    u5, u5, t+        mul     t, a1, b5+        adcs    u6, u6, t+        mul     t, a1, b6+        adcs    u7, u7, t+        mul     t, a1, b7+        adcs    u8, u8, t+        mul     t, a1, b8+        adcs    u9, u9, t+        cset    u10, cs++        umulh   t, a1, b0+        adds    u2, u2, t+        umulh   t, a1, b1+        adcs    u3, u3, t+        umulh   t, a1, b2+        adcs    u4, u4, t+        umulh   t, a1, b3+        adcs    u5, u5, t+        umulh   t, a1, b4+        adcs    u6, u6, t+        umulh   t, a1, b5+        adcs    u7, u7, t+        umulh   t, a1, b6+        adcs    u8, u8, t+        umulh   t, a1, b7+        adcs    u9, u9, t+        umulh   t, a1, b8+        adc     u10, u10, t++        stp     u0, u1, [sp]++// Row 2 = [u11;...;u0] = [a2;a1;a0] * [b8;...;b0]++        ldp     a2, a3, [x, #16]++        mul     t, a2, b0+        adds    u2, u2, t+        mul     t, a2, b1+        adcs    u3, u3, t+        mul     t, a2, b2+        adcs    u4, u4, t+        mul     t, a2, b3+        adcs    u5, u5, t+        mul     t, a2, b4+        adcs    u6, u6, t+        mul     t, a2, b5+        adcs    u7, u7, t+        mul     t, a2, b6+        adcs    u8, u8, t+        mul     t, a2, b7+        adcs    u9, u9, t+        mul     t, a2, b8+        adcs    u10, u10, t+        cset    u11, cs++        umulh   t, a2, b0+        adds    u3, u3, t+        umulh   t, a2, b1+        adcs    u4, u4, t+        umulh   t, a2, b2+        adcs    u5, u5, t+        umulh   t, a2, b3+        adcs    u6, u6, t+        umulh   t, a2, b4+        adcs    u7, u7, t+        umulh   t, a2, b5+        adcs    u8, u8, t+        umulh   t, a2, b6+        adcs    u9, u9, t+        umulh   t, a2, b7+        adcs    u10, u10, t+        umulh   t, a2, b8+        adc     u11, u11, t++// Row 3 = [u12;...;u0] = [a3;a2;a1;a0] * [b8;...;b0]++        mul     t, a3, b0+        adds    u3, u3, t+        mul     t, a3, b1+        adcs    u4, u4, t+        mul     t, a3, b2+        adcs    u5, u5, t+        mul     t, a3, b3+        adcs    u6, u6, t+        mul     t, a3, b4+        adcs    u7, u7, t+        mul     t, a3, b5+        adcs    u8, u8, t+        mul     t, a3, b6+        adcs    u9, u9, t+        mul     t, a3, b7+        adcs    u10, u10, t+        mul     t, a3, b8+        adcs    u11, u11, t+        cset    u12, cs++        umulh   t, a3, b0+        adds    u4, u4, t+        umulh   t, a3, b1+        adcs    u5, u5, t+        umulh   t, a3, b2+        adcs    u6, u6, t+        umulh   t, a3, b3+        adcs    u7, u7, t+        umulh   t, a3, b4+        adcs    u8, u8, t+        umulh   t, a3, b5+        adcs    u9, u9, t+        umulh   t, a3, b6+        adcs    u10, u10, t+        umulh   t, a3, b7+        adcs    u11, u11, t+        umulh   t, a3, b8+        adc     u12, u12, t++        stp     u2, u3, [sp, #16]++// Row 4 = [u13;...;u0] = [a4;a3;a2;a1;a0] * [b8;...;b0]++        ldp     a4, a5, [x, #32]++        mul     t, a4, b0+        adds    u4, u4, t+        mul     t, a4, b1+        adcs    u5, u5, t+        mul     t, a4, b2+        adcs    u6, u6, t+        mul     t, a4, b3+        adcs    u7, u7, t+        mul     t, a4, b4+        adcs    u8, u8, t+        mul     t, a4, b5+        adcs    u9, u9, t+        mul     t, a4, b6+        adcs    u10, u10, t+        mul     t, a4, b7+        adcs    u11, u11, t+        mul     t, a4, b8+        adcs    u12, u12, t+        cset    u13, cs++        umulh   t, a4, b0+        adds    u5, u5, t+        umulh   t, a4, b1+        adcs    u6, u6, t+        umulh   t, a4, b2+        adcs    u7, u7, t+        umulh   t, a4, b3+        adcs    u8, u8, t+        umulh   t, a4, b4+        adcs    u9, u9, t+        umulh   t, a4, b5+        adcs    u10, u10, t+        umulh   t, a4, b6+        adcs    u11, u11, t+        umulh   t, a4, b7+        adcs    u12, u12, t+        umulh   t, a4, b8+        adc     u13, u13, t++// Row 5 = [u14;...;u0] = [a5;a4;a3;a2;a1;a0] * [b8;...;b0]++        mul     t, a5, b0+        adds    u5, u5, t+        mul     t, a5, b1+        adcs    u6, u6, t+        mul     t, a5, b2+        adcs    u7, u7, t+        mul     t, a5, b3+        adcs    u8, u8, t+        mul     t, a5, b4+        adcs    u9, u9, t+        mul     t, a5, b5+        adcs    u10, u10, t+        mul     t, a5, b6+        adcs    u11, u11, t+        mul     t, a5, b7+        adcs    u12, u12, t+        mul     t, a5, b8+        adcs    u13, u13, t+        cset    u14, cs++        umulh   t, a5, b0+        adds    u6, u6, t+        umulh   t, a5, b1+        adcs    u7, u7, t+        umulh   t, a5, b2+        adcs    u8, u8, t+        umulh   t, a5, b3+        adcs    u9, u9, t+        umulh   t, a5, b4+        adcs    u10, u10, t+        umulh   t, a5, b5+        adcs    u11, u11, t+        umulh   t, a5, b6+        adcs    u12, u12, t+        umulh   t, a5, b7+        adcs    u13, u13, t+        umulh   t, a5, b8+        adc     u14, u14, t++        stp     u4, u5, [sp, #32]++// Row 6 = [u15;...;u0] = [a6;a5;a4;a3;a2;a1;a0] * [b8;...;b0]++        ldp     a6, a7, [x, #48]++        mul     t, a6, b0+        adds    u6, u6, t+        mul     t, a6, b1+        adcs    u7, u7, t+        mul     t, a6, b2+        adcs    u8, u8, t+        mul     t, a6, b3+        adcs    u9, u9, t+        mul     t, a6, b4+        adcs    u10, u10, t+        mul     t, a6, b5+        adcs    u11, u11, t+        mul     t, a6, b6+        adcs    u12, u12, t+        mul     t, a6, b7+        adcs    u13, u13, t+        mul     t, a6, b8+        adcs    u14, u14, t+        cset    u15, cs++        umulh   t, a6, b0+        adds    u7, u7, t+        umulh   t, a6, b1+        adcs    u8, u8, t+        umulh   t, a6, b2+        adcs    u9, u9, t+        umulh   t, a6, b3+        adcs    u10, u10, t+        umulh   t, a6, b4+        adcs    u11, u11, t+        umulh   t, a6, b5+        adcs    u12, u12, t+        umulh   t, a6, b6+        adcs    u13, u13, t+        umulh   t, a6, b7+        adcs    u14, u14, t+        umulh   t, a6, b8+        adc     u15, u15, t++// Row 7 = [u16;...;u0] = [a7;a6;a5;a4;a3;a2;a1;a0] * [b8;...;b0]++        mul     t, a7, b0+        adds    u7, u7, t+        mul     t, a7, b1+        adcs    u8, u8, t+        mul     t, a7, b2+        adcs    u9, u9, t+        mul     t, a7, b3+        adcs    u10, u10, t+        mul     t, a7, b4+        adcs    u11, u11, t+        mul     t, a7, b5+        adcs    u12, u12, t+        mul     t, a7, b6+        adcs    u13, u13, t+        mul     t, a7, b7+        adcs    u14, u14, t+        mul     t, a7, b8+        adcs    u15, u15, t+        cset    u16, cs++        umulh   t, a7, b0+        adds    u8, u8, t+        umulh   t, a7, b1+        adcs    u9, u9, t+        umulh   t, a7, b2+        adcs    u10, u10, t+        umulh   t, a7, b3+        adcs    u11, u11, t+        umulh   t, a7, b4+        adcs    u12, u12, t+        umulh   t, a7, b5+        adcs    u13, u13, t+        umulh   t, a7, b6+        adcs    u14, u14, t+        umulh   t, a7, b7+        adcs    u15, u15, t+        umulh   t, a7, b8+        adc     u16, u16, t++        stp     u6, u7, [sp, #48]++// Row 8 = [u16;...;u0] = [a8;a7;a6;a5;a4;a3;a2;a1;a0] * [b8;...;b0]++        ldr     a8, [x, #64]++        mul     t, a8, b0+        adds    u8, u8, t+        mul     t, a8, b1+        adcs    u9, u9, t+        mul     t, a8, b2+        adcs    u10, u10, t+        mul     t, a8, b3+        adcs    u11, u11, t+        mul     t, a8, b4+        adcs    u12, u12, t+        mul     t, a8, b5+        adcs    u13, u13, t+        mul     t, a8, b6+        adcs    u14, u14, t+        mul     t, a8, b7+        adcs    u15, u15, t+        mul     t, a8, b8+        adc     u16, u16, t++        umulh   t, a8, b0+        adds    u9, u9, t+        umulh   t, a8, b1+        adcs    u10, u10, t+        umulh   t, a8, b2+        adcs    u11, u11, t+        umulh   t, a8, b3+        adcs    u12, u12, t+        umulh   t, a8, b4+        adcs    u13, u13, t+        umulh   t, a8, b5+        adcs    u14, u14, t+        umulh   t, a8, b6+        adcs    u15, u15, t+        umulh   t, a8, b7+        adc     u16, u16, t++// Now we have the full product, which we consider as+// 2^521 * h + l. Form h + l + 1++        subs    xzr, xzr, xzr+        ldp     b0, b1, [sp]+        extr    t, u9, u8, #9+        adcs    b0, b0, t+        extr    t, u10, u9, #9+        adcs    b1, b1, t+        ldp     b2, b3, [sp, #16]+        extr    t, u11, u10, #9+        adcs    b2, b2, t+        extr    t, u12, u11, #9+        adcs    b3, b3, t+        ldp     b4, b5, [sp, #32]+        extr    t, u13, u12, #9+        adcs    b4, b4, t+        extr    t, u14, u13, #9+        adcs    b5, b5, t+        ldp     b6, b7, [sp, #48]+        extr    t, u15, u14, #9+        adcs    b6, b6, t+        extr    t, u16, u15, #9+        adcs    b7, b7, t+        orr     b8, u8, #~0x1FF+        lsr     t, u16, #9+        adcs    b8, b8, t++// Now CF is set if h + l + 1 >= 2^521, which means it's already+// the answer, while if ~CF the answer is h + l so we should subtract+// 1 (all considered in 521 bits). Hence subtract ~CF and mask.++        sbcs    b0, b0, xzr+        sbcs    b1, b1, xzr+        sbcs    b2, b2, xzr+        sbcs    b3, b3, xzr+        sbcs    b4, b4, xzr+        sbcs    b5, b5, xzr+        sbcs    b6, b6, xzr+        sbcs    b7, b7, xzr+        sbc     b8, b8, xzr+        and     b8, b8, #0x1FF++// Store back digits of final result++        stp     b0, b1, [z]+        stp     b2, b3, [z, #16]+        stp     b4, b5, [z, #32]+        stp     b6, b7, [z, #48]+        str     b8, [z, #64]++// Restore registers++        CFI_INC_SP(64)+        CFI_POP2(x25,x26)+        CFI_POP2(x23,x24)+        CFI_POP2(x21,x22)+        CFI_POP2(x19,x20)++        CFI_RET++S2N_BN_SIZE_DIRECTIVE(bignum_mul_p521_alt)++#if defined(__linux__) && defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/s2n/arm/bignum_neg_p256.S view
@@ -0,0 +1,72 @@+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0++// ----------------------------------------------------------------------------+// Negate modulo p_256, z := (-x) mod p_256, assuming x reduced+// Input x[4]; output z[4]+//+//    extern void bignum_neg_p256(uint64_t z[static 4], const uint64_t x[static 4]);+//+// Standard ARM ABI: X0 = z, X1 = x+// ----------------------------------------------------------------------------++#include "_internal_s2n_bignum_arm.h"++        S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_neg_p256)+        S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_neg_p256)+        S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_neg_p256)+        .text+        .balign 4++#define z x0+#define x x1++#define p x2+#define t x3++#define d0 x4+#define d1 x5+#define d2 x6+#define d3 x7+++S2N_BN_SYMBOL(bignum_neg_p256):+        CFI_START++// Load the 4 digits of x++        ldp     d0, d1, [x]+        ldp     d2, d3, [x, #16]++// Set a bitmask p for the input being nonzero, so that we avoid doing+// -0 = p_256 and hence maintain strict modular reduction++        orr     t, d0, d1+        orr     p, d2, d3+        orr     p, p, t+        cmp     p, #0+        csetm   p, ne++// Mask the nontrivial words of p_256 = [n3;0;n1;-1] and subtract++        subs    d0, p, d0+        and     t, p, #0x00000000ffffffff+        sbcs    d1, t, d1+        sbcs    d2, xzr, d2+        and     t, p, #0xffffffff00000001+        sbc     d3, t, d3++// Write back the result++        stp     d0, d1, [z]+        stp     d2, d3, [z, #16]++// Return++        CFI_RET++S2N_BN_SIZE_DIRECTIVE(bignum_neg_p256)++#if defined(__linux__) && defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/s2n/arm/bignum_sqr_p521.S view
@@ -0,0 +1,1125 @@+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.+// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0++// ----------------------------------------------------------------------------+// Square modulo p_521, z := (x^2) mod p_521, assuming x reduced+// Input x[9]; output z[9]+//+//    extern void bignum_sqr_p521(uint64_t z[static 9], const uint64_t x[static 9]);+//+// Standard ARM ABI: X0 = z, X1 = x+// ----------------------------------------------------------------------------++#include "_internal_s2n_bignum_arm.h"++// bignum_sqr_p521 is functionally equivalent to unopt/bignum_sqr_p521_base.+// It is written in a way that+// 1. A subset of scalar multiplications in bignum_montmul_p384 are carefully+//    chosen and vectorized+// 2. The vectorized assembly is rescheduled using the SLOTHY superoptimizer.+//    https://github.com/slothy-optimizer/slothy+//+// The output program of step 1. is as follows:+//+//        stp     x19, x20, [sp, #-16]!+//        stp     x21, x22, [sp, #-16]!+//        stp     x23, x24, [sp, #-16]!+//        ldp x20, x19, [x1]+//        ldr q23, [x1]+//        ldr q1, [x1]+//        ldr q16, [x1]+//        ldp x14, x12, [x1, #16]+//        ldr q28, [x1, #16]+//        ldr q31, [x1, #16]+//        ldp x9, x2, [x1, #32]+//        ldr q29, [x1, #32]+//        ldr q4, [x1, #32]+//        ldr q5, [x1]+//        ldr q2, [x1, #32]+//        ldp x6, x13, [x1, #48]+//        ldr q24, [x1, #48]+//        ldr q27, [x1, #48]+//        ldr q0, [x1, #16]+//        ldr q30, [x1, #48]+//        mul x17, x9, x6+//        mul x10, x2, x13+//        umulh x24, x9, x6+//        subs x4, x9, x2+//        cneg x4, x4, cc+//        csetm x16, cc+//        subs x3, x13, x6+//        cneg x23, x3, cc+//        mul x3, x4, x23+//        umulh x4, x4, x23+//        cinv x22, x16, cc+//        eor x23, x3, x22+//        eor x16, x4, x22+//        adds x3, x17, x24+//        adc x24, x24, xzr+//        umulh x4, x2, x13+//        adds x3, x3, x10+//        adcs x24, x24, x4+//        adc x4, x4, xzr+//        adds x24, x24, x10+//        adc x10, x4, xzr+//        cmn x22, #0x1+//        adcs x4, x3, x23+//        adcs x24, x24, x16+//        adc x10, x10, x22+//        adds x8, x17, x17+//        adcs x22, x4, x4+//        adcs x5, x24, x24+//        adcs x11, x10, x10+//        adc x23, xzr, xzr+//        movi v25.2D, #0xffffffff+//        uzp2 v19.4S, v4.4S, v4.4S+//        xtn v26.2S, v29.2D+//        xtn v22.2S, v4.2D+//        rev64 v4.4S, v4.4S+//        umull v7.2D, v26.2S, v22.2S+//        umull v21.2D, v26.2S, v19.2S+//        uzp2 v17.4S, v29.4S, v29.4S+//        mul v4.4S, v4.4S, v29.4S+//        usra v21.2D, v7.2D, #32+//        umull v18.2D, v17.2S, v19.2S+//        uaddlp v4.2D, v4.4S+//        and v7.16B, v21.16B, v25.16B+//        umlal v7.2D, v17.2S, v22.2S+//        shl v4.2D, v4.2D, #32+//        usra v18.2D, v21.2D, #32+//        umlal v4.2D, v26.2S, v22.2S+//        usra v18.2D, v7.2D, #32+//        mov x15, v4.d[0]+//        mov x16, v4.d[1]+//        mul x3, x9, x2+//        mov x10, v18.d[0]+//        mov x17, v18.d[1]+//        umulh x4, x9, x2+//        adds x24, x10, x3+//        adcs x10, x16, x4+//        adc x17, x17, xzr+//        adds x7, x24, x3+//        adcs x10, x10, x4+//        adc x17, x17, xzr+//        adds x8, x8, x10+//        adcs x22, x22, x17+//        adcs x21, x5, xzr+//        adcs x5, x11, xzr+//        adc x11, x23, xzr+//        movi v25.2D, #0xffffffff+//        uzp2 v19.4S, v27.4S, v27.4S+//        xtn v26.2S, v24.2D+//        xtn v22.2S, v27.2D+//        rev64 v4.4S, v27.4S+//        umull v7.2D, v26.2S, v22.2S+//        umull v21.2D, v26.2S, v19.2S+//        uzp2 v17.4S, v24.4S, v24.4S+//        mul v4.4S, v4.4S, v24.4S+//        usra v21.2D, v7.2D, #32+//        umull v18.2D, v17.2S, v19.2S+//        uaddlp v4.2D, v4.4S+//        and v7.16B, v21.16B, v25.16B+//        umlal v7.2D, v17.2S, v22.2S+//        shl v4.2D, v4.2D, #32+//        usra v18.2D, v21.2D, #32+//        umlal v4.2D, v26.2S, v22.2S+//        usra v18.2D, v7.2D, #32+//        mov x23, v4.d[0]+//        mov x16, v4.d[1]+//        mul x3, x6, x13+//        mov x10, v18.d[0]+//        mov x17, v18.d[1]+//        umulh x4, x6, x13+//        adds x24, x10, x3+//        adcs x10, x16, x4+//        adc x17, x17, xzr+//        adds x24, x24, x3+//        adcs x10, x10, x4+//        adc x17, x17, xzr+//        adds x23, x23, x21+//        adcs x16, x24, x5+//        adcs x3, x10, x11+//        adc x21, x17, xzr+//        ldr x17, [x1, #64]+//        add x5, x17, x17+//        mul x11, x17, x17+//        and x17, x20, #0xfffffffffffff+//        mul x4, x5, x17+//        extr x17, x19, x20, #52+//        and x17, x17, #0xfffffffffffff+//        mul x10, x5, x17+//        lsr x17, x4, #52+//        add x24, x10, x17+//        lsl x17, x4, #12+//        extr x17, x24, x17, #12+//        adds x15, x15, x17+//        extr x17, x14, x19, #40+//        and x17, x17, #0xfffffffffffff+//        mul x10, x5, x17+//        lsr x17, x24, #52+//        add x4, x10, x17+//        lsl x17, x24, #12+//        extr x17, x4, x17, #24+//        adcs x7, x7, x17+//        extr x17, x12, x14, #28+//        and x17, x17, #0xfffffffffffff+//        mul x10, x5, x17+//        lsr x17, x4, #52+//        add x24, x10, x17+//        lsl x17, x4, #12+//        extr x17, x24, x17, #36+//        adcs x8, x8, x17+//        extr x17, x9, x12, #16+//        and x17, x17, #0xfffffffffffff+//        mul x10, x5, x17+//        lsr x17, x24, #52+//        add x4, x10, x17+//        lsl x17, x24, #12+//        extr x17, x4, x17, #48+//        adcs x22, x22, x17+//        lsr x17, x9, #4+//        and x17, x17, #0xfffffffffffff+//        mul x10, x5, x17+//        lsr x17, x4, #52+//        add x24, x10, x17+//        lsl x17, x4, #12+//        extr x4, x24, x17, #60+//        extr x17, x2, x9, #56+//        and x17, x17, #0xfffffffffffff+//        mul x10, x5, x17+//        lsr x17, x24, #52+//        add x24, x10, x17+//        lsl x17, x4, #8+//        extr x17, x24, x17, #8+//        adcs x23, x23, x17+//        extr x17, x6, x2, #44+//        and x17, x17, #0xfffffffffffff+//        mul x10, x5, x17+//        lsr x17, x24, #52+//        add x4, x10, x17+//        lsl x17, x24, #12+//        extr x17, x4, x17, #20+//        adcs x16, x16, x17+//        extr x17, x13, x6, #32+//        and x17, x17, #0xfffffffffffff+//        mul x10, x5, x17+//        lsr x17, x4, #52+//        add x24, x10, x17+//        lsl x17, x4, #12+//        extr x17, x24, x17, #32+//        adcs x3, x3, x17+//        lsr x17, x13, #20+//        mul x10, x5, x17+//        lsr x17, x24, #52+//        add x10, x10, x17+//        lsl x17, x24, #12+//        extr x17, x10, x17, #44+//        adcs x4, x21, x17+//        lsr x17, x10, #44+//        adc x24, x11, x17+//        extr x10, x7, x15, #9+//        extr x17, x8, x7, #9+//        stp x10, x17, [x0]                       // @slothy:writes=buffer0+//        extr x10, x22, x8, #9+//        extr x17, x23, x22, #9+//        stp x10, x17, [x0, #16]                  // @slothy:writes=buffer16+//        extr x10, x16, x23, #9+//        extr x17, x3, x16, #9+//        stp x10, x17, [x0, #32]                  // @slothy:writes=buffer32+//        extr x10, x4, x3, #9+//        extr x17, x24, x4, #9+//        stp x10, x17, [x0, #48]                  // @slothy:writes=buffer48+//        and x10, x15, #0x1ff+//        lsr x17, x24, #9+//        add x17, x10, x17+//        str x17, [x0, #64]                       // @slothy:writes=buffer64+//        uzp1 v17.4S, v28.4S, v23.4S+//        rev64 v4.4S, v28.4S+//        uzp1 v7.4S, v23.4S, v23.4S+//        mul v4.4S, v4.4S, v23.4S+//        uaddlp v4.2D, v4.4S+//        shl v4.2D, v4.2D, #32+//        umlal v4.2D, v7.2S, v17.2S+//        mov x8, v4.d[0]+//        mov x22, v4.d[1]+//        umulh x23, x20, x14+//        subs x17, x20, x19+//        cneg x4, x17, cc+//        csetm x24, cc+//        subs x17, x12, x14+//        cneg x17, x17, cc+//        mul x10, x4, x17+//        umulh x17, x4, x17+//        cinv x16, x24, cc+//        eor x3, x10, x16+//        eor x4, x17, x16+//        adds x24, x8, x23+//        adc x10, x23, xzr+//        umulh x17, x19, x12+//        adds x24, x24, x22+//        adcs x10, x10, x17+//        adc x17, x17, xzr+//        adds x10, x10, x22+//        adc x17, x17, xzr+//        cmn x16, #0x1+//        adcs x24, x24, x3+//        adcs x10, x10, x4+//        adc x17, x17, x16+//        adds x15, x8, x8+//        adcs x7, x24, x24+//        adcs x8, x10, x10+//        adcs x22, x17, x17+//        adc x23, xzr, xzr+//        movi v25.2D, #0xffffffff+//        uzp2 v19.4S, v16.4S, v16.4S+//        xtn v26.2S, v1.2D+//        xtn v22.2S, v16.2D+//        rev64 v4.4S, v16.4S+//        umull v7.2D, v26.2S, v22.2S+//        umull v21.2D, v26.2S, v19.2S+//        uzp2 v17.4S, v1.4S, v1.4S+//        mul v4.4S, v4.4S, v1.4S+//        usra v21.2D, v7.2D, #32+//        umull v18.2D, v17.2S, v19.2S+//        uaddlp v4.2D, v4.4S+//        and v7.16B, v21.16B, v25.16B+//        umlal v7.2D, v17.2S, v22.2S+//        shl v4.2D, v4.2D, #32+//        usra v18.2D, v21.2D, #32+//        umlal v4.2D, v26.2S, v22.2S+//        usra v18.2D, v7.2D, #32+//        mov x21, v4.d[0]+//        mov x16, v4.d[1]+//        mul x3, x20, x19+//        mov x10, v18.d[0]+//        mov x17, v18.d[1]+//        umulh x4, x20, x19+//        adds x24, x10, x3+//        adcs x10, x16, x4+//        adc x17, x17, xzr+//        adds x5, x24, x3+//        adcs x10, x10, x4+//        adc x17, x17, xzr+//        adds x11, x15, x10+//        adcs x15, x7, x17+//        adcs x7, x8, xzr+//        adcs x8, x22, xzr+//        adc x22, x23, xzr+//        xtn v7.2S, v31.2D+//        shrn v4.2S, v31.2D, #32+//        umull v4.2D, v7.2S, v4.2S+//        shl v4.2D, v4.2D, #33+//        umlal v4.2D, v7.2S, v7.2S+//        mov x23, v4.d[0]+//        mov x16, v4.d[1]+//        mul x3, x14, x12+//        umulh x10, x14, x14+//        umulh x17, x12, x12+//        umulh x4, x14, x12+//        adds x24, x10, x3+//        adcs x10, x16, x4+//        adc x17, x17, xzr+//        adds x24, x24, x3+//        adcs x10, x10, x4+//        adc x17, x17, xzr+//        adds x16, x23, x7+//        adcs x3, x24, x8+//        adcs x4, x10, x22+//        adc x24, x17, xzr+//        ldp x10, x17, [x0]                       // @slothy:reads=buffer0+//        adds x10, x10, x21+//        adcs x17, x17, x5+//        stp x10, x17, [x0]                       // @slothy:writes=buffer0+//        ldp x10, x17, [x0, #16]                  // @slothy:reads=buffer16+//        adcs x10, x10, x11+//        adcs x17, x17, x15+//        stp x10, x17, [x0, #16]                  // @slothy:writes=buffer16+//        ldp x10, x17, [x0, #32]                  // @slothy:reads=buffer32+//        adcs x10, x10, x16+//        adcs x17, x17, x3+//        stp x10, x17, [x0, #32]                  // @slothy:writes=buffer32+//        ldp x10, x17, [x0, #48]                  // @slothy:reads=buffer48+//        adcs x10, x10, x4+//        adcs x17, x17, x24+//        stp x10, x17, [x0, #48]                  // @slothy:writes=buffer48+//        ldr x17, [x0, #64]                       // @slothy:reads=buffer64+//        adc x17, x17, xzr+//        str x17, [x0, #64]                       // @slothy:writes=buffer64+//        movi v25.2D, #0xffffffff+//        uzp2 v19.4S, v2.4S, v2.4S+//        xtn v26.2S, v5.2D+//        xtn v22.2S, v2.2D+//        rev64 v4.4S, v2.4S+//        umull v7.2D, v26.2S, v22.2S+//        umull v21.2D, v26.2S, v19.2S+//        uzp2 v17.4S, v5.4S, v5.4S+//        mul v4.4S, v4.4S, v5.4S+//        usra v21.2D, v7.2D, #32+//        umull v18.2D, v17.2S, v19.2S+//        uaddlp v4.2D, v4.4S+//        and v7.16B, v21.16B, v25.16B+//        umlal v7.2D, v17.2S, v22.2S+//        shl v4.2D, v4.2D, #32+//        usra v18.2D, v21.2D, #32+//        umlal v4.2D, v26.2S, v22.2S+//        usra v18.2D, v7.2D, #32+//        mov x5, v4.d[0]+//        mov x4, v4.d[1]+//        movi v25.2D, #0xffffffff+//        uzp2 v17.4S, v30.4S, v30.4S+//        xtn v19.2S, v0.2D+//        xtn v26.2S, v30.2D+//        rev64 v4.4S, v30.4S+//        umull v7.2D, v19.2S, v26.2S+//        umull v22.2D, v19.2S, v17.2S+//        uzp2 v21.4S, v0.4S, v0.4S+//        mul v4.4S, v4.4S, v0.4S+//        usra v22.2D, v7.2D, #32+//        umull v17.2D, v21.2S, v17.2S+//        uaddlp v4.2D, v4.4S+//        and v7.16B, v22.16B, v25.16B+//        umlal v7.2D, v21.2S, v26.2S+//        shl v4.2D, v4.2D, #32+//        usra v17.2D, v22.2D, #32+//        umlal v4.2D, v19.2S, v26.2S+//        usra v17.2D, v7.2D, #32+//        mov x24, v4.d[0]+//        mov x10, v4.d[1]+//        mov x17, v18.d[0]+//        adds x4, x4, x17+//        mov x17, v18.d[1]+//        adcs x24, x24, x17+//        mov x17, v17.d[0]+//        adcs x10, x10, x17+//        mov x17, v17.d[1]+//        adc x17, x17, xzr+//        adds x15, x4, x5+//        adcs x4, x24, x4+//        adcs x24, x10, x24+//        adcs x10, x17, x10+//        adc x17, xzr, x17+//        adds x7, x4, x5+//        adcs x8, x24, x15+//        adcs x22, x10, x4+//        adcs x23, x17, x24+//        adcs x16, xzr, x10+//        adc x3, xzr, x17+//        subs x17, x14, x12+//        cneg x24, x17, cc+//        csetm x4, cc+//        subs x17, x13, x6+//        cneg x10, x17, cc+//        mul x17, x24, x10+//        umulh x24, x24, x10+//        cinv x10, x4, cc+//        cmn x10, #0x1+//        eor x17, x17, x10+//        adcs x23, x23, x17+//        eor x17, x24, x10+//        adcs x16, x16, x17+//        adc x3, x3, x10+//        subs x17, x20, x19+//        cneg x24, x17, cc+//        csetm x4, cc+//        subs x17, x2, x9+//        cneg x10, x17, cc+//        mul x17, x24, x10+//        umulh x24, x24, x10+//        cinv x10, x4, cc+//        cmn x10, #0x1+//        eor x17, x17, x10+//        adcs x11, x15, x17+//        eor x17, x24, x10+//        adcs x15, x7, x17+//        adcs x7, x8, x10+//        adcs x22, x22, x10+//        adcs x23, x23, x10+//        adcs x16, x16, x10+//        adc x3, x3, x10+//        subs x17, x19, x12+//        cneg x24, x17, cc+//        csetm x4, cc+//        subs x17, x13, x2+//        cneg x10, x17, cc+//        mul x17, x24, x10+//        umulh x24, x24, x10+//        cinv x10, x4, cc+//        cmn x10, #0x1+//        eor x17, x17, x10+//        adcs x8, x22, x17+//        eor x17, x24, x10+//        adcs x23, x23, x17+//        adcs x16, x16, x10+//        adc x3, x3, x10+//        subs x17, x20, x14+//        cneg x24, x17, cc+//        csetm x4, cc+//        subs x17, x6, x9+//        cneg x10, x17, cc+//        mul x17, x24, x10+//        umulh x24, x24, x10+//        cinv x10, x4, cc+//        cmn x10, #0x1+//        eor x17, x17, x10+//        adcs x22, x15, x17+//        eor x17, x24, x10+//        adcs x4, x7, x17+//        adcs x24, x8, x10+//        adcs x23, x23, x10+//        adcs x16, x16, x10+//        adc x3, x3, x10+//        subs x12, x20, x12+//        cneg x10, x12, cc+//        csetm x17, cc+//        subs x12, x13, x9+//        cneg x9, x12, cc+//        mul x12, x10, x9+//        umulh x13, x10, x9+//        cinv x9, x17, cc+//        cmn x9, #0x1+//        eor x12, x12, x9+//        adcs x4, x4, x12+//        eor x12, x13, x9+//        adcs x24, x24, x12+//        adcs x10, x23, x9+//        adcs x17, x16, x9+//        adc x13, x3, x9+//        subs x19, x19, x14+//        cneg x12, x19, cc+//        csetm x9, cc+//        subs x6, x6, x2+//        cneg x14, x6, cc+//        mul x19, x12, x14+//        umulh x12, x12, x14+//        cinv x14, x9, cc+//        cmn x14, #0x1+//        eor x19, x19, x14+//        adcs x23, x4, x19+//        eor x19, x12, x14+//        adcs x16, x24, x19+//        adcs x6, x10, x14+//        adcs x2, x17, x14+//        adc x9, x13, x14+//        ldp x12, x14, [x0]                       // @slothy:reads=buffer0+//        extr x19, x6, x16, #8+//        adds x10, x19, x12+//        extr x19, x2, x6, #8+//        adcs x17, x19, x14+//        ldp x14, x12, [x0, #16]                  // @slothy:reads=buffer16+//        extr x19, x9, x2, #8+//        adcs x13, x19, x14+//        and x14, x17, x13+//        lsr x19, x9, #8+//        adcs x6, x19, x12+//        and x9, x14, x6+//        ldp x14, x12, [x0, #32]                  // @slothy:reads=buffer32+//        lsl x19, x5, #1+//        adcs x2, x19, x14+//        and x14, x9, x2+//        extr x19, x11, x5, #63+//        adcs x3, x19, x12+//        and x9, x14, x3+//        ldp x14, x12, [x0, #48]                  // @slothy:reads=buffer48+//        extr x19, x22, x11, #63+//        adcs x4, x19, x14+//        and x14, x9, x4+//        extr x19, x23, x22, #63+//        adcs x24, x19, x12+//        and x12, x14, x24+//        ldr x14, [x0, #64]                       // @slothy:reads=buffer64+//        extr x19, x16, x23, #63+//        and x19, x19, #0x1ff+//        adc x19, x14, x19+//        lsr x14, x19, #9+//        orr x19, x19, #0xfffffffffffffe00+//        cmp xzr, xzr+//        adcs xzr, x10, x14+//        adcs xzr, x12, xzr+//        adcs xzr, x19, xzr+//        adcs x10, x10, x14+//        adcs x17, x17, xzr+//        adcs x13, x13, xzr+//        adcs x6, x6, xzr+//        adcs x2, x2, xzr+//        adcs x9, x3, xzr+//        adcs x12, x4, xzr+//        adcs x14, x24, xzr+//        adc x19, x19, xzr+//        and x19, x19, #0x1ff+//        stp x10, x17, [x0]                       // @slothy:writes=buffer0+//        stp x13, x6, [x0, #16]                   // @slothy:writes=buffer16+//        stp x2, x9, [x0, #32]                    // @slothy:writes=buffer32+//        stp x12, x14, [x0, #48]                  // @slothy:writes=buffer48+//        str x19, [x0, #64]                       // @slothy:writes=buffer64+//        ldp     x23, x24, [sp], #16+//        ldp     x21, x22, [sp], #16+//        ldp     x19, x20, [sp], #16+//        ret+//+// The bash script used for step 2 is as follows:+//+//        # Store the assembly instructions except the last 'ret',+//        # callee-register store/loads as, say, 'input.S'.+//        export OUTPUTS="[hint_buffer0,hint_buffer16,hint_buffer32,hint_buffer48,hint_buffer64]"+//        export RESERVED_REGS="[x18,x25,x26,x27,x28,x29,x30,sp,q8,q9,q10,q11,q12,q13,q14,q15,v8,v9,v10,v11,v12,v13,v14,v15]"+//        <s2n-bignum>/tools/external/slothy.sh input.S my_out_dir+//        # my_out_dir/3.opt.s is the optimized assembly. Its output may differ+//        # from this file since the sequence is non-deterministically chosen.+//        # Please add 'ret' at the end of the output assembly.++        S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_sqr_p521)+        S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_sqr_p521)+        S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_sqr_p521)+        .text+        .balign 4++S2N_BN_SYMBOL(bignum_sqr_p521):+        CFI_START++// Save registers++        CFI_PUSH2(x19,x20)+        CFI_PUSH2(x21,x22)+        CFI_PUSH2(x23,x24)++        ldr q23, [x1, #32]+        ldp x9, x2, [x1, #32]+        ldr q16, [x1, #32]+        ldr q20, [x1, #48]+        ldp x6, x13, [x1, #48]+        rev64 v2.4S, v23.4S+        mul x14, x9, x2+        ldr q31, [x1, #48]+        subs x22, x9, x2+        uzp2 v26.4S, v23.4S, v23.4S+        mul v30.4S, v2.4S, v16.4S+        xtn v0.2S, v20.2D+        csetm x12, cc+        xtn v21.2S, v16.2D+        xtn v23.2S, v23.2D+        umulh x10, x9, x6+        rev64 v27.4S, v31.4S+        umull v2.2D, v21.2S, v26.2S+        cneg x23, x22, cc+        uaddlp v25.2D, v30.4S+        umull v18.2D, v21.2S, v23.2S+        mul x22, x9, x6+        mul v6.4S, v27.4S, v20.4S+        uzp2 v17.4S, v20.4S, v20.4S+        shl v20.2D, v25.2D, #32+        uzp2 v27.4S, v31.4S, v31.4S+        mul x16, x2, x13+        umlal v20.2D, v21.2S, v23.2S+        usra v2.2D, v18.2D, #32+        adds x8, x22, x10+        umull v25.2D, v17.2S, v27.2S+        xtn v31.2S, v31.2D+        movi v1.2D, #0xffffffff+        adc x3, x10, xzr+        umulh x21, x2, x13+        uzp2 v21.4S, v16.4S, v16.4S+        umull v18.2D, v0.2S, v27.2S+        subs x19, x13, x6+        and v7.16B, v2.16B, v1.16B+        umull v27.2D, v0.2S, v31.2S+        cneg x20, x19, cc+        movi v30.2D, #0xffffffff+        umull v16.2D, v21.2S, v26.2S+        umlal v7.2D, v21.2S, v23.2S+        mul x19, x23, x20+        cinv x7, x12, cc+        uaddlp v6.2D, v6.4S+        eor x12, x19, x7+        adds x11, x8, x16+        umulh x10, x23, x20+        ldr q1, [x1]+        usra v16.2D, v2.2D, #32+        adcs x19, x3, x21+        shl v2.2D, v6.2D, #32+        adc x20, x21, xzr+        adds x17, x19, x16+        usra v18.2D, v27.2D, #32+        adc x19, x20, xzr+        cmn x7, #0x1+        umlal v2.2D, v0.2S, v31.2S+        umulh x16, x9, x2+        adcs x8, x11, x12+        usra v16.2D, v7.2D, #32+        ldr x12, [x1, #64]+        eor x20, x10, x7+        umulh x10, x6, x13+        mov x23, v2.d[0]+        mov x3, v2.d[1]+        adcs x21, x17, x20+        usra v25.2D, v18.2D, #32+        and v23.16B, v18.16B, v30.16B+        adc x7, x19, x7+        adds x22, x22, x22+        ldr q7, [x1, #16]+        adcs x17, x8, x8+        umlal v23.2D, v17.2S, v31.2S+        mov x19, v16.d[0]+        mul x11, x12, x12+        ldr q4, [x1]+        usra v25.2D, v23.2D, #32+        add x5, x12, x12+        adcs x15, x21, x21+        ldr q28, [x1]+        mov x12, v20.d[1]+        adcs x24, x7, x7+        mov x21, v16.d[1]+        adc x4, xzr, xzr+        adds x19, x19, x14+        ldr q18, [x1, #16]+        xtn v26.2S, v1.2D+        adcs x8, x12, x16+        adc x21, x21, xzr+        adds x7, x19, x14+        xtn v23.2S, v7.2D+        rev64 v21.4S, v28.4S+        adcs x12, x8, x16+        ldp x20, x19, [x1]+        mov x16, v25.d[1]+        xtn v22.2S, v28.2D+        adc x14, x21, xzr+        adds x8, x22, x12+        uzp2 v24.4S, v28.4S, v28.4S+        rev64 v28.4S, v18.4S+        mul x12, x6, x13+        mul v16.4S, v21.4S, v1.4S+        shrn v31.2S, v7.2D, #32+        adcs x22, x17, x14+        mov x14, v25.d[0]+        and x21, x20, #0xfffffffffffff+        umull v17.2D, v26.2S, v24.2S+        ldr q2, [x1, #32]+        adcs x17, x15, xzr+        ldr q30, [x1, #48]+        umull v7.2D, v26.2S, v22.2S+        adcs x15, x24, xzr+        ldr q0, [x1, #16]+        movi v6.2D, #0xffffffff+        adc x4, x4, xzr+        adds x14, x14, x12+        uzp1 v27.4S, v18.4S, v4.4S+        uzp2 v19.4S, v1.4S, v1.4S+        adcs x24, x3, x10+        mul x3, x5, x21+        umull v29.2D, v23.2S, v31.2S+        ldr q5, [x1]+        adc x21, x16, xzr+        adds x16, x14, x12+        extr x12, x19, x20, #52+        umull v18.2D, v19.2S, v24.2S+        adcs x24, x24, x10+        and x10, x12, #0xfffffffffffff+        ldp x14, x12, [x1, #16]+        usra v17.2D, v7.2D, #32+        adc x21, x21, xzr+        adds x23, x23, x17+        mul x17, x5, x10+        shl v21.2D, v29.2D, #33+        lsl x10, x3, #12+        lsr x1, x3, #52+        rev64 v29.4S, v2.4S+        uaddlp v25.2D, v16.4S+        add x17, x17, x1+        adcs x16, x16, x15+        extr x3, x14, x19, #40+        mov x15, v20.d[0]+        extr x10, x17, x10, #12+        and x3, x3, #0xfffffffffffff+        shl v3.2D, v25.2D, #32+        and v6.16B, v17.16B, v6.16B+        mul x1, x5, x3+        usra v18.2D, v17.2D, #32+        adcs x3, x24, x4+        extr x4, x12, x14, #28+        umlal v6.2D, v19.2S, v22.2S+        xtn v20.2S, v2.2D+        umlal v3.2D, v26.2S, v22.2S+        movi v26.2D, #0xffffffff+        lsr x24, x17, #52+        and x4, x4, #0xfffffffffffff+        uzp2 v19.4S, v2.4S, v2.4S+        add x1, x1, x24+        mul x24, x5, x4+        lsl x4, x17, #12+        xtn v24.2S, v5.2D+        extr x17, x1, x4, #24+        adc x21, x21, xzr+        umlal v21.2D, v23.2S, v23.2S+        adds x4, x15, x10+        lsl x10, x1, #12+        adcs x15, x7, x17+        mul v23.4S, v28.4S, v4.4S+        and x7, x4, #0x1ff+        lsr x17, x1, #52+        umulh x1, x19, x12+        uzp2 v17.4S, v5.4S, v5.4S+        extr x4, x15, x4, #9+        add x24, x24, x17+        mul v29.4S, v29.4S, v5.4S+        extr x17, x24, x10, #36+        extr x10, x9, x12, #16+        uzp1 v28.4S, v4.4S, v4.4S+        adcs x17, x8, x17+        and x8, x10, #0xfffffffffffff+        umull v16.2D, v24.2S, v20.2S+        extr x10, x17, x15, #9+        mul x15, x5, x8+        stp x4, x10, [x0]+        lsl x4, x24, #12+        lsr x8, x9, #4+        uaddlp v4.2D, v23.4S+        and x8, x8, #0xfffffffffffff+        umull v23.2D, v24.2S, v19.2S+        mul x8, x5, x8+        extr x10, x2, x9, #56+        lsr x24, x24, #52+        and x10, x10, #0xfffffffffffff+        add x15, x15, x24+        extr x4, x15, x4, #48+        mul x24, x5, x10+        lsr x10, x15, #52+        usra v23.2D, v16.2D, #32+        add x10, x8, x10+        shl v4.2D, v4.2D, #32+        adcs x22, x22, x4+        extr x4, x6, x2, #44+        lsl x15, x15, #12+        lsr x8, x10, #52+        extr x15, x10, x15, #60+        and x10, x4, #0xfffffffffffff+        umlal v4.2D, v28.2S, v27.2S+        add x8, x24, x8+        extr x4, x13, x6, #32+        mul x24, x5, x10+        uzp2 v16.4S, v30.4S, v30.4S+        lsl x10, x15, #8+        rev64 v28.4S, v30.4S+        and x15, x4, #0xfffffffffffff+        extr x4, x8, x10, #8+        mul x10, x5, x15+        lsl x15, x8, #12+        adcs x23, x23, x4+        lsr x4, x8, #52+        lsr x8, x13, #20+        add x4, x24, x4+        mul x8, x5, x8+        lsr x24, x4, #52+        extr x15, x4, x15, #20+        lsl x4, x4, #12+        add x10, x10, x24+        adcs x15, x16, x15+        extr x4, x10, x4, #32+        umulh x5, x20, x14+        adcs x3, x3, x4+        usra v18.2D, v6.2D, #32+        lsl x16, x10, #12+        extr x24, x15, x23, #9+        lsr x10, x10, #52+        uzp2 v27.4S, v0.4S, v0.4S+        add x8, x8, x10+        extr x10, x3, x15, #9+        extr x4, x22, x17, #9+        and v25.16B, v23.16B, v26.16B+        lsr x17, x8, #44+        extr x15, x8, x16, #44+        extr x16, x23, x22, #9+        xtn v7.2S, v30.2D+        mov x8, v4.d[0]+        stp x24, x10, [x0, #32]+        uaddlp v30.2D, v29.4S+        stp x4, x16, [x0, #16]+        umulh x24, x20, x19+        adcs x15, x21, x15+        adc x16, x11, x17+        subs x11, x20, x19+        xtn v5.2S, v0.2D+        csetm x17, cc+        extr x3, x15, x3, #9+        mov x22, v4.d[1]+        cneg x21, x11, cc+        subs x10, x12, x14+        mul v31.4S, v28.4S, v0.4S+        cneg x10, x10, cc+        cinv x11, x17, cc+        shl v4.2D, v30.2D, #32+        umull v28.2D, v5.2S, v16.2S+        extr x23, x16, x15, #9+        adds x4, x8, x5+        mul x17, x21, x10+        umull v22.2D, v5.2S, v7.2S+        adc x15, x5, xzr+        adds x4, x4, x22+        uaddlp v2.2D, v31.4S+        lsr x5, x16, #9+        adcs x16, x15, x1+        mov x15, v18.d[0]+        adc x1, x1, xzr+        umulh x10, x21, x10+        adds x22, x16, x22+        umlal v4.2D, v24.2S, v20.2S+        umull v30.2D, v27.2S, v16.2S+        stp x3, x23, [x0, #48]+        add x3, x7, x5+        adc x16, x1, xzr+        usra v28.2D, v22.2D, #32+        mul x23, x20, x19+        eor x1, x17, x11+        cmn x11, #0x1+        mov x17, v18.d[1]+        umull v18.2D, v17.2S, v19.2S+        adcs x7, x4, x1+        eor x1, x10, x11+        umlal v25.2D, v17.2S, v20.2S+        movi v16.2D, #0xffffffff+        adcs x22, x22, x1+        usra v18.2D, v23.2D, #32+        umulh x4, x14, x14+        adc x1, x16, x11+        adds x10, x8, x8+        shl v23.2D, v2.2D, #32+        str x3, [x0, #64]+        adcs x5, x7, x7+        and v16.16B, v28.16B, v16.16B+        usra v30.2D, v28.2D, #32+        adcs x7, x22, x22+        mov x21, v3.d[1]+        adcs x11, x1, x1+        umlal v16.2D, v27.2S, v7.2S+        adc x22, xzr, xzr+        adds x16, x15, x23+        mul x8, x14, x12+        umlal v23.2D, v5.2S, v7.2S+        usra v18.2D, v25.2D, #32+        umulh x15, x14, x12+        adcs x21, x21, x24+        usra v30.2D, v16.2D, #32+        adc x1, x17, xzr+        adds x3, x16, x23+        adcs x21, x21, x24+        adc x1, x1, xzr+        adds x24, x10, x21+        umulh x21, x12, x12+        adcs x16, x5, x1+        adcs x10, x7, xzr+        mov x17, v21.d[1]+        adcs x23, x11, xzr+        adc x5, x22, xzr+        adds x1, x4, x8+        adcs x22, x17, x15+        ldp x17, x4, [x0]+        mov x11, v21.d[0]+        adc x21, x21, xzr+        adds x1, x1, x8+        adcs x15, x22, x15+        adc x8, x21, xzr+        adds x22, x11, x10+        mov x21, v3.d[0]+        adcs x11, x1, x23+        ldp x1, x10, [x0, #16]+        adcs x15, x15, x5+        adc x7, x8, xzr+        adds x8, x17, x21+        mov x23, v4.d[1]+        ldp x5, x21, [x0, #32]+        adcs x17, x4, x3+        ldr x4, [x0, #64]+        mov x3, v18.d[0]+        adcs x24, x1, x24+        stp x8, x17, [x0]+        adcs x17, x10, x16+        ldp x1, x16, [x0, #48]+        adcs x5, x5, x22+        adcs x8, x21, x11+        stp x5, x8, [x0, #32]+        adcs x1, x1, x15+        mov x15, v23.d[1]+        adcs x21, x16, x7+        stp x1, x21, [x0, #48]+        adc x10, x4, xzr+        subs x7, x14, x12+        mov x16, v18.d[1]+        cneg x5, x7, cc+        csetm x4, cc+        subs x11, x13, x6+        mov x8, v23.d[0]+        cneg x7, x11, cc+        cinv x21, x4, cc+        mov x11, v30.d[0]+        adds x4, x23, x3+        mul x22, x5, x7+        mov x23, v30.d[1]+        adcs x8, x8, x16+        adcs x16, x15, x11+        adc x11, x23, xzr+        umulh x3, x5, x7+        stp x24, x17, [x0, #16]+        mov x5, v4.d[0]+        subs x15, x20, x19+        cneg x7, x15, cc+        str x10, [x0, #64]+        csetm x1, cc+        subs x24, x2, x9+        cneg x17, x24, cc+        cinv x15, x1, cc+        adds x23, x4, x5+        umulh x1, x7, x17+        adcs x24, x8, x4+        adcs x10, x16, x8+        eor x8, x22, x21+        adcs x16, x11, x16+        mul x22, x7, x17+        eor x17, x1, x15+        adc x1, xzr, x11+        adds x11, x24, x5+        eor x7, x3, x21+        adcs x3, x10, x23+        adcs x24, x16, x24+        adcs x4, x1, x10+        eor x10, x22, x15+        adcs x16, xzr, x16+        adc x1, xzr, x1+        cmn x21, #0x1+        adcs x8, x4, x8+        adcs x22, x16, x7+        adc x7, x1, x21+        subs x21, x19, x12+        csetm x4, cc+        cneg x1, x21, cc+        subs x21, x13, x2+        cinv x16, x4, cc+        cneg x4, x21, cc+        cmn x15, #0x1+        adcs x21, x23, x10+        mul x23, x1, x4+        adcs x11, x11, x17+        adcs x3, x3, x15+        umulh x1, x1, x4+        adcs x24, x24, x15+        adcs x8, x8, x15+        adcs x22, x22, x15+        eor x17, x23, x16+        adc x15, x7, x15+        subs x7, x20, x14+        cneg x7, x7, cc+        csetm x4, cc+        subs x10, x20, x12+        cneg x23, x10, cc+        csetm x10, cc+        subs x12, x6, x9+        cinv x20, x4, cc+        cneg x12, x12, cc+        cmn x16, #0x1+        eor x1, x1, x16+        adcs x17, x24, x17+        mul x4, x7, x12+        adcs x8, x8, x1+        umulh x1, x7, x12+        adcs x24, x22, x16+        adc x7, x15, x16+        subs x12, x13, x9+        cneg x12, x12, cc+        cinv x13, x10, cc+        subs x19, x19, x14+        mul x9, x23, x12+        cneg x19, x19, cc+        csetm x10, cc+        eor x16, x1, x20+        subs x22, x6, x2+        umulh x12, x23, x12+        eor x1, x4, x20+        cinv x4, x10, cc+        cneg x22, x22, cc+        cmn x20, #0x1+        adcs x15, x11, x1+        eor x6, x12, x13+        adcs x10, x3, x16+        adcs x17, x17, x20+        eor x23, x9, x13+        adcs x2, x8, x20+        mul x11, x19, x22+        adcs x24, x24, x20+        adc x7, x7, x20+        cmn x13, #0x1+        adcs x3, x10, x23+        umulh x22, x19, x22+        adcs x17, x17, x6+        eor x12, x22, x4+        extr x22, x15, x21, #63+        adcs x8, x2, x13+        extr x21, x21, x5, #63+        ldp x16, x23, [x0]+        adcs x20, x24, x13+        eor x1, x11, x4+        adc x6, x7, x13+        cmn x4, #0x1+        ldp x2, x7, [x0, #16]+        adcs x1, x3, x1+        extr x19, x1, x15, #63+        adcs x14, x17, x12+        extr x1, x14, x1, #63+        lsl x17, x5, #1+        adcs x8, x8, x4+        extr x12, x8, x14, #8+        ldp x15, x11, [x0, #32]+        adcs x9, x20, x4+        adc x3, x6, x4+        adds x16, x12, x16+        extr x6, x9, x8, #8+        ldp x14, x12, [x0, #48]+        extr x8, x3, x9, #8+        adcs x20, x6, x23+        ldr x24, [x0, #64]+        lsr x6, x3, #8+        adcs x8, x8, x2+        and x2, x1, #0x1ff+        and x1, x20, x8+        adcs x4, x6, x7+        adcs x3, x17, x15+        and x1, x1, x4+        adcs x9, x21, x11+        and x1, x1, x3+        adcs x6, x22, x14+        and x1, x1, x9+        and x21, x1, x6+        adcs x14, x19, x12+        adc x1, x24, x2+        cmp xzr, xzr+        orr x12, x1, #0xfffffffffffffe00+        lsr x1, x1, #9+        adcs xzr, x16, x1+        and x21, x21, x14+        adcs xzr, x21, xzr+        adcs xzr, x12, xzr+        adcs x21, x16, x1+        adcs x1, x20, xzr+        adcs x19, x8, xzr+        stp x21, x1, [x0]+        adcs x1, x4, xzr+        adcs x21, x3, xzr+        stp x19, x1, [x0, #16]+        adcs x1, x9, xzr+        stp x21, x1, [x0, #32]+        adcs x21, x6, xzr+        adcs x1, x14, xzr+        stp x21, x1, [x0, #48]+        adc x1, x12, xzr+        and x1, x1, #0x1ff+        str x1, [x0, #64]++// Restore regs and return++        CFI_POP2(x23,x24)+        CFI_POP2(x21,x22)+        CFI_POP2(x19,x20)++        CFI_RET++S2N_BN_SIZE_DIRECTIVE(bignum_sqr_p521)++#if defined(__linux__) && defined(__ELF__)+.section .note.GNU-stack,"",%progbits+#endif
+ cbits/s2n/arm/bignum_sqr_p521_alt.S view

file too large to diff

+ cbits/s2n/arm/bignum_tomont_p256.S view

file too large to diff

+ cbits/s2n/arm/bignum_tomont_p384.S view

file too large to diff

+ cbits/s2n/arm/curve25519_x25519.S view

file too large to diff

+ cbits/s2n/arm/curve25519_x25519_alt.S view

file too large to diff

+ cbits/s2n/arm/curve25519_x25519base.S view

file too large to diff

+ cbits/s2n/arm/curve25519_x25519base_alt.S view

file too large to diff

+ cbits/s2n/arm/edwards25519_encode.S view

file too large to diff

+ cbits/s2n/arm/edwards25519_scalarmulbase.S view

file too large to diff

+ cbits/s2n/arm/edwards25519_scalarmulbase_alt.S view

file too large to diff

+ cbits/s2n/arm/p256_montjadd.S view

file too large to diff

+ cbits/s2n/arm/p256_montjadd_alt.S view

file too large to diff

+ cbits/s2n/arm/p256_montjdouble.S view

file too large to diff

+ cbits/s2n/arm/p256_montjdouble_alt.S view

file too large to diff

+ cbits/s2n/arm/p256_montjmixadd.S view

file too large to diff

+ cbits/s2n/arm/p256_montjmixadd_alt.S view

file too large to diff

+ cbits/s2n/arm/p256_scalarmul.S view

file too large to diff

+ cbits/s2n/arm/p256_scalarmul_alt.S view

file too large to diff

+ cbits/s2n/arm/p256_scalarmulbase.S view

file too large to diff

+ cbits/s2n/arm/p256_scalarmulbase_alt.S view

file too large to diff

+ cbits/s2n/arm/p384_montjscalarmul.S view

file too large to diff

+ cbits/s2n/arm/p384_montjscalarmul_alt.S view

file too large to diff

+ cbits/s2n/arm/p521_jscalarmul.S view

file too large to diff

+ cbits/s2n/arm/p521_jscalarmul_alt.S view

file too large to diff

+ cbits/s2n/import.sh view

file too large to diff

+ cbits/s2n/include/_internal_s2n_bignum_arm.h view

file too large to diff

+ cbits/s2n/include/_internal_s2n_bignum_x86_att.h view

file too large to diff

+ cbits/s2n/x86_att/_internal_s2n_bignum_x86_att.h view

file too large to diff

+ cbits/s2n/x86_att/bignum_deamont_p384.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_deamont_p384_alt.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_demont_p256.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_demont_p256_alt.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_emontredc_8n.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_inv_p521.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_kmul_16_32.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_kmul_32_64.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_ksqr_16_32.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_ksqr_32_64.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_modinv.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_montifier.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_montinv_p384.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_montmul_p384.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_montmul_p384_alt.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_montsqr_p384.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_montsqr_p384_alt.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_mul.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_mul_8_16.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_mul_p521.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_mul_p521_alt.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_neg_p256.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_sqr.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_sqr_8_16.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_sqr_p521.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_sqr_p521_alt.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_tomont_p256.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_tomont_p256_alt.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_tomont_p384.S view

file too large to diff

+ cbits/s2n/x86_att/bignum_tomont_p384_alt.S view

file too large to diff

+ cbits/s2n/x86_att/curve25519_x25519.S view

file too large to diff

+ cbits/s2n/x86_att/curve25519_x25519_alt.S view

file too large to diff

+ cbits/s2n/x86_att/curve25519_x25519base.S view

file too large to diff

+ cbits/s2n/x86_att/curve25519_x25519base_alt.S view

file too large to diff

+ cbits/s2n/x86_att/edwards25519_encode.S view

file too large to diff

+ cbits/s2n/x86_att/edwards25519_scalarmulbase.S view

file too large to diff

+ cbits/s2n/x86_att/edwards25519_scalarmulbase_alt.S view

file too large to diff

+ cbits/s2n/x86_att/p256_montjadd.S view

file too large to diff

+ cbits/s2n/x86_att/p256_montjadd_alt.S view

file too large to diff

+ cbits/s2n/x86_att/p256_montjdouble.S view

file too large to diff

+ cbits/s2n/x86_att/p256_montjdouble_alt.S view

file too large to diff

+ cbits/s2n/x86_att/p256_montjmixadd.S view

file too large to diff

+ cbits/s2n/x86_att/p256_montjmixadd_alt.S view

file too large to diff

+ cbits/s2n/x86_att/p256_scalarmul.S view

file too large to diff

+ cbits/s2n/x86_att/p256_scalarmul_alt.S view

file too large to diff

+ cbits/s2n/x86_att/p256_scalarmulbase.S view

file too large to diff

+ cbits/s2n/x86_att/p256_scalarmulbase_alt.S view

file too large to diff

+ cbits/s2n/x86_att/p384_montjscalarmul.S view

file too large to diff

+ cbits/s2n/x86_att/p384_montjscalarmul_alt.S view

file too large to diff

+ cbits/s2n/x86_att/p521_jscalarmul.S view

file too large to diff

+ cbits/s2n/x86_att/p521_jscalarmul_alt.S view

file too large to diff

cbits/sha1_armv8.c view

file too large to diff

cbits/sha1_x86.c view

file too large to diff

cbits/sha256_armv8.c view

file too large to diff

cbits/sha3_armv8.c view

file too large to diff

cbits/sha512_armv8.c view

file too large to diff

+ cbits/tests/bearssl_diff.c view

file too large to diff

+ cbits/tests/ct/README view

file too large to diff

+ cbits/tests/ct/cpu_stub.c view

file too large to diff

+ cbits/tests/ct/ct.h view

file too large to diff

+ cbits/tests/ct/ct_aes.c view

file too large to diff

+ cbits/tests/ct/ct_aes_armv8.c view

file too large to diff

+ cbits/tests/ct/ct_canary.c view

file too large to diff

+ cbits/tests/ct/ct_chapoly.c view

file too large to diff

+ cbits/tests/ct/ct_decaf.c view

file too large to diff

+ cbits/tests/ct/ct_ed25519.c view

file too large to diff

+ cbits/tests/ct/ct_mlkem.c view

file too large to diff

+ cbits/tests/ct/ct_p256.c view

file too large to diff

+ cbits/tests/ct/ct_powm.c view

file too large to diff

+ cbits/tests/ct/ct_x25519.c view

file too large to diff

+ cbits/tests/ct/known.txt view

file too large to diff

+ cbits/tests/ct/run.sh view

file too large to diff

+ cbits/tests/endian/README view

file too large to diff

+ cbits/tests/endian/endian.c view

file too large to diff

+ cbits/tests/endian/run.sh view

file too large to diff

+ cbits/tests/endian/vectors.txt view

file too large to diff

+ cbits/tests/fuzz/README view

file too large to diff

+ cbits/tests/fuzz/corpus/aead-authentic.bin view

file too large to diff

+ cbits/tests/fuzz/corpus/ed25519-tampered.bin view

file too large to diff

+ cbits/tests/fuzz/corpus/ed25519-valid.bin view

file too large to diff

+ cbits/tests/fuzz/corpus/p256-on-curve.bin view

file too large to diff

+ cbits/tests/fuzz/fuzz.h view

file too large to diff

+ cbits/tests/fuzz/fuzz_aead.c view

file too large to diff

+ cbits/tests/fuzz/fuzz_canary.c view

file too large to diff

+ cbits/tests/fuzz/fuzz_decaf.c view

file too large to diff

+ cbits/tests/fuzz/fuzz_ed25519.c view

file too large to diff

+ cbits/tests/fuzz/fuzz_p256.c view

file too large to diff

+ cbits/tests/fuzz/run.sh view

file too large to diff

+ cbits/tests/fuzz/standalone.c view

file too large to diff

+ cbits/tests/perf/floors.txt view

file too large to diff

+ cbits/tests/perf/run.sh view

file too large to diff

+ cbits/tests/perf/throughput.c view

file too large to diff

+ cbits/tests/ppc8_diff.c view

file too large to diff

+ cbits/tests/scrub/README view

file too large to diff

+ cbits/tests/scrub/known.txt view

file too large to diff

+ cbits/tests/scrub/run.sh view

file too large to diff

+ cbits/tests/scrub/scrub.c view

file too large to diff

+ cbits/tests/sysdrg/run.sh view

file too large to diff

+ cbits/tests/sysdrg/sysdrg.c view

file too large to diff

+ cbits/tests/width/ed448_width.c view

file too large to diff

+ cbits/tests/width/p256_width.c view

file too large to diff

+ cbits/tests/width/run.sh view

file too large to diff

+ cbits/tests/width/x25519_width.c view

file too large to diff

crypton.cabal view

file too large to diff

tests/BlockCipher/AESSpec.hs view

file too large to diff

tests/ECDSASpec.hs view

file too large to diff

+ tests/EntropySpec.hs view

file too large to diff

tests/HashSpec.hs view

file too large to diff

tests/KDF/PBKDF2Spec.hs view

file too large to diff

tests/NumberSpec.hs view

file too large to diff

tests/PubKey/DSASpec.hs view

file too large to diff

tests/PubKey/ElGamalSpec.hs view

file too large to diff

tests/PubKey/MGF1Spec.hs view

file too large to diff

+ tests/PubKey/MLDSASpec.hs view

file too large to diff

+ tests/PubKey/MLDSAVectors.hs view

file too large to diff

+ tests/PubKey/MLKEMSpec.hs view

file too large to diff

+ tests/PubKey/MLKEMVectors.hs view

file too large to diff

tests/PubKey/P256Spec.hs view

file too large to diff

tests/PubKey/PSSSpec.hs view

file too large to diff

tests/PubKey/RSASpec.hs view

file too large to diff

tests/PubKey/RabinSpec.hs view

file too large to diff

tests/RuntimeSpec.hs view

file too large to diff

tests/StreamCipher/ChaChaPoly1305Spec.hs view

file too large to diff

+ tests/SysDRGSpec.hs view

file too large to diff

+ tests/forkprocess/ForkProcess.hs view

file too large to diff

+ tests/tutorial/extract.awk view

file too large to diff

+ tests/tutorial/run.sh view

file too large to diff